Voice detection method, device, equipment and storage medium

Abstract

The present application relates to the field of computer processing technology, and provides a speech detection method, apparatus, device, readable storage medium and program product, which can accurately confirm the position of keywords in speech while ensuring the efficiency of keyword detection, including: candidate phoneme probability corresponding to each speech frame in the speech to be detected; filtering the speech frames with blank phonemes based on the candidate phoneme probability corresponding to the speech frames, and obtaining the recognition phoneme sequence corresponding to the speech to be detected based on the candidate phoneme probability corresponding to the speech frames with existing phonemes; determining the extracted speech frame sequence corresponding to the target detection word in the speech to be detected based on the position of the speech frame sequence corresponding to the phoneme sequence of the target detection word in the speech to be detected; obtaining the distribution of the phoneme probability corresponding to each extracted speech frame in the extracted speech frame sequence; and obtaining the speech detection result corresponding to the speech to be detected based on the distribution of the phoneme probability corresponding to the extracted speech frames.

Description

Technical Field

[0001] The present application relates to the field of computer processing technology, and in particular to a speech detection method, apparatus, computer equipment, computer-readable storage medium and computer program product. Background Art

[0002] With the advent of the intelligent era, artificial intelligence technology has been widely used in various fields, such as image recognition and speech recognition. One of the technical branches of speech recognition based on artificial intelligence technology is keyword spotting; keyword spotting can be applied to voice wake-up systems, without the need for users to manually control the device, to achieve intelligent control. For example, after the voice wake-up system detects that the user has issued a voice corresponding to the keyword, it directly controls the vehicle-mounted device or home appliance according to the user's voice command.

[0003] When speech detection is required, features of the speech can be extracted and speech detection results can be obtained based on the extracted features. However, current language detection methods have the problems of low detection efficiency and low speech detection accuracy. Summary of the invention

[0004] Based on this, it is necessary to provide a speech detection method, apparatus, computer device, computer-readable storage medium and computer program product to address the above technical problems.

[0005] A speech detection method, the method comprising: extracting features from each speech frame in the speech to be detected, obtaining feature vectors corresponding to the speech frame, and sorting the feature vectors corresponding to each speech frame according to the corresponding speech frame order to obtain a feature vector sequence; performing phoneme recognition based on the feature vector sequence to obtain the candidate phoneme probability corresponding to each speech frame; filtering speech frames with blank phonemes based on the candidate phoneme probability corresponding to the speech frames, and obtaining a recognition phoneme sequence corresponding to the speech to be detected based on the candidate phoneme probability corresponding to the speech frames with existing phonemes; using the phoneme sequence corresponding to the target detection word selected from the recognition phoneme sequence as the detection word phoneme sequence, and determining the extracted speech frame sequence corresponding to the target detection word in the speech to be detected based on the position of the speech frame sequence corresponding to the detection word phoneme sequence in the speech frame sequence of the speech to be detected; obtaining the phoneme probability distribution corresponding to each extracted speech frame in the extracted speech frame sequence; the phoneme probability distribution corresponding to the extracted speech frame is obtained by performing speech frame phoneme detection according to the feature vectors corresponding to the extracted speech frame; and obtaining the speech detection result corresponding to the speech to be detected based on the phoneme probability distribution corresponding to the extracted speech frame.

[0006] A speech detection device, the device comprising: a feature vector processing module, used to extract features from each speech frame in the speech to be detected, obtain feature vectors corresponding to the speech frame, and sort the feature vectors corresponding to each speech frame according to the corresponding speech frame sequence to obtain a feature vector sequence; a candidate phoneme possibility acquisition module, used to perform phoneme recognition based on the feature vector sequence to obtain candidate phoneme possibility corresponding to each speech frame; a phoneme sequence acquisition module, used to filter speech frames with blank phonemes based on the candidate phoneme possibility corresponding to the speech frames, and obtain a recognized phoneme sequence corresponding to the speech to be detected based on the candidate phoneme possibility corresponding to the speech frames with phonemes; a speech frame extraction module, used to extract the candidate phoneme possibility corresponding to the speech frames from the speech frame A phoneme sequence corresponding to a target detection word selected from a recognition phoneme sequence is used as a detection word phoneme sequence, and based on the position of a speech frame sequence corresponding to the detection word phoneme sequence in the speech frame sequence of the speech to be detected, an extracted speech frame sequence corresponding to the target detection word in the speech to be detected is determined; a phoneme possibility distribution acquisition module is used to obtain the phoneme possibility distribution corresponding to each extracted speech frame in the extracted speech frame sequence; the phoneme possibility distribution corresponding to the extracted speech frame is obtained by performing speech frame phoneme detection respectively according to the feature vectors corresponding to the extracted speech frame; a speech detection result acquisition module is used to obtain a speech detection result corresponding to the speech to be detected based on the phoneme possibility distribution corresponding to the extracted speech frame.

[0007] In some embodiments, the speech frame extraction module is also used to determine the starting position and ending position of the speech frame sequence corresponding to the detection word phoneme sequence of the speech to be detected; the forward position corresponding to the starting position is used as the first extraction position, and the backward position corresponding to the ending position is used as the second extraction position; the speech frame sequence located between the first extraction position and the second extraction position in the speech to be detected is used as the extracted speech frame sequence corresponding to the target detection word.

[0008] In some embodiments, the speech frame extraction module is also used to take the number of speech frames with blank phonemes in the speech to be detected as the number of blank frames; determine the number of speech frame extensions based on the number of blank frames; the number of speech frame extensions is positively correlated with the number of blank frames; take the forward position whose distance from the starting position is the number of speech frame extensions as the first extraction position, and take the backward position whose distance from the ending position is the number of speech frame extensions as the second extraction position.

[0009] In some embodiments, the possibility of the candidate phonemes includes the possibility corresponding to phoneme blanks; the phoneme sequence acquisition module is also used to take the speech frames whose possibility corresponding to phoneme blanks is greater than the possibility threshold as phoneme blank speech frames; filter the phoneme blank speech frames to obtain decoded speech frames, and obtain a speech decoding network formed based on the possibility of the candidate phonemes corresponding to the decoded speech frames; decode based on the speech decoding network to obtain a target decoding path, arrange the phonemes passed by the target decoding path in path order, and obtain the recognition phoneme sequence corresponding to the speech to be detected.

[0010] In some embodiments, the feature vector processing module is also used to input each speech frame in the speech to be detected into the feature extraction submodel of the trained speech detection model for feature extraction, so as to obtain the feature vector corresponding to the speech frame; the trained speech detection model includes a feature extraction submodel and a speech sequence recognition submodel and a speech frame phoneme detection submodel respectively connected to the feature extraction submodel; the candidate phoneme possibility acquisition module is also used to input the feature vector sequence into the speech sequence recognition submodel for phoneme recognition, so as to obtain the candidate phoneme possibility corresponding to each of the speech frames; the phoneme possibility distribution acquisition module is also used to extract the feature vector corresponding to the extracted speech frame from the feature vector sequence, and input the feature vector corresponding to the extracted speech frame into the speech frame phoneme detection submodel for phoneme recognition; the possibility distribution of each type of phoneme output by the speech frame phoneme detection submodel in each extracted speech frame is used as the phoneme possibility distribution corresponding to each extracted speech frame.

[0011] In some embodiments, the device also includes a speech detection model training module for obtaining training speech; inputting the training speech into a feature extraction submodel to be trained for feature extraction to obtain training feature vectors corresponding to each training speech frame, and sorting the training feature vectors corresponding to each training speech frame in accordance with the corresponding speech frame sequence to obtain a training vector sequence; inputting the training vector sequence into a speech sequence recognition submodel to be trained for phoneme recognition to obtain a phoneme sequence recognition result; obtaining a first model loss value based on a first difference between the phoneme sequence recognition result and a standard recognition result corresponding to the training speech; the first model loss value is positively correlated with the first difference; inputting each of the training feature vectors into the speech frame phoneme detection submodel to be trained; The sub-model performs phoneme recognition to obtain the speech frame phoneme detection result corresponding to the speech frame phoneme detection sub-model; a second model loss value is obtained based on the second difference between the speech frame phoneme detection result and the standard recognition result corresponding to the training speech; the first model loss value is positively correlated with the second difference; a target model loss value is obtained based on the first model loss value and the second model loss value; based on the target model loss value, parameters of the feature extraction sub-model to be trained, the speech frame phoneme detection sub-model to be trained and the speech sequence recognition sub-model to be trained are adjusted, and the feature extraction sub-model after parameter adjustment, the speech frame phoneme detection sub-model after parameter adjustment and the speech sequence recognition sub-model after parameter adjustment form the trained speech detection model.

[0012] In some embodiments, the candidate phoneme possibility acquisition module is also used to sequentially acquire the current feature vector in the feature vector sequence in the order of the feature vector sequence; acquire the phoneme representation vector corresponding to the previous feature vector corresponding to the current feature vector; the phoneme representation vector corresponding to the previous feature vector is the representation vector corresponding to the phoneme obtained by inputting the previous feature vector into the speech sequence recognition sub-model for phoneme recognition; the phoneme representation vector corresponding to the previous feature vector and the current feature vector are input into the speech sequence recognition sub-model for phoneme recognition to obtain the candidate phoneme possibility corresponding to the current speech frame corresponding to the current feature vector.

[0013] In some embodiments, the device also includes a phoneme representation vector acquisition module, which is used to input the previous feature vector into the speech sequence recognition sub-model for phoneme recognition to obtain the candidate phoneme possibility corresponding to the previous feature vector; the candidate phoneme possibility corresponding to the previous feature vector includes the possibility corresponding to each candidate phoneme in the candidate phoneme set; select the candidate phoneme with the largest corresponding possibility from the candidate phoneme set as the target phoneme; use the phoneme representation vector corresponding to the target phoneme as the phoneme representation vector corresponding to the previous feature vector.

[0014] In some embodiments, the device also includes a phoneme recognition continuation module, which is used to return to the step of sequentially acquiring the current feature vector in the feature vector sequence in the order of the feature vector sequence when the feature vector in the feature vector sequence has not been phoneme recognized, so as to perform phoneme recognition on the feature vector that has not been phoneme recognized until the phoneme recognition of the feature vector in the feature vector sequence is completed.

[0015] In some embodiments, the speech detection result acquisition module is also used to obtain the detection word position of the target detection word in the speech to be detected based on the phoneme probability distribution corresponding to the extracted speech frame; obtain the speech in the speech to be detected that is located at the backward position corresponding to the detection word position as the instruction detection speech; perform speech command detection on the shown instruction detection speech to obtain the target speech command corresponding to the instruction detection speech, so as to perform voice control based on the target speech command.

[0016] A computer device comprises a memory and a processor, wherein the memory stores a computer program and the processor executes the above method.

[0017] A computer-readable storage medium stores a computer program, wherein the computer program is executed by a processor to execute the above method.

[0018] A computer program product comprises a computer program, wherein the computer program implements the above method when executed by a processor.

[0019] The above-mentioned speech detection method, device, computer equipment, computer-readable storage medium and computer program product include: extracting features from each speech frame in the speech to be detected to obtain feature vectors corresponding to the speech frame, and sorting the feature vectors corresponding to each speech frame according to the corresponding speech frame order to obtain a feature vector sequence; performing phoneme recognition based on the feature vector sequence to obtain the candidate phoneme possibility corresponding to each speech frame; filtering the speech frames with blank phonemes based on the candidate phoneme possibility corresponding to the speech frames, and obtaining the recognition phoneme sequence corresponding to the speech to be detected based on the candidate phoneme possibility corresponding to the speech frames with existing phonemes; using the phoneme sequence corresponding to the target detection word selected from the recognition phoneme sequence as the detection word phoneme sequence, and determining the extracted speech frame sequence corresponding to the target detection word in the speech to be detected based on the position of the speech frame sequence corresponding to the detection word phoneme sequence in the speech frame sequence of the speech to be detected; obtaining the phoneme possibility distribution corresponding to each extracted speech frame in the extracted speech frame sequence; the phoneme possibility distribution corresponding to the extracted speech frame is obtained by performing speech frame phoneme detection according to the feature vectors corresponding to the extracted speech frame; based on the phoneme possibility distribution corresponding to the extracted speech frame, obtaining the speech detection result corresponding to the speech to be detected. Since speech frames with blank phonemes are filtered based on the possibility of candidate phonemes corresponding to the speech frames, and the recognized phoneme sequence is obtained based on the possibility of candidate phonemes corresponding to the speech frames with existing phonemes, the efficiency of obtaining the recognized phoneme sequence is improved, and the efficiency of obtaining the phoneme sequence corresponding to the target detection word from the recognized phoneme sequence is also improved; then, based on the position of the speech frame sequence corresponding to the detection word phoneme sequence in the speech frame sequence of the speech to be detected, the extracted speech frame sequence corresponding to the target detection word is determined, and there is no need to perform phoneme detection on all speech frames subsequently, thereby improving efficiency; and since the extracted speech frame sequence is subjected to frame-by-frame phoneme detection, the speech detection result obtained according to the distribution of phoneme possibility of each extracted speech frame obtained by frame-by-frame phoneme detection can accurately reflect the position of the keyword in the speech, thereby improving accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] FIG. 1( a ) is a schematic diagram of an application scenario of a speech detection method in some embodiments;

[0021] FIG1( b ) is a schematic flow chart of a speech detection method in some embodiments;

[0022] Figure 2 A schematic diagram of a speech decoding network for non-frame-by-frame decoding search in some embodiments;

[0023] Figure 3 A schematic diagram of a speech decoding network for frame-by-frame decoding search in some embodiments;

[0024] Figure 4 is a flowchart of a speech detection method in some embodiments;

[0025] Figure 5 A schematic diagram of a model architecture in some embodiments;

[0026] Figure 6 A schematic diagram of a model architecture in some embodiments;

[0027] Figure 7 is a flowchart of a speech detection method in some embodiments;

[0028] Figure 8 A schematic diagram of phoneme state transition corresponding to a keyword in some embodiments;

[0029] Fig. 9 A schematic diagram of state transition of each speech frame in some embodiments;

[0030] Fig.10 A schematic diagram of model forward inference and model deployment in some embodiments;

[0031] Fig.11 is a structural block diagram of a speech detection device in some embodiments;

[0032] Fig.12 1 is a diagram of the internal structure of a computer device in some embodiments. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0034] Reference to "embodiments" in this application means that the specific features, structures, or characteristics described in conjunction with the embodiments may be included in at least some embodiments of the present application. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments.

[0035] First, let’s introduce the terms involved in the solution provided by this application:

[0036] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0037] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. Basic artificial intelligence technologies generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operating / interactive systems, mechatronics and other technologies. Artificial intelligence software technologies mainly include computer vision technology, speech processing technology, natural language processing technology, as well as machine learning / deep learning, autonomous driving, smart transportation and other major directions.

[0038] The key technologies of speech technology include automatic speech recognition technology, speech synthesis technology and voiceprint recognition technology. Enabling computers to listen, see, speak and feel is the development direction of human-computer interaction in the future, among which speech has become one of the most promising human-computer interaction methods in the future.

[0039] Keyword Spotting (KWS) is a branch of speech recognition technology. The keyword detection task mainly includes: identifying whether there are keywords in the speech and confirming the exact position of the keywords in the speech; the position of the keyword in the speech includes the starting position or the ending position of the keyword in the speech. The keyword can be a wake-up word. After the user issues the voice corresponding to the wake-up word, the speech recognition system is awakened, and the speech recognition system begins to detect the voice after the user issues the voice corresponding to the wake-up word to determine the operation instruction issued by the user; for example, if the wake-up word is "Xiao Ming Xiao Ming", after the user issues the voice corresponding to "Xiao Ming Xiao Ming", the speech recognition system is awakened and begins to detect whether the user issues voice operation instructions such as "turn on the air conditioner". Keywords can also be voice operation instructions. Users can directly say voice operation instructions without saying the wake-up word first and then the voice operation instruction.

[0040] Natural language processing (NLP) is an important direction in the fields of computer science and artificial intelligence. It studies various theories and methods that can achieve effective communication between people and computers using natural language. Natural language processing is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field will involve natural language, that is, the language people use in daily life, so it is closely related to the study of linguistics. Natural language processing technology usually includes text processing, semantic understanding, machine translation, robot question answering, knowledge graph and other technologies.

[0041] Machine Learning (ML) is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include deep neural networks (DNN), belief networks, reinforcement learning, transfer learning, inductive learning, and learning by teaching. Among them, deep neural networks include feedforward sequential memory networks (FSMN) and convolutional neural networks (CNN).

[0042] In the process of deep learning, the cross entropy loss function (CE loss function) can be used for model training. According to the number of tasks that the model needs to perform, deep learning can be divided into single-task learning and multi-task learning (MTL); single-task learning mainly learns one task at a time, and multi-task learning learns multiple related tasks together and learns multiple tasks at the same time.

[0043] Speech recognition model training includes acoustic model training, which can be sequence-to-sequence model training. There is no need to pre-align the frames of the training speech (that is, there is no need to perform phoneme annotation on each training speech frame) for model training. Model training can be performed with only a training speech frame sequence as input and a training phoneme sequence as output.

[0044] Hidden Markov Model (HMM) is a kind of Markov chain whose state cannot be observed directly, but can be observed through a sequence of observation vectors. Each observation vector is represented by various states through certain probability density distributions, and each observation vector is generated by a sequence of states with corresponding probability density distributions. Therefore, the hidden Markov model is a dual random process - a hidden Markov chain with a certain number of states and a set of explicit random functions. In the hidden Markov model, the state is not directly visible, but the output depends on the state and is visible. Each state has a possible probability distribution through possible output tokens. Therefore, generating a sequence of tokens through an HMM provides some information about the sequence of states. Note that "hidden" refers to the sequence of states that the model passes through, not the parameters of the model. Hidden Markov models are known for their pattern recognition in time, such as speech, handwriting, gesture recognition, word class labeling, music scores, partial discharge and bioinformatics applications.

[0045] The solution provided in the embodiments of the present application involves speech recognition technology based on artificial intelligence, which can be applied to driving scenarios, smart home scenarios, robot control scenarios and other scenarios. If the computing power of devices such as vehicles, smart home devices and robots is strong, the speech detection method can be directly executed by devices such as vehicles, smart home devices and robots; if the computing power of vehicles, smart home devices and robots is weak, the collected voice can be sent to the server by the vehicle, smart home device or robot, so that the server executes the speech detection method, and the server feeds back the results obtained after executing the speech detection method to the vehicle, smart home device or robot, as shown in Figure 1 (a). In the introduction of the following embodiments, the device that executes the speech detection method is referred to as a computer device, which can be a vehicle, a smart home device, a robot and a server.

[0046] The "sequence-to-sequence" model focuses on the result from an input sequence to an output sequence, that is, the "sequence-to-sequence" model focuses on whether the predicted sequence output by the model itself is the same as the actual sequence. In speech recognition, for speech frames whose corresponding phonemes are difficult to determine, the "sequence-to-sequence" model will assign a phoneme blank label to the speech frame, rather than forcibly assigning a certain type of phoneme label to the speech frame; based on the feature that the "sequence-to-sequence" model can assign phoneme blank labels to speech frames whose corresponding phonemes are difficult to determine, the "sequence-to-sequence" model can be used to predict whether there is a target detection word (i.e., keyword) in the speech. As shown in Figure 1(b), after obtaining the speech uttered by the user, the computer device can perform frame processing on the speech to obtain N0 to N 1000speech frames, and input the speech frame sequence into the "sequence to sequence" model; use the "sequence to sequence" model to predict the existence of phonemes in the speech frame sequence, and obtain the possibility of the phonemes existing in each speech frame output by the "sequence to sequence" model; if the possibility of the phoneme existing in the speech frame is less than a preset value, it is considered that the possibility of the phoneme existing in the speech frame is low, and the speech frame is a speech frame with blank phonemes; if the possibility of the phoneme existing in the speech frame is greater than or equal to the preset value, it is considered that the possibility of the phoneme existing in the speech frame is high, and the speech frame is a speech frame with non-empty phonemes.

[0047] After using the "sequence to sequence" model to predict the existence of phonemes in each speech frame of the speech frame sequence, the probability of the existence of phonemes in each speech frame is obtained, and the speech frames with blank phonemes and the speech frames with non-blank phonemes are determined. As shown in Figure 1(b), the speech frames with blank phonemes include N0, N1, N2 and N 1000 The speech frames with non-empty phonemes include N3, N4 and N 999 wait.

[0048] Next, the computer device can filter the speech frames with blank phonemes, retain the speech frames with non-blank phonemes, obtain a sequence of speech frames with non-blank phonemes, and perform decoding search based on the probability distribution of various types of phonemes in the speech frames with non-blank phonemes, such as the probability distribution of various types of phonemes in N3, N4, N5, N6, N7, N8, N9, N10, N11, N12, N13, N14, N15, N16, N17, N18, N19, N20, N21, N22, N30, N40, N50, N60, N70, N80, N9 995 、N 996 、N 998 and N 999 The decoding search is performed based on the probability distribution of each phoneme non-empty speech frame. Since the decoding search at this stage is for phoneme non-empty speech frames and does not include phoneme blank speech frames, the decoding search at this stage is called non-frame-by-frame decoding search.

[0049] Based on the above-mentioned non-frame-by-frame decoding search, the computer device can determine whether there is a target detection word in the speech; for example, the phoneme sequence corresponding to the target detection word "turn on the air conditioner" is "dak ai k ong t iao". Through the non-frame-by-frame decoding search, it is determined that the phoneme sequence "dak ai k ong t iao" exists in the non-empty speech frame sequence of the speech, then it can be considered that the speech contains the target detection word "turn on the air conditioner".

[0050] After determining that the target detection word exists in the speech, the phoneme non-empty speech frame sequence in which the above-mentioned phoneme sequence exists can be determined. Since the above-mentioned non-frame-by-frame decoding search can only determine the phonemes corresponding to the phoneme non-empty speech frames, and cannot determine the phonemes corresponding to the phoneme blank speech frames, it can only determine the approximate position of the above-mentioned target detection word in the speech (the approximate position of the target detection word in the speech can be understood as: the approximate moment when the target detection word exists in the speech), and it cannot be determined whether the multiple phoneme blank speech frames adjacent to the phoneme non-empty speech frames in which the phonemes of the target detection word exist, that is, it is impossible to accurately determine the position of the above-mentioned target detection word in the speech. Therefore, the computer device can determine the approximate moment when the target detection word exists in the speech based on the phoneme non-empty speech frame sequence in which the phoneme sequence of the target detection word exists, and use the speech frames at multiple consecutive moments adjacent to the approximate moment as the speech frames in which the phonemes of the target detection word may exist.

[0051] For example, if there is a phoneme non-empty speech frame sequence of the phoneme sequence "dak ai kong tiao" is N 100 To N 150 The phoneme is not empty speech frame, then it can be determined that the approximate time when the target detection word exists in the speech is N 100 and N 150 Therefore, it can be considered that the 100 The previous consecutive speech frames may contain the phonemes of the target detection word, such as N 80 To N 100 Each speech frame between may contain the phoneme of the target word; it can also be considered that the phonemes between N 150 The following consecutive speech frames may contain the phonemes of the target detection word, such as N 150 To N 180 Each speech frame between may contain the phoneme of the target detection word. Therefore, it is determined that the speech frame sequence in which the phoneme of the target detection word may exist is N 80 To N 180 As shown in Figure 1(b), in N 80 To N 180 In the speech frame sequence between, some speech frames are speech frames with blank phonemes, and some speech frames are speech frames with non-empty phonemes.

[0052] After determining the speech frame sequence in which the phonemes of the target detection word may exist, the computer device obtains the possible degree distribution of each speech frame of each type of phoneme in the speech frame sequence (which can be called the phoneme possible degree distribution of each speech frame). For example, for N 100 , you can get the phonemes "a", "o", "e" and "d" and other types of phonemes in N 100 The probability of existence forms a probability distribution; among them, the phoneme "a" in N100 The possibility of existence is represented by N 100 The probability that the corresponding actual phoneme is "a".

[0053] In some embodiments, a pre-built phoneme category detection model can be used to determine the probability of each type of phoneme existing in each speech frame. The labels used when training the phoneme category detection model may include labels for each type of phoneme, but do not include labels representing phoneme blanks. Therefore, the phoneme category detection model can be used to compulsorily predict the phonemes corresponding to each speech frame, and no speech frame will be predicted as a phoneme blank. It should be noted that the labels used when training the above-mentioned "sequence to sequence" model include labels representing phoneme blanks. For speech frames that are difficult to predict the corresponding phonemes, the "sequence to sequence" model will label them as phoneme blanks, and will not compulsorily assign a label of a certain type of phoneme to the speech frame, so as to reduce the model loss value.

[0054] The computer device obtains the phoneme probability distribution of each speech frame in the speech frame sequence where the phoneme of the target detection word may exist, and then performs decoding search based on the phoneme probability distribution of each speech frame to determine the phoneme corresponding to each speech frame in the speech frame sequence where the phoneme of the target detection word may exist. It should be noted that the decoding search at this stage is for each continuous speech frame, so the decoding search at this stage is called frame-by-frame decoding search.

[0055] Combination Figure 2 Introduce the above frame-by-frame decoding search process: based on N 80 To N 180 The possible distribution of phonemes in each speech frame of the speech frame sequence between and the possible transition of phonemes between adjacent speech frames (the possible transition can be given by the language model) can be constructed. Figure 2 The speech decoding network (also called decoding graph) shown in FIG. 1 is then decoded and searched to obtain the maximum probability path, and the phoneme located on the maximum probability path is used as the phoneme corresponding to the speech frame. For example, if the first phoneme of the maximum probability path is "d", the phoneme "d" can be used as the phoneme corresponding to N 80 Similarly, for other phonemes of the maximum probability path, type processing can be performed to determine N 80 With N 180 The phonemes corresponding to each speech frame between the two can accurately determine the position of the target detection word in the speech and realize forced alignment of the speech.

[0056] It should be noted that the process of non-frame-by-frame decoding search is similar to that of frame-by-frame decoding search. The main difference is that the speech decoding network used for decoding search is different. The speech decoding network used for non-frame-by-frame decoding search is based on speech frames with non-empty phonemes (such as N3, N4, N 995、N 996 、N 998 and N 999 The phoneme possibility distribution (such as the probability distribution of the phonemes) and the possibility of phoneme transfer between adjacent speech frames are constructed.

[0057] In addition, if the "sequence to sequence" model outputs the probability of each speech frame being a phoneme blank and the probability of the existence of various types of phonemes, and the probability of a certain speech frame being a phoneme blank is lower than a preset value, then the speech frame can be considered to be a speech frame with non-empty phonemes, and the probability of the existence of various types of phonemes in the speech frame output by the "sequence to sequence" model can be used as the phoneme probability distribution of the speech frame, and combined with the phoneme blank ( Figure 3 The probability of state transition between adjacent speech frames is constructed by using the probability of state transition between adjacent speech frames. Figure 3 The speech decoding network shown in FIG. The state transition between adjacent speech frames includes: phoneme state to phoneme state, phoneme state to phoneme blank state, and phoneme blank state to phoneme blank state; the state transition possibility between adjacent speech frames can be given by the language model.

[0058] In some embodiments, Figure 4 As shown, a voice detection method is provided, which is described by taking the method applied to a computer device as an example, and includes the following steps:

[0059] Step S402, extracting features from each speech frame in the speech to be detected to obtain feature vectors corresponding to the speech frames, and sorting the feature vectors corresponding to each speech frame according to the corresponding speech frame sequence to obtain a feature vector sequence.

[0060] Wherein, the voice to be detected whether there is a target detection word can be used as the voice to be detected, and the voice to be detected can be uttered by a user or by a pronunciation device. The voice to be detected is subjected to frame processing, and a plurality of voice frames can be obtained. The voice frame can be represented by at least one of MFCC features (Mel Frequency Cepstrum Coefficient), FBank features (FilterBank) or energy features. The feature vector of the voice frame can be a feature vector for describing the voice frame, and can be obtained by extracting at least one of MFCC features, FBank features or energy features using a feature extraction model.

[0061] The computer device can perform frame processing on the speech to be detected according to a preset frame length to obtain multiple speech frames, and the preset frame length can be 20 milliseconds. Then, the computer device can sort the feature vectors of each speech frame according to the order of the speech frames to obtain a feature vector sequence.

[0062] Step S404: Perform phoneme recognition based on the feature vector sequence to obtain the possibility of candidate phonemes corresponding to each speech frame.

[0063] Among them, phoneme refers to the smallest speech unit divided according to the natural attributes of speech. According to the analysis of pronunciation movements, the pronunciation of a word can be composed of one or more phonemes; for example, for Chinese characters, each Chinese character can have a corresponding relationship with the phoneme in Chinese phonetic symbols.

[0064] Phoneme recognition mainly detects the possibility of the existence of phonemes in each speech frame. The possibility of the candidate phonemes corresponding to a speech frame refers to the possibility of the candidate phonemes existing in the speech frame and the possibility of the candidate phonemes not existing in the speech frame. The possibility can be expressed as probability; the candidate phoneme possibility can include the possibility of the speech frame being phoneme-free and the possibility of various types of phonemes existing in the speech frame. Among the candidate phoneme possibilities of a speech frame, if the possibility of the phoneme-free is less than the possibility threshold, the speech frame can be considered to be a speech frame with phoneme-free. If the possibility of the phoneme-free is greater than or equal to the possibility threshold, the speech frame can be considered to be a speech frame with non-empty phonemes (i.e., a speech frame with phonemes).

[0065] In this step, after the computer device obtains the feature vector sequence corresponding to the speech frame sequence, it performs phoneme recognition based on the feature vector sequence to detect the possibility of the existence of phonemes in each speech frame, obtain the possibility of each type of phoneme existing in the speech frame and the possibility of the speech frame being a phoneme blank, and construct the possibility of candidate phonemes for each speech frame.

[0066] Step S406, filtering the speech frames without phonemes based on the possibility of the candidate phonemes corresponding to the speech frames, and obtaining the recognition phoneme sequence corresponding to the speech to be detected based on the possibility of the candidate phonemes corresponding to the speech frames with phonemes.

[0067] Among them, the speech frame with phoneme blank refers to the speech whose probability of phoneme blank is greater than or equal to the probability threshold, and the speech frame with phoneme (that is, the phoneme non-blank speech frame) refers to the speech whose probability of phoneme blank is less than the probability threshold; for example, as shown in Figure 1(b), in the range of N0 to N 1000 Among the speech frames, the speech frames with phoneme blanks include N0, N1, N2 and N 1000 The speech frames with non-empty phonemes include N3, N4 and N 999 etc. The recognition phoneme sequence includes phonemes of non-empty speech frames, such as N3, N4 and N 999 The phonemes corresponding to the non-empty speech frames of the same phonemes.

[0068] Computer equipment can filter in N0 to N 1000The phoneme-free speech frames in the speech frames are retained, and the speech frames with non-empty phonemes are obtained to obtain a sequence of speech frames with non-empty phonemes. The decoding search is performed based on the possible degree distribution of various types of phonemes in the non-empty phoneme speech frames, such as the possible degree distribution of various types of phonemes in N3, N4, N 995 、N 996 、N 998 and N 999 The possible degree distribution of each phoneme on the non-empty speech frame is Figure 3 The decoding search shown obtains the phonemes corresponding to the non-empty speech frames and forms a recognized phoneme sequence; since the decoding search at this stage is for the non-empty speech frames of the phonemes and does not include the speech frames with blank phonemes, the decoding search at this stage is called a non-frame-by-frame decoding search.

[0069] In step S408, the phoneme sequence corresponding to the target detection word selected from the recognition phoneme sequence is used as the detection word phoneme sequence, and based on the position of the speech frame sequence corresponding to the detection word phoneme sequence in the speech frame sequence of the speech to be detected, the extracted speech frame sequence corresponding to the target detection word in the speech to be detected is determined.

[0070] The target detection word is a keyword. The voice operation instruction can be used as the target detection word, such as the voice operation instruction "turn on the air conditioner" as the target detection word; the wake-up word can also be used as the target detection word, such as "Xiao Ming Xiao Ming" as the target detection word. The detection word phoneme sequence is the phoneme sequence corresponding to the target detection word. For example, if the target detection word is "turn on the air conditioner", the detection word phoneme sequence is "dak ai k ong t iao".

[0071] Extracting a speech frame sequence refers to extracting a plurality of continuous speech frames from the speech to be detected, wherein the plurality of continuous speech frames include a phoneme non-empty speech frame sequence corresponding to the detection word phoneme sequence, and a phoneme blank speech frame adjacent to the phoneme non-empty speech frame sequence corresponding to the detection word phoneme sequence.

[0072] After the computer device obtains the recognition phoneme sequence formed by the phonemes corresponding to the non-empty speech frame, if it is determined that the recognition phoneme sequence includes the detection word phoneme sequence "dak ai k ong t iao", then it can be determined that the speech frame sequence corresponding to the detection word phoneme sequence is N 100 To N 150 The sequence of non-empty speech frames between phonemes.

[0073] Next, based on N 100 To N 150 The phoneme non-empty speech frame sequence between N0 and N 1000 The position in the speech frame sequence can be determined to be located at N 100 The previous consecutive speech frames may contain the phonemes of the target detection word, or may be located in N150 The subsequent multiple consecutive speech frames may contain the phonemes of the target detection word; therefore, the phonemes located at N 100 The previous continuous multiple speech frames, N 100 To N 150 Each speech frame between and between N 150 The subsequent multiple continuous speech frames are used as extracted speech frames to obtain an extracted speech frame sequence.

[0074] Step S410, obtaining the phoneme probability distribution corresponding to each extracted speech frame in the extracted speech frame sequence.

[0075] The possible distribution of the phonemes corresponding to the extracted speech frame refers to the possible distribution of each type of phoneme in the extracted speech frame. For example, if the extracted speech frame is N 100 , then the phonemes "a", "o", "e" and "d" are all in N 100 The probability of existence is N 100 The probability distribution of the phonemes corresponding to the extracted speech frames is obtained by performing speech frame phoneme detection (ie, phoneme category detection) on the feature vectors corresponding to the extracted speech frames.

[0076] In some embodiments, a phoneme category detection model can be used to determine the probability of each type of phoneme existing in each extracted speech frame, and obtain the phoneme probability distribution of each extracted speech frame. The labels used when training the phoneme category detection model can include labels for each type of phoneme, but do not include labels representing phoneme blanks. Therefore, the phoneme category detection model can be used to compulsorily predict the phoneme corresponding to each speech frame, and a certain speech frame will not be predicted as a phoneme blank.

[0077] After the computer device performs phoneme detection on the speech frame according to the feature vector corresponding to the extracted speech frame, the possible degree distribution of the phonemes corresponding to each speech frame can be obtained.

[0078] Step S412: obtaining a speech detection result corresponding to the speech to be detected based on the phoneme probability distribution corresponding to the extracted speech frame.

[0079] The computer device obtains the phoneme probability distribution of each speech frame in each extracted speech frame sequence, and then performs decoding search based on the phoneme probability distribution of each extracted speech frame to determine the phoneme corresponding to each extracted speech frame. It should be noted that the decoding search at this stage is for each continuous speech frame, so the decoding search at this stage is called frame-by-frame decoding search.

[0080] Combination Figure 2 Introduce the above frame-by-frame decoding search process: If the extracted speech frame sequence is N 80 To N 180The computer device can generate a speech frame sequence based on N 80 To N 180 The phoneme probability distribution of each speech frame in the speech frame sequence between and the transition probability of the phonemes between adjacent speech frames (the transition probability can be given by the language model) can be constructed Figure 2 The speech decoding network (also called decoding graph) shown in FIG. 1 is then decoded and searched to obtain the maximum probability path, and the phoneme located on the maximum probability path is used as the phoneme corresponding to the speech frame. For example, if the first phoneme of the maximum probability path is "d", the phoneme "d" can be used as the phoneme corresponding to N 80 Similarly, for other phonemes of the maximum probability path, type processing can be performed to determine N 80 With N 180 The phonemes corresponding to each speech frame between the two can accurately determine the position of the target detection word in the speech and realize forced alignment of the speech.

[0081] In the above-mentioned speech detection method, since speech frames with blank phonemes are filtered based on the possibility of candidate phonemes corresponding to the speech frames, and the recognized phoneme sequence is obtained based on the possibility of candidate phonemes corresponding to the speech frames with existing phonemes, there is no need to perform subsequent processing on the speech frames with blank phonemes, thereby improving the efficiency of obtaining the recognized phoneme sequence, and also improving the efficiency of obtaining the phoneme sequence corresponding to the target detection word from the recognized phoneme sequence; then, based on the position of the speech frame sequence corresponding to the detection word phoneme sequence in the speech frame sequence of the speech to be detected, the extracted speech frame sequence corresponding to the target detection word is determined, and there is no need to perform phoneme detection on all speech frames subsequently, thereby improving efficiency; and since the extracted speech frame sequence is subjected to frame-by-frame phoneme detection, the speech detection result obtained according to the distribution of the phoneme possibility of each extracted speech frame obtained by frame-by-frame phoneme detection can accurately reflect the position of the keyword in the speech, thereby improving accuracy.

[0082] In some embodiments, the above-mentioned step S408 may include the following steps: determining the speech frame sequence corresponding to the detection word phoneme sequence at the starting position and the ending position of the speech to be detected; taking the forward position corresponding to the starting position as the first extraction position, and taking the backward position corresponding to the ending position as the second extraction position; taking the speech frame sequence located between the first extraction position and the second extraction position in the speech to be detected as the extracted speech frame sequence corresponding to the target detection word.

[0083] Among them, the starting position can be understood as the frame time corresponding to the first speech frame in the speech frame sequence corresponding to the detection word phoneme sequence in the speech to be detected, and the ending position can be understood as the frame time corresponding to the last speech frame in the speech frame sequence corresponding to the detection word phoneme sequence in the speech to be detected.

[0084] The first extraction position is the forward position of the starting position, that is, the frame time of the first speech frame in the speech to be detected that is earlier than the speech frame sequence corresponding to the detection word phoneme sequence; the second extraction position is the backward position of the ending position, that is, the frame time of the last speech frame in the speech to be detected that is later than the speech frame sequence corresponding to the detection word phoneme sequence.

[0085] For example, if the speech frame sequence corresponding to the detection word phoneme sequence is N 100 To N 150 The phoneme non-empty speech frame sequence between, where N 100 is the first speech frame in the speech frame sequence corresponding to the detection word phoneme sequence, N 150 is the last speech frame in the speech frame sequence corresponding to the detection word phoneme sequence; 100 The frame time of N, such as the 80th frame time, is used as the first extraction position; 150 The frame time, such as the 180th frame time, is used as the second extraction position, and the continuous N 80 To N 100 Each speech frame between is taken as an extracted speech frame to obtain an extracted speech frame sequence.

[0086] In the above method, the corresponding extraction position is determined based on the start and end positions of the speech frame sequence corresponding to the detection word phoneme sequence at the speech to be detected, while taking into account the phoneme detection efficiency and ensuring the accuracy of the extracted speech frame sequence.

[0087] In some embodiments, the above-mentioned first extraction position can also be the frame time of a phoneme non-empty speech frame in the speech to be detected that is earlier than and adjacent to the first speech frame in the speech frame sequence corresponding to the detection word phoneme sequence; the number of phoneme non-empty speech frames between the frame time corresponding to the first extraction position and the frame time of the first speech frame in the speech frame sequence corresponding to the detection word phoneme sequence is less than or equal to a preset value.

[0088] For example, in the speech to be detected, after performing a non-frame-by-frame decoding search for the phoneme non-empty speech frame, it is determined that the speech frame sequence corresponding to the detection word phoneme sequence is N 1100 To N 1150 The phoneme non-empty speech frame sequence between, where N 1100 is the first speech frame in the speech frame sequence corresponding to the detected word phoneme sequence.

[0089] If the preset value is 1, and the phoneme of the speech to be detected is not in the empty speech frame sequence, N 1100 The previous non-empty phoneme speech frame is N 1079 , N 1079 To N 1100 The speech frames between are all phoneme blank speech frames, N 1079The corresponding 1079th frame time and N 1100 The number of non-empty speech frames between the corresponding 1100th frame time is 0, which is less than the preset value, so it can be considered that N 1079 The corresponding 1079th frame time is earlier than and adjacent to N 1100 The frame time of the image is selected, and the 1079th frame time is used as the first extraction position.

[0090] If the preset value is 1, and the phoneme of the speech to be detected is not in the empty speech frame sequence, N 1100 The first three phonemes of the non-empty speech frame are N 1068 、N 1070 and N 1079 , N 1068 To N 1070 The speech frames between 1070 To N 1079 The speech frames between 1079 To N 1100 The speech frames between are all phoneme blank speech frames; N 1068 The corresponding 1068th frame time and N 1100 The corresponding number of non-empty phoneme speech frames between the 1100th frame time is N 1070 and N 1079 , the number of non-empty phonemes is 2, which is greater than the preset value. In this case, N 1068 The corresponding 1068th frame time is adjacent to N 1100 The frame time. N 1070 The corresponding 1070th frame time and N 1100 The corresponding number of non-empty phoneme speech frames between the 1100th frame time is N 1079 , the corresponding number of non-empty phonemes is 1, which is exactly equal to the preset value. At this time, N 1070 The corresponding 1070th frame time is adjacent to N 1100 The frame time. N 1079 The corresponding 1079th frame time and N 1100 There is no non-empty speech frame between the corresponding 1100th frame time, and the corresponding number of non-empty speech is 0, which is less than the preset value. At this time, N 1079 The corresponding 1079th frame time is adjacent to N 1100 Frame time.

[0091] In some embodiments, if there are multiple frame times of non-empty phoneme voice frames in the speech to be detected that are earlier than and adjacent to the first voice frame in the speech frame sequence corresponding to the detection word phoneme sequence, then any frame time can be selected and used as the first extraction position. For example, the frame time of non-empty phoneme voice frames in the speech to be detected that are earlier than and adjacent to the first voice frame in the speech frame sequence corresponding to the detection word phoneme sequence is N. 1070 and N 1079 At this time, you can select any frame time as the first extraction position.

[0092] In the above method, the frame time of the phoneme non-empty speech frame in the speech to be detected that is earlier than and adjacent to the first speech frame in the speech frame sequence corresponding to the detection word phoneme sequence is taken as the first extraction position, and the number of phoneme non-empty speech frames between the frame time corresponding to the first extraction position and the frame time of the first speech frame in the speech frame sequence corresponding to the detection word phoneme sequence is less than or equal to a preset value. This can avoid taking the frame time corresponding to the phoneme non-empty speech frame that is far away from the first speech frame as the first extraction position, avoid selecting too many extracted speech frames, and reduce the complexity of frame-by-frame decoding search.

[0093] In some embodiments, the second extraction position may also be a frame time of a phoneme non-empty speech frame in the speech to be detected that is later than and adjacent to the last speech frame in the speech frame sequence corresponding to the detection word phoneme sequence; the number of phoneme non-empty speech frames between the frame time corresponding to the second extraction position and the frame time of the last speech frame in the speech frame sequence corresponding to the detection word phoneme sequence is less than or equal to a preset value.

[0094] For example, in the speech to be detected, after performing a non-frame-by-frame decoding search for the phoneme non-empty speech frame, it is determined that the speech frame sequence corresponding to the detection word phoneme sequence is N 1100 To N 1150 The phoneme non-empty speech frame sequence between, where N 1500 It is the last speech frame in the speech frame sequence corresponding to the detected word phoneme sequence.

[0095] If the preset value is 1, and the phoneme of the speech to be detected is not in the empty speech frame sequence, N 1500 The next non-empty speech frame of the phoneme is N 1550 , N 1500 To N 1550 The speech frames between are all phoneme blank speech frames, N 1500 The corresponding 1500th frame time and N 1550 The number of non-empty speech frames between the corresponding 1550th frame time is 0, which is less than the preset value, so it can be considered that N 1550 The corresponding 1550th frame time is later than and adjacent to N 1550The frame time of 1550th frame is taken as the second extraction position.

[0096] If the preset value is 1, and the phoneme of the speech to be detected is not in the empty speech frame sequence, N 1500 The last three phonemes of the non-empty speech frame are N 1550 、N 1567 and N 1579 , N 1500 To N 1550 The speech frames between 1550 To N 1567 The speech frames between 1567 To N 1579 The speech frames between are all phoneme blank speech frames; N 1500 The corresponding 1500th frame time and N 1550 There is no non-empty speech frame between the corresponding 1550th frame time, and the corresponding number of non-empty speech is 0, which is less than the preset value. At this time, N 1550 The corresponding 1550th frame time is adjacent to N 1500 The frame time. N 1500 The corresponding 1500th frame time and N 1567 The number of non-empty phoneme speech frames between the corresponding 1567th frame time is N 1550 , the corresponding number of non-empty phonemes is 1, which is exactly equal to the preset value. At this time, N 1567 The corresponding 1567th frame time is adjacent to N 1500 The frame time. N 1500 The corresponding 1500th frame time and N 1579 The number of non-empty phoneme speech frames between the corresponding 1579th frame time is N 1550 and N 1567 , the number of non-empty phonemes is 2, which is greater than the preset value. In this case, N 1579 The corresponding 1579th frame time is adjacent to N 1500 Frame time.

[0097] In some embodiments, if there are multiple frame times of non-empty phoneme voice frames in the speech to be detected that are later than and adjacent to the last voice frame in the speech frame sequence corresponding to the detection word phoneme sequence, then any frame time can be selected and used as the second extraction position. For example, the frame time of non-empty phoneme voice frames in the speech to be detected that are later than and adjacent to the last voice frame in the speech frame sequence corresponding to the detection word phoneme sequence is N. 1550 and N 1567 At this time, you can select any frame time as the second extraction position.

[0098] In the above method, the frame time of the phoneme non-empty speech frame in the speech to be detected that is later than and adjacent to the last speech frame in the speech frame sequence corresponding to the detection word phoneme sequence, and the number of phoneme non-empty speech frames between the frame time corresponding to the second extraction position and the frame time of the last speech frame in the speech frame sequence corresponding to the detection word phoneme sequence is less than or equal to a preset value, which can avoid taking the frame time corresponding to the phoneme non-empty speech frame far away from the last speech frame as the second extraction position, avoid selecting too many extracted speech frames, and reduce the complexity of frame-by-frame decoding search.

[0099] In some embodiments, the above-mentioned step of taking the forward position corresponding to the starting position as the first extraction position and the backward position corresponding to the ending position as the second extraction position can specifically include: taking the number of speech frames with blank phonemes in the speech to be detected as the number of blank frames; determining the number of speech frame extensions based on the number of blank frames; the number of speech frame extensions is positively correlated with the number of blank frames; taking the forward position whose distance from the starting position is the number of speech frame extensions as the first extraction position, and taking the backward position whose distance from the ending position is the number of speech frame extensions as the second extraction position.

[0100] The number of blank frames refers to the number of speech frames with phoneme blanks between the frame time corresponding to the first speech frame and the frame time corresponding to the last speech frame, wherein the first speech frame is the first speech frame in the speech frame sequence corresponding to the detection word phoneme sequence, and the last speech frame is the last speech frame in the speech frame sequence corresponding to the detection word phoneme sequence. For example, the speech frame sequence corresponding to the detection word phoneme sequence is N 1100 To N 1150 The phoneme non-empty speech frame sequence between, where N 1100 is the first speech frame in the speech frame sequence corresponding to the detection word phoneme sequence, N 1500 is the last speech frame in the speech frame sequence corresponding to the detection word phoneme sequence; correspondingly, N 1100 The corresponding 1100th frame time to N 1150 The number of phoneme blank speech frames between the corresponding 1500th frame moments is taken as the number of blank frames.

[0101] The extended number of speech frames can be the number of speech frames between the frame time corresponding to the first extraction position and the frame time corresponding to the above-mentioned first speech frame; since the frame time corresponding to the first extraction position is earlier than the frame time corresponding to the first speech frame, the speech frame corresponding to the first extraction position is earlier than the first speech frame, the extracted speech frame is equivalent to the speech frame forward expanded on the basis of the first speech frame, and therefore, the number of speech frames can be called the extended number of speech frames.

[0102] The number of extended speech frames may also be the number of speech frames between the frame time corresponding to the second extraction position and the frame time corresponding to the last speech frame mentioned above; since the frame time corresponding to the second extraction position is later than the frame time corresponding to the last speech frame, the speech frame corresponding to the second extraction position is later than the last speech frame, and the extracted speech frame is equivalent to the speech frame obtained by backward expansion based on the last speech frame. Therefore, the number of speech frames may be called the number of extended speech frames.

[0103] The number of voice frame extensions is positively correlated with the number of blank frames. The larger the number of blank frames, the larger the number of voice frame extensions. For example, if the number of blank frames is 5 and 10 respectively, according to the positive correlation between the number of blank frames and the number of voice frame extensions, the number of voice frames corresponding to the number of blank frames being 5 is less than the number of voice frames corresponding to the number of blank frames being 10.

[0104] For example, if the number of speech frame extensions is 10, and the speech frame sequence corresponding to the detection word phoneme sequence is N 100 To N 150 The phoneme non-empty speech frame sequence between, where N 100 is the first speech frame in the speech frame sequence corresponding to the detection word phoneme sequence, N 150 is the last speech frame in the speech frame sequence corresponding to the detection word phoneme sequence; the forward position with a distance from the starting position equal to the number of speech frame extensions can be used as the first extraction position, that is, N 100 The 10th speech frame N before 90 The corresponding position is taken as the forward position with a distance from the starting position equal to the number of speech frame extensions, and N 90 The corresponding position is taken as the first extraction position; the backward position with a distance from the end position equal to the number of speech frame extensions can also be taken as the second extraction position, that is, N 150 The 10th speech frame N 160 The corresponding position is taken as the backward position with a distance from the end position equal to the number of speech frame extensions, and N 160 The corresponding position is used as the second extraction position.

[0105] In the above method, the more blank frames there are, the more speech frames the "sequence to sequence" model cannot judge. Therefore, the more speech frames are extended, the more phoneme blank speech frames are forcibly assigned phoneme category labels, thereby improving the accuracy of forced speech alignment.

[0106] In some embodiments, the possibility of candidate phonemes includes the possibility corresponding to phoneme blanks; the above step S406 includes: taking speech frames whose possibility corresponding to phoneme blanks is greater than a possibility threshold as speech frames with phoneme blanks; filtering speech frames with phoneme blanks to obtain decoded speech frames, and obtaining a speech decoding network formed based on the possibility of candidate phonemes corresponding to the decoded speech frames; decoding based on the speech decoding network to obtain a target decoding path, arranging the phonemes passed by the target decoding path in the order of the path, and obtaining a recognition phoneme sequence corresponding to the speech to be detected.

[0107] Among them, the decoded speech frame is a phoneme non-empty speech frame. Since in the non-frame-by-frame decoding search stage, only the phoneme non-empty speech frame is decoded and searched, and the phoneme blank speech frame is not decrypted and searched, the phoneme non-empty speech frame can be called a decoded speech frame.

[0108] The probability of the candidate phonemes corresponding to the decoded speech frame can be output by the "sequence to sequence" model. If it is output by the "sequence to sequence" model, the probability of the candidate phonemes includes the probability that each type of phoneme exists in the decoded speech frame and the probability that the decoded speech frame is a phoneme-free speech frame.

[0109] Based on the probability of candidate phonemes corresponding to the decoded speech frame, the following Figure 3 The speech decoding network shown in the figure, the probability corresponding to the directed edges between the states connecting different decoded speech frames in the speech decoding network can be given by the language model. The target decoding path is the path with the highest probability in the speech decoding network.

[0110] The computer device forms a method based on the probability of candidate phonemes corresponding to the decoded speech frame. Figure 3 After the speech decoding network shown in the figure is constructed, a decoding search can be performed on the speech decoding network, the maximum probability path is taken as the target decoding path, the phonemes passed by the target decoding path are arranged in the order of the path, and the recognition phoneme sequence corresponding to the speech to be detected is obtained.

[0111] In the above manner, a speech decoding network is constructed according to the probability of candidate phonemes of the decoded speech frame obtained by filtering the phoneme blank speech frame, thereby reducing the decoding complexity and improving the recognition efficiency of the target detection word.

[0112] In some embodiments, the above step of extracting features from each speech frame in the speech to be detected to obtain feature vectors corresponding to the speech frames may specifically include: inputting each speech frame in the speech to be detected into a feature extraction submodel of a trained speech detection model for feature extraction to obtain feature vectors corresponding to the speech frames. The above step of performing phoneme recognition based on a feature vector sequence to obtain the possibility of candidate phonemes corresponding to each speech frame may specifically include: inputting the feature vector sequence into a speech sequence recognition submodel for phoneme recognition to obtain the possibility of candidate phonemes corresponding to each speech frame. The above step of obtaining the possibility distribution of phonemes corresponding to each extracted speech frame in the extracted speech frame sequence may specifically include: extracting feature vectors corresponding to the extracted speech frames from the feature vector sequence, inputting the feature vectors corresponding to the extracted speech frames into the speech frame phoneme detection submodel for phoneme recognition; using the possibility distribution of each type of phonemes output by the speech frame phoneme detection submodel in each extracted speech frame as the possibility distribution of phonemes corresponding to each extracted speech frame.

[0113] Among them, Figure 5 As shown, the trained speech detection model includes a feature extraction sub-model, a speech sequence recognition sub-model and a speech frame phoneme detection sub-model respectively connected to the feature extraction sub-model.

[0114] The feature extraction sub-model is used to extract features from speech frames to obtain feature vectors of speech frames.

[0115] The speech sequence recognition submodel is a "sequence to sequence" model, and the speech frame phoneme detection submodel is a phoneme category detection model. The training of these two models is introduced as follows: the labels used in training the above-mentioned "sequence to sequence" model include labels that represent phoneme blanks. For speech frames that are difficult to predict the corresponding phonemes, the "sequence to sequence" model will be labeled with phoneme blanks, and will not forcibly assign a label of a certain type of phoneme to the speech frame to reduce the model loss value; the labels used in training the phoneme category detection model can include labels of various types of phonemes, but do not include labels that represent phoneme blanks. Therefore, the phoneme category detection model can be used to forcibly predict the phoneme corresponding to each speech frame, and a certain speech frame will not be predicted as a speech frame with a phoneme blank. For other contents of the two models, the "sequence to sequence" model and the phoneme category detection model, please refer to the above introduction, which will not be repeated here.

[0116] The computer device inputs each speech frame in the speech to be detected into the feature extraction submodel of the trained speech detection model for feature extraction, and obtains the feature vector corresponding to the speech frame; and the computer device can also input the feature vector sequence into the speech sequence recognition submodel for phoneme recognition, and obtain the possibility of the candidate phoneme corresponding to each speech frame. In addition, the computer device can extract the feature vector corresponding to the extracted speech frame from the feature vector sequence, and input the feature vector corresponding to the extracted speech frame into the speech frame phoneme detection submodel for phoneme recognition; the possibility distribution of each type of phoneme output by the speech frame phoneme detection submodel in each extracted speech frame is used as the phoneme possibility distribution corresponding to each extracted speech frame.

[0117] In the above method, the same feature extraction sub-model performs feature extraction on the speech frame to obtain the feature vector of the speech frame, and the "sequence to sequence" model and the phoneme category detection model perform phoneme processing on the feature vector of the speech frame respectively to improve processing efficiency.

[0118] In some embodiments, the step of obtaining a trained speech detection model may specifically include: obtaining training speech; inputting the training speech into a feature extraction submodel to be trained for feature extraction, obtaining training feature vectors corresponding to each training speech frame, and sorting the training feature vectors corresponding to each training speech frame in the order of the corresponding speech frames to obtain a training vector sequence; inputting the training vector sequence into a speech sequence recognition submodel to be trained for phoneme recognition, and obtaining a phoneme sequence recognition result; obtaining a first model loss value based on a first difference between the phoneme sequence recognition result and a standard recognition result corresponding to the training speech; the first model loss value is positively correlated with the first difference; inputting each training feature vector into the speech frame audio to be trained; The phoneme detection submodel performs phoneme recognition to obtain the speech frame phoneme detection result corresponding to the speech frame phoneme detection submodel; the second model loss value is obtained based on the second difference between the speech frame phoneme detection result and the standard recognition result corresponding to the training speech; the first model loss value is positively correlated with the second difference; the target model loss value is obtained based on the first model loss value and the second model loss value; based on the target model loss value, the parameters of the feature extraction submodel to be trained, the speech frame phoneme detection submodel to be trained and the speech sequence recognition submodel to be trained are adjusted, and the feature extraction submodel after the parameter adjustment, the speech frame phoneme detection submodel after the parameter adjustment and the speech sequence recognition submodel after the parameter adjustment form a trained speech detection model.

[0119] Among them, when training the speech sequence recognition sub-model, the input used can be the training vector sequence corresponding to the speech frame sequence, and the output used can be the training phoneme sequence corresponding to the speech frame sequence. The training vector sequence and the training phoneme sequence do not need to correspond one-to-one in time, that is, there is no need to assign a corresponding phoneme label to each speech frame in the speech frame sequence.

[0120] The phoneme sequence recognition result obtained by the speech sequence recognition submodel to be trained for phoneme recognition is the phoneme sequence existing in the training speech; the standard recognition result of the training speech compared with the phoneme sequence recognition result is the training phoneme sequence, and the first difference characterizes the degree of difference between the phoneme sequence output by the speech sequence recognition submodel and the training phonemes.

[0121] When training the speech frame phoneme detection sub-model, the training vector sequence and the training phoneme sequence need to correspond one-to-one in time, that is, each speech frame in the speech frame sequence needs to be assigned a corresponding phoneme label; the loss function of the speech frame phoneme detection sub-model can be a CE function.

[0122] The speech frame phoneme detection result obtained by the speech frame phoneme detection submodel to be trained for phoneme recognition is the predicted phoneme label assigned to each training speech frame; the standard recognition result of the training speech compared with the speech frame phoneme detection result is the true phoneme label of each training speech frame, and the second difference characterizes the degree of difference between the speech frame phoneme detection result output by the speech frame phoneme detection submodel and the true phoneme label of the training speech frame.

[0123] The target model loss value may be the sum of the first model loss value and the second model loss value; the target model loss value may also be the sum of the first model loss value and the weighted second model loss value. For example, if the speech sequence recognition submodel is a Transducer, and the speech frame detection submodel is a model using the CE function as the loss function, then the target model loss value may be L=L Transducer +αL CE , α represents the weight assigned to the loss value of the second model.

[0124] The computer device inputs the training speech into the feature extraction submodel to be trained for feature extraction, obtains the training feature vectors corresponding to each training speech frame, and sorts the training feature vectors corresponding to each training speech frame according to the corresponding speech frame order to obtain a training vector sequence; inputs the training vector sequence into the speech sequence recognition submodel to be trained for phoneme recognition, and obtains the phoneme sequence recognition result; obtains the first model loss value based on the first difference between the phoneme sequence recognition result and the standard recognition result corresponding to the training speech; the first model loss value is positively correlated with the first difference; inputs each training feature vector into the speech frame phoneme detection submodel to be trained for phoneme recognition, and obtains the speech sequence. The speech frame phoneme detection result corresponding to the speech frame phoneme detection submodel; the second model loss value is obtained based on the second difference between the speech frame phoneme detection result and the standard recognition result corresponding to the training speech; the first model loss value is positively correlated with the second difference; the target model loss value is obtained based on the first model loss value and the second model loss value; based on the target model loss value, the parameters of the feature extraction submodel to be trained, the speech frame phoneme detection submodel to be trained and the speech sequence recognition submodel to be trained are adjusted, and the feature extraction submodel after the parameter adjustment, the speech frame phoneme detection submodel after the parameter adjustment and the speech sequence recognition submodel after the parameter adjustment form a trained speech detection model.

[0125] In the above method, the training vector sequence is used to construct a speech sequence recognition sub-model and a speech frame detection sub-model respectively, and the model loss value is determined according to the difference between the corresponding prediction result and the standard recognition result. The parameters of the speech sequence recognition sub-model and the speech frame detection sub-model are adjusted according to the target model loss value determined by the first model loss value and the second model loss value to improve the accuracy of the model prediction.

[0126] In some embodiments, the above-mentioned step of inputting the feature vector sequence into the speech sequence recognition sub-model for phoneme recognition to obtain the possibility of candidate phonemes corresponding to each speech frame can specifically include: obtaining the current feature vector in the feature vector sequence in sequence according to the order of the feature vector sequence; obtaining the phoneme representation vector corresponding to the previous feature vector corresponding to the current feature vector; inputting the phoneme representation vector corresponding to the previous feature vector and the current feature vector into the speech sequence recognition sub-model for phoneme recognition to obtain the possibility of candidate phonemes corresponding to the current speech frame corresponding to the current feature vector.

[0127] The phoneme representation vector corresponding to the previous feature vector is the representation vector corresponding to the phoneme obtained by inputting the previous feature vector into the speech sequence recognition sub-model for phoneme recognition.

[0128] The feature extraction sub-model can be built based on the Encoder, the speech frame phoneme detection sub-model can be built based on the Linear layer, and the speech sequence recognition sub-model can be built based on the Predictor and the Joint layer. Figure 6 The model shown in the figure. The encoder can be a recurrent neural network, such as LSTM (Long Short-Term Memory), which accepts the audio feature input at time t and outputs the acoustic hidden layer representation. The encoder can also be a network based on FSMN. The predictor can be a recurrent neural network, such as LSTM, which accepts the non-empty output label of the previous model. Output text hidden layer representation, Predictor can also be CNN network (Convolutional Neural Networks) to improve computing efficiency. Joint can be a fully connected neural network, such as linear layer + activation unit, which sums the acoustic representation and text representation after linear transformation, outputs the hidden unit representation, and finally converts it into probability distribution through softmax.

[0129] The computer device sequentially obtains the current feature vector in the feature vector sequence in the order of the feature vector sequence, and then obtains the phoneme representation vector corresponding to the previous feature vector corresponding to the current feature vector; the phoneme representation vector corresponding to the previous feature vector and the current feature vector are input into the speech sequence recognition submodel for phoneme recognition, and the possibility of candidate phonemes corresponding to the current speech frame corresponding to the current feature vector is obtained.

[0130] In the above manner, the phoneme representation vector corresponding to the previous feature vector and the current feature vector are used to perform phoneme recognition, which can improve the prediction accuracy of the possibility of the candidate phoneme corresponding to the current speech frame corresponding to the current feature vector.

[0131] In some embodiments, the step of obtaining a phoneme representation vector corresponding to a previous feature vector includes: inputting the previous feature vector into a speech sequence recognition sub-model for phoneme recognition to obtain the possibility of a candidate phoneme corresponding to the previous feature vector; selecting the candidate phoneme with the largest corresponding possibility from the candidate phoneme set as the target phoneme; and using the phoneme representation vector corresponding to the target phoneme as the phoneme representation vector corresponding to the previous feature vector.

[0132] The possibility of the candidate phoneme corresponding to the previous feature vector includes the possibility of each candidate phoneme in the candidate phoneme set, and the phoneme representation vector corresponding to the target phoneme includes the phonemes of the accumulated multiple feature vectors, such as Figure 6 shown

[0133] After the computer device inputs the previous feature vector into the speech sequence recognition sub-model for phoneme recognition, it uses the candidate phoneme possibility output by the speech sequence recognition sub-model as the candidate phoneme possibility of the previous feature vector, selects the corresponding candidate phoneme with the largest possibility from the candidate phoneme set as the target phoneme, and uses the phoneme representation vector corresponding to the target phoneme as the phoneme representation vector corresponding to the previous feature vector.

[0134] In the above method, the candidate phoneme with the greatest probability in the candidate phoneme set is used as the target phoneme, and the phoneme representation vector corresponding to the previous feature vector is obtained to improve the prediction accuracy of the current feature vector.

[0135] In some embodiments, after the phoneme representation vector corresponding to the previous feature vector and the current feature vector are input into the speech sequence recognition sub-model for phoneme recognition and the possibility of the candidate phoneme corresponding to the current speech frame corresponding to the current feature vector is obtained, the computer device can also perform the following steps: when the feature vector in the feature vector sequence has not been phonetically recognized, return to the step of sequentially acquiring the current feature vector in the feature vector sequence in the order of the feature vector sequence to perform phoneme recognition on the feature vector that has not been phonetically recognized until the phoneme recognition of the feature vector in the feature vector sequence is completed.

[0136] If there are still some feature vectors in the feature vector sequence that have not been input into the speech sequence recognition sub-model for phoneme recognition, then the feature vectors in the feature vector sequence have not been fully recognized by phonemes. At this time, the feature vectors that have not been recognized by phonemes can be used as the current feature vectors and processed according to the above-mentioned processing method for the current feature vector until all the feature vectors in the feature vector sequence have been recognized by phonemes.

[0137] In the above method, if the feature vector in the feature vector sequence has not been phoneme recognized, the feature vector that has not been phoneme recognized will be used as the current feature vector and phoneme recognition will be performed to ensure that all feature vectors in the feature vector sequence are recognized, avoid omissions, and improve the accuracy of forced speech alignment.

[0138] In some embodiments, the above-mentioned step of obtaining a speech detection result corresponding to the speech to be detected based on the extracted phoneme possibility distribution corresponding to the speech frame can specifically include: obtaining the detection word position of the target detection word in the speech to be detected based on the extracted phoneme possibility distribution corresponding to the speech frame; obtaining the speech at the backward position corresponding to the detection word position in the speech to be detected as the instruction detection speech; performing speech instruction detection on the shown instruction detection speech to obtain the target speech instruction corresponding to the instruction detection speech, so as to perform speech control based on the target speech instruction.

[0139] The target detection word may be a wake-up word. The computer device may obtain the voice corresponding to the wake-up word issued by the user and perform voice command detection to determine the voice command issued by the user. The target voice command may be a voice control command, such as "turn on the air conditioner" and "send a message".

[0140] The detection word position is the position of the target detection word in the speech, which can be reflected as the temporal correspondence between the phoneme sequence of the target detection word and the speech frame sequence. The command detection speech is the speech to be detected that may contain the target speech command.

[0141] After the computer device determines the position of the target detection word in the speech by frame-by-frame decoding search, it uses the speech at the backward position after the detection word position as the command detection speech to determine whether the command detection speech includes the target voice command; if the computer device determines that the command detection speech includes the target voice command, voice control is performed according to the target voice command, such as turning on the air conditioner.

[0142] In the above method, the speech at the backward position corresponding to the position of the target detection word in the speech to be detected is used as the command detection speech, avoiding the detection of other invalid speech and reducing the energy consumption of the device.

[0143] The present application also provides a voice detection method, which mainly includes the following steps:

[0144] Step S1, inputting each speech frame in the speech to be detected into the feature extraction submodel of the trained speech detection model to extract features, and obtaining a feature vector corresponding to the speech frame;

[0145] Step S2: sort the feature vectors corresponding to each speech frame according to the corresponding speech frame order to obtain a feature vector sequence

[0146] Step S3, inputting the feature vector sequence into the speech sequence recognition sub-model to perform phoneme recognition, and obtaining the possibility of candidate phonemes corresponding to each speech frame;

[0147] Step S4, taking the speech frames whose probability corresponding to the phoneme blank is greater than the probability threshold as the speech frames with the phoneme blank;

[0148] Step S5, filtering speech frames with blank phonemes to obtain decoded speech frames, and acquiring a speech decoding network formed based on the probability of candidate phonemes corresponding to the decoded speech frames;

[0149] Step S6, decoding is performed based on the speech decoding network to obtain a target decoding path, and the phonemes passed by the target decoding path are arranged in the order of the path to obtain a recognition phoneme sequence corresponding to the speech to be detected;

[0150] Step S7: the phoneme sequence corresponding to the target detection word selected from the recognition phoneme sequence is used as the detection word phoneme sequence;

[0151] Step S8, determining the starting position and the ending position of the speech frame sequence corresponding to the detection word phoneme sequence in the speech to be detected;

[0152] Step S9, taking the number of speech frames with blank phonemes in the speech to be detected as the number of blank frames;

[0153] Step S10, determining the number of voice frame extensions based on the number of blank frames; the number of voice frame extensions is positively correlated with the number of blank frames;

[0154] Step S11, taking a forward position whose distance from the start position is the number of speech frame extensions as a first extraction position, and taking a backward position whose distance from the end position is the number of speech frame extensions as a second extraction position;

[0155] Step S12, taking the speech frame sequence located between the first extraction position and the second extraction position in the speech to be detected as the extracted speech frame sequence corresponding to the target detection word;

[0156] Step S13, extracting a feature vector corresponding to the extracted speech frame from the feature vector sequence, and inputting the feature vector corresponding to the extracted speech frame into the speech frame phoneme detection sub-model for phoneme recognition;

[0157] Step S14, using the probability distribution of each type of phoneme in each extracted speech frame output by the speech frame phoneme detection sub-model as the phoneme probability distribution corresponding to each extracted speech frame;

[0158] Step S15, obtaining the detection word position of the target detection word in the to-be-detected speech based on the possible degree distribution of the phonemes corresponding to the extracted speech frame;

[0159] Step S16, obtaining the speech at the backward position corresponding to the detection word position in the speech to be detected as the command detection speech;

[0160] Step S17, performing voice command detection on the command detection voice to obtain a target voice command corresponding to the command detection voice, so as to perform voice control based on the target voice command.

[0161] In the present embodiment, since speech frames with blank phonemes are filtered based on the possibility of candidate phonemes corresponding to the speech frames, and the recognized phoneme sequence is obtained based on the possibility of candidate phonemes corresponding to the speech frames with existing phonemes, there is no need to perform subsequent processing on the speech frames with blank phonemes, thereby improving the efficiency of obtaining the recognized phoneme sequence, and also improving the efficiency of obtaining the phoneme sequence corresponding to the target detection word from the recognized phoneme sequence; then, based on the position of the speech frame sequence corresponding to the detection word phoneme sequence in the speech frame sequence of the speech to be detected, the extracted speech frame sequence corresponding to the target detection word is determined, and there is no need to perform phoneme detection on all the speech frames subsequently, thereby improving efficiency; and since the extracted speech frame sequence is subjected to frame-by-frame phoneme detection, the speech detection result obtained according to the distribution of the phoneme possibility of each extracted speech frame obtained by frame-by-frame phoneme detection can accurately reflect the position of the keyword in the speech, thereby improving accuracy.

[0162] In order to better understand the above method, an application example of the speech detection method of the present application is described in detail below. The speech sequence recognition submodel of this application example is a Transducer model, which belongs to a keyword (i.e., target detection word) detection system based on the Transducer model. In this application example, the model training of the keyword detection system adopts a multi-task learning (MTL) strategy, that is, simultaneously minimizing the loss value (Transducer Loss) of the loss function of the Transducer model and the loss value of the CE loss function, taking into account both accuracy and efficiency while ensuring the accuracy and real-time performance of the keyword detection system.

[0163] This application example mainly includes:

[0164] (1) Adopting the MTL strategy, optimizing Transducer Loss and CE Loss simultaneously, it realizes “sequence to sequence” modeling and frame-by-frame correspondence modeling;

[0165] (2) Combining the linear search graph and the frame-by-frame phoneme posterior probability, the speech-text alignment is efficiently completed and the keyword location is extracted.

[0166] The solution provided in this application example has the following characteristics:

[0167] (1) Using a larger granularity modeling unit, the phoneme, the output of the acoustic model has actual pronunciation meaning;

[0168] (2) Since the output of Transducer has the concept of "empty output", when decoding, speech frames with empty output are directly skipped for decoding, which improves decoding efficiency;

[0169] (3) The training process is simple, forced alignment is an optional process, and Transducer training does not require a temporal correspondence between speech and text.

[0170] This application example can be used in the in-vehicle offline voice custom wake-up system. Custom wake-up plays an important role in in-vehicle voice assistants, and also has high requirements for the accuracy of keyword detection. Through custom wake-up, some quick tasks and control instructions in the voice assistant can be quickly implemented, such as sending and canceling information and controlling in-vehicle equipment. The entire process of this application example is as follows Figure 7 As shown, the following steps are included:

[0171] Step S702, the vehicle voice wake-up system collects the voice (i.e., audio) in the vehicle in real time;

[0172] Step S704, the vehicle-mounted voice wake-up system performs frame processing on the voice to obtain multiple voice frames and extract features of each voice frame;

[0173] In step S706, the vehicle voice wake-up system uses the transducer to perform acoustic calculations on each voice frame to obtain an acoustic score output by the transducer. The acoustic score includes the probability that the voice frame is a phoneme blank and the probability that each type of phoneme exists in the voice frame.

[0174] Step S708: The vehicle voice wake-up system constructs a decoding graph (equivalent to a voice decoding network) based on the acoustic scores of the voice frames with non-empty phonemes, and performs a non-frame-by-frame decoding search.

[0175] In step S710, the vehicle-mounted voice wake-up system determines whether there are keywords in the voice based on the non-frame-by-frame decoding search results to make a wake-up judgment; if there are keywords in the voice, wake-up is performed and the process proceeds to step S712; if there are no keywords in the voice, wake-up is not performed and the process returns to step S702 for voice collection.

[0176] Step S712: After the vehicle-mounted voice wake-up system determines that there are keywords in the voice, it uses the model constructed by the CE loss function to predict the acoustic scores of multiple consecutive voice frames, and constructs a decoding graph based on the acoustic scores of multiple consecutive voice frames to perform frame-by-frame decoding search.

[0177] Among them, for the Transducer model, given the input sequence: x = (x1, x2, ..., x T )∈x * And the output sequence: y = (y1, y2, ..., y U )∈y * .

[0178] x * and * The set of input sequence and output sequence, x t ∈x and y u∈y are all real vectors, and x and y represent the input and output spaces respectively. For example, in this application example, the modeling unit of the Transducer model is the phoneme, the input sequence x is a feature vector sequence (such as FBank feature, MFCC feature), x t represents the feature vector at time t; the output sequence y is the phoneme sequence, y u Represents the phoneme at step u.

[0179] The ultimate goal of Transducer model training is to maximize the following objective function:

[0180]

[0181] That is, given the input and output, maximize the sum of all possible path probabilities. The loss function for Transducer model training is: L Transducer =-p(y|x).

[0182] As mentioned above, the custom wake-up task is to detect whether there is a wake-up word in the speech, and on the other hand, it is also necessary to accurately obtain the position where the wake-up word appears and ends in the speech. Due to the output delay problem of the Transducer model, the timestamp of the phoneme output by the network cannot correspond to the actual position of the speech, and the delay is random. Therefore, when the output of the Transducer model is used for forced alignment and speech extraction, large errors will occur. In order to achieve the purpose of accurately extracting the keyword position, this application example adopts Figure 6 In the model structure shown, a linear layer is set after the encoder output to predict the posterior probability of a phoneme generated in the current frame.

[0183] The model in this application example can be regarded as a multi-task learning model, which simultaneously learns frame-by-frame phoneme prediction and sequence-level phoneme prediction. The sub-model corresponding to the linear layer uses the frame-level CE loss function L CE Finally, the model is trained by interpolating two loss functions: L = L Transducer +αL CE .

[0184] In general, the decoding and forced alignment of this application example mainly includes three parts: wake-up registration, wake-up determination and wake-up alignment. The following is a detailed introduction to each part. Among them, the wake-up word is a keyword.

[0185] 1. Wake up registration:

[0186] Viterbi decoding search usually uses WFST and acoustic probability as input, and performs Viterbi search and beamcutoff on the acoustic probability frame by frame on WFST (Weighted Finite State Transducer), which can quickly find the path with the best acoustic and linguistic probabilities. This path is called BEST-LATTICE.

[0187] In this application example, the wake-up-free engine uses a registration method to achieve the purpose of quickly switching the wake-up word, and divides the ROOT-WFST required for the wake-up-free into GARBAGE (such as Figure 8 The GARBAGE part is used to absorb speech frames without wake-up words, while the KEYWORDS part is used to absorb speech frames with wake-up words. When training WFST, it is difficult to determine what wake-up words there will be, so a slot of $KEYWORDS will be left in ROOT-WFST for registering wake-up words. When the KEYWORDS-WFST represented by the wake-up word is connected to the entire ROOT-WFST, as shown in the figure. Figure 8 shown.

[0188] In addition, when registering a wake-up word, in order to distinguish between wake-up words and non-wake-up words, the part of the output word list that is greater than or equal to KEYWORD_START is called the output of the wake-up word, and the part that is less than KEYWORD_START is called GARGAGE output.

[0189] 2. Wake-up judgment:

[0190] The output of Transducer is the probability of the existence of various types of phonemes in the speech frame and the probability of the speech frame being blank with phonemes. This application example inputs the probability distribution of speech frames with non-empty phonemes into Viterbi search decoding, and can obtain the optimal result sequence of acoustic and language scores, namely BEST-LATTICE. In the LATTICE sequence, the element at each position consists of three parts: ILABEL (input symbol), OLABEL (output symbol) and WEIGHT (weight), where ILABEL represents the corresponding phoneme, OLABEL represents the decoding result, and WEIGHT represents the overall cost of the sequence.

[0191] Through the optimal LATTICE, it is determined whether the part OLABEL>=KEYWORD_START appears. If so, the part is taken out to obtain the result of the wake-up word.

[0192] In the actual process, it is also necessary to judge the confidence of the wake-up result, that is, to check whether the ratio of the acoustic probability of the wake-up word part to the optimal acoustic probability meets the set threshold.

[0193] 3. Wake-up alignment:

[0194] In voice wake-up, accurate alignment information of the wake-up word is required to segment the voice, but due to the large uncertain delay of the Transducer output, it cannot meet the alignment requirements. Therefore, this application example uses the frame-level CE function output, which can output the optimal acoustic sequence with a high probability, and because the CE output is frame-aligned, in an ideal case, the alignment information of the frame-level CE output will be controlled within the frame duration.

[0195] In order to take into account both accuracy and alignment accuracy, when being awakened, by obtaining the awakened BEST-LATTICE, an ALIGNMENT-WFST with SIL ("SIL" means silence) and spin is constructed, and the phoneme probability distribution sequence of multiple consecutive speech frames of the CE model and ALIGNMENT-WFST are input into the Viterbi decoding, and a wake-up LATTICE can also be obtained. Through the wake-up word extraction method mentioned in the second part, accurate wake-up word alignment information can be obtained.

[0196] ALIGNMENT-WFST is based on BEST-LATTICE to build WFST. Fig. 9 As shown, BEST-LATTICE is divided into two parts: GARBAGE (GA) and KEYWORDS. GARBAGE will form several GA GROUPs according to the pronunciation relationship, and KEYWORDS will also become a GROUP. The GROUPs are connected by spinning SIL, and the GA will also perform spin operations inside to absorb the same pronunciation. KEYWORDS will also be divided into multiple GROUPs, and the GROUPs are connected by spinning SIL, and the GROUPs will also perform spin operations inside to absorb the same pronunciation. Fig. 9 The dashed spin in the figure refers to the spin of the elements in the GROUP.

[0197] After the model training is completed, libtorch is used to quantize and deploy the model. The Android version of libtorch uses the QNNPACK library (Quantized Neural Networks PACKage) to perform INT8 matrix calculations, which greatly accelerates the matrix operation speed. The model is trained using pytorch, and then the model is quantized after training, that is, the model parameters are quantized to INT8, and INT8 matrix multiplication is used to accelerate the calculation. The quantized model is exported for forward inference in the C++ environment, such as Fig.10 shown.

[0198] Compared with traditional voice wake-up systems, the voice wake-up system provided in this application example has advantages in accuracy and CPU (central processing unit) occupancy rate:

[0199] (1) In terms of accuracy, in a relatively noisy environment, the solution proposed in this application example significantly exceeds the existing model;

[0200] (2) In terms of CPU usage, the quantification and deployment method adopted in this application example also reduces the CPU utilization.

[0201] Among them, Table 1 shows the wake-up rate (%) comparison results of the voice wake-up system based on DNN-HMM and the voice wake-up system based on the improved Transducer model provided by this application example:

[0202] Model Test set 1 Test Set 2 Test set 3 DNN-HMM 91.08 84.44 82.52 Transducer 96.49 96.33 95

[0203] Table 1

[0204] Table 2 shows the CPU usage comparison results of the voice wake-up system based on DNN-HMM and the voice wake-up system based on the improved Transducer model provided in this application example:

[0205] Model CPU usage (peak) DNN-HMM 21.31％ Transducer 19.94％

[0206] Table 2

[0207] It should be understood that although Figure 1(b) to Figure 10 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1(b) to Figure 10 At least part of the steps may include multiple steps or multiple stages. These steps or stages are not necessarily performed at the same time, but can be performed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed in turn or alternately with other steps or at least part of the steps or stages in other steps.

[0208] In some embodiments, Fig.11 As shown, a speech detection device is provided, comprising:

[0209] The feature vector processing module 1102 is used to extract features from each speech frame in the speech to be detected, obtain feature vectors corresponding to the speech frames, and sort the feature vectors corresponding to each speech frame according to the corresponding speech frame sequence to obtain a feature vector sequence;

[0210] A candidate phoneme possibility acquisition module 1104 is used to perform phoneme recognition based on the feature vector sequence to obtain the candidate phoneme possibility corresponding to each of the speech frames;

[0211] The phoneme sequence acquisition module 1106 is used to filter the speech frames without phonemes based on the possibility of the candidate phonemes corresponding to the speech frames, so as to obtain the recognition phoneme sequence corresponding to the speech to be detected based on the possibility of the candidate phonemes corresponding to the speech frames with phonemes;

[0212] The speech frame extraction module 1108 is used to use the phoneme sequence corresponding to the target detection word selected from the recognition phoneme sequence as the detection word phoneme sequence, and determine the extracted speech frame sequence corresponding to the target detection word in the speech to be detected based on the position of the speech frame sequence corresponding to the detection word phoneme sequence in the speech frame sequence of the speech to be detected;

[0213] The phoneme probability distribution acquisition module 1110 is used to acquire the phoneme probability distribution corresponding to each extracted speech frame in the extracted speech frame sequence; the phoneme probability distribution corresponding to the extracted speech frame is obtained by performing speech frame phoneme detection according to the feature vectors corresponding to the extracted speech frame;

[0214] The speech detection result acquisition module 1112 is used to obtain the speech detection result corresponding to the speech to be detected based on the phoneme probability distribution corresponding to the extracted speech frame.

[0215] In some embodiments, the speech frame extraction module 1108 is also used to determine the starting position and the ending position of the speech frame sequence corresponding to the detection word phoneme sequence of the speech to be detected; the forward position corresponding to the starting position is used as the first extraction position, and the backward position corresponding to the ending position is used as the second extraction position; the speech frame sequence located between the first extraction position and the second extraction position in the speech to be detected is used as the extracted speech frame sequence corresponding to the target detection word.

[0216] In some embodiments, the speech frame extraction module 1108 is also used to take the number of speech frames with blank phonemes in the speech to be detected as the number of blank frames; determine the number of speech frame extensions based on the number of blank frames; the number of speech frame extensions is positively correlated with the number of blank frames; take the forward position whose distance from the starting position is the number of speech frame extensions as the first extraction position, and take the backward position whose distance from the ending position is the number of speech frame extensions as the second extraction position.

[0217] In some embodiments, the candidate phoneme likelihood includes the likelihood corresponding to a phoneme blank; the phoneme sequence acquisition module 1106 is further used to take a speech frame whose likelihood corresponding to a phoneme blank is greater than a likelihood threshold as a phoneme blank speech frame; filter the phoneme blank speech frames to obtain decoded speech frames, and obtain a speech decoding network formed based on the candidate phoneme likelihood corresponding to the decoded speech frames; perform decoding based on the speech decoding network to obtain a target decoding path, arrange the phonemes passed by the target decoding path in path order, and obtain a recognition phoneme sequence corresponding to the speech to be detected.

[0218] In some embodiments, the feature vector processing module 1102 is also used to input each speech frame in the speech to be detected into the feature extraction submodel of the trained speech detection model for feature extraction, so as to obtain the feature vector corresponding to the speech frame; the trained speech detection model includes a feature extraction submodel and a speech sequence recognition submodel and a speech frame phoneme detection submodel respectively connected to the feature extraction submodel; the candidate phoneme possibility acquisition module is also used to input the feature vector sequence into the speech sequence recognition submodel for phoneme recognition, so as to obtain the candidate phoneme possibility corresponding to each of the speech frames; the phoneme possibility distribution acquisition module is also used to extract the feature vector corresponding to the extracted speech frame from the feature vector sequence, and input the feature vector corresponding to the extracted speech frame into the speech frame phoneme detection submodel for phoneme recognition; the possibility distribution of each type of phoneme output by the speech frame phoneme detection submodel in each extracted speech frame is used as the phoneme possibility distribution corresponding to each extracted speech frame.

[0219] In some embodiments, the device also includes a speech detection model training module for obtaining training speech; inputting the training speech into a feature extraction submodel to be trained for feature extraction to obtain training feature vectors corresponding to each training speech frame, and sorting the training feature vectors corresponding to each training speech frame in accordance with the corresponding speech frame sequence to obtain a training vector sequence; inputting the training vector sequence into a speech sequence recognition submodel to be trained for phoneme recognition to obtain a phoneme sequence recognition result; obtaining a first model loss value based on a first difference between the phoneme sequence recognition result and a standard recognition result corresponding to the training speech; the first model loss value is positively correlated with the first difference; inputting each of the training feature vectors into the speech frame phoneme detection submodel to be trained; The sub-model performs phoneme recognition to obtain the speech frame phoneme detection result corresponding to the speech frame phoneme detection sub-model; a second model loss value is obtained based on the second difference between the speech frame phoneme detection result and the standard recognition result corresponding to the training speech; the first model loss value is positively correlated with the second difference; a target model loss value is obtained based on the first model loss value and the second model loss value; based on the target model loss value, parameters of the feature extraction sub-model to be trained, the speech frame phoneme detection sub-model to be trained and the speech sequence recognition sub-model to be trained are adjusted, and the feature extraction sub-model after parameter adjustment, the speech frame phoneme detection sub-model after parameter adjustment and the speech sequence recognition sub-model after parameter adjustment form the trained speech detection model.

[0220] In some embodiments, the candidate phoneme probability acquisition module 1104 is also used to sequentially acquire the current feature vector in the feature vector sequence in the order of the feature vector sequence; acquire the phoneme representation vector corresponding to the previous feature vector corresponding to the current feature vector; the phoneme representation vector corresponding to the previous feature vector is the representation vector corresponding to the phoneme obtained by inputting the previous feature vector into the speech sequence recognition sub-model for phoneme recognition; the phoneme representation vector corresponding to the previous feature vector and the current feature vector are input into the speech sequence recognition sub-model for phoneme recognition to obtain the candidate phoneme probability corresponding to the current speech frame corresponding to the current feature vector.

[0221] In some embodiments, the device also includes a phoneme representation vector acquisition module, which is used to input the previous feature vector into the speech sequence recognition sub-model for phoneme recognition to obtain the candidate phoneme possibility corresponding to the previous feature vector; the candidate phoneme possibility corresponding to the previous feature vector includes the possibility corresponding to each candidate phoneme in the candidate phoneme set; select the candidate phoneme with the largest corresponding possibility from the candidate phoneme set as the target phoneme; use the phoneme representation vector corresponding to the target phoneme as the phoneme representation vector corresponding to the previous feature vector.

[0222] In some embodiments, the device also includes a phoneme recognition continuation module, which is used to return to the step of sequentially acquiring the current feature vector in the feature vector sequence in the order of the feature vector sequence when the feature vector in the feature vector sequence has not been phoneme recognized, so as to perform phoneme recognition on the feature vector that has not been phoneme recognized until the phoneme recognition of the feature vector in the feature vector sequence is completed.

[0223] In some embodiments, the speech detection result acquisition module 1112 is also used to obtain the detection word position of the target detection word in the speech to be detected based on the phoneme probability distribution corresponding to the extracted speech frame; obtain the speech in the speech to be detected that is located at the backward position corresponding to the detection word position as the command detection speech; perform speech command detection on the command detection speech to obtain the target speech command corresponding to the command detection speech, so as to perform voice control based on the target speech command.

[0224] For the specific definition of the speech detection device, please refer to the definition of the speech detection method above, which will not be repeated here. Each module in the above-mentioned speech detection device can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0225] In some embodiments, a computer device is provided, whose internal structure diagram can be as follows: Fig.12 As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store voice detection data. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a voice detection method is implemented.

[0226] Those skilled in the art will understand that Fig.12 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0227] In some embodiments, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0228] In some embodiments, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0229] In some embodiments, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned various method embodiments are implemented.

[0230] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the above-mentioned computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0231] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0232] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A speech detection method, characterized in that: The method comprises: Extracting features from each speech frame in the speech to be detected to obtain a feature vector corresponding to the speech frame, and sorting the feature vectors corresponding to each speech frame according to the corresponding speech frame sequence to obtain a feature vector sequence; Perform phoneme recognition based on the feature vector sequence to obtain the possibility of candidate phonemes corresponding to each of the speech frames; Filtering the speech frames without phonemes based on the possibility of the candidate phonemes corresponding to the speech frames, and obtaining the recognition phoneme sequence corresponding to the speech to be detected based on the possibility of the candidate phonemes corresponding to the speech frames with phonemes; The phoneme sequence corresponding to the target detection word selected from the recognition phoneme sequence is used as the detection word phoneme sequence, and based on the position of the speech frame sequence corresponding to the detection word phoneme sequence in the speech frame sequence of the speech to be detected, the extracted speech frame sequence corresponding to the target detection word in the speech to be detected is determined; Obtaining a phoneme probability distribution corresponding to each extracted speech frame in the extracted speech frame sequence; the phoneme probability distribution corresponding to the extracted speech frame is obtained by performing speech frame phoneme detection respectively according to the feature vectors corresponding to the extracted speech frame; Obtaining the detection word position of the target detection word in the to-be-detected speech based on the phoneme probability distribution corresponding to the extracted speech frame; Acquire the speech at the backward position corresponding to the position of the detection word in the speech to be detected as the command detection speech; Perform voice command detection on the command detection voice to obtain a target voice command corresponding to the command detection voice, so as to perform voice control based on the target voice command.

2. The method according to claim 1, characterized in that The step of determining the extracted speech frame sequence corresponding to the target detection word in the speech to be detected based on the position of the speech frame sequence corresponding to the detection word phoneme sequence in the speech frame sequence of the speech to be detected comprises: Determine the starting position and the ending position of the speech frame sequence corresponding to the detection word phoneme sequence at the speech to be detected; The forward position corresponding to the starting position is used as the first extraction position, and the backward position corresponding to the ending position is used as the second extraction position; The speech frame sequence located between the first extraction position and the second extraction position in the speech to be detected is used as the extracted speech frame sequence corresponding to the target detection word.

3. The method according to claim 2, characterized in that The method of taking the forward position corresponding to the starting position as the first extraction position and taking the backward position corresponding to the ending position as the second extraction position includes: The number of speech frames with blank phonemes in the speech to be detected is taken as the number of blank frames; Based on the number of blank frames, determining the number of voice frame extensions; the number of voice frame extensions is positively correlated with the number of blank frames; A forward position whose distance from the starting position is the extension number of the speech frame is used as a first extraction position, and a backward position whose distance from the ending position is the extension number of the speech frame is used as a second extraction position.

4. The method according to claim 1, characterized in that: The candidate phoneme possibility includes the possibility corresponding to the phoneme blank; the filtering of the phoneme blank speech frame based on the candidate phoneme possibility corresponding to the speech frame to obtain the recognition phoneme sequence corresponding to the speech to be detected based on the candidate phoneme possibility corresponding to the speech frame with the phoneme, comprises: The speech frame whose probability corresponding to the phoneme blank is greater than the probability threshold is regarded as the speech frame of the phoneme blank; Filtering speech frames with blank phonemes to obtain decoded speech frames, and obtaining a speech decoding network formed based on the probability of candidate phonemes corresponding to the decoded speech frames; Decoding is performed based on the speech decoding network to obtain a target decoding path, and the phonemes passed by the target decoding path are arranged in a path order to obtain a recognition phoneme sequence corresponding to the speech to be detected.

5. The method according to claim 1, characterized in that The step of extracting features from each speech frame in the speech to be detected to obtain a feature vector corresponding to the speech frame includes: Input each speech frame in the speech to be detected into a feature extraction submodel of a trained speech detection model for feature extraction, and obtain a feature vector corresponding to the speech frame; the trained speech detection model includes a feature extraction submodel and a speech sequence recognition submodel and a speech frame phoneme detection submodel respectively connected to the feature extraction submodel; The performing phoneme recognition based on the feature vector sequence to obtain the possibility of candidate phonemes corresponding to each of the speech frames includes: Inputting the feature vector sequence into the speech sequence recognition sub-model to perform phoneme recognition, and obtaining the possibility of candidate phonemes corresponding to each speech frame; The obtaining of the phoneme probability distribution corresponding to each extracted speech frame in the extracted speech frame sequence includes: Extracting the feature vector corresponding to the extracted speech frame from the feature vector sequence, and inputting the feature vector corresponding to the extracted speech frame into the speech frame phoneme detection submodel for phoneme recognition; The probability distribution of each type of phoneme output by the speech frame phoneme detection sub-model in each extracted speech frame is used as the phoneme probability distribution corresponding to each extracted speech frame.

6. The method according to claim 5, characterized in that The step of obtaining the trained speech detection model comprises: Get the training voice; Inputting the training speech into the feature extraction submodel to be trained to perform feature extraction, obtaining training feature vectors corresponding to each training speech frame, and sorting the training feature vectors corresponding to each training speech frame according to the corresponding speech frame order to obtain a training vector sequence; Inputting the training vector sequence into the speech sequence recognition sub-model to be trained to perform phoneme recognition, and obtaining a phoneme sequence recognition result; Obtaining a first model loss value based on a first difference between the phoneme sequence recognition result and a standard recognition result corresponding to the training speech; the first model loss value is positively correlated with the first difference; Inputting each of the training feature vectors into the speech frame phoneme detection sub-model to be trained for phoneme recognition, and obtaining a speech frame phoneme detection result corresponding to the speech frame phoneme detection sub-model; Obtaining a second model loss value based on a second difference between the speech frame phoneme detection result and the standard recognition result corresponding to the training speech; the first model loss value is positively correlated with the second difference; Obtaining a target model loss value based on the first model loss value and the second model loss value; Based on the target model loss value, parameters of the feature extraction sub-model to be trained, the speech frame phoneme detection sub-model to be trained and the speech sequence recognition sub-model to be trained are adjusted, and the feature extraction sub-model after parameter adjustment, the speech frame phoneme detection sub-model after parameter adjustment and the speech sequence recognition sub-model after parameter adjustment form the trained speech detection model.

7. The method according to claim 5, characterized in that The step of inputting the feature vector sequence into the speech sequence recognition sub-model to perform phoneme recognition to obtain the possibility of candidate phonemes corresponding to each speech frame includes: Obtaining the current feature vector in the feature vector sequence in sequence according to the order of the feature vector sequence; Obtaining a phoneme representation vector corresponding to a previous feature vector corresponding to the current feature vector; the phoneme representation vector corresponding to the previous feature vector is a representation vector corresponding to the phoneme obtained by inputting the previous feature vector into the speech sequence recognition sub-model for phoneme recognition; The phoneme representation vector corresponding to the previous feature vector and the current feature vector are input into the speech sequence recognition sub-model for phoneme recognition, and the possibility of candidate phonemes corresponding to the current speech frame corresponding to the current feature vector is obtained.

8. The method according to claim 7, characterized in that The step of obtaining the phoneme representation vector corresponding to the previous feature vector comprises: Inputting the previous feature vector into the speech sequence recognition submodel to perform phoneme recognition, and obtaining the possibility degree of the candidate phoneme corresponding to the previous feature vector; the possibility degree of the candidate phoneme corresponding to the previous feature vector includes the possibility degree corresponding to each candidate phoneme in the candidate phoneme set; Selecting a candidate phoneme with the largest corresponding probability from the candidate phoneme set as the target phoneme; The phoneme representation vector corresponding to the target phoneme is used as the phoneme representation vector corresponding to the previous feature vector.

9. The method according to claim 7, characterized in that: After inputting the phoneme representation vector corresponding to the previous feature vector and the current feature vector into the speech sequence recognition submodel for phoneme recognition and obtaining the possibility of candidate phonemes corresponding to the current speech frame corresponding to the current feature vector, the method further includes: When the feature vectors in the feature vector sequence have not been phoneme-recognized, the step of sequentially acquiring the current feature vector in the feature vector sequence in the order of the feature vector sequence is returned to perform phoneme recognition on the feature vectors that have not been phoneme-recognized until the feature vectors in the feature vector sequence have been phoneme-recognized.

10. A speech detection device, characterized in that: The device comprises: A feature vector processing module is used to extract features from each speech frame in the speech to be detected, obtain feature vectors corresponding to the speech frames, and sort the feature vectors corresponding to each speech frame according to the corresponding speech frame sequence to obtain a feature vector sequence; A candidate phoneme possibility acquisition module, used to perform phoneme recognition based on the feature vector sequence to obtain the candidate phoneme possibility corresponding to each of the speech frames; A phoneme sequence acquisition module, configured to filter the speech frames without phonemes based on the possibility of the candidate phonemes corresponding to the speech frames, so as to obtain the recognition phoneme sequence corresponding to the speech to be detected based on the possibility of the candidate phonemes corresponding to the speech frames with phonemes; A speech frame extraction module, used to use the phoneme sequence corresponding to the target detection word selected from the recognition phoneme sequence as the detection word phoneme sequence, and determine the extracted speech frame sequence corresponding to the target detection word in the speech to be detected based on the position of the speech frame sequence corresponding to the detection word phoneme sequence in the speech frame sequence of the speech to be detected; A phoneme probability distribution acquisition module is used to acquire the phoneme probability distribution corresponding to each extracted speech frame in the extracted speech frame sequence; the phoneme probability distribution corresponding to the extracted speech frame is obtained by performing speech frame phoneme detection respectively according to the feature vectors corresponding to the extracted speech frame; A speech detection result acquisition module is used to obtain the detection word position of the target detection word in the speech to be detected based on the phoneme probability distribution corresponding to the extracted speech frame; obtain the speech at the backward position corresponding to the detection word position in the speech to be detected as the command detection speech; perform speech command detection on the command detection speech to obtain the target speech command corresponding to the command detection speech, so as to perform speech control based on the target speech command.

11. The device according to claim 10, characterized in that The speech frame extraction module is further used for: Determine the starting position and the ending position of the speech frame sequence corresponding to the detection word phoneme sequence at the speech to be detected; The forward position corresponding to the starting position is used as the first extraction position, and the backward position corresponding to the ending position is used as the second extraction position; The speech frame sequence located between the first extraction position and the second extraction position in the speech to be detected is used as the extracted speech frame sequence corresponding to the target detection word.

12. The device according to claim 11, characterized in that The speech frame extraction module is further used for: The number of speech frames with blank phonemes in the speech to be detected is taken as the number of blank frames; Based on the number of blank frames, determining the number of voice frame extensions; the number of voice frame extensions is positively correlated with the number of blank frames; A forward position whose distance from the starting position is the extension number of the speech frame is used as a first extraction position, and a backward position whose distance from the ending position is the extension number of the speech frame is used as a second extraction position.

13. The device according to claim 10, characterized in that The possibility of the candidate phonemes includes the possibility corresponding to the phoneme blank; the phoneme sequence acquisition module is also used for: The speech frame whose probability corresponding to the phoneme blank is greater than the probability threshold is regarded as the speech frame of the phoneme blank; Filtering speech frames with blank phonemes to obtain decoded speech frames, and obtaining a speech decoding network formed based on the probability of candidate phonemes corresponding to the decoded speech frames; Decoding is performed based on the speech decoding network to obtain a target decoding path, and the phonemes passed by the target decoding path are arranged in a path order to obtain a recognition phoneme sequence corresponding to the speech to be detected.

14. The device according to claim 10, characterized in that The feature vector processing module is further used for: Input each speech frame in the speech to be detected into a feature extraction submodel of a trained speech detection model for feature extraction, and obtain a feature vector corresponding to the speech frame; the trained speech detection model includes a feature extraction submodel and a speech sequence recognition submodel and a speech frame phoneme detection submodel respectively connected to the feature extraction submodel; The candidate phoneme possibility acquisition module is further used for: Inputting the feature vector sequence into the speech sequence recognition sub-model to perform phoneme recognition, and obtaining the possibility of candidate phonemes corresponding to each speech frame; The phoneme possible degree distribution acquisition module is also used for: Extracting the feature vector corresponding to the extracted speech frame from the feature vector sequence, and inputting the feature vector corresponding to the extracted speech frame into the speech frame phoneme detection submodel for phoneme recognition; The probability distribution of each type of phoneme output by the speech frame phoneme detection sub-model in each extracted speech frame is used as the phoneme probability distribution corresponding to each extracted speech frame.

15. The device according to claim 14, characterized in that The device also includes a speech detection model training module, which is used to: Get the training voice; Inputting the training speech into the feature extraction submodel to be trained to perform feature extraction, obtaining training feature vectors corresponding to each training speech frame, and sorting the training feature vectors corresponding to each training speech frame according to the corresponding speech frame order to obtain a training vector sequence; Inputting the training vector sequence into the speech sequence recognition sub-model to be trained to perform phoneme recognition, and obtaining a phoneme sequence recognition result; Obtaining a first model loss value based on a first difference between the phoneme sequence recognition result and a standard recognition result corresponding to the training speech; the first model loss value is positively correlated with the first difference; Inputting each of the training feature vectors into the speech frame phoneme detection sub-model to be trained for phoneme recognition, and obtaining a speech frame phoneme detection result corresponding to the speech frame phoneme detection sub-model; Obtaining a second model loss value based on a second difference between the speech frame phoneme detection result and the standard recognition result corresponding to the training speech; the first model loss value is positively correlated with the second difference; Obtaining a target model loss value based on the first model loss value and the second model loss value; Based on the target model loss value, parameters of the feature extraction sub-model to be trained, the speech frame phoneme detection sub-model to be trained and the speech sequence recognition sub-model to be trained are adjusted, and the feature extraction sub-model after parameter adjustment, the speech frame phoneme detection sub-model after parameter adjustment and the speech sequence recognition sub-model after parameter adjustment form the trained speech detection model.

16. The device according to claim 14, characterized in that The candidate phoneme possibility acquisition module is further used for: Obtaining the current feature vector in the feature vector sequence in sequence according to the order of the feature vector sequence; Obtaining a phoneme representation vector corresponding to a previous feature vector corresponding to the current feature vector; the phoneme representation vector corresponding to the previous feature vector is a representation vector corresponding to the phoneme obtained by inputting the previous feature vector into the speech sequence recognition sub-model for phoneme recognition; The phoneme representation vector corresponding to the previous feature vector and the current feature vector are input into the speech sequence recognition sub-model for phoneme recognition, and the possibility of candidate phonemes corresponding to the current speech frame corresponding to the current feature vector is obtained.

17. The device according to claim 16, characterized in that The device also includes a phoneme representation vector acquisition module, which is used to: Inputting the previous feature vector into the speech sequence recognition submodel to perform phoneme recognition, and obtaining the possibility degree of the candidate phoneme corresponding to the previous feature vector; the possibility degree of the candidate phoneme corresponding to the previous feature vector includes the possibility degree corresponding to each candidate phoneme in the candidate phoneme set; Selecting a candidate phoneme with the largest corresponding probability from the candidate phoneme set as the target phoneme; The phoneme representation vector corresponding to the target phoneme is used as the phoneme representation vector corresponding to the previous feature vector.

18. The device according to claim 16, characterized in that The device also includes a phoneme recognition continuous module, which is used to: When the feature vectors in the feature vector sequence have not been phoneme-recognized, the step of sequentially acquiring the current feature vector in the feature vector sequence in the order of the feature vector sequence is returned to perform phoneme recognition on the feature vectors that have not been phoneme-recognized until the feature vectors in the feature vector sequence have been phoneme-recognized.

19. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 9 is implemented.

20. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

21. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images