Method and apparatus for detecting human voice

Abstract

This application provides a method and apparatus for human voice detection, relating to the field of Artificial Intelligence (AI) and related to preprocessing for Automatic Speech Recognition (ASR). The method includes: acquiring a speech signal; acquiring the power spectrum of each speech frame based on the speech signal; dividing the power spectrum of each speech frame into frequency bands in the frequency domain to obtain m sub-power spectra, where m is an integer greater than or equal to 2; and obtaining the probability that each speech frame in the speech signal contains human voice based on the m sub-power spectra. By dividing the entire power spectrum into frequency bands to separate low-frequency and high-frequency bands into different bands, human voice and noise can be accurately identified for the power spectra corresponding to different frequency bands, improving the noise resistance of human voice detection and increasing the accuracy of human voice detection.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method and device for human voice detection. Background Technology

[0002] With the continuous development of Artificial Intelligence (AI) technology, speech recognition, which enables human-machine interaction through voice signals, is becoming increasingly important. Automatic Speech Recognition (ASR), as a means of human-computer interaction, can convert input speech into corresponding text sequences and is widely used in voice assistants, smart speakers, and smart conferencing systems. However, ASR processes input speech indiscriminately. For example, a speech segment may contain segments without human voices, but ASR will still process these segments into text sequences, resulting in high power consumption.

[0003] Therefore, a Voice Activity Detection (VAD) method has been proposed. Before performing ASR, it identifies the speech segments containing human voices in advance and then performs ASR processing on the speech segments containing human voices, thereby reducing the power consumption of ASR.

[0004] However, the above method may detect some noise as speech segments containing human voices, such as the sound of a door closing or a table being slammed, resulting in low accuracy in human voice detection and affecting the power consumption of the ASR. Summary of the Invention

[0005] This application provides a method and device for human voice detection, which can accurately identify human voice and noise, improve the noise resistance of human voice detection, and thus improve the accuracy of human voice detection.

[0006] In a first aspect, embodiments of this application provide a human voice detection method, including:

[0007] Acquire audio signals;

[0008] Obtain the power spectrum of each speech frame in the speech signal;

[0009] The power spectrum of each speech frame is divided into frequency bands in the frequency domain to obtain m sub-power spectra, where m is an integer greater than or equal to 2.

[0010] Based on the sub-power spectra of m frequency bands, the probability that each speech frame in the speech signal contains human voice is obtained.

[0011] In one possible implementation, the power spectrum of each speech frame is divided into frequency bands in the frequency domain to obtain m sub-power spectra, including:

[0012] According to the Mel scale, the power spectrum is divided into frequency bands in the frequency domain to obtain sub-power spectra of m frequency bands;

[0013] Each frequency band has an equal length of Mel scale markings.

[0014] In one possible implementation, the power spectrum of each speech frame is divided into frequency bands in the frequency domain to obtain sub-power spectra of multiple frequency bands, including:

[0015] According to the Mel scale, the frequency domain of the power spectrum is divided into frequency bands to obtain m intermediate frequency bands, and the Mel scale of each intermediate frequency band is of equal length;

[0016] Based on the power spectrum, obtain the sub-power spectrum of the highest frequency band among the m intermediate frequency bands, and increase the upper limit frequency of the m-1 intermediate frequency bands excluding the highest frequency band to obtain the sub-power spectrum of the m-1 bands; or,

[0017] Based on the power spectrum, obtain the sub-power spectrum of the lowest frequency band among the m intermediate frequency bands, and reduce the lower limit frequency of the m-1 intermediate frequency bands excluding the lowest frequency band to obtain the sub-power spectrum of the m-1 frequency bands.

[0018] Among them, two adjacent frequency bands partially overlap in the frequency domain.

[0019] In one possible implementation, the probability that each speech frame in the speech signal contains human voice is obtained based on the sub-power spectra of m frequency bands, including:

[0020] For each frequency band's sub-power spectrum, extract K harmonic components of different orders, where K is greater than or equal to 1;

[0021] Based on the sub-power spectrum of each frequency band and K different harmonic components, the probability that each speech frame in the speech signal contains human voice is obtained.

[0022] In one possible implementation, the probability that each speech frame in the speech signal contains human voice is obtained based on the sub-power spectrum of each frequency band and K different harmonic components, including:

[0023] Extract the features corresponding to the sub-power spectrum of each frequency band, as well as the features corresponding to the K harmonic components of different orders;

[0024] The features corresponding to the sub-power spectrum of each frequency band and the features corresponding to the K harmonic components of different orders are spliced ​​together to obtain the spectral enhancement features of each speech frame.

[0025] Based on the spectral enhancement features of each speech frame, the probability that each speech frame in the speech signal contains human voices is obtained.

[0026] In one possible implementation, the method further includes:

[0027] Obtain the logarithmic average energy of the sub-power spectrum of each frequency band and the logarithmic average energy of the K different harmonic components;

[0028] The log-average energy corresponding to the sub-power spectrum of each frequency band and the log-average energy corresponding to the K different harmonic components are spliced ​​together to obtain the energy enhancement features of each speech frame.

[0029] Based on the spectral enhancement features of each speech frame, the probability that each speech frame in the speech signal contains human voice is obtained, including:

[0030] Based on the spectral enhancement features and energy enhancement features of each speech frame, the probability that each speech frame in the speech signal contains human voices is obtained.

[0031] In one possible implementation, the probability that each speech frame in the speech signal contains human voice is obtained based on the sub-power spectra of m frequency bands, including:

[0032] Based on the sub-power spectra of m frequency bands, the spectral enhancement features of each speech frame are obtained;

[0033] Obtain the energy enhancement features of each speech frame;

[0034] Based on the spectral enhancement features and energy enhancement features of each speech frame, the probability that each speech frame in the speech signal contains human voices is obtained.

[0035] In one possible implementation, the energy enhancement features of each speech frame are obtained, including:

[0036] Based on the sub-power spectra of m frequency bands, the energy enhancement features of each speech frame are obtained.

[0037] In one possible implementation, based on the sub-power spectra of m frequency bands, the spectral enhancement features and energy enhancement features of each speech frame are obtained, including:

[0038] For each frequency band's sub-power spectrum, extract K harmonic components of different orders, where K is greater than or equal to 1;

[0039] Based on the sub-power spectrum of each frequency band and K different harmonic components, the spectral enhancement features and energy enhancement features of each speech frame are obtained.

[0040] In one possible implementation, the K harmonic components of different orders include the 2nd harmonic component, the 3rd harmonic component, and the 5th harmonic component.

[0041] In one possible implementation, the probability that each speech frame in the speech signal contains human voice is obtained based on the spectral enhancement features and energy enhancement features of each speech frame, including:

[0042] Based on the spectral enhancement features of multiple consecutive speech frames, the first output result corresponding to each frame in the multiple consecutive frames is obtained. The first output result of the multiple consecutive frames is used to represent the temporal correlation of the spectrum of different frequency bands.

[0043] Based on the energy enhancement features of multiple consecutive speech frames, a second output result corresponding to each frame in the multiple consecutive frames is obtained. The second output result of the multiple consecutive frames is used to represent the temporal correlation of energy in different frequency bands.

[0044] Based on the first and second output results corresponding to each frame, the probability that each speech frame contains human voice is obtained.

[0045] In one possible implementation, based on the spectral enhancement features of multiple consecutive speech frames, a first output result corresponding to each frame in the multiple consecutive frames is obtained, including:

[0046] The spectral enhancement features of multiple consecutive speech frames are input into a neural network employing speech activation detection to obtain the first output result corresponding to each of the multiple consecutive frames.

[0047] In one possible implementation, based on the energy enhancement features of multiple consecutive speech frames, a second output result corresponding to each frame in the multiple consecutive frames is obtained, including:

[0048] The energy enhancement features of multiple consecutive speech frames are input into a neural network employing an attention mechanism to obtain a second output result for each frame in the multiple consecutive frames.

[0049] In one possible implementation, the first output and the second output are vectors of the same dimension.

[0050] In one possible implementation, the probability that each speech frame contains human voice is obtained based on the first output result and the second output result corresponding to each frame, including:

[0051] The first and second output results are input into the result fusion network to obtain the probability that each speech frame in the output speech signal contains human voice.

[0052] The result fusion network includes a point-by-point multiplication computation layer and at least one fully connected layer.

[0053] In one possible implementation, the result fusion network outputs a 2D vector, which includes the probability that each speech frame contains human voice and the probability that each speech frame contains non-human voice.

[0054] Secondly, embodiments of this application provide a human voice detection device, comprising:

[0055] The acquisition module is used to acquire the speech signal and the power spectrum of each speech frame in the speech signal.

[0056] The partitioning module is used to divide the power spectrum of each speech frame into frequency bands in the frequency domain to obtain m sub-power spectra of frequency bands, where m is an integer greater than or equal to 2.

[0057] The processing module is used to obtain the probability that each speech frame in the speech signal contains human voice based on the sub-power spectra of m frequency bands.

[0058] Thirdly, embodiments of this application provide a human voice detection device, including: a memory, a processor, and a communication interface;

[0059] Memory is used to store program instructions;

[0060] The processor is used to invoke program instructions in memory to execute the human voice detection method as described in any of the first aspects.

[0061] Fourthly, embodiments of this application provide a chip including at least one processor and a communication interface, the communication interface and at least one processor being interconnected via a line, the at least one processor being used to run a computer program or instructions to perform the human voice detection method as described in any of the first aspects.

[0062] Fifthly, embodiments of this application provide a computer-readable storage medium including instructions that, when executed on a computer, cause the computer to perform the human voice detection method as described in any of the first aspects.

[0063] In a sixth aspect, embodiments of this application provide a computer program product containing instructions that, when executed on a computer, cause the computer to perform the human voice detection method as described in any of the first aspects.

[0064] In summary, the voice detection method and device provided in this application, by dividing the power spectrum of the entire frequency band into frequency bands in the frequency domain, obtains the probability that each frame of the speech signal contains a human voice. This allows for accurate identification of human voices and noise, improving the noise resistance of voice detection and increasing its accuracy. Furthermore, since near-field and far-field human voices differ in energy, their energy enhancement characteristics can distinguish between them, thereby improving resistance to far-field human voices and increasing the accuracy of voice detection for users. Attached Figure Description

[0065] Figure 1 A schematic diagram of a speech recognition system provided in an embodiment of this application;

[0066] Figure 2 A flowchart of a voice detection method provided in an embodiment of this application;

[0067] Figure 3 A schematic diagram illustrating the division of the power spectrum into m frequency bands according to an embodiment of this application;

[0068] Figure 4 A schematic diagram illustrating the extraction of the 2nd, 3rd, and 5th harmonic components according to an embodiment of this application;

[0069] Figure 5 A schematic diagram illustrating the use of CNN to extract spectral enhancement features according to an embodiment of this application;

[0070] Figure 6 A flowchart of a voice detection method provided in another embodiment of this application;

[0071] Figure 7 A block diagram illustrating a voice detection method provided in another embodiment of this application;

[0072] Figure 8 A schematic diagram illustrating the acquisition of energy enhancement features according to an embodiment of this application;

[0073] Figure 9 A flowchart of a voice detection method provided in another embodiment of this application;

[0074] Figure 10 A schematic diagram illustrating the acquisition of a first output result by a neural network employing a speech activation detection method, according to an embodiment of this application;

[0075] Figure 11 A schematic diagram illustrating the second output result obtained by a neural network employing an attention mechanism according to an embodiment of this application;

[0076] Figure 12 A schematic diagram of the result fusion network provided in an embodiment of this application;

[0077] Figure 13 A schematic diagram of the hardware structure of a neural network processor provided in an embodiment of this application;

[0078] Figure 14 This is a schematic diagram of the structure of a human voice detection device provided in an embodiment of this application;

[0079] Figure 15 This is a schematic diagram of the structure of a human voice detection device provided in another embodiment of this application;

[0080] Figure 16 This is a schematic diagram of the structure of a chip provided in an embodiment of this application. Detailed Implementation

[0081] Figure 1 This is a schematic diagram of a speech recognition system provided in an embodiment of this application, as shown below. Figure 1 As shown, the application scenarios of this application are based on Figure 1 The example shown illustrates a speech recognition system that may include multiple terminal devices 110. A user 120 can speak to each of these terminal devices 110. Correspondingly, each terminal device 110 collects the user 120's speech signal and performs speech recognition processing on it. For instance, it first performs human voice detection on the speech signal to obtain the probability of human voices in each speech frame, thereby distinguishing between human voice and non-human voice regions. Then, it performs ASR processing on the pure human voice region to obtain the information sent by the user 120. The terminal device 110 then performs corresponding processing based on the ASR processing result. For example, the terminal device 110 may respond accordingly based on the result, or the terminal device 110 may send the result to a server for the server to respond accordingly.

[0082] Terminal device 110 can install and run relevant applications (or apps). The terminal device provides related services to the user through the installed applications. Terminal device 110 can be, but is not limited to, any type of vehicle, handheld electronic device, etc., and can interact with the user through input devices such as keyboards, virtual keyboards, touchpads, touchscreens, and voice control devices. Vehicles can be, for example, intelligent driving vehicles, or ordinary vehicles equipped with voice processing applications. Electronic devices can be, for example, smartphones, tablets, smart speakers, etc.

[0083] User 120 is the speaker or issuer of the voice. This issuer could be, for example, a robot or other intelligent voice-generating device.

[0084] In this application scenario, speech recognition processing is completed by the terminal device 110 itself, without interaction with the server. Furthermore, this application can also be applied to another application scenario where speech recognition processing is completed through interaction between the terminal device 110 and the server. For example, each terminal device 110 collects the speech signal from user 120 and sends it to the server. The server then performs speech recognition processing on the speech signal, such as first performing human voice detection to obtain the human voice probability of each speech frame, thereby distinguishing between human voice and non-human voice intervals, and then performing ASR processing on the speech signal in the pure human voice interval. Alternatively, each terminal device 110 collects the speech signal from user 120, first performs human voice detection to obtain the human voice probability of each speech frame, thereby distinguishing between human voice and non-human voice intervals, and then the terminal device 110 sends the speech signal in the pure human voice interval to the server, where the server performs ASR processing on the speech signal in the pure human voice interval.

[0085] The following explanation uses a vehicle as an example of a terminal device 110. When driving, the driver's eyes should be on the road and hands should be on the steering wheel. The driver can operate the vehicle through voice interaction, such as voice navigation and voice calls. For human-machine interaction to occur via voice signals, the voice recognition system needs to be able to identify the specific meaning of the voice signals. For example, if the driver says "Drive to destination A" to the vehicle, the vehicle will collect the voice signal, perform human voice detection, obtain the probability of human voices in each voice frame, thus distinguishing between human voice and non-human voice regions. The pure human voice region will then undergo ASR processing to obtain the driver's message "Drive to destination A." The vehicle will then navigate to the destination A from its current location and drive according to that route.

[0086] The following explanation uses a smart speaker as an example of a terminal device 110. For instance, when a user wants the smart speaker to play a song, they can control it via voice interaction. The smart speaker needs a voice recognition system capable of recognizing the specific meaning of the voice signal. For example, if a user says "Play song B" to the smart speaker, the smart speaker will collect the voice signal, perform human voice detection, obtain the probability of human voices in each voice frame, thus distinguishing between human voice and non-human voice ranges. Then, the voice signal in the pure human voice range will undergo ASR processing to obtain the user's message "Play song B." The smart speaker will then play song B.

[0087] The following explanation uses a smartphone as an example of a terminal device 110. For instance, when a user needs to launch an application on their smartphone, they can control it via voice interaction. The smartphone's voice recognition system must be able to identify the specific meaning of the voice signal. For example, if the user says "Open application C" to the smartphone, the smartphone will collect the voice signal, perform human voice detection, obtain the probability of human voices in each voice frame, thus distinguishing between human voice and non-human voice ranges. Then, the pure human voice range will undergo ASR processing to obtain the user's message "Open application C". The smartphone will then launch application C and display its launch interface.

[0088] The specific implementation process of detecting human voices in the speech signal and obtaining the probability of human voices in each speech frame can be found in the following embodiments of this application.

[0089] In current technologies, after acquiring a speech signal, the power spectrum is extracted, and then the entire power spectrum is subjected to voice activity detection (VAD) to obtain the probability of human voice. However, in existing technologies, VAD detection is performed on the entire power spectrum at the same time. In reality, VAD detection mainly detects low-frequency bands, but some noise is very similar to human voice in the low-frequency band, so VAD detection can easily misidentify noise as human voice. Therefore, this application provides a human voice detection method. Since noise and human voice have different characteristics in the high-frequency band, the entire power spectrum is segmented into frequency bands to divide the low-frequency and high-frequency bands into different bands. This allows for human voice detection based on the power spectrum corresponding to different frequency bands, which can accurately identify human voice and noise, improving the noise resistance of human voice detection.

[0090] In addition, users usually speak to the terminal device at close range. During this process, they are likely to be interfered with by voices from a distance. The energy characteristics of voices spoken at close range are different from those spoken at a distance. Therefore, the energy characteristics of voice signals can be used to distinguish between near-field and far-field voices, identify near-field voices as human voices, and not identify far-field voices as human voices, thereby improving the ability to resist far-field voices.

[0091] To facilitate understanding, the relevant terms used in this application will be explained first.

[0092] Automatic Speech Recognition (ASR), or speech recognition for short, refers to the technology that automatically converts speech signals into corresponding text or commands using a computer. It has wide applications in voice assistants, smart speakers, and smart conferencing systems. Simply put, speech recognition gives computers hearing, enabling them to accurately recognize and understand the information carried by speech signals, much like human communication. Speech recognition technology uses speech signals as input. Compared to traditional input methods such as keyboards and mice, it is simple to operate, requires no special training, and is easy to use. Furthermore, people speak very quickly, at the millisecond level, which is about 3 to 4 times faster than a typewriter, and a microphone can be used directly as input.

[0093] Voice Activity Detection (VAD), also known as voice detection or endpoint detection, is a technique for detecting the presence of human voices in audio and is widely used in the preprocessing of speech signals.

[0094] Figure 2 A flowchart of a voice detection method provided in an embodiment of this application is shown below. Figure 2 As shown, the method in this embodiment may include:

[0095] S201, Acquire voice signal.

[0096] In this embodiment, the speech signal is a one-dimensional time series, which can be a real-time acquired speech signal or a pre-recorded speech signal. This speech signal is the most original obtained speech signal, which refers to the speech signal to be recognized by the human voice detection method.

[0097] In one possible implementation, the voice signal can be acquired in real time by the terminal device. In another possible implementation, the voice signal can be pre-acquired, such as by acquiring voice signals stored locally on the terminal device, receiving voice signals sent from other terminal devices, or receiving voice signals sent from a server, and so on.

[0098] S202. Obtain the power spectrum of each speech frame in the speech signal.

[0099] In this embodiment, the speech signal is inherently a non-stationary signal, but it is generally assumed that the speech signal is a stationary signal between 10ms and 30ms, that is, it has short-time stationary characteristics. This means that its spectral characteristics and physical features can be approximated as constant within the 10ms to 30ms time period. Therefore, a short-time analysis method is used to divide the speech signal to be identified into frames for processing. Each frame's duration is within the 10ms to 30ms time period, but this embodiment is not limited to this. For example, a frame length of 20ms can be used, meaning that a 20ms speech signal is divided into one speech frame.

[0100] In this embodiment, the power spectrum of each speech frame describes the power of a speech frame (typically between 10ms and 30ms) in the frequency domain. After obtaining each speech frame, its power spectrum is then acquired. Specifically, each speech frame is windowed to make them continuous. Then, a Fourier transform is performed on each windowed speech frame to transform it from the time domain to the frequency domain, obtaining its spectrum relative to the time domain. After obtaining the spectrum relative to the time domain, the power spectrum of the speech frame can be obtained from this spectrum.

[0101] Optionally, before obtaining the power spectrum of each speech frame in the speech signal, the speech signal can be pre-emphasized to compensate for high-frequency components and make the spectrum of the speech signal flat.

[0102] S203. Divide the power spectrum of each speech frame into frequency bands in the frequency domain to obtain m sub-power spectra.

[0103] In this embodiment, the power spectrum of each speech frame refers to the power spectrum of each frame of the speech signal after being transformed from the time domain to the frequency domain, representing the power spectrum of the entire frequency band in the frequency domain. The power spectrum of each speech frame is divided into frequency bands in the frequency domain, such as... Figure 3 As shown, sub-power spectra of m frequency bands are obtained, where m is an integer greater than or equal to 2. This means dividing the power spectrum of the entire frequency band into m bands, and then obtaining the power spectrum of each band from the power spectrum of the speech frame for each of these m bands. The power spectrum of each band is a partial power spectrum of the speech frame's power spectrum; therefore, the power spectrum of each band is called a sub-power spectrum. For example, the frequency domain of the power spectrum of each speech frame can be divided into three bands, such as low, mid, and high frequency bands, and then sub-power spectra of the low, mid, and high frequency bands are obtained from the power spectrum.

[0104] In one implementation, there are no overlapping frequencies between adjacent frequency bands. In another implementation, there are overlapping frequencies between adjacent frequency bands.

[0105] S204. Based on the sub-power spectra of m frequency bands, obtain the probability that each speech frame in the speech signal contains human voice.

[0106] In this embodiment, the results of voice detection include two categories: speech frames containing human voices and speech frames not containing human voices. Therefore, the voice detection result for each speech frame in the speech signal includes two probability values: one is the probability value containing human voices, and the other is the probability value not containing human voices. The probability values ​​for containing human voices and not containing human voices are both less than or equal to 1, and the sum of the probability values ​​for containing human voices and not containing human voices equals 1.

[0107] Optionally, after obtaining the probability that each speech frame in the speech signal contains human voice, a threshold judgment and smoothing process can be performed on the probability to obtain the human voice interval.

[0108] In one implementation, a predetermined threshold can be selected. If the probability of each speech frame containing human voice is greater than the predetermined threshold, the output is determined to be human voice; otherwise, it is considered non-human voice. For example, if the predetermined threshold is 0.5, the speech frame is considered to contain human voice if the probability of containing human voice is greater than 0.5; otherwise, the speech frame is considered not to contain human voice. The thresholding and smoothing of the human voice probability can be found in descriptions of related technologies and will not be elaborated here.

[0109] The voice detection method provided in this embodiment divides the power spectrum of the entire frequency band into frequency bands in the frequency domain to obtain sub-power spectra for each frequency band. Then, based on the sub-power spectra of each frequency band, the probability that each frame of the speech signal contains human voice is obtained. Since the power spectra of noise and human voice differ in the high-frequency band, the entire power spectrum is divided into frequency bands to separate the low-frequency and high-frequency bands into different frequency bands. This allows for accurate identification of human voice and noise based on the power spectra corresponding to different frequency bands, improving the noise resistance and accuracy of voice detection.

[0110] In the above Figure 2 Based on the illustrated embodiment, in some embodiments, a possible implementation of S203 is as follows: the power spectrum is divided into frequency bands in the frequency domain according to the Mel scale to obtain m sub-power spectra. The Mel scale values ​​for each frequency band are of equal length.

[0111] Specifically, the power spectrum of each speech frame is divided into frequency bands using the Mel scale as follows:

[0112] Assuming that the number of points for calculating the Fourier transform (fft) in each speech frame is n, and the sampling rate is fs, and assuming that the number of frequency bands after frequency segmentation in the frequency domain is m, according to the frequency-Mel frequency transform formula, see Formula 1 below:

[0113] B = 1125ln(1 + f / 700) (Formula 1)

[0114] Where B is the Mel frequency and f is the frequency.

[0115] According to the sampling theorem, when the sampling frequency is greater than twice the highest frequency of the signal, no information will be lost during the sampling process. Combining this with Formula 1 above, the highest Mel frequency can be obtained as:

[0116] B max =1125ln(1+fs / 1400)

[0117] Among them, B max This is the highest Mel frequency.

[0118] Then obtain the frequency point f after dividing the Mel scale into m equal parts. i The following formula two is satisfied:

[0119]

[0120] From Formula 2 above, we can obtain f i =700[(1+fs / 1400)] i / m -1],i=1,2,...,m-1

[0121] Since the power spectrum has a length of Given a vector where each value represents the resolution fr = fs / (n-1), the frequency point f can be obtained. i The corresponding power spectrum index values ​​are shown in Formula 3:

[0122] index i =f i / fr=700[(1+fs / 1400) i / m Formula 3

[0123] Therefore, the frequency domain of the entire power spectrum is divided into m intervals, namely:

[0124] [0,index1],[index1 index2],…[index (m-1) ,n / 2-1].

[0125] For example, if the number of points for calculating the FFT is 400, the sampling rate is 16kHz, and the power spectrum is divided into 3 frequency bands, then the range of the 3 frequency bands is [0:23], [23:76], and [76:200].

[0126] In the above Figure 2Based on the illustrated embodiment, in some other embodiments, a possible implementation of S203 is as follows: The power spectrum is divided into frequency bands according to a Mel scale to obtain m intermediate frequency bands, each with an equal Mel scale length. Based on the power spectrum, the sub-power spectrum of the highest frequency band among the m intermediate frequency bands is obtained, and the upper limit frequency of the m-1 intermediate frequency bands (excluding the highest frequency band) is increased to obtain the sub-power spectrum of the m-1 frequency bands; or, based on the power spectrum, the sub-power spectrum of the lowest frequency band among the m intermediate frequency bands is obtained, and the lower limit frequency of the m-1 intermediate frequency bands (excluding the lowest frequency band) is decreased to obtain the sub-power spectrum of the m-1 frequency bands; wherein adjacent frequency bands partially overlap in the frequency domain.

[0127] According to the Mel scale, the frequency domain of the power spectrum is divided into frequency bands to obtain m intermediate frequency bands. The specific implementation process can be found in the above description and will not be repeated here.

[0128] For example, if the number of points for calculating the FFT is 400, the sampling rate is 16kHz, and the power spectrum is divided into 3 frequency bands, then the range of the 3 intermediate frequency bands obtained are [0:23], [23:76], and [76:200].

[0129] Among the three intermediate frequency bands mentioned above, the highest frequency band is [76:200]. The other frequency bands besides the highest frequency band are [0:23] and [23:76]. Increasing the upper limit frequency 23 of [0:23] by 10 values ​​yields the frequency band [0:33], and increasing the upper limit frequency 76 of [23:76] by 10 values ​​yields the frequency band [23:86]. The three frequency bands obtained are [0:33], [23:86], and [76:200]. The sub-power spectra of [0:33], [23:86], and [76:200] are obtained from the power spectrum.

[0130] For example, if the number of points for calculating the FFT is 400, the sampling rate is 16kHz, and the power spectrum is divided into 3 frequency bands, then the range of the 3 intermediate frequency bands obtained are [0:23], [23:76], and [76:200].

[0131] Among the three intermediate frequency bands mentioned above, the lowest frequency band is [0:23]. The other frequency bands besides the lowest frequency band are [23:76] and [76:200]. Reducing the lower limit frequency 23 of [23:76] by 10 values ​​yields the frequency band [13:76], and reducing the lower limit frequency 76 of [76:200] by 10 values ​​yields the frequency band [66:200]. The final three frequency bands obtained are [0:23], [13:76], and [66:200]. Sub-power spectra of [0:23], [13:76], and [66:200] are obtained from the power spectrum.

[0132] Therefore, except for the last frequency band, the upper limit of each other frequency band can be extended upwards, or except for the first frequency band, the lower limit of each other frequency band can be extended downwards by x values, where x is an integer greater than 0. That is, the number of overlapping sampling points between adjacent frequency bands is x. This ensures the stability of the transition between two adjacent frequency bands and improves the recognition effect of human voice.

[0133] Based on the above embodiments, as an optional embodiment, one possible implementation of S204 is as follows: extract K harmonic components of different orders for the sub-power spectrum of each frequency band, where K is greater than or equal to 1; obtain the probability that each frame of the speech signal contains human voice based on the sub-power spectrum of each frequency band and the K harmonic components of different orders.

[0134] In this embodiment, harmonic components refer to components whose Fourier series number is greater than 1 within one period. That is, the frequency of a harmonic component is higher than the fundamental frequency of the signal; none are lower than the fundamental frequency. Harmonic components can be divided into two different types according to frequency: one type has frequencies that are 1, 3, 5, etc., multiples of the fundamental frequency, called odd harmonics; the other type has frequencies that are 2, 4, 6, etc., multiples of the fundamental frequency, called even harmonics. After dividing the input power spectrum into frequency bands to obtain m frequency bands, harmonic components of different orders are extracted from each of the m frequency bands. In this embodiment, the value of the order is not limited.

[0135] In one feasible scheme, K is set to 3, and the K different harmonic components include the 2nd, 3rd, and 5th harmonic components, such as... Figure 4 As shown.

[0136] For example, assuming the spectrum of each input frequency band is represented by spec, and three different harmonic components are extracted, with the harmonic order represented by h, then the extracted harmonic results can be expressed as:

[0137] harmonic=stridedSlice(spec,h),h=2,3,5

[0138] Where stridedSlice(spec,h) means that in spec, one value is taken for every h values. That is, when h=2, the calculation starts from the beginning of each of the m frequency bands, and one component is taken for every 2 values. When h=3, the calculation starts from the beginning of each of the m frequency bands, and one component is taken for every 3 values. When h=5, the calculation starts from the beginning of each of the m frequency bands, and one component is taken for every 5 values.

[0139] It should be noted that, since the first value of frequency band 1 in the m frequency bands represents the DC component and there is no harmonic component, the input of frequency band 1 is modified to spec[1:].

[0140] Based on the above embodiments, as an optional embodiment, one possible way to obtain the probability that each speech frame in the speech signal contains human voice based on the sub-power spectrum of each frequency band and K different harmonic components is as follows: extract the features corresponding to the sub-power spectrum of each frequency band and the features corresponding to the K different harmonic components respectively; concatenate the features corresponding to the sub-power spectrum of each frequency band and the features corresponding to the K different harmonic components respectively to obtain the spectral enhancement features of each speech frame; and obtain the probability that each speech frame in the speech signal contains human voice based on the spectral enhancement features of each speech frame.

[0141] In this embodiment, since the outputs between each frequency band and between the K different harmonic components corresponding to each frequency band are isolated, the feature outputs are also independent after extracting the features corresponding to the sub-power spectrum of each frequency band and the features corresponding to the K different harmonic components. Therefore, the features corresponding to the sub-power spectrum of each frequency band and the features corresponding to the K different harmonic components are spliced ​​together to obtain the spectral enhancement features of each speech frame.

[0142] In one feasible approach, convolutional neural networks (CNNs) can be used to extract features from the power spectrum of each frequency band and the harmonic components of different orders within each band. For example... Figure 5 As shown, Figure 5 This diagram illustrates the extraction of spectral enhancement features using a CNN according to an embodiment of this application. Assuming there are m frequency bands, each band is connected to a CNN network. The input to the CNN is the power spectrum value of each frequency band, and the output is the feature corresponding to a sub-power spectrum of each frequency band. For different harmonic components of each of the m frequency bands, each harmonic component is connected to a CNN. The input to the CNN is the harmonic component, and the output is the feature corresponding to the harmonic component. The features output by all CNNs are concatenated (e.g., concat) to obtain the spectral enhancement features of the speech frame.

[0143] Based on the above embodiments, after obtaining the power spectrum of each speech frame in the speech signal, the energy enhancement features of each speech frame can also be obtained. Then, based on the obtained spectral enhancement features and energy enhancement features of each speech frame, the probability that each speech frame contains human voices is obtained. Generally, when a user interacts with a terminal device, the user will speak to the terminal device at close range. Due to the different environments in which the user is located, it is possible that other people are also making sounds at the same time, and these sounds are actually background human voices. Compared with the user, these other people are generally farther away from the terminal device, and there is a difference in energy between distant human voices and nearby human voices. Therefore, this application distinguishes between near-field human voices and far-field human voices by obtaining the energy enhancement features of the speech frames.

[0144] Figure 6 A flowchart of a voice detection method provided in another embodiment of this application. Figure 7 A block diagram of a voice detection method provided in another embodiment of this application, as shown below. Figure 6 and Figure 7 As shown, the method in this embodiment may include:

[0145] S601, Acquire voice signal.

[0146] S602. Obtain the power spectrum of each speech frame in the speech signal.

[0147] S603. Divide the power spectrum of each speech frame into frequency bands in the frequency domain to obtain m sub-power spectra.

[0148] In this embodiment, S601-S603 can be referred to the relevant descriptions in the above embodiments, and will not be repeated here.

[0149] S604. Based on the sub-power spectra of m frequency bands, obtain the spectral enhancement features and energy enhancement features of each speech frame.

[0150] The spectral enhancement features of each speech frame obtained based on the sub-power spectra of m frequency bands can be found in the relevant descriptions in the above embodiments, and will not be repeated here.

[0151] One possible way to obtain energy enhancement features is as follows: After obtaining the K harmonic components of different orders for each frequency band based on the sub-power spectra of m frequency bands, obtain the logarithmic average energy corresponding to the sub-power spectrum of each frequency band and the logarithmic average energy corresponding to the K harmonic components of different orders respectively; concatenate the logarithmic average energy corresponding to the sub-power spectrum of each frequency band and the logarithmic average energy corresponding to the K harmonic components of different orders respectively to obtain the energy enhancement features of each speech frame.

[0152] In this embodiment, the expression for the logarithmic average energy is LogMeanSquare(), and the specific calculation process is as follows: Square each value of the power spectrum or its harmonic components, then calculate the mean by frame dimension, and finally perform a logarithmic operation. Obtain the logarithmic average energy corresponding to the sub-power spectrum of each frequency band and the logarithmic average energy corresponding to K different harmonic components. This means inputting the sub-power spectrum of each frequency band into LogMeanSquare(), and inputting the different harmonic components corresponding to each frequency band into LogMeanSquare().

[0153] Since the outputs of each frequency band and the K different harmonic components corresponding to each frequency band are isolated, the log average energy outputs of the sub-power spectrum of each frequency band and the K different harmonic components are also independent. Therefore, the log average energy of the sub-power spectrum of each frequency band and the log average energy of the K different harmonic components are concatenated to obtain the energy enhancement features of each speech frame.

[0154] like Figure 8 As shown, Figure 8 This is a schematic diagram illustrating the acquisition of energy enhancement features according to an embodiment of this application. Assuming there are m frequency bands, for each frequency band, it is connected to a LogMeanSquare() function. The input of LogMeanSquare() is the power spectrum value of each frequency band, and the output of LogMeanSquare() is the log-mean energy corresponding to the sub-power spectrum of each frequency band. For different harmonic components of each of the m frequency bands, each harmonic component is connected to a LogMeanSquare() function. The input of LogMeanSquare() is the harmonic component, and the output of LogMeanSquare() is the log-mean energy corresponding to the harmonic component. The log-mean energies output from all LogMeanSquare() functions are concatenated (e.g., concat) to obtain the energy enhancement features of that speech frame.

[0155] It should be noted that since the first value of frequency band 1 represents the DC component, it is not included in the energy calculation.

[0156] S605. Based on the spectral enhancement features and energy enhancement features of each speech frame, obtain the probability that each speech frame in the speech signal contains human voice.

[0157] In this embodiment, after obtaining the spectral enhancement features and energy enhancement features of each speech frame, the probability that each speech frame contains human voice is obtained based on the spectral enhancement features and energy enhancement features of each speech frame.

[0158] Therefore, since noise and human voice have different power spectra in the high-frequency range, their spectral enhancement characteristics also differ. Thus, by dividing the entire power spectrum into frequency bands to separate low-frequency and high-frequency bands, spectral enhancement characteristics can be obtained for the power spectra corresponding to different frequency bands. Quantization can accurately distinguish between human voice and noise, improving the noise immunity and accuracy of human voice detection. Furthermore, since near-field and far-field human voices differ in energy, energy enhancement characteristics can differentiate between near-field and far-field human voices, thereby improving resistance to far-field human voices and increasing the accuracy of human voice detection for users.

[0159] Figure 9 A flowchart of a voice detection method provided in another embodiment of this application is shown below. Figure 9 As shown, the method in this embodiment may include:

[0160] S901, Acquire voice signal.

[0161] S902. Obtain the power spectrum of each speech frame in the speech signal.

[0162] S903. Divide the power spectrum of each speech frame into frequency bands in the frequency domain to obtain m sub-power spectra.

[0163] S904. For the sub-power spectrum of each frequency band, extract K harmonic components of different orders.

[0164] S905. Extract the features corresponding to the sub-power spectrum of each frequency band and the features corresponding to the K harmonic components of different orders.

[0165] S906. The features corresponding to the sub-power spectra of each frequency band and the features corresponding to the K harmonic components of different orders are spliced ​​together to obtain the spectral enhancement features of each speech frame.

[0166] In this embodiment, steps S901-S906 can be referred to the relevant descriptions in the above embodiments, and will not be repeated here.

[0167] S907. Based on the spectral enhancement features of multiple consecutive speech frames, obtain the first output result corresponding to each frame in the multiple consecutive frames.

[0168] In this embodiment, the first output results of multiple consecutive frames are used to represent the temporal correlation of the spectrum of different frequency bands.

[0169] For example, a series of consecutive speech frames can refer to 10 consecutive speech frames; this application does not limit the number. In this embodiment, based on the spectral enhancement features of the consecutive speech frames, a first output result corresponding to each frame in the series is obtained. The first output result corresponding to each frame is obtained by referring to the spectral enhancement features of the consecutive speech frames. Since the consecutive speech frames are continuous in time, the first output result of the consecutive frames can represent the temporal correlation of the spectra of different frequency bands.

[0170] As an optional implementation, S907 may include: inputting the spectral enhancement features of multiple consecutive speech frames into a neural network employing a speech activation detection method to obtain a first output result corresponding to each of the multiple consecutive frames.

[0171] In this embodiment, the first output result is the output result of the neural network using the Voice Activation Detection (VAD) method, such as... Figure 10 As shown, Figure 10 This is a schematic diagram illustrating the first output result obtained by a neural network employing a speech activation detection method according to an embodiment of this application.

[0172] The neural network that uses speech activation detection takes as input the spectral enhancement features of multiple consecutive speech frames. Since the input is the spectral enhancement features of multiple consecutive speech frames, the first output result corresponding to each frame can be obtained.

[0173] Optionally, in one implementation, the neural network employing speech activation detection can be a fully connected neural network, a CNN network, or a Long Short Term Memory network (LSTM).

[0174] For example, using a CNN network as a neural network employing speech activation detection, assuming the speech sampling rate is 16kHz, a short-time Fourier transform with a window length of 320 sampling points and a window shift of 160 sampling points is performed on the speech signal before input to extract its frequency features. Since it is a complete sentence, each frame is not isolated and has certain contextual information. Therefore, taking the current frame as the center, the previous 7 frames and the next 8 frames, a total of 16 frames, are taken as the input of the convolutional neural network. That is, the input of the convolutional neural network is an image in the range [16, 160]. Then, after passing through the convolutional neural network, the first output result is the output result of 16 consecutive frames. It should be noted that the specific implementation process of the neural network using speech activation detection can be found in the description of related technologies, and will not be repeated here.

[0175] S908. Obtain the logarithmic average energy corresponding to the sub-power spectrum of each frequency band and the logarithmic average energy corresponding to the K different harmonic components.

[0176] S909. The logarithmic average energy corresponding to the sub-power spectrum of each frequency band and the logarithmic average energy corresponding to the K harmonic components of different orders are spliced ​​together to obtain the energy enhancement features of each speech frame.

[0177] S910. Based on the energy enhancement features of multiple consecutive speech frames, obtain the second output result corresponding to each frame in the multiple consecutive frames.

[0178] In this embodiment, the second output results of multiple consecutive frames are used to represent the temporal correlation of energy in different frequency bands.

[0179] For example, a series of consecutive speech frames can refer to 10 consecutive speech frames; this application does not limit the number. In this embodiment, based on the energy enhancement characteristics of the consecutive speech frames, a second output result is obtained for each frame within the series. The second output result for each frame is obtained by referring to the energy enhancement characteristics of the consecutive speech frames. Since the consecutive speech frames are temporally continuous, the second output result can represent the temporal correlation of energy in different frequency bands. Therefore, it can effectively identify voiced / unvoiced sounds and the audio energy relationship between near and far fields.

[0180] Optionally, as an optional embodiment, the above S910 may include: inputting the energy enhancement features of multiple consecutive speech frames into a neural network employing an attention mechanism to obtain a second output result corresponding to each of the multiple consecutive frames.

[0181] In this embodiment, the second output result is the output result of the neural network using the attention mechanism. The input of the neural network using the attention mechanism is the energy enhancement features of multiple consecutive speech frames. Since the input is the energy enhancement features of multiple consecutive speech frames, the second output result corresponding to each frame can be obtained.

[0182] In one possible implementation, the neural network employing the attention mechanism uses a three-layer transformer structure, such as... Figure 11 As shown, Figure 11 This is a schematic diagram of a neural network using an attention mechanism to obtain a second output result according to an embodiment of this application. Its input is energy enhancement features of multiple consecutive frames, and its output is the second output result of each frame in the multiple consecutive frames.

[0183] It should be noted that this embodiment does not limit the execution order of S905-S907 and S908-S910.

[0184] S911. Based on the first and second output results corresponding to each frame, obtain the probability that each speech frame contains human voice.

[0185] In this embodiment, after obtaining the first output result and the second output result corresponding to each frame, the first output result and the second output result of each frame are fused to obtain the probability that each speech frame contains human voice. As an optional implementation, S911 may include: inputting the first output result and the second output result into a result fusion network to obtain the probability that each speech frame in the output speech signal contains human voice; wherein the result fusion network includes a pointwise multiplication calculation layer and at least one fully connected layer. Taking a result fusion network including a pointwise multiplication calculation layer and two fully connected layers as an example, such as... Figure 12 As shown, Figure 12 This is a schematic diagram of a result fusion network provided in an embodiment of this application. The result fusion network has two inputs: a first output result and a second output result. The output result includes the probability that each speech frame contains human voice.

[0186] Optionally, the output of the result fusion network can be a 2-dimensional vector. One dimension of the vector represents the probability that each speech frame contains human voice, and the other dimension represents the probability that each speech frame contains non-human voice. The probability values ​​for containing human voice and not containing human voice are both less than or equal to 1, and the sum of the probabilities of containing human voice and not containing human voice equals 1.

[0187] Optionally, the first output and the second output can be vectors of the same dimension.

[0188] For example, the aforementioned consecutive frames are T frames, m is 3, and the K different harmonic components include the 2nd, 3rd, and 5th harmonic components. The extracted power spectrum features corresponding to each frequency band and the features corresponding to each harmonic component have dimensions [T, 32]. Therefore, the input to the neural network for speech activation detection is the concatenation of 12 [T, 32] values, which is [T, 384]. Figure 10 As shown; the input to the neural network using the attention mechanism is the concatenation of 12 [T,1] values, which is [T,12], as shown. Figure 11 As shown. The output of the neural network for speech activation detection is [T, 64], and the output of the neural network using the attention mechanism is also [T, 64], thus ensuring that the first and second outputs are vectors of the same dimension, as shown. Figure 12 As shown. The final output of the fusion network is [T,2], which represents the probability that each speech frame contains human voice and the probability that each speech frame contains non-human voice.

[0189] It should be noted that any of the above embodiments can be implemented alone, or at least two of the above embodiments can be combined in any way, and there is no limitation thereto.

[0190] In one implementation, the voice detection method provided in this application relates to the field of artificial intelligence and is related to the preprocessing of ASR. It requires the use of multiple neural networks to calculate and process the relevant data. Therefore, the voice detection method provided in this application can be completed by a neural network processing unit (NPU) assisting a central processing unit (CPU). Figure 13 This is a schematic diagram of the hardware structure of a neural network processor provided in an embodiment of this application.

[0191] exist Figure 13 In this system, the neural network processor 1300 is mounted on the CPU as a coprocessor, and the CPU allocates tasks to it. The core of the neural network processor 1300 is the arithmetic circuit 1301, which is controlled by the controller 1302 to retrieve data from the memory for computation.

[0192] In some implementations, the arithmetic circuit 1301 includes multiple process engines (PEs). In some implementations, the arithmetic circuit 1301 may be an electronic circuit capable of performing digital operations such as multiplication and addition, or it may be a general-purpose matrix processor.

[0193] For example, suppose we have an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit 1301 retrieves the corresponding data for matrix B from the weight memory 1303 and caches it in each PE of the arithmetic circuit. The arithmetic circuit retrieves the data for matrix A from the input memory 1304 and performs matrix operations with matrix B. The partial or final result of the obtained matrix is ​​stored in the accumulator 1305.

[0194] For example, in the embodiments of this application, the power spectrum of the speech signal and the point-by-point multiplication operation can be performed by the operation circuit 1301.

[0195] The storage unit access controller 1306 is mainly used to move input data from external memory to unified memory 1307, or to move weight data to weight memory 1303, or to move input data to input memory 1304.

[0196] Unified memory 1307 is used to store input and output data. Weight data is transferred to weight memory 1303 via memory access controller 1306. Input data is also transferred to unified memory 1307 via memory access controller 1306. Bus interface unit 1308 is used for interaction between the advanced extensible interface (AXI) bus and memory access controller 1306 and instruction fetch memory 1309.

[0197] The bus interface unit 1308 is also used by the instruction fetch memory 1309 to fetch instructions from the external memory, and by the memory access controller 1306 to fetch the original data of the input matrix A or the weight matrix B from the external memory.

[0198] In this embodiment of the application, the data that needs to be obtained from the external memory may be, for example, the power spectrum of the speech signal.

[0199] The vector computation unit 1310 includes multiple arithmetic processing units that, when needed, further process the output of the arithmetic circuit, such as vector multiplication, vector addition, exponential operations, logarithmic operations, size comparisons, etc. It is mainly used for computation in non-convolutional or fully connected layers of neural networks, such as pooling and batch normalization.

[0200] In some implementations, vector computation unit 1310 stores the processed output vector in unified memory 1307. For example, vector computation unit 1310 can apply a nonlinear function to the output of arithmetic circuit 1301, such as a vector of accumulated values, to generate activation values. In some implementations, vector computation unit 1310 generates normalized values, merged values, or both. In some implementations, the processed output vector can be used as activation input to arithmetic circuit 1301, for example, for use in subsequent layers of a neural network.

[0201] The instruction fetch memory 1309 connected to the controller 1302 is used to store the instructions used by the controller.

[0202] Unified memory 1307, input memory 1304, weighted memory 1303, and instruction fetch memory 1309 are all internal memories of the NPU. External memories are proprietary to this NPU hardware architecture.

[0203] Figure 14 This is a schematic diagram of the structure of a voice detection device provided in an embodiment of this application, as shown below. Figure 14As shown, the voice detection device can be a terminal device, or a component of a terminal device (e.g., an integrated circuit, a chip, etc.), or it can be a server, or a component of a server (e.g., an integrated circuit, a chip, etc.). The voice detection device 1400 in this embodiment includes: an acquisition module 1401, a division module 1402, and a processing module 1403.

[0204] The acquisition module 1401 is used to acquire the speech signal and the power spectrum of each speech frame in the speech signal.

[0205] The segmentation module 1402 is used to divide the power spectrum of each speech frame into frequency bands in the frequency domain to obtain sub-power spectra of m frequency bands, where m is an integer greater than or equal to 2.

[0206] The processing module 1403 is used to obtain the probability that each speech frame in the speech signal contains human voice based on the sub-power spectra of m frequency bands.

[0207] In one possible implementation, the partitioning module 1402 is specifically used to: divide the power spectrum into frequency bands in the frequency domain according to the Mel scale to obtain m sub-power spectra; wherein the Mel scale of each frequency band is of equal length.

[0208] In one possible implementation, the partitioning module 1402 is specifically used to: divide the frequency domain of the power spectrum into frequency bands according to the Mel scale to obtain m intermediate frequency bands, each intermediate frequency band having an equal Mel scale length. Based on the power spectrum, obtain the sub-power spectrum of the highest frequency band among the m intermediate frequency bands and increase the upper limit frequency of the m-1 intermediate frequency bands excluding the highest frequency band to obtain the sub-power spectrum of the m-1 frequency bands; or, based on the power spectrum, obtain the sub-power spectrum of the lowest frequency band among the m intermediate frequency bands and decrease the lower limit frequency of the m-1 intermediate frequency bands excluding the lowest frequency band to obtain the sub-power spectrum of the m-1 frequency bands. Wherein, adjacent frequency bands partially overlap in the frequency domain.

[0209] In one possible implementation, the processing module 1403 is specifically configured to: extract K harmonic components of different orders from the sub-power spectrum of each frequency band, where K is greater than or equal to 1; and obtain the probability that each speech frame in the speech signal contains human voice based on the sub-power spectrum of each frequency band and the K harmonic components of different orders.

[0210] In one possible implementation, the processing module 1403 is specifically used to: extract the features corresponding to the sub-power spectrum of each frequency band, and the features corresponding to the K harmonic components of different orders.

[0211] The features corresponding to the sub-power spectra of each frequency band and the features corresponding to the K harmonic components of different orders are spliced ​​together to obtain the spectral enhancement features of each speech frame.

[0212] Based on the spectral enhancement features of each speech frame, the probability that each speech frame in the speech signal contains human voices is obtained.

[0213] In one possible implementation, the processing module 1403 is further configured to: obtain the logarithmic average energy corresponding to the sub-power spectrum of each frequency band and the logarithmic average energy corresponding to the K harmonic components of different orders.

[0214] The log-average energy corresponding to the sub-power spectrum of each frequency band and the log-average energy corresponding to the K different harmonic components are concatenated to obtain the energy enhancement features of each speech frame.

[0215] Specifically, when processing module 1403 obtains the probability that each speech frame in the speech signal contains human voice based on the spectral enhancement characteristics of each speech frame, it is used for:

[0216] Based on the spectral enhancement features and energy enhancement features of each speech frame, the probability that each speech frame in the speech signal contains human voices is obtained.

[0217] In one possible implementation, the processing module 1403 is specifically used for:

[0218] Based on the sub-power spectra of m frequency bands, the spectral enhancement features of each speech frame are obtained.

[0219] Obtain the energy enhancement features of each speech frame.

[0220] Based on the spectral enhancement features and energy enhancement features of each speech frame, the probability that each speech frame in the speech signal contains human voices is obtained.

[0221] In one possible implementation, the processing module 1403 is specifically used for:

[0222] Based on the sub-power spectra of m frequency bands, the energy enhancement features of each speech frame are obtained.

[0223] In one possible implementation, the processing module 1403 is specifically used for:

[0224] For each frequency band's sub-power spectrum, extract K harmonic components of different orders, where K is greater than or equal to 1.

[0225] Based on the sub-power spectrum of each frequency band and K different harmonic components, the spectral enhancement features and energy enhancement features of each speech frame are obtained.

[0226] Optionally, the K harmonic components of different orders include the 2nd harmonic component, the 3rd harmonic component, and the 5th harmonic component.

[0227] In one possible implementation, the processing module 1403 is specifically used for:

[0228] Based on the spectral enhancement features of multiple consecutive speech frames, the first output result corresponding to each frame in the multiple consecutive frames is obtained. The first output result of the multiple consecutive frames is used to represent the temporal correlation of the spectra of different frequency bands.

[0229] Based on the energy enhancement features of multiple consecutive speech frames, a second output result is obtained for each frame in the multiple consecutive frames. The second output result of the multiple consecutive frames is used to represent the temporal correlation of energy in different frequency bands.

[0230] Based on the first and second output results corresponding to each frame, the probability that each speech frame contains human voice is obtained.

[0231] In one possible implementation, the processing module 1403 is specifically used for:

[0232] The spectral enhancement features of multiple consecutive speech frames are input into a neural network employing speech activation detection to obtain the first output result corresponding to each of the multiple consecutive frames.

[0233] In one possible implementation, the processing module 1403 is specifically used for:

[0234] The energy enhancement features of multiple consecutive speech frames are input into a neural network employing an attention mechanism to obtain a second output result for each frame in the multiple consecutive frames.

[0235] Optionally, the first output and the second output can be vectors of the same dimension.

[0236] In one possible implementation, the processing module 1403 is specifically used for:

[0237] The first and second output results are input into the result fusion network to obtain the probability that each speech frame in the output speech signal contains human voice.

[0238] The result fusion network includes a point-by-point multiplication computation layer and at least one fully connected layer.

[0239] Optionally, the result of the result fusion network is output as a 2D vector, which includes the probability that each speech frame contains human voice and the probability that each speech frame contains non-human voice.

[0240] The human voice detection device in this embodiment can implement the solution of any of the above method embodiments, and its implementation principle and technical effect are similar, so it will not be described again here.

[0241] Figure 15 This is a schematic diagram of the structure of a voice detection device provided in another embodiment of this application, as shown below. Figure 15As shown, the voice detection device can be the terminal device in the above method embodiments, or it can be a component of the terminal device (e.g., an integrated circuit, a chip, etc.). The voice detection device 1500 in this embodiment includes: a memory 1501, a processor 1502, and a communication interface 1503. The memory 1501, processor 1502, and communication interface 1503 can be interconnected or interconnected via a bus 1504.

[0242] Optionally, the memory 1501 is used to store program instructions. The memory 1501 may be, but is not limited to, random access memory, read-only memory, erasable programmable read-only memory, or portable read-only memory. The communication interface 1503 is used to support communication between the voice detection device 1500 and the device.

[0243] Optionally, the processor 1502 may be a central processing unit, a general-purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. The processor may also be a combination that implements computational functions, such as a combination of one or more microprocessors, a combination of a digital signal processor and a microprocessor, etc. The processor 1502 can be used to read the program stored in the memory 1501 and execute the scheme of any of the above method embodiments.

[0244] Figure 16 This is a schematic diagram of the structure of a chip provided in one embodiment of this application, as shown below. Figure 16 As shown. Chip 1600 may include: a processor 1601, and a communication interface 1602 interconnected with the processor via a line. Wherein:

[0245] The processor 1601 is used to run computer programs or instructions. In its implementation, the processor 1601 mainly includes a controller, an arithmetic logic unit (ALU), and registers. The controller is primarily responsible for instruction decoding and issuing control signals for the operations corresponding to the instructions. The ALU is primarily responsible for performing fixed-point or floating-point arithmetic operations, shift operations, and logical operations, and can also perform address and translation operations. Registers are primarily responsible for storing register operands and intermediate operation results temporarily stored during instruction execution. In its implementation, the processor 1601's hardware architecture can be an application-specific integrated circuit (ASIC) architecture, a microprocessor architecture without interlocked pipeline stages, an advanced reduced instruction set machine (RISC) architecture, or an NP architecture, etc. The processor 1601 can be single-core or multi-core.

[0246] The communication interface 1602 can be used to input data to be processed into the processor 1601 and to output the processing results from the processor 1601. In one specific implementation, the communication interface 1602 can be a general-purpose input / output interface that can connect to multiple peripheral devices (such as displays, RF modules, etc.). The communication interface 1602 can be connected to the processor 1601 via bus 1603.

[0247] In this application, the processor 1601 can be used to call the implementation program of the human voice detection method provided in one or more embodiments of this application on the terminal side from the memory, and execute the instructions contained in the program. The memory can be integrated with the processor 1601, in which case the memory is part of the chip 1600. Alternatively, the memory is an external component of the chip 1600, and the processor 1601 calls the instructions or data stored in the memory through the communication interface 1602.

[0248] The communication interface 1602 can be used to output the execution results of the processor 1601. For details regarding the human voice detection method provided in one or more embodiments of this application, please refer to the foregoing embodiments; further details will not be repeated here.

[0249] In one possible embodiment, the chip 1600 described above can be a system-on-a-chip (SoC). It should be noted that the functions of the processor 1601 and the communication interface 1602 interconnected with the processor via a line can be implemented through hardware design, software design, or a combination of both; no limitation is imposed here.

[0250] This application also provides a computer-readable storage medium, which includes instructions that, when executed on a computer, will perform any of the method embodiments.

[0251] It should be noted that the module division in this embodiment is illustrative and represents only one logical functional division; in actual implementation, there may be other division methods. The functional modules in this embodiment can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated modules can be implemented in hardware or as software functional modules.

[0252] If the integrated module is implemented as a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0253] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

Claims

1. A human voice detection method, characterized by, include: Acquire audio signals; Obtain the power spectrum of each speech frame in the speech signal; The power spectrum of each speech frame is divided into frequency bands in the frequency domain to obtain m sub-power spectra, where m is an integer greater than or equal to 2; Based on the sub-power spectra of the m frequency bands, the probability that each speech frame in the speech signal contains human voice is obtained; The step of dividing the power spectrum of each speech frame into frequency bands in the frequency domain to obtain sub-power spectra of multiple frequency bands includes: According to the Mel scale, the frequency domain of the power spectrum is divided into frequency bands to obtain m intermediate frequency bands, each of which has an equal Mel scale length. Based on the power spectrum, obtain the sub-power spectrum of the highest frequency band among the m intermediate frequency bands, and increase the upper limit frequency of the m-1 intermediate frequency bands excluding the highest frequency band to obtain the sub-power spectrum of the m-1 frequency bands; or, Based on the power spectrum, the sub-power spectrum of the lowest frequency band among the m intermediate frequency bands is obtained, and the lower limit frequency of the m-1 intermediate frequency bands excluding the lowest frequency band is reduced to obtain the sub-power spectrum of the m-1 frequency bands. Among them, two adjacent frequency bands partially overlap in the frequency domain.

2. The method of claim 1, wherein, The step of dividing the power spectrum of each speech frame into frequency bands in the frequency domain to obtain m sub-power spectra includes: According to the Mel scale, the power spectrum is divided into frequency bands in the frequency domain to obtain m sub-power spectra. Each frequency band has an equal length of Mel scale markings.

3. The method according to claim 1 or 2, characterized in that, The step of obtaining the probability that each speech frame in the speech signal contains human voice based on the sub-power spectra of the m frequency bands includes: For each frequency band's sub-power spectrum, extract K harmonic components of different orders, where K is greater than or equal to 1; Based on the sub-power spectrum of each frequency band and K different harmonic components, the probability that each speech frame in the speech signal contains human voice is obtained.

4. The method of claim 3, wherein, The step of obtaining the probability that each speech frame in the speech signal contains human voice based on the sub-power spectrum of each frequency band and K different harmonic components includes: Extract the features corresponding to the sub-power spectrum of each frequency band, as well as the features corresponding to the K harmonic components of different orders; The features corresponding to the sub-power spectrum of each frequency band and the features corresponding to the K harmonic components of different orders are spliced ​​together to obtain the spectral enhancement features of each speech frame. Based on the spectral enhancement features of each speech frame, the probability that each speech frame in the speech signal contains human voice is obtained.

5. The method of claim 4, wherein, The method further includes: Obtain the logarithmic average energy of the sub-power spectrum of each frequency band and the logarithmic average energy of the K different harmonic components; The log-average energy corresponding to the sub-power spectrum of each frequency band and the log-average energy corresponding to the K different harmonic components are spliced ​​together to obtain the energy enhancement features of each speech frame. The step of obtaining the probability that each speech frame in the speech signal contains human voice based on the spectral enhancement features of each speech frame includes: Based on the spectral enhancement features and energy enhancement features of each speech frame, the probability that each speech frame in the speech signal contains human voice is obtained.

6. The method according to claim 1 or 2, characterized in that, The step of obtaining the probability that each speech frame in the speech signal contains human voice based on the sub-power spectra of the m frequency bands includes: Based on the sub-power spectra of the m frequency bands, the spectral enhancement features of each speech frame are obtained; Obtain the energy enhancement features of each speech frame; Based on the spectral enhancement features and energy enhancement features of each speech frame, the probability that each speech frame in the speech signal contains human voice is obtained.

7. The method of claim 6, wherein, The acquisition of energy enhancement features for each speech frame includes: Based on the sub-power spectra of the m frequency bands, the energy enhancement features of each speech frame are obtained.

8. The method according to claim 7, characterized in that, Based on the sub-power spectra of the m frequency bands, the spectral enhancement features and energy enhancement features of each speech frame are obtained, including: For each frequency band's sub-power spectrum, extract K harmonic components of different orders, where K is greater than or equal to 1; Based on the sub-power spectrum of each frequency band and K different harmonic components, the spectral enhancement features and energy enhancement features of each speech frame are obtained.

9. The method according to claim 3, characterized in that, The K harmonic components of different orders include the 2nd harmonic component, the 3rd harmonic component, and the 5th harmonic component.

10. The method according to any one of claims 4-5 and 8, characterized in that, The K harmonic components of different orders include the 2nd harmonic component, the 3rd harmonic component, and the 5th harmonic component.

11. The method according to claim 5, characterized in that, The step of obtaining the probability that each speech frame in the speech signal contains human voice based on the spectral enhancement features and energy enhancement features of each speech frame includes: Based on the spectral enhancement features of multiple consecutive speech frames, a first output result corresponding to each frame in the multiple consecutive frames is obtained. The first output result of the multiple consecutive frames is used to represent the temporal correlation of the spectra of different frequency bands. Based on the energy enhancement features of multiple consecutive speech frames, a second output result corresponding to each frame in the multiple consecutive frames is obtained. The second output result of the multiple consecutive frames is used to represent the temporal correlation of energy in different frequency bands. Based on the first and second output results corresponding to each frame, the probability that each speech frame contains human voice is obtained.

12. The method according to claim 6, characterized in that, The step of obtaining the probability that each speech frame in the speech signal contains human voice based on the spectral enhancement features and energy enhancement features of each speech frame includes: Based on the spectral enhancement features of multiple consecutive speech frames, a first output result corresponding to each frame in the multiple consecutive frames is obtained. The first output result of the multiple consecutive frames is used to represent the temporal correlation of the spectra of different frequency bands. Based on the energy enhancement features of multiple consecutive speech frames, a second output result corresponding to each frame in the multiple consecutive frames is obtained. The second output result of the multiple consecutive frames is used to represent the temporal correlation of energy in different frequency bands. Based on the first and second output results corresponding to each frame, the probability that each speech frame contains human voice is obtained.

13. The method according to claim 11, characterized in that, The step of obtaining the first output result corresponding to each frame in a series of consecutive speech frames based on the spectral enhancement features includes: The spectral enhancement features of multiple consecutive speech frames are input into a neural network employing speech activation detection to obtain the first output result corresponding to each of the multiple consecutive frames.

14. The method according to claim 12, characterized in that, The step of obtaining the first output result corresponding to each frame in a series of consecutive speech frames based on the spectral enhancement features includes: The spectral enhancement features of multiple consecutive speech frames are input into a neural network employing speech activation detection to obtain the first output result corresponding to each of the multiple consecutive frames.

15. The method according to claim 11, characterized in that, The step of obtaining the second output result corresponding to each frame in a series of consecutive speech frames based on the energy enhancement features includes: The energy enhancement features of multiple consecutive audio frames are input into a neural network employing an attention mechanism to obtain a second output result corresponding to each of the multiple consecutive frames.

16. The method according to claim 12, characterized in that, The step of obtaining the second output result corresponding to each frame in a series of consecutive speech frames based on the energy enhancement features includes: The energy enhancement features of multiple consecutive audio frames are input into a neural network employing an attention mechanism to obtain a second output result corresponding to each of the multiple consecutive frames.

17. The method according to claim 13, characterized in that, The step of obtaining the second output result corresponding to each frame in a series of consecutive speech frames based on the energy enhancement features includes: The energy enhancement features of multiple consecutive audio frames are input into a neural network employing an attention mechanism to obtain a second output result corresponding to each of the multiple consecutive frames.

18. The method according to claim 14, characterized in that, The step of obtaining the second output result corresponding to each frame in a series of consecutive speech frames based on the energy enhancement features includes: The energy enhancement features of multiple consecutive audio frames are input into a neural network employing an attention mechanism to obtain a second output result corresponding to each of the multiple consecutive frames.

19. The method according to any one of claims 11-18, characterized in that, The first output result and the second output result are vectors with the same dimension.

20. The method according to any one of claims 11-18, characterized in that, Based on the first output result and the second output result corresponding to each frame, the probability that each audio frame contains human voice is obtained, including: The first output result and the second output result are input into the result fusion network to obtain the probability that each speech frame in the output speech signal contains human voice; The result fusion network includes a point-by-point multiplication computation layer and at least one fully connected layer.

21. The method according to claim 19, characterized in that, Based on the first output result and the second output result corresponding to each frame, the probability that each audio frame contains human voice is obtained, including: The first output result and the second output result are input into the result fusion network to obtain the probability that each speech frame in the output speech signal contains human voice; The result fusion network includes a point-by-point multiplication computation layer and at least one fully connected layer.

22. The method according to any one of claims 11-18, characterized in that, The result of the fusion network is a 2D vector, which includes the probability that each speech frame contains human voice and the probability that each speech frame contains non-human voice.

23. The method according to claim 19, characterized in that, The result of the fusion network is a 2D vector, which includes the probability that each speech frame contains human voice and the probability that each speech frame contains non-human voice.

24. The method according to claim 20, characterized in that, The result of the fusion network is a 2D vector, which includes the probability that each speech frame contains human voice and the probability that each speech frame contains non-human voice.

25. The method according to claim 21, characterized in that, The result of the fusion network is a 2D vector, which includes the probability that each speech frame contains human voice and the probability that each speech frame contains non-human voice.

26. A human voice detection device, characterized in that, include: An acquisition module is used to acquire a speech signal and acquire the power spectrum of each speech frame in the speech signal; The partitioning module is used to divide the power spectrum of each speech frame into frequency bands in the frequency domain to obtain m sub-power spectra of frequency bands, where m is an integer greater than or equal to 2; The processing module is used to obtain the probability that each speech frame in the speech signal contains human voice based on the sub-power spectra of the m frequency bands. The partitioning module is specifically used for: dividing the frequency domain of the power spectrum into frequency bands according to the Mel scale to obtain m intermediate frequency bands, each intermediate frequency band having an equal Mel scale length; obtaining the sub-power spectrum of the highest frequency band among the m intermediate frequency bands and increasing the upper limit frequency of the m-1 intermediate frequency bands excluding the highest frequency band among the m intermediate frequency bands to obtain the sub-power spectrum of the m-1 frequency bands; or, obtaining the sub-power spectrum of the lowest frequency band among the m intermediate frequency bands and decreasing the lower limit frequency of the m-1 intermediate frequency bands excluding the lowest frequency band among the m intermediate frequency bands to obtain the sub-power spectrum of the m-1 frequency bands; wherein, adjacent frequency bands partially overlap in the frequency domain.

27. A human voice detection device, characterized in that, include: Memory, processor, and communication interface; The memory is used to store program instructions; The processor is used to invoke program instructions in the memory to execute the human voice detection method as described in any one of claims 1-25.

28. A chip, characterized in that, It includes at least one processor and a communication interface, the communication interface and the at least one processor being interconnected via a line, the at least one processor being used to run a computer program or instructions to perform the human voice detection method as described in any one of claims 1-25.

29. A computer-readable storage medium, characterized in that, Includes instructions that, when executed on a computer, cause the computer to perform the human voice detection method as described in any one of claims 1-25.

30. A computer program product, characterized in that, The computer program product includes computer program code that, when run on a computer, causes the computer to perform the method as described in any one of claims 1-25.

Images