Method, device and computer readable storage medium for generating comfort noise

Abstract

The application discloses a comfort noise generation method, device, equipment and computer readable storage medium, and relates to the technical field of communication. The method is applied to a vehicle-mounted system, and comprises the following steps: obtaining a random noise initial value, and generating a random noise based on the random noise initial value; receiving a voice signal to which comfort noise is to be added, wherein the voice signal comprises background noise, or the voice signal comprises background noise and a human voice signal; obtaining the background noise in the voice signal; and processing the random noise and the background noise, and obtaining the comfort noise for adding to the voice signal according to a processing result. Since the comfort noise is obtained by processing the random noise and the background noise, the energy difference between the comfort noise and the background noise is small, so that the energy of the noise in the obtained voice signal is stable after the comfort noise is added to the voice signal, and the quality of the obtained voice signal is high.

Description

Technical Field

[0001] This application relates to the field of communication technology, and in particular to a method, apparatus, device, and computer-readable storage medium for generating comfort noise. Background Technology

[0002] During a call, there may be periods when no one is speaking. To prevent both parties from mistaking these periods of silence for a call interruption, noise needs to be generated and added to these periods of silence. This noise is called comfort noise.

[0003] In related technologies, web real-time communication (WebRTC) technology is used to generate comfortable noise. For example, a random noise matrix is ​​generated based on the set [0, 1], random noise is obtained based on the random noise matrix, the amplitude of the random noise is adjusted according to the power spectrum of the random noise, and the adjusted random noise is used as the generated comfortable noise.

[0004] However, the comfort noise generated by WebRTC technology often differs significantly in energy from the background noise that originally existed during the call. This results in unstable noise energy in the obtained speech signal after adding comfort noise to the speech signal collected during the call, leading to poor speech signal quality. Summary of the Invention

[0005] This application provides a method, apparatus, device, and computer-readable storage medium for generating comfort noise, which can be used to generate comfort noise with a small energy difference from background noise. The technical solution is as follows:

[0006] On the one hand, this application provides a method for generating comfort noise, which is applied to an in-vehicle terminal. The method includes: obtaining an initial value of random noise, generating random noise based on the initial value of random noise; receiving a speech signal to which comfort noise is to be added, wherein the speech signal includes background noise, or the speech signal includes background noise and human voice signal; obtaining the background noise in the speech signal; processing the random noise and background noise, and obtaining the comfort noise to be added to the speech signal based on the processing result.

[0007] In one possible implementation, the speech signal includes multiple speech frames. Obtaining background noise in the speech signal includes: for any one of the multiple speech frames, obtaining the power spectrum and amplitude spectrum of at least one speech block associated with that speech frame, and the power spectrum of that speech frame; obtaining the probability of human voice presence in at least one speech block based on the power spectrum and amplitude spectrum of the at least one speech block; and obtaining the background noise in that speech frame based on the probability of human voice presence in at least one speech block and the power spectrum of that speech frame.

[0008] In one possible implementation, the background noise in a given speech frame is obtained based on the probability of human voice presence in at least one speech block and the power spectrum of that speech frame. This includes: obtaining a stationary noise portion of the background noise in the given speech frame based on the power spectrum of speech blocks with a human voice presence probability below a reference threshold; obtaining a speech frame associated with the given speech frame based on the relationship between the human voice presence probability of at least one speech block and the reference threshold; obtaining a noise-compensated portion of the background noise in the given speech frame based on the power spectrum of the given speech frame and the power spectrum of the speech frame associated with the given speech frame; and obtaining the background noise in the given speech frame based on the stationary noise portion and the noise-compensated portion of the background noise in the given speech frame.

[0009] In one possible implementation, at least one speech block associated with any speech frame includes speech blocks preceding and / or following the speech frame. Obtaining speech frames associated with the speech frame based on the relationship between the probability of human voice presence in at least one speech block and a reference threshold includes: obtaining frame count indicators corresponding to each speech block based on the relationship between the probability of human voice presence in at least one speech block and the reference threshold; obtaining a first number of speech frames preceding and adjacent to the speech frame based on the sum of the frame count indicators corresponding to the speech blocks preceding the speech frame; and obtaining a second number of speech frames following and adjacent to the speech frame based on the sum of the frame count indicators corresponding to the speech blocks following the speech frame.

[0010] In one possible implementation, after obtaining the speech frame associated with any speech frame based on the relationship between the probability of human voice presence in at least one speech block and a reference threshold, the method further includes: determining the energy state of any speech frame based on the energy of the speech frame and the energy of at least one speech block associated with the speech frame; adjusting the number of speech frames associated with the speech frame based on the energy state of the speech frame to obtain an adjusted speech frame associated with the speech frame, wherein the adjusted speech frame associated with the speech frame is used to obtain the noise compensation portion of the background noise in the speech frame.

[0011] In one possible implementation, the noise compensation portion of the background noise in any given speech frame is obtained based on the power spectrum of the given speech frame and the power spectrum of the speech frames associated with the given speech frame. This includes: obtaining the minimum power spectrum in the power spectrum of the given speech frame and the power spectrum of the speech frames associated with the given speech frame; adjusting the size of the minimum power spectrum based on an adjustment factor; and obtaining the noise compensation portion of the background noise in the given speech frame based on the adjustment result.

[0012] In one possible implementation, the processing result includes initial comfort noise, and obtaining comfort noise for adding the speech signal based on the processing result, including: obtaining a gain factor corresponding to the initial comfort noise; performing gain processing on the initial comfort noise based on the gain factor to obtain comfort noise for adding the speech signal.

[0013] In one possible implementation, obtaining the gain factor corresponding to the initial comfort noise includes: obtaining the autocorrelation vector and reflection coefficient of the background noise, wherein the autocorrelation vector is used to indicate the degree of autocorrelation of the background noise and the reflection coefficient is used to indicate the amount of background noise reflected in the audio channel; and obtaining the gain factor corresponding to the initial comfort noise based on the autocorrelation vector and the reflection coefficient.

[0014] On the other hand, a device for generating comfort noise is provided, which is applied to an in-vehicle terminal. The device includes: a generation module for acquiring an initial value of random noise and generating random noise based on the initial value of random noise; a first acquisition module for receiving a speech signal to which comfort noise is to be added, wherein the speech signal includes background noise, or the speech signal includes background noise and human voice signal; a second acquisition module for acquiring the background noise in the speech signal; and a third acquisition module for processing the random noise and background noise, and obtaining the comfort noise to be added to the speech signal based on the processing result.

[0015] In one possible implementation, the speech signal includes multiple speech frames. The second acquisition module is configured to, for any one of the multiple speech frames, acquire the power spectrum and amplitude spectrum of at least one speech block associated with that speech frame, as well as the power spectrum of that speech frame; acquire the probability of human presence in at least one speech block based on the power spectrum and amplitude spectrum of the at least one speech block; and acquire the background noise in that speech frame based on the probability of human presence in at least one speech block and the power spectrum of that speech frame.

[0016] In one possible implementation, the second acquisition module is configured to: acquire a stationary noise portion of the background noise in any given speech frame based on the power spectrum of speech blocks whose human voice presence probability is below a reference threshold; acquire a speech frame associated with the given speech frame based on the relationship between the human voice presence probability of at least one speech block and the reference threshold; acquire a noise-compensated portion of the background noise in the given speech frame based on the power spectrum of the given speech frame and the power spectrum of the speech frame associated with the given speech frame; and acquire the background noise in the given speech frame based on the stationary noise portion and the noise-compensated portion of the background noise in the given speech frame.

[0017] In one possible implementation, at least one speech block associated with any speech frame includes speech blocks preceding and / or following the speech frame; the second acquisition module is configured to acquire frame count indication values ​​corresponding to each speech block based on the relationship between the probability of human voice presence in at least one speech block and a reference threshold; acquire a first number of speech frames preceding and adjacent to the speech frame based on the sum of the frame count indication values ​​corresponding to the speech blocks preceding the speech frame; and acquire a second number of speech frames following and adjacent to the speech frame based on the sum of the frame count indication values ​​corresponding to the speech blocks following the speech frame.

[0018] In one possible implementation, the second acquisition module is further configured to, after acquiring the speech frames associated with the speech frame, determine the energy state of the speech frame based on the energy of the speech frame and the energy of at least one speech block associated with the speech frame; and adjust the number of speech frames associated with the speech frame based on the energy state of the speech frame to obtain an adjusted speech frame associated with the speech frame, wherein the adjusted speech frame associated with the speech frame is used to acquire the noise compensation portion of the background noise in the speech frame.

[0019] In one possible implementation, the second acquisition module is used to acquire the power spectrum of any given speech frame and the minimum power spectrum of the power spectrum of the speech frames associated with the given speech frame; adjust the size of the minimum power spectrum based on an adjustment factor, and acquire the noise compensation part of the background noise in the given speech frame according to the adjustment result.

[0020] In one possible implementation, the processing result includes initial comfort noise, a third acquisition module for acquiring the gain factor corresponding to the initial comfort noise, and gain processing of the initial comfort noise based on the gain factor to obtain the comfort noise used to add the speech signal.

[0021] In one possible implementation, the third acquisition module is used to acquire the autocorrelation vector and reflection coefficient of the background noise, wherein the autocorrelation vector is used to indicate the degree of autocorrelation of the background noise and the reflection coefficient is used to indicate the amount of background noise reflected in the audio channel; and based on the autocorrelation vector and reflection coefficient, the gain factor corresponding to the initial comfort noise is acquired.

[0022] On the other hand, a computer device is provided, comprising a processor and a memory, wherein at least one computer program is stored in the memory, and the at least one computer program is loaded and executed by the processor to enable the computer device to implement any of the above-described methods for generating comfortable noise.

[0023] On the other hand, a computer-readable storage medium is also provided, in which at least one computer program is stored, the at least one computer program being loaded and executed by a processor to enable a computer to implement any of the above-described methods for generating comfortable noise.

[0024] On the other hand, a computer program product or computer program is also provided, comprising computer instructions stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform any of the aforementioned methods for generating comfortable noise.

[0025] The technical solution provided in this application brings at least the following beneficial effects:

[0026] The technical solution provided in this application is applied to in-vehicle terminals, thus enabling its application in in-vehicle communication scenarios. In this solution, background noise is acquired from the speech signal to which comfort noise is to be added. The generated random noise and the acquired background noise are processed, and the comfort noise used to add to the speech signal is obtained based on the processing result. Since the comfort noise is obtained by processing random noise and background noise, the energy difference between the comfort noise and background noise is small. Therefore, after adding the comfort noise to the speech signal, the noise energy in the resulting speech signal is more stable, and the quality of the resulting speech signal is higher. Consequently, after playing the speech signal with added comfort noise, the noise heard by the listener is more stable, resulting in a better auditory experience. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 This is a schematic diagram of an implementation environment provided in an embodiment of this application;

[0029] Figure 2 This is a flowchart of a method for generating comfort noise provided in an embodiment of this application;

[0030] Figure 3 This is a schematic diagram illustrating a process for acquiring a speech signal with added comfort noise, provided in an embodiment of this application.

[0031] Figure 4 This is a schematic diagram of the structure of a comfort noise generation device provided in an embodiment of this application;

[0032] Figure 5 This is a schematic diagram of the structure of a server provided in an embodiment of this application;

[0033] Figure 6 This is a schematic diagram of the structure of a terminal provided in an embodiment of this application. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0035] This application provides a method for generating comfort noise, which can be applied to... Figure 1 The implementation environment is shown. For example... Figure 1 As shown, the implementation environment may include: a first terminal 11, a second terminal 12, and a server 13.

[0036] The first terminal 11 is the terminal where the vehicle's in-vehicle system is located; that is, the first terminal 11 is an in-vehicle terminal. The second terminal 12 is an in-vehicle terminal of another vehicle not connected to the first terminal 11, or a terminal of another type besides an in-vehicle terminal. Both the first terminal 11 and the second terminal 12 include a sound acquisition device and a sound playback device, so that, based on the sound acquisition device and the sound playback device, the user of the first terminal 11 can communicate with the user of the second terminal 12. For example, the sound acquisition device includes a microphone, and the sound playback device includes a speaker.

[0037] The first terminal 11 and the second terminal 12 are respectively communicatively connected to the server 13. Exemplarily, the first terminal 11 is used to send a first voice signal with added comfort noise to the server 13, the server 13 is used to send the first voice signal to the second terminal 12, and the second terminal 12 is used to play the received first voice signal. The second terminal 12 is also used to send a second voice signal with added comfort noise to the server 13, the server 13 is used to send the second voice signal to the first terminal 11, and the first terminal 11 is used to play the received second voice signal.

[0038] For example, server 13 is a single server, a server cluster consisting of multiple servers, or a cloud computing service center. The first terminal 11 and the second terminal 12 can establish a communication connection with server 13 via wired or wireless network. Those skilled in the art should understand that the first terminal 11, the second terminal 12, and server 13 are merely examples; other existing or future terminals or servers that are applicable to this application should also be included within the scope of protection of this application and are incorporated herein by reference.

[0039] It should be noted that, Figure 1This is merely an example of an implementation environment in which the method provided in this application embodiment is applied. The method provided in this application embodiment can also be applied to other implementation environments. For example, the method can be applied to an implementation environment that does not include a server but only includes a first terminal and a second terminal, that is, a scenario where the terminals directly conduct voice communication.

[0040] The method for generating comfort noise provided in this application embodiment can be as follows: Figure 2 As shown, next, combined with Figure 1 The implementation environment is shown to illustrate the method. For example, the method is applied to... Figure 1 The first terminal 11 is shown. (As shown) Figure 2 As shown, the method includes, but is not limited to, steps 201 to 204.

[0041] In step 201, an initial value of random noise is obtained, and random noise is generated based on the initial value of random noise.

[0042] For example, the initial value of random noise is stored in the first terminal 11 or another device besides the first terminal 11. When the initial value of random noise is stored in the first terminal 11, the first terminal 11 obtains the initial value of random noise by retrieving the stored initial value. When the initial value of random noise is stored in another device, the first terminal 11 sends a retrieval request to the other device, requesting the retrieval of the initial value of random noise, and the first terminal 11 receives the initial value of random noise sent by the other device based on the retrieval request. The value of the initial value of random noise can be determined based on experience or actual needs; for example, the initial value of random noise can be between 0 and 2. 16 One of the values ​​in the random noise initial value is called the seed.

[0043] In this embodiment, random noise can be generated based on the initial value of random noise using the Mason rotation algorithm. For example, a Mason rotation chain is first obtained based on the initial value of random noise, then the Mason rotation chain is processed, and random noise is generated based on the processing result. Exemplarily, the Mason rotation chain is obtained based on the initial value of random noise according to the following expression 1.

[0044] NT[i+1]=f*(NT[i]+(NT[i]>>t))+i+1 i∈[0,r] (Expression 1)

[0045] In this context, the initial value of the random noise is equal to NT[0]. The values ​​of f, t, and r can be determined according to requirements, for example, f equals 1812433253, t equals 30, and r equals 623. In the above expression 1, NT[i] >> t means shifting the binary representation of NT[i] to the right by t bits. According to the content of expression 1, the value of NT[i+1] can be obtained based on the value of NT[i]. Thus, the values ​​of NT[0] to NT[r] can be arranged into a Mason slew chain in the order of 0 to r.

[0046] For example, processing the Mason swivel chain includes updating the values ​​of each position of the Mason swivel chain based on a first mask to obtain an updated Mason swivel chain. The first mask can be determined based on experience or actual needs. The value of the i-th position of the Mason swivel chain is NT[i] in the above expression 1. In one possible implementation, the process of updating the values ​​of each position of the Mason swivel chain based on the first mask includes updating the values ​​of each position of the Mason swivel chain at least once based on the first mask. For example, first update NT[i] according to the following expression 2, and then update NT[i] again according to the following expression 3, thereby realizing the update of NT[i] based on the first mask. By updating NT[i] at least once based on the first mask, the randomness of the updated Mason swivel chain can be improved, thereby improving the randomness of the random noise obtained based on the updated Mason swivel chain.

[0047] NT[i]=NT[i+m]+((NT[i]&up)+(NT[(i+1)%N]&low)) (Expression 2)

[0048] The values ​​of N and m can be determined according to requirements, for example, N equals 512 and m equals 2. up represents the high h bits of the first mask, and low represents the low g bits of the first mask. The values ​​of h and g can be determined according to requirements, for example, h and g are both 100. NT[i]&up means performing a bitwise AND operation between NT[i] and up, NT[(i+1)%N]&low means performing a bitwise AND operation between NT[(i+1)%N] and low, and (i+1)%N means taking i+1 modulo N.

[0049] NT[i] = NT[i+m] % N^XA (Expression 3)

[0050] Here, NT[i+m]%N^XA represents taking the modulo of N with NT[i+m] and then performing a bitwise XOR operation with XA. The value of XA can be shown in expression 4 below.

[0051]

[0052] Where X represents the sequence of NT[0] to NT[r] in order from 0 to r after updating NT[i] according to expression 2, and x0 represents the first value of the sequence. X>>a means shifting the binary representation of X to the right by a bits, and (X>>a)^b means performing a bitwise XOR operation between the result of shifting the binary representation of X to the right by a bits and b. a and b can be determined according to the requirements. For example, a equals 1, and b is obtained based on the elements in the reference matrix. b can be a sequence consisting of at least one element in the reference matrix, or b equals one element in the reference matrix. The reference matrix can be represented as matrix 1 below.

[0053]

[0054] Referring to matrix 1, the reference matrix can be a c x k matrix, where c and k can be determined according to requirements; for example, both c and k can be equal to 128. (a0 to a...) k-1 The value can be determined according to requirements, and the embodiments of this application do not limit it.

[0055] For example, processing the Mason swirl chain yields an updated Mason swirl chain. Generating random noise based on this updated chain includes: transforming the updated Mason swirl chain into the frequency domain to obtain random noise. In one possible implementation, before transforming the updated Mason swirl chain into the frequency domain, the updated Mason swirl chain is updated at least once more to further enhance its randomness, thereby increasing the randomness of the generated random noise. For example, the updated Mason swirl chain is updated according to expressions 5 to 8 below.

[0056]

[0057] Where y0 represents the sequence of NT[0] to NT[r] in order from 0 to r after updating NT[i] according to expression 3, u0 represents the first bit shift, y0 >> u0 means shifting the binary representation of y0 to the right by u0 bits, and d0 represents the second mask. (y0 >> u0) & d0 means performing a bitwise AND operation between the right-shifted binary representation of y0 and d0. This indicates that y0 is XORed with (y0 >> u0) & d0.

[0058]

[0059] Where u1 represents the second displacement and d1 represents the third mask. The operation method of expression 6 is the same as that of expression 5, and will not be repeated here.

[0060]

[0061] Where u2 represents the third displacement and d2 represents the fourth mask. The operation method of expression 7 is the same as that of expression 5, and will not be repeated here.

[0062]

[0063] Where u3 represents the fourth displacement, the operation method of expression 8 is the same as the operation method of expression 5, and will not be repeated here.

[0064] In this embodiment, the updated Mason slew chain can be transformed into the frequency domain using Euler's formula to obtain random noise. Random noise includes a real part and an imaginary part. The real part X(k,l).re of random noise is shown in expression 9 below, and the imaginary part X(k,l).im of random noise is shown in expression 10 below.

[0065] X(k0, l).re = coS(f(2π·z(k0, l) / N)) (Expression 9)

[0066] X(k0, l).im = -sin(f(2π·z(k0, l) / N)) (Expression 10)

[0067] In expressions 9 and 10, f(2π·z(k0, l) / N) represents the value corresponding to the k0th frequency point after performing a Fast Fourier Transform on the updated Mason slew chain. This value is used to add noise to the k0th frequency point of the l-th speech frame. For example, the value of k0 ranges from 1 to N-1. The value used to add noise to the 0th frequency point of the l-th speech frame is 0, thus avoiding the addition of noise at the 0th frequency point of the speech frame. By avoiding the addition of noise at the 0th frequency point of the speech frame, it is possible to avoid noise at the beginning of the speech frame, which would negatively impact the auditory experience.

[0068] Figure 3 This is a schematic diagram of a process for acquiring a speech signal with added comfort noise provided in an embodiment of this application. The above step 201 can correspond to... Figure 3 The operation for obtaining random noise is shown.

[0069] In step 202, a speech signal to which comfort noise is to be added is received, the speech signal including background noise, or the speech signal including background noise and human voice signal.

[0070] Because the background noise in a vehicle environment is relatively high, the received voice signal to which comfort noise is to be added can be a noise-reduced signal. In this embodiment, the received voice signal to which comfort noise is to be added can also be a signal that has undergone noise reduction and echo cancellation processing, thereby resulting in a high signal-to-noise ratio and a better call experience for both parties.

[0071] Step 202 can correspond to Figure 3 The illustration shows the operation of receiving a speech signal to which comfort noise is to be added. In the embodiments of this application, the execution order of steps 201 and 202 is not limited. Step 201 can be executed first and then step 202, or step 202 can be executed first and then step 201, or steps 201 and 202 can be executed simultaneously.

[0072] Step 203: Obtain the background noise in the speech signal.

[0073] Since people's speaking frequency is usually between 500 Hz and 3000 Hz, and the human ear is more sensitive to sounds above 200 Hz, when acquiring background noise in a speech signal, the speech signal can be filtered first to remove low-frequency noise signals, thereby improving the signal-to-noise ratio of the filtered speech signal and ensuring the auditory effect. Then, the background noise in the filtered speech signal can be acquired.

[0074] In one possible implementation, the speech signal to which comfort noise is to be added is input into a filter, and the output speech signal of the filter is obtained, thereby achieving the filtering processing of the speech signal to which comfort noise is to be added. The filter can be an infinite impulse response (IIR) filter, and the transfer factor H(Z) of the IIR filter can be shown in Equation 1 below.

[0075]

[0076] Where Z equals 2πnk1 / N, and the values ​​of n and k1 both range from 0 to N-1. e1 to e6 are determined according to the type of IIR filter.

[0077] For example, N also represents the number of points used in the Fast Fourier Transform (FFT) when obtaining the frequency domain representation of the speech signal to which comfort noise is to be added. For instance, the speech signal to which comfort noise is to be added is windowed, and then the windowed speech signal to which comfort noise is to be added is subjected to a Fast Fourier Transform (FFT) to obtain the frequency domain representation of the speech signal to which comfort noise is to be added, wherein the number of points used in the Fast Fourier Transform is N.

[0078] The window used for windowing can be a Hamming window. The speech signal to which comfort noise is to be added after windowing consists of multiple speech frames, each with a frame length of 8 milliseconds and a corresponding frequency point count of 128. The Fast Fourier Transform uses 512 points, and the frame shift corresponding to the speech signal to which comfort noise is to be added after windowing is 384 frequency points. The frequency domain representation Y(k, l) of the speech signal to which comfort noise is to be added can be shown in the following expression 11.

[0079]

[0080] Where w(n) represents the window function used for windowing, x(n) represents the speech signal to which comfort noise is to be added, e represents the natural constant, and N represents the number of points used in the Fast Fourier Transform. The values ​​of n and k are both in the range of 0 to N-1, and l represents the sequence number of the speech frames included in the speech signal to which comfort noise is to be added.

[0081] In one possible implementation, when the speech signal comprises multiple speech frames, background noise in the speech signal is obtained, including but not limited to steps 2031 to 2033.

[0082] Step 2031: For any one of the multiple speech frames, obtain the power spectrum and amplitude spectrum of at least one speech block associated with that speech frame, as well as the power spectrum of that speech frame.

[0083] For example, for any one of a plurality of speech frames, at least one speech block associated with that speech frame includes a speech block located before that speech frame and / or a speech block located after that speech frame. For instance, for the first speech frame in the plurality of speech frames, at least one speech block associated with the first speech frame includes at least one speech block located after and adjacent to the first speech frame. For the last speech frame in the plurality of speech frames, at least one speech block associated with the last speech frame includes at least one speech block located before and adjacent to the last speech frame.

[0084] For any speech frame located between the first and last speech frames in a plurality of speech frames, the at least one speech block associated with that speech frame includes at least one speech block located before and adjacent to that speech frame, and at least one speech block located after and adjacent to that speech frame. The number of speech blocks associated with each speech frame can be determined based on experience or actual needs.

[0085] Step 2032: Obtain the probability of human voice presence in at least one speech block based on the power spectrum and amplitude spectrum of at least one speech block.

[0086] For example, a speech block includes at least one speech frame. When a speech block includes multiple speech frames, the probability of human voice presence is determined by treating the multiple speech frames as a whole, without having to determine the probability of human voice presence for each individual speech frame. This improves the efficiency of determining the probability of human voice presence, and consequently, reduces the time required to execute the comfort noise generation method provided in this application embodiment, resulting in higher efficiency in generating comfort noise.

[0087] In one possible implementation, for any speech block within at least one speech block, the power spectrum of that speech block is divided by its amplitude spectrum to obtain the probability of human presence in that speech block. By dividing the power spectrum of the speech block by its amplitude spectrum, the relationship between power and amplitude at each frequency point of the speech block becomes relatively intuitive. When the probability of human presence is obtained based on the relationship between power and amplitude at each frequency point, the reliability of the obtained probability of human presence is relatively high.

[0088] Step 2033: Obtain the background noise in any given speech frame based on the probability of human voice presence in at least one speech block and the power spectrum of the given speech frame.

[0089] For example, the background noise in any given speech frame is obtained based on the probability of human voice presence in at least one speech block and the power spectrum of the given speech frame, including steps A1 to A3.

[0090] Step A1: Based on the power spectrum of speech blocks with a human voice presence probability below a reference threshold, obtain the stationary noise portion of the background noise in any given speech frame.

[0091] The reference threshold can be set based on experience or actual needs. For any given speech block, if the probability of human voice presence in that speech block is lower than the reference threshold, then that speech block is considered to contain only background noise. Since the stationary noise portion of the background noise is the same across multiple speech frames included in the speech signal, the power-stable portion of the power spectrum of that speech block can be obtained, and the speech signal corresponding to this portion can be used as the stationary noise portion of the background noise in any given speech frame. The power-stable portion can refer to the portion where the power variation does not exceed a reference value, which can be set based on experience or actual needs.

[0092] Step A2: Based on the relationship between the probability of human voice presence in at least one speech block and a reference threshold, obtain the speech frame associated with any given speech frame; based on the power spectrum of any given speech frame and the power spectrum of the speech frame associated with any given speech frame, obtain the noise compensation part of the background noise in any given speech frame.

[0093] Since background noise may vary, meaning that the background noise in a speech frame includes noise other than the stationary noise component, it is necessary to obtain the noise compensation component of the background noise. The background noise is then obtained based on the stationary noise component and the noise compensation component to improve the accuracy of the obtained background noise.

[0094] Based on the content of step 2031 above, it is known that at least one speech block associated with any speech frame may include a speech block located before and / or a speech block located after the speech frame. In one possible implementation, the speech frame associated with the speech frame is obtained based on the relationship between the probability of human voice presence in at least one speech block and a reference threshold, including but not limited to steps A21 and A22.

[0095] Step A21: Based on the relationship between the probability of human voice presence in at least one speech block and a reference threshold, obtain the frame number indicator value corresponding to each speech block.

[0096] For example, for any one of the at least one speech blocks, if the probability of human voice presence in that speech block is lower than a reference threshold, the frame count indicator value corresponding to that speech block is a first value; if the probability of human voice presence in that speech block is greater than or equal to the reference threshold, the frame count indicator value corresponding to that speech block is a second value. For example, the first value is 0 and the second value is 1.

[0097] Step A22: Based on the sum of the frame count indication values ​​corresponding to the speech blocks preceding the speech frame, obtain a first number of speech frames that are preceding and adjacent to the speech frame; based on the sum of the frame count indication values ​​corresponding to the speech blocks following the speech frame, obtain a second number of speech frames that are following and adjacent to the speech frame.

[0098] For example, the sum of the frame number indication values ​​W_L corresponding to the speech blocks preceding any given speech frame. pre The sum of the frame number indication values ​​W_L corresponding to the speech block following any given speech frame is obtained using Formula 2 below. aft It can be obtained according to Formula 3 below.

[0099]

[0100] Among them, P I (k,l) represents the frame number indicator value corresponding to the i-th speech block before any given speech frame, and w1 represents the number of speech blocks before any given speech frame.

[0101]

[0102] Among them, P J (k,l) represents the frame number indicator value corresponding to the J-th speech block after any given speech frame, and w2 represents the number of speech blocks after any given speech frame.

[0103] For ease of explanation, the speech frame that is before and adjacent to any given speech frame is called the preceding speech frame, and the speech frame that is after and adjacent to any given speech frame is called the following speech frame.

[0104] For example, in W_L pre If the value is less than or equal to the third value, the first quantity is the first reference quantity, and the first reference quantity of previous audio frames is obtained. In W_L pre If the first quantity is greater than the third value, then the first quantity is equal to W_L. pre Get W_L pre One previous audio frame. In W_L aft If the value is less than or equal to the fourth value, the second quantity is the second reference quantity, and the second reference quantity of subsequent speech frames are obtained. In W_L aft If the value is greater than the fourth value, the second quantity equals W_L. aft Get W_L aft The next audio frame.

[0105] The third value is less than the first reference quantity, and the fourth value is less than the second reference quantity. For example, both the third and fourth values ​​are 1, and both the first and second reference quantities are 2. This is achieved through W_L pre If the value is less than or equal to the third value, obtain the first reference number of speech frames, and in W_L aft When the value is less than or equal to the fourth value, obtaining the second reference number of voice frames can result in a larger number of voice frames being obtained.

[0106] When subsequently obtaining the minimum power spectrum from the power spectrum of any given speech frame and the power spectra of related speech frames, the minimum power spectrum obtained from the power spectra of a larger number of speech frames is closer to the power spectrum containing only background noise than the minimum power spectrum obtained from the power spectra of fewer speech frames. Therefore, when obtaining the noise compensation portion of the background noise based on the obtained minimum power spectrum, the accuracy of the obtained noise compensation portion is higher.

[0107] In one possible implementation, after obtaining the speech frames associated with any given speech frame, the number of speech frames associated with that speech frame can be adjusted according to the energy state of that speech frame. For example, after obtaining the speech frames associated with any given speech frame based on the relationship between the probability of human voice presence in at least one speech block and a reference threshold, the method further includes: determining the energy state of the given speech frame based on the energy of the given speech frame and the energy of at least one speech block associated with the given speech frame; adjusting the number of speech frames associated with the given speech frame based on the energy state of the given speech frame to obtain adjusted speech frames associated with the given speech frame, wherein the adjusted speech frames associated with the given speech frame are used to obtain the noise compensation portion of the background noise in the given speech frame.

[0108] In this embodiment, the energy of a speech frame can be calculated based on its amplitude spectrum, and the energy of a speech block can be calculated based on the amplitude spectrum of the speech frames included in the speech block. Taking a speech block associated with a certain speech frame, comprising 15 speech frames preceding and 15 speech frames following that speech frame, as an example, the energy of these 31 speech frames can be represented as: E(k, L-15), E(k, L-14), ..., E(k, L), ..., E(k, L+14), E(k, L+15). Here, E(k, L) represents the energy of the speech frame, and E(k, L-15) represents the energy of the 15th speech frame preceding it. The energy representation of the remaining speech frames follows the same principle and will not be elaborated further here.

[0109] Depending on the location of the speech block associated with any given speech frame, the energy state of that speech frame is described through the following cases 1 to 3.

[0110] Case 1: The speech block associated with any given speech frame includes only at least one speech block located before that given speech frame.

[0111] For case 1, if the energy of any speech frame is higher than the energy of at least one speech block, the energy state of that speech frame is rising. If the energy of any speech frame is equal to the energy of at least one speech block, the energy state of that speech frame is balanced. If the energy of any speech frame is lower than the energy of at least one speech block, the energy state of that speech frame is falling.

[0112] For example, when the energy state of any given speech frame is in an upward state, the power spectrum of that speech frame is higher than the power spectrum of the speech frames associated with that speech frame, thus increasing the number of preceding speech frames associated with that speech frame. Therefore, when subsequently obtaining the minimum power spectrum from the power spectrum of any given speech frame and the power spectra of the speech frames associated with that speech frame, the obtained minimum power spectrum is closer to the power spectrum containing only background noise. Consequently, when obtaining the noise compensation portion of the background noise based on the obtained minimum power spectrum, the accuracy of the obtained noise compensation portion is higher.

[0113] When the energy state of any given speech frame is in equilibrium, the power spectrum of that speech frame is equal to the power spectrum of all speech frames associated with it. The number of speech frames associated with that speech frame remains constant or decreases. By reducing the number of speech frames associated with that speech frame, the efficiency of obtaining the minimum power spectrum when subsequently extracting it from the power spectrum of that speech frame and the power spectra of its associated speech frames is higher, thus resulting in higher efficiency in generating comfortable noise.

[0114] If the energy state of any given speech frame is in a decreasing state, and the power spectrum of that speech frame is lower than the power spectrum of the speech frames associated with that speech frame, then the number of preceding speech frames associated with that speech frame is reduced, or the number of preceding speech frames remains unchanged.

[0115] Case 2: The speech block associated with any given speech frame includes only at least one speech block located after that given speech frame.

[0116] For case 2, if the energy of any speech frame is higher than the energy of at least one speech block, the energy state of that speech frame is decreasing. If the energy of any speech frame is equal to the energy of at least one speech block, the energy state of that speech frame is balanced. If the energy of any speech frame is lower than the energy of at least one speech block, the energy state of that speech frame is increasing.

[0117] For example, when the energy state of any given speech frame is decreasing, the number of subsequent speech frames associated with that speech frame is increased. When subsequently obtaining the minimum power spectrum from the power spectrum of any given speech frame and the power spectra of the speech frames associated with that speech frame, the obtained minimum power spectrum is closer to the power spectrum containing only background noise. Therefore, when obtaining the noise compensation portion of the background noise based on the obtained minimum power spectrum, the accuracy of the obtained noise compensation portion is higher.

[0118] When the energy state of any given speech frame is in equilibrium, the number of speech frames associated with that speech frame remains constant or decreases. When the energy state of any given speech frame is rising, the number of subsequent speech frames associated with that speech frame decreases or remains constant. By reducing the number of subsequent speech frames associated with that speech frame, the efficiency of obtaining the minimum power spectrum when subsequently extracting it from the power spectrum of any given speech frame and the power spectra of the speech frames associated with it is higher, thus resulting in higher efficiency in generating comfortable noise.

[0119] Case 3: The speech block associated with any speech frame includes at least one speech block located before any speech frame and at least one speech block located after any speech frame.

[0120] For scenario 3, if the energy of any given speech frame is higher than the energy of all speech blocks associated with that speech frame, the energy state of that speech frame first increases and then decreases. If the energy of any given speech frame is equal to the energy of all speech blocks associated with that speech frame, the energy state of that speech frame is balanced. If the energy of any given speech frame is lower than the energy of all speech blocks associated with that speech frame, the energy state of that speech frame first decreases and then increases.

[0121] If the energy of any speech frame is higher than the energy of at least one speech block preceding it and not higher than the energy of at least one speech block following it, the energy state of that speech frame is rising. If the energy of any speech frame is lower than the energy of at least one speech block preceding it and not lower than the energy of at least one speech block following it, the energy state of that speech frame is falling.

[0122] For example, if the energy state of any given speech frame is initially rising and then falling, increase the number of preceding and following speech frames associated with that speech frame. If the energy state of any given speech frame is rising, increase the number of preceding speech frames associated with that speech frame and decrease the number of following speech frames associated with that speech frame, or keep the number of following speech frames unchanged.

[0123] When the energy state of any given speech frame is in a balanced state, the number of speech frames associated with that speech frame remains unchanged or decreases. When the energy state of any given speech frame is in a state of first decreasing and then increasing, at least one of the number of preceding or following speech frames associated with that speech frame is decreased, or the number of preceding and following speech frames remains unchanged.

[0124] If the energy state of any given speech frame is decreasing, increase the number of subsequent speech frames associated with that speech frame, decrease the number of preceding speech frames associated with that speech frame, or keep the number of preceding speech frames unchanged.

[0125] In one possible implementation, the noise compensation portion of the background noise in any given speech frame is obtained based on the power spectrum of the given speech frame and the power spectrum of the speech frames associated with the given speech frame. This includes: obtaining the minimum power spectrum in the power spectrum of the given speech frame and the power spectrum of the speech frames associated with the given speech frame; adjusting the size of the minimum power spectrum based on an adjustment factor; and obtaining the noise compensation portion of the background noise in the given speech frame based on the adjustment result.

[0126] The embodiments of this application do not limit the method for obtaining the power spectrum of any given speech frame and the minimum power spectrum among all speech frames associated with that speech frame. For example, the power spectrum of any given speech frame and the minimum power spectrum among all speech frames associated with that speech frame can be obtained directly. Taking a speech frame associated with a certain speech frame including 15 speech frames before that speech frame and 15 speech frames after that speech frame as an example, the obtained minimum power spectrum S min (k, L) can be represented by the following expression 12.

[0127] S min (k, L) = min{S(k, L-15), S(k, L-14), ..., S(k, L), ..., S(k, L+14), S(k, L+15)} (Expression 12)

[0128] Where S(k, L) represents the power spectrum of the speech frame, and S(k, L-15) represents the power spectrum of the 15th speech frame preceding the speech frame. The power spectrum of the remaining speech frames is represented in the same way and will not be repeated here. S(k, L) can be calculated according to the following formula 4.

[0129] S(k,L)=αS(k,L-1)+(1-α)Y(k,L) (Formula 4)

[0130] Where α is set according to experience or actual needs, S(k, 0) can be a preset value, and Y(k, L) is the frequency domain representation of the speech frame.

[0131] For example, first obtain the minimum power spectrum of the preceding speech frame associated with any given speech frame, and the minimum power spectrum of the following speech frame associated with any given speech frame. Then obtain the minimum power spectrum of the preceding speech frame, the minimum power spectrum of the following speech frame, and the minimum power spectrum of the given speech frame. When performing the operations of obtaining the minimum power spectrum of the preceding speech frame and obtaining the minimum power spectrum of the following speech frame in parallel, obtaining the power spectrum of the given speech frame and the minimum power spectrum of the speech frames associated with the given speech frame is more efficient.

[0132] Taking a specific speech frame as an example, which includes 15 speech frames preceding and 15 speech frames following that speech frame, the minimum power spectrum S among these 31 speech frames... min (k, L) can be obtained according to expressions 13 to 15 below.

[0133] S1 = min{S(k, L-15), S(k, L-14), ..., S(k, L-1)} (Expression 13)

[0134] S2 = min{S(k, L+1), ..., S(k, L+14), S(k, L+15)} (Expression 14)

[0135] S min (k, L) = min{S1, S2, S(k, L)} (Expression 15)

[0136] In one possible implementation, when adjusting the magnitude of the minimum power spectrum based on an adjustment factor, the adjustment factor can be multiplied by the power corresponding to each frequency point in the minimum power spectrum, and the result of the multiplication can be used as the noise compensation part of the background noise in any given speech frame. For example, the noise compensation part λ leak (k, L) is shown in expression 16 below.

[0137] λ leak (k, L)=θ∑ p S min (k, L) (Expression 16)

[0138] Where θ represents the adjustment factor, θ∑ p S min (k, L) represents multiplying the adjustment factor by the power corresponding to each frequency point in the minimum power spectrum. For example, θ ranges from 0.5 to 5.

[0139] Step A3: Based on the stationary noise portion and noise compensation portion of the background noise in any given speech frame, obtain the background noise in that given speech frame.

[0140] For example, the stationary noise portion and the noise-compensated portion of the background noise in any given speech frame are added together, and the result is taken as the background noise of that speech frame. That is, the background noise λ(k, L) of any given speech frame is shown in the following expression 17.

[0141] λ(k,L)=λ stat (k, L) + λ leak (k, L) (Expression 17)

[0142] Where, λ stat (k, L) represents the stationary noise portion of the background noise in any given speech frame.

[0143] Step 203 can correspond to Figure 3 The operation shown is for obtaining background noise in the speech signal.

[0144] In step 204, random noise and background noise are processed, and comfortable noise for adding to the speech signal is obtained based on the processing results.

[0145] Since the background noise obtained in step 203 may have a large power, when processing random noise and background noise, the power spectrum of the background noise can be adjusted first, and then the random noise and the adjusted background noise can be processed. By adjusting the power spectrum of the background noise, it is possible to avoid the power spectrum of the background noise being greater than that of the speech signal, thereby preventing the power spectrum of the comfort noise obtained based on the background noise from being too large, which would result in a poor listening experience.

[0146] For example, for any given speech frame, if the power spectrum of the background noise in that speech frame is less than or equal to the square of the amplitude spectrum of that speech frame, the power spectrum of the background noise remains unchanged; if the power spectrum of the background noise is greater than the square of the amplitude spectrum of that speech frame, the power spectrum of the background noise is reduced. The reduced power spectrum λ1(k, L) of the background noise can be represented by the following expression 18.

[0147]

[0148] Where λ0(k,L) represents the unreduced power spectrum of the background noise in any given speech frame, and Y 2 (k, L) represents the square of Y(k, L), and λ(k, L-1) represents the power spectrum of the background noise of the previous speech frame for any given speech frame.

[0149] In one possible implementation, the processing of random noise and background noise is a convolution process, resulting in a convolution result. Exemplarily, regardless of the method used to process the random noise and background noise, the processing is performed in the frequency domain, and the resulting result includes the initial comfort noise. The real part X0(k0, l).re of the initial comfort noise can be represented by expression 19 below, and the imaginary part X0(k0, l).im of the initial comfort noise can be represented by expression 20 below.

[0150] X0(k0, l).re=cos(f(2π·z(k0, l) / N))*λ2(k, L) (Expression 19)

[0151] X0(k0, l).im=-sin(f(2π·z(k0, l) / N))*λ2(k, L) (Expression 20)

[0152] Wherein, if the background noise is not adjusted, λ2(k,L) is λ0(k,L), and if the background noise is adjusted, λ2(k,L) is λ1(k,L).

[0153] For example, if the processing result includes initial comfort noise, this initial comfort noise can be directly used as the comfort noise for adding the speech signal, or the initial comfort noise can be adjusted to obtain the comfort noise for adding the speech signal. For example, adjusting the initial comfort noise can be done by performing gain processing on the initial comfort noise based on the gain factor corresponding to the initial comfort noise. That is, obtaining the comfort noise for adding the speech signal based on the processing result includes: obtaining the gain factor corresponding to the initial comfort noise; and performing gain processing on the initial comfort noise based on the gain factor to obtain the comfort noise for adding the speech signal.

[0154] In this embodiment of the application, obtaining the gain factor corresponding to the initial comfort noise includes, but is not limited to: obtaining the autocorrelation vector and reflection coefficient of the background noise, wherein the autocorrelation vector is used to indicate the degree of autocorrelation of the background noise and the reflection coefficient is used to indicate the amount of background noise reflected in the audio channel; and obtaining the gain factor corresponding to the initial comfort noise based on the autocorrelation vector and the reflection coefficient.

[0155] For example, the autocorrelation vector of the background noise is obtained based on the order of the linear predicted coding (LPC) used in the speech signal. The reflection coefficient of the background noise is obtained by solving the autocorrelation vector using the Schur algorithm. Based on the autocorrelation vector and the reflection coefficient, the initial gain factor corresponding to the comfort noise is obtained, including but not limited to: obtaining the maximum value between the reflection coefficient and the fifth value, and using the quotient of the autocorrelation vector and this maximum value as the initial gain factor corresponding to the comfort noise. That is, the initial gain factor gain corresponding to the comfort noise can be expressed as shown in Expression 21 below.

[0156] gain = auto_corr / max(reflect, D) (Expression 21)

[0157] Here, auto_corr represents the autocorrelation vector, reflect represents the reflection coefficient, and D represents the fifth value. For example, the fifth value is 1, and the fifth value can be represented in floating-point form.

[0158] For example, gain processing of the initial comfort noise is achieved by multiplying the gain factor with the initial comfort noise, and the result is used as the comfort noise to be added to the speech signal.

[0159] The above operations for processing random noise and background noise can be corresponding to Figure 3 The operation shown for obtaining the initial comfort noise, and the operation for obtaining the gain factor corresponding to the initial comfort noise, can be described as follows: Figure 3 The illustrated operation for obtaining the gain factor, which performs gain processing on the initial comfort noise based on the gain factor, yields the comfort noise used to add the speech signal. Figure 3 The operation for obtaining comfortable noise is shown.

[0160] In one possible implementation, the comfort noise used to incorporate the speech signal is synthesized with the speech signal in the frequency domain to obtain a speech signal with added comfort noise. This operation can correspond to... Figure 3 The operation for acquiring a speech signal with added comfort noise is shown. The real part of the speech signal with added comfort noise, out(k0, l).re, can be represented by the following expression 22, and the imaginary part of the speech signal with added comfort noise, out(k0, l).im, can be represented by the following expression 23.

[0161] out(k0, l).re = Y(k0, l).re + gain·X0(k0, l).re (Expression 22)

[0162] out(k0, l).im = Y(k0, l).im + gain·X0(k0, l).im (Expression 23)

[0163] The above Y(k0, l).re represents the real part of the speech signal, and Y(k0, l).im represents the imaginary part of the speech signal.

[0164] After obtaining the speech signal with added comfort noise, it can be converted into a time-domain representation using an inverse Fourier transform. This time-domain representation of the speech signal with added comfort noise is then sent to the server, allowing the other party's terminal to receive the speech signal sent by the server.

[0165] The method provided in this application is applied to an in-vehicle terminal, thus enabling its application in in-vehicle communication scenarios. In this method, background noise is acquired from the speech signal to which comfort noise is to be added. The generated random noise and the acquired background noise are processed, and the comfort noise used to add to the speech signal is obtained based on the processing result. Since the comfort noise is obtained by processing random noise and background noise, the energy difference between the comfort noise and background noise is small. Therefore, after adding the comfort noise to the speech signal, the energy of the noise in the resulting speech signal is more stable, and the quality of the resulting speech signal is higher. Consequently, after playing the speech signal with added comfort noise, the noise heard by the listener is more stable, resulting in a better auditory experience.

[0166] See Figure 4 This application provides a comfort noise generation device, which is applied to an in-vehicle terminal. The device includes a generation module 401, a first acquisition module 402, a second acquisition module 403, and a third acquisition module 404.

[0167] The generation module 401 is used to obtain an initial value of random noise and generate random noise based on the initial value of random noise; the first acquisition module 402 is used to receive a speech signal to which comfort noise is to be added, the speech signal including background noise, or the speech signal including background noise and human voice signal; the second acquisition module 403 is used to acquire the background noise in the speech signal; the third acquisition module 404 is used to process the random noise and background noise, and obtain the comfort noise to be added to the speech signal based on the processing result.

[0168] For example, the speech signal includes multiple speech frames. The second acquisition module 403 is used to acquire, for any one of the multiple speech frames, the power spectrum and amplitude spectrum of at least one speech block associated with the speech frame, and the power spectrum of the speech frame; acquire the human voice presence probability of at least one speech block based on the power spectrum and amplitude spectrum of the at least one speech block; and acquire the background noise in the speech frame based on the human voice presence probability of the at least one speech block and the power spectrum of the speech frame.

[0169] For example, the second acquisition module 403 is configured to acquire the stationary noise portion of the background noise in any given speech frame based on the power spectrum of speech blocks whose human voice presence probability is lower than a reference threshold; acquire the speech frame associated with the given speech frame based on the relationship between the human voice presence probability of at least one speech block and the reference threshold; acquire the noise compensation portion of the background noise in the given speech frame based on the power spectrum of the given speech frame and the power spectrum of the speech frame associated with the given speech frame; and acquire the background noise in the given speech frame based on the stationary noise portion and the noise compensation portion of the background noise in the given speech frame.

[0170] For example, at least one speech block associated with any speech frame includes speech blocks located before and / or after the speech frame; the second acquisition module 403 is configured to acquire frame count indication values ​​corresponding to each speech block based on the relationship between the probability of human voice presence in at least one speech block and a reference threshold; acquire a first number of speech frames located before and adjacent to the speech frame based on the sum of the frame count indication values ​​corresponding to the speech blocks located before the speech frame; and acquire a second number of speech frames located after and adjacent to the speech frame based on the sum of the frame count indication values ​​corresponding to the speech blocks located after the speech frame.

[0171] For example, the second acquisition module 403 is further configured to, after acquiring the speech frames associated with the speech frame, determine the energy state of the speech frame based on the energy of the speech frame and the energy of at least one speech block associated with the speech frame; and adjust the number of speech frames associated with the speech frame based on the energy state of the speech frame to obtain the adjusted speech frame associated with the speech frame, wherein the adjusted speech frame associated with the speech frame is used to acquire the noise compensation portion of the background noise in the speech frame.

[0172] For example, the second acquisition module 403 is used to acquire the power spectrum of the arbitrary speech frame and the minimum power spectrum of the power spectrum of the speech frame associated with the arbitrary speech frame; adjust the size of the minimum power spectrum based on the adjustment factor, and acquire the noise compensation part of the background noise in the arbitrary speech frame according to the adjustment result.

[0173] For example, the processing result includes initial comfort noise, and a third acquisition module 404 is used to acquire the gain factor corresponding to the initial comfort noise; and to perform gain processing on the initial comfort noise based on the gain factor to obtain comfort noise for adding the speech signal.

[0174] For example, the third acquisition module 404 is used to acquire the autocorrelation vector and reflection coefficient of the background noise, wherein the autocorrelation vector is used to indicate the degree of autocorrelation of the background noise and the reflection coefficient is used to indicate the amount of background noise reflected in the audio channel; and based on the autocorrelation vector and reflection coefficient, the gain factor corresponding to the initial comfort noise is acquired.

[0175] The device provided in this application embodiment is applied to an in-vehicle terminal, thus enabling its use in in-vehicle communication scenarios. In this device, background noise is acquired from the speech signal to which comfort noise is to be added. The generated random noise and the acquired background noise are processed, and the comfort noise used to add to the speech signal is obtained based on the processing result. Since the comfort noise is obtained by processing random noise and background noise, the energy difference between the comfort noise and background noise is small. Therefore, after adding the comfort noise to the speech signal, the energy of the noise in the resulting speech signal is more stable, and the quality of the resulting speech signal is higher. Consequently, after playing the speech signal with added comfort noise, the noise heard by the listener is more stable, resulting in a better auditory experience.

[0176] It should be noted that the apparatus provided in the above embodiments is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0177] Figure 5This is a schematic diagram of a server structure provided in an embodiment of this application. Servers can vary significantly due to differences in configuration or performance. They may include one or more processors 501 and one or more memories 502. The processor 501 may be a Central Processing Unit (CPU). The one or more memories 502 store at least one computer program, which is loaded and executed by the one or more processors 501 to enable the server to implement the comfortable noise generation method provided in the above-described method embodiments. Of course, the server may also have wired or wireless network interfaces, a keyboard, and input / output interfaces for input and output. The server may also include other components for implementing device functions, which will not be elaborated here.

[0178] Figure 6 This is a schematic diagram of the structure of a terminal provided in an embodiment of this application. The terminal can be a vehicle-mounted terminal. Typically, the terminal includes a processor 601 and a memory 602.

[0179] Processor 601 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 601 may be implemented using at least one hardware form selected from Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), and Programmable Logic Array (PLA). Processor 601 may also include a main processor and a coprocessor. The main processor, also known as the CPU, is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 601 may integrate a Graphics Processing Unit (GPU), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 601 may also include an Artificial Intelligence (AI) processor, which is used to handle computational operations related to machine learning.

[0180] The memory 602 may include one or more computer-readable storage media, which may be non-transitory. The memory 602 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 602 are used to store at least one instruction, which is executed by the processor 601 to cause the terminal to implement the method for generating comfortable noise provided in the method embodiments of this application.

[0181] In some embodiments, the terminal may also optionally include: a peripheral device interface 603 and at least one peripheral device. The processor 601, memory 602, and peripheral device interface 603 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface 603 via a bus, signal line, or circuit board. Specifically, the peripheral device includes at least one of: a radio frequency circuit 604, a display screen 605, a camera assembly 606, an audio circuit 607, a positioning assembly 608, and a power supply 609.

[0182] Peripheral interface 603 can be used to connect at least one input / output (I / O) related peripheral device to processor 601 and memory 602. In some embodiments, processor 601, memory 602 and peripheral interface 603 are integrated on the same chip or circuit board; in some other embodiments, any one or two of processor 601, memory 602 and peripheral interface 603 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0183] The radio frequency (RF) circuit 604 is used to receive and transmit radio frequency (RF) signals, also known as electromagnetic signals. The RF circuit 604 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 604 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals back into electrical signals. Optionally, the RF circuit 604 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc. The RF circuit 604 can communicate with other terminals through at least one wireless communication protocol. This wireless communication protocol includes, but is not limited to: metropolitan area networks (MANs), various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks (WLANs), and / or Wireless Fidelity (WiFi) networks. In some embodiments, the RF circuit 604 may also include circuitry related to Near Field Communication (NFC), which is not limited in this application.

[0184] Display screen 605 is used to display a user interface (UI). The UI may include graphics, text, icons, videos, and any combination thereof. When display screen 605 is a touch display screen, it also has the ability to collect touch signals on or above its surface. Touch signals can be input as control signals to processor 601 for processing. In this case, display screen 605 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there may be one display screen 605, disposed on the front panel of the terminal; in other embodiments, there may be at least two display screens, disposed on different surfaces of the terminal or in a folded design; in still other embodiments, display screen 605 may be a flexible display screen, disposed on a curved or folded surface of the terminal. Furthermore, display screen 605 may be configured as a non-rectangular irregular shape, i.e., a non-rectangular screen. Display screen 605 may be made of materials such as Liquid Crystal Display (LCD) or Organic Light-Emitting Diode (OLED).

[0185] The camera assembly 606 is used to acquire images or videos. Optionally, the camera assembly 606 includes a front-facing camera and a rear-facing camera. Typically, the front-facing camera is located on the front panel of the terminal, and the rear-facing camera is located on the back of the terminal. In some embodiments, there are at least two rear-facing cameras, which are any one of a main camera, a depth-sensing camera, a wide-angle camera, and a telephoto camera, to achieve background blurring by fusion of the main camera and the depth-sensing camera, panoramic shooting by fusion of the main camera and the wide-angle camera, virtual reality (VR) shooting, or other fusion shooting functions. In some embodiments, the camera assembly 606 may also include a flash. The flash can be a single-color temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm-light flash and a cool-light flash, which can be used for light compensation at different color temperatures.

[0186] The audio circuit 607 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, converting them into electrical signals that are input to the processor 601 for processing, or to the radio frequency circuit 604 for voice communication. For stereo sound acquisition or noise reduction purposes, multiple microphones may be used, each positioned at a different location on the terminal. The microphone may also be an array microphone or an omnidirectional microphone. The speaker is used to convert electrical signals from the processor 601 or the radio frequency circuit 604 into sound waves. The speaker may be a traditional diaphragm speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can convert electrical signals not only into audible sound waves but also into inaudible sound waves for purposes such as distance measurement. In some embodiments, the audio circuit 607 may also include a headphone jack.

[0187] The positioning component 608 is used to locate the current geographic location of the terminal in order to enable navigation or location-based services (LBS).

[0188] Power supply 609 is used to power the various components in the terminal. Power supply 609 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery. When power supply 609 includes a rechargeable battery, the rechargeable battery can support wired or wireless charging. The rechargeable battery can also be used to support fast charging technology.

[0189] In some embodiments, the terminal further includes one or more sensors 610. The one or more sensors 610 include, but are not limited to: an accelerometer 611, a gyroscope 612, a pressure sensor 613, a fingerprint sensor 614, an optical sensor 615, and a proximity sensor 616.

[0190] Accelerometer 611 can detect the magnitude of acceleration along the three coordinate axes of a coordinate system established by the terminal. For example, accelerometer 611 can be used to detect the components of gravitational acceleration along the three coordinate axes. Processor 601 can control display screen 605 to display the user interface in either a landscape or portrait view based on the gravitational acceleration signal acquired by accelerometer 611. Accelerometer 611 can also be used for games or for acquiring user motion data.

[0191] The gyroscope sensor 612 can detect the terminal's orientation and rotation angle. The gyroscope sensor 612, in conjunction with the accelerometer sensor 611, can collect the user's 3D movements on the terminal. Based on the data collected by the gyroscope sensor 612, the processor 601 can perform the following functions: motion sensing (e.g., changing the UI based on the user's tilt), image stabilization during shooting, game control, and inertial navigation.

[0192] The pressure sensor 613 can be disposed on the side bezel of the terminal and / or on the lower layer of the display screen 605. When the pressure sensor 613 is disposed on the side bezel of the terminal, it can detect the user's grip signal on the terminal, and the processor 601 can perform left / right hand recognition or quick operation based on the grip signal collected by the pressure sensor 613. When the pressure sensor 613 is disposed on the lower layer of the display screen 605, the processor 601 can control the operable controls on the UI interface based on the user's pressure operation on the display screen 605. The operable controls include at least one of button controls, scroll bar controls, icon controls, and menu controls.

[0193] The fingerprint sensor 614 is used to collect a user's fingerprint. The processor 601 identifies the user based on the fingerprint collected by the fingerprint sensor 614, or vice versa. When the user's identity is identified as trusted, the processor 601 authorizes the user to perform relevant sensitive operations, including unlocking the screen, viewing encrypted information, downloading software, making payments, and changing settings. The fingerprint sensor 614 can be located on the front, back, or side of the terminal. When the terminal has physical buttons or a manufacturer's logo, the fingerprint sensor 614 can be integrated with the physical buttons or the manufacturer's logo.

[0194] An optical sensor 615 is used to collect ambient light intensity. In one embodiment, the processor 601 can control the display brightness of the display screen 605 based on the ambient light intensity collected by the optical sensor 615. Specifically, when the ambient light intensity is high, the display brightness of the display screen 605 is increased; when the ambient light intensity is low, the display brightness of the display screen 605 is decreased. In another embodiment, the processor 601 can also dynamically adjust the shooting parameters of the camera assembly 606 based on the ambient light intensity collected by the optical sensor 615.

[0195] The proximity sensor 616, also known as a distance sensor, is typically mounted on the front panel of the terminal. The proximity sensor 616 is used to detect the distance between the user and the front of the terminal. In one embodiment, when the proximity sensor 616 detects that the distance between the user and the front of the terminal is gradually decreasing, the processor 601 controls the display screen 605 to switch from a screen-on state to a screen-off state; when the proximity sensor 616 detects that the distance between the user and the front of the terminal is gradually increasing, the processor 601 controls the display screen 605 to switch from a screen-off state to a screen-on state.

[0196] Those skilled in the art will understand that Figure 6 The structure shown does not constitute a limitation on the terminal and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0197] In an exemplary embodiment, a computer device is also provided, comprising a processor and a memory, wherein at least one computer program is stored in the memory. The at least one computer program is loaded and executed by one or more processors to enable the computer device to implement any of the aforementioned methods for generating comfortable noise.

[0198] In an exemplary embodiment, a computer-readable storage medium is also provided, which stores at least one computer program, which is loaded and executed by a processor of a computer device to enable the computer to implement any of the above-described methods for generating comfortable noise.

[0199] In one possible implementation, the aforementioned computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, floppy disk, and optical data storage device, etc.

[0200] In an exemplary embodiment, a computer program product or computer program is also provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform any of the aforementioned methods for generating comfortable noise.

[0201] It should be noted that all information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals involved in this application have been authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the first adjustment strategy involved in this application was obtained with full authorization.

[0202] It should be understood that "multiple" as used in this article refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0203] It should be noted that the terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the above exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0204] The above are merely exemplary embodiments of this application and are not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the principles of this application should be included within the protection scope of this application.

Claims

1. A method for generating comfort noise, characterized in that, The method is applied to an in-vehicle terminal, and the method includes: Obtain an initial value for random noise, and generate random noise based on the initial value for random noise; Receive a speech signal to which comfort noise is to be added, the speech signal including background noise, or the speech signal including the background noise and human voice signal, the speech signal including multiple speech frames; For any one of the plurality of speech frames, obtain the power spectrum and amplitude spectrum of at least one speech block associated with the speech frame, as well as the power spectrum of the speech frame; based on the power spectrum and amplitude spectrum of the at least one speech block, obtain the probability of human voice presence in the at least one speech block; based on the probability of human voice presence in the at least one speech block and the power spectrum of the speech frame, obtain the background noise in the speech frame. The random noise and background noise in the speech signal are processed, and a comfortable noise is obtained based on the processing result to be added to the speech signal.

2. The method according to claim 1, characterized in that, The step of obtaining the background noise in any given speech frame based on the probability of human voice presence in the at least one speech block and the power spectrum of any given speech frame includes: Based on the power spectrum of speech blocks with a human voice presence probability below a reference threshold, the stationary noise portion of the background noise in any given speech frame is obtained. Based on the relationship between the probability of human voice presence in the at least one speech block and the reference threshold, obtain the speech frame associated with any speech frame; based on the power spectrum of any speech frame and the power spectrum of the speech frame associated with any speech frame, obtain the noise compensation part of the background noise in any speech frame. Based on the stationary noise portion and noise compensation portion of the background noise in any given speech frame, the background noise in any given speech frame is obtained.

3. The method according to claim 2, characterized in that, The at least one speech block associated with any given speech frame includes a speech block located before the given speech frame and / or a speech block located after the given speech frame; obtaining the speech frame associated with any given speech frame based on the relationship between the probability of human voice presence in the at least one speech block and the reference threshold includes: Based on the relationship between the probability of human voice presence in the at least one speech block and the reference threshold, obtain the frame number indication value corresponding to each speech block; Based on the sum of the frame number indication values ​​corresponding to the speech blocks located before any speech frame, obtain a first number of speech frames located before and adjacent to any speech frame. Based on the sum of the frame number indication values ​​corresponding to the speech blocks located after any one speech frame, a second number of speech frames located after and adjacent to any one speech frame are obtained.

4. The method according to claim 2 or 3, characterized in that, After obtaining the speech frame associated with any speech frame based on the relationship between the probability of human voice presence in the at least one speech block and the reference threshold, the method further includes: The energy state of any given speech frame is determined based on the energy of any given speech frame and the energy of at least one speech block associated with the given speech frame. Based on the energy state of any given speech frame, the number of speech frames associated with that speech frame is adjusted to obtain an adjusted speech frame associated with that speech frame. The adjusted speech frame associated with that speech frame is used to obtain the noise compensation portion of the background noise in that speech frame.

5. The method according to claim 2, characterized in that, The step of obtaining the noise compensation portion of the background noise in any given speech frame based on the power spectrum of any given speech frame and the power spectrum of speech frames associated with that given speech frame includes: Obtain the power spectrum of any given speech frame and the minimum power spectrum among the power spectra of the speech frames associated with that given speech frame; The magnitude of the minimum power spectrum is adjusted based on the adjustment factor, and the noise compensation part of the background noise in any speech frame is obtained according to the adjustment result.

6. The method according to any one of claims 1-3 and 5, characterized in that, The processing result includes initial comfort noise, and the process of obtaining comfort noise for incorporating the speech signal based on the processing result includes: Obtain the gain factor corresponding to the initial comfort noise; The initial comfort noise is amplified based on the gain factor to obtain the comfort noise used to incorporate the speech signal.

7. The method according to claim 6, characterized in that, The step of obtaining the gain factor corresponding to the initial comfort noise includes: Obtain the autocorrelation vector and reflection coefficient of the background noise in the speech signal, wherein the autocorrelation vector is used to indicate the degree of autocorrelation of the background noise, and the reflection coefficient is used to indicate the amount of background noise reflected in the vocal tract; The gain factor corresponding to the initial comfort noise is obtained based on the autocorrelation vector and the reflection coefficient.

8. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing at least one computer program, the at least one computer program being loaded and executed by the processor to enable the computer device to implement the method for generating comfort noise as described in any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one computer program, which is loaded and executed by a processor to enable the computer to implement the method for generating comfort noise as described in any one of claims 1-7.

Images