An audio extraction method, device and electronic equipment

Abstract

The application provides an audio extraction method and device and electronic equipment, and relates to the technical field of communication. The method comprises the following steps: obtaining a mixed signal to be processed and a voiceprint feature vector of target human voice to be extracted; inputting the mixed signal and the voiceprint feature vector into a complex long short-term memory (LSTM) network model to obtain first output data; performing compensation processing on the first output data according to the mixed signal and the voiceprint feature vector to obtain second output data; and obtaining audio data of the target human voice by using the second output data. The scheme of the application can solve the problem of poor quality of extracted audio information of target human voice in the current target speaker voice enhancement technology.

Description

Technical Field

[0001] This invention relates to the field of communication technology, and in particular to an audio extraction method, apparatus, and electronic device. Background Technology

[0002] With the development of artificial intelligence technology, the requirements for voice quality in communication are becoming increasingly stringent. In real-life scenarios, voice audio is often filled with many interfering sound sources, affecting human perception, understanding, and the efficiency of human-computer interaction. The task of highlighting the target speaker's voice and suppressing noise is called speech enhancement technology. Depending on the application scenario, speech enhancement technology can be further subdivided, such as noise reduction algorithms for simple noise reduction, echo cancellation technology for removing echoes, beamforming technology for highlighting audio from a specific direction, and target speaker voice enhancement technology for audio from multiple speakers. Currently, target speaker voice enhancement technology mainly utilizes the amplitude information of the mixed signal to obtain the audio data of the target speaker to be extracted, but it suffers from the problem of poor quality of the extracted audio information. Summary of the Invention

[0003] The purpose of this invention is to provide an audio extraction method, apparatus, and electronic device to solve the problem that current target speaker voice enhancement technologies suffer from poor quality in extracting audio information of the target speaker's voice.

[0004] To achieve the above objectives, embodiments of the present invention provide an audio extraction method, comprising:

[0005] Obtain the mixed signal to be processed and the voiceprint feature vector of the target human voice to be extracted;

[0006] The mixed signal and the voiceprint feature vector are input into a complex long short-term memory (LSTM) network model to obtain the first output data;

[0007] The first output data is compensated based on the mixed signal and the voiceprint feature vector to obtain the second output data; wherein, the second output data is frequency domain data;

[0008] The audio data of the target human voice is obtained using the second output data; wherein the audio data is time-domain data.

[0009] Optionally, the voiceprint feature vector of the target human voice to be extracted is obtained, including:

[0010] Obtain the registered voice data of the target human voice to be extracted;

[0011] The registered voice data is input into the auxiliary network model to obtain the voiceprint feature vector of the target human voice to be extracted.

[0012] Optionally, the step of inputting the mixed signal and the voiceprint feature vector into a complex LSTM network model to obtain the first output data includes:

[0013] The mixed signal is divided into a real part and an imaginary part;

[0014] The real and imaginary parts of the mixed signal and the voiceprint feature vector are input into the first LSTM layer of the complex LSTM network model to obtain the first sub-output data;

[0015] The first sub-output data and the voiceprint feature vector are input into the second LSTM layer of the complex LSTM network model to obtain the first output data.

[0016] Optionally, the step of compensating the first output data based on the mixed signal and the voiceprint feature vector to obtain the second output data includes:

[0017] According to the channel dimension, the mixed signal, the voiceprint feature vector and the first output data are concatenated to obtain the concatenated pre-extracted data;

[0018] The pre-extracted data is divided into M sub-bands across the entire frequency band; where M is a positive integer.

[0019] The M sub-band data are respectively input into a Convolutional Neural Network (CNN) model to obtain the second output data.

[0020] Optionally, the step of inputting the M sub-band data into the CNN model respectively to obtain the second output data includes:

[0021] For each of the M sub-band data, the sub-band data is input into the first CNN layer of the CNN model to obtain the second sub-output data with a channel dimension of N1;

[0022] The second sub-output data corresponding to each sub-band data is sequentially input into the second CNN layer in the CNN model to obtain the third sub-output data with a channel dimension of N2; wherein, the third sub-output data is determined by concatenating the output data of the M sub-band data through the second CNN layer according to the frequency band dimension;

[0023] The third sub-output data is input into the third CNN layer of the CNN model to obtain the second output data with a channel dimension of 2; wherein, one channel dimension is the real part data and the other channel dimension is the imaginary part data;

[0024] Where N1 and N2 are both positive integers.

[0025] Optionally, the step of sequentially inputting the second sub-output data corresponding to each sub-band data into the second CNN layer of the CNN model to obtain the third sub-output data with a channel dimension of N2 includes:

[0026] The following steps are executed sequentially according to the descending order of the frequency points corresponding to the M sub-band data:

[0027] Step 1: Input the second sub-output data corresponding to the i-th sub-band data into the first CNN unit in the second CNN layer to obtain the output data of the first CNN unit with a channel dimension of N2;

[0028] Step 2: Input the output data of the first CNN unit into the second CNN unit in the second CNN layer to obtain the output data of the second CNN unit with a channel dimension of N1; wherein, the output data of the second CNN unit is used to input the second sub-output data corresponding to the (i+1)th sub-band data into the second CNN layer together;

[0029] Step 3: According to the frequency band dimension, the output data of the first CNN unit corresponding to the M sub-band data are concatenated to obtain the concatenated output data with a channel dimension of N2;

[0030] Step 4: Input the spliced ​​output data with channel dimension N2 into the third CNN unit in the second CNN layer to obtain the output data of the third CNN unit with channel dimension N1, and use the output data of the third CNN unit as the input data of the first CNN unit, and return to step 1 to repeat the process multiple times;

[0031] Step 5: Determine the output data with channel dimension N2 obtained from the last splicing as the third sub-output data.

[0032] Optionally, obtaining the audio data of the target human voice based on the second output data includes:

[0033] The product of the second output data and the mixed signal is subjected to inverse Fourier transform processing to obtain the audio data of the target human voice.

[0034] To achieve the above objectives, embodiments of the present invention provide an audio extraction device, comprising:

[0035] The acquisition module is used to acquire the mixed signal to be processed and the voiceprint feature vector of the target human voice to be extracted;

[0036] The first processing module is used to input the mixed signal and the voiceprint feature vector into a complex LSTM network model to obtain the first output data;

[0037] The second processing module is used to perform compensation processing on the first output data according to the mixed signal and the voiceprint feature vector to obtain the second output data; wherein, the second output data is frequency domain data;

[0038] An extraction module is used to obtain audio data of the target human voice using the second output data; wherein the audio data is time-domain data.

[0039] Optionally, the acquisition module includes:

[0040] The acquisition unit is used to acquire the registered voice data of the target human voice to be extracted;

[0041] The first processing unit is used to input the registered voice data into the auxiliary network model to obtain the voiceprint feature vector of the target human voice to be extracted.

[0042] Optionally, the first processing module includes:

[0043] The first partitioning unit is used to divide the mixed signal into a real part and an imaginary part;

[0044] The second processing unit is used to input the real and imaginary parts of the mixed signal and the voiceprint feature vector into the first LSTM layer of the complex LSTM network model to obtain the first sub-output data.

[0045] The third processing unit is used to input the first sub-output data and the voiceprint feature vector into the second LSTM layer of the complex LSTM network model to obtain the first output data.

[0046] Optionally, the second processing module includes:

[0047] The splicing unit is used to splice the mixed signal, the voiceprint feature vector and the first output data according to the channel dimension to obtain the spliced ​​pre-extracted data;

[0048] The second partitioning unit is used to divide the pre-extracted data into M sub-bands across the entire frequency band; where M is a positive integer.

[0049] The fourth processing unit is used to input the M sub-band data into the CNN model respectively to obtain the second output data.

[0050] Optionally, the fourth processing unit is further configured to:

[0051] For each of the M sub-band data, the sub-band data is input into the first CNN layer of the CNN model to obtain the second sub-output data with a channel dimension of N1;

[0052] The second sub-output data corresponding to each sub-band data is sequentially input into the second CNN layer in the CNN model to obtain the third sub-output data with a channel dimension of N2; wherein, the third sub-output data is determined by concatenating the output data of the M sub-band data through the second CNN layer according to the frequency band dimension;

[0053] The third sub-output data is input into the third CNN layer of the CNN model to obtain the second output data with a channel dimension of 2; wherein, one channel dimension is the real part data and the other channel dimension is the imaginary part data;

[0054] Where N1 and N2 are both positive integers.

[0055] Optionally, the fourth processing unit is further configured to:

[0056] The following steps are executed sequentially according to the descending order of the frequency points corresponding to the M sub-band data:

[0057] Step 1: Input the second sub-output data corresponding to the i-th sub-band data into the first CNN unit in the second CNN layer to obtain the output data of the first CNN unit with a channel dimension of N2;

[0058] Step 2: Input the output data of the first CNN unit into the second CNN unit in the second CNN layer to obtain the output data of the second CNN unit with a channel dimension of N1; wherein, the output data of the second CNN unit is used to input the second sub-output data corresponding to the (i+1)th sub-band data into the second CNN layer together;

[0059] Step 3: According to the frequency band dimension, the output data of the first CNN unit corresponding to the M sub-band data are concatenated to obtain the concatenated output data with a channel dimension of N2;

[0060] Step 4: Input the spliced ​​output data with channel dimension N2 into the third CNN unit in the second CNN layer to obtain the output data of the third CNN unit with channel dimension N1, and use the output data of the third CNN unit as the input data of the first CNN unit, and return to step 1 to repeat the process multiple times;

[0061] Step 5: Determine the output data with channel dimension N2 obtained from the last splicing as the third sub-output data.

[0062] Optionally, the acquisition module includes:

[0063] The transformation unit is used to perform inverse Fourier transform processing on the product of the second output data and the mixed signal to obtain the audio data of the target human voice.

[0064] To achieve the above objectives, embodiments of the present invention provide a mobile terminal, including a transceiver, a processor, a memory, and a program or instructions stored in the memory and executable on the processor; when the processor executes the program or instructions, it implements the steps of the audio extraction method described above.

[0065] To achieve the above objectives, embodiments of the present invention provide a readable storage medium having a program or instructions stored thereon, which, when executed by a processor, implement the steps of the audio extraction method described above.

[0066] The beneficial effects of the above-described technical solution of the present invention are as follows:

[0067] In this embodiment of the invention, by acquiring the mixed signal to be processed and the voiceprint feature vector of the target human voice to be extracted, the mixed signal and the voiceprint feature vector are input into a complex LSTM network model to obtain first output data. The first output data is then compensated based on the mixed signal and the voiceprint feature vector to obtain second output data. This ensures that the audio data of the target human voice obtained using the second output data has high quality, and can solve the problem of poor quality of audio information extracted from the target human voice in current target speaker speech enhancement technologies. Attached Figure Description

[0068] Figure 1 This is a flowchart of the audio extraction method according to an embodiment of the present invention;

[0069] Figure 2 This is a schematic diagram of the audio extraction architecture according to an embodiment of the present invention;

[0070] Figure 3 This is a schematic diagram of the CNN architecture according to an embodiment of the present invention;

[0071] Figure 4 This is a block diagram of the audio extraction device according to an embodiment of the present invention;

[0072] Figure 5 This is a block diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0073] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0074] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of the invention. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification do not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments.

[0075] In various embodiments of the present invention, it should be understood that the sequence number of each process described below does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0076] In addition, the terms "system" and "network" are often used interchangeably in this article.

[0077] In the embodiments provided in this application, it should be understood that "B corresponding to A" means that B is associated with A, and B can be determined based on A. However, it should also be understood that determining B based on A does not mean determining B solely based on A; B can also be determined based on A and / or other information.

[0078] Target speaker speech enhancement technology, also known as target speaker speech extraction, refers to extracting the speech content of a target speaker from pre-recorded registered speech information when multiple speakers coexist, treating other voices as interference sources. Significant progress has been made in research on target speaker speech extraction, with numerous research results in areas such as network structure, optimized input features, and loss function values, further promoting the development of this field. A relatively mature speech filter algorithm is applied, whose network framework mainly consists of two parts: an auxiliary network structure for extracting the target speaker's voiceprint information, or speaker-embedding, and a main network structure that uses the speaker-embedding as a reference. During training, the auxiliary network and the main network can be trained jointly or separately. The network input can be a time-frequency signal, such as the power spectrum after Fourier transform, or a time-domain signal. The Spex and Spex+ algorithms are typical end-to-end processing models, with both input and output being time-domain signals. The network framework mainly includes an encoder that encodes the time-domain signal, a speaker extractor (the main module) that extracts the signal, a decoder that performs the inverse transform of the time-domain signal encoding, and a speaker embedding extractor that extracts the speaker's registered speech's voiceprint feature vector. The difference between Spex and Spex+ is that the encoder weights are shared between the main and auxiliary networks.

[0079] like Figure 1 As shown, an audio extraction method according to an embodiment of the present invention includes the following steps:

[0080] Step 11: Obtain the mixed signal to be processed and the voiceprint feature vector of the target human voice to be extracted.

[0081] Optionally, the mixed signal to be processed may include audio data of the target human voice to be extracted, or it may not include audio data of the target human voice to be extracted.

[0082] Optionally, the voiceprint feature vector of the target human voice to be extracted can be used as a reference for the main network to extract the speech of the target speaker.

[0083] Step 12: Input the mixed signal and the voiceprint feature vector into the complex LSTM network model to obtain the first output data.

[0084] Step 13: Perform compensation processing on the first output data based on the mixed signal and the voiceprint feature vector to obtain the second output data.

[0085] The second output data is frequency domain data.

[0086] Optionally, the main network includes the complex LSTM network model and is capable of compensating the first output data based on the mixed signal and the voiceprint feature vector to obtain the second output data.

[0087] Step 14: Use the second output data to obtain the audio data of the target human voice.

[0088] The audio data is time-domain data.

[0089] In the above scheme, by acquiring the mixed signal to be processed and the voiceprint feature vector of the target human voice to be extracted, the mixed signal and the voiceprint feature vector are input into a complex LSTM network model to obtain the first output data. The first output data is then compensated based on the mixed signal and the voiceprint feature vector to obtain the second output data. This ensures that the audio data of the target human voice obtained using the second output data has high quality, and can solve the problem of poor quality of audio information extracted from the target human voice in current target speaker speech enhancement technologies.

[0090] like Figure 2 As shown, the architecture for audio extraction is presented. Specifically, the speech signal is first mathematically modeled: the mathematical model of a single-channel noise signal can be expressed as mix(t) = s1(t) + s2(t). After performing a short-time Fourier transform on the noise signal, it is expressed as Mix(t,f) = S1(t,f) + S2(t,f), where each specific t and f value is called a time-frequency unit. s1 represents sound source 1, and s2 represents sound source 2; mix may or may not contain the target speaker's speech. Here, the number of Fourier transform points is 256. The audio extraction architecture based on mathematical modeling extracts the audio data of the target human voice.

[0091] Optionally, the voiceprint feature vector of the target human voice to be extracted is obtained, including:

[0092] Obtain the registered voice data of the target human voice to be extracted;

[0093] The registered voice data is input into the auxiliary network model to obtain the voiceprint feature vector of the target human voice to be extracted.

[0094] Specifically, the requirements for obtaining the registered voice corresponding to the target human voice are as follows: the voice content can be any content, but repeated words should be avoided, and more factors should be included, such as the voice spoken at the most common speaking speed and tone. This embodiment of the invention is not limited to these.

[0095] Furthermore, the voice recognition network (ECAPA-TDNN) model, i.e., the auxiliary network model, is used to obtain the speaker-embedding of the target speaker's voice, which serves as a reference for the main network model to extract the target speaker's speech. The input data for the auxiliary network model comes from the target speaker's registered speech (Enroll_speech) data. To keep the input features within the same feature range as the main network model, a mixed power spectrum, such as Mix_power_spectrum = |Enroll_speech(t,f)|, can be used instead of the frequency domain features (Fbank) used by the ECAPA-TDNN model. During training, the ECAPA-TDNN model can be trained separately, where the loss function can be calculated using AAMsoftmax; the auxiliary network model extracts a 256-dimensional speaker-embedding.

[0096] When the main network model extracts the audio data of the target human voice based on the speaker-embedding, it first obtains the mixed signal to be processed (Mixtrue_speech). Mixtrue_speech may contain audio data of the target human voice or may not contain audio data of the target human voice. Then, Mixtrue_speech and speaker-embedding are input together into the extractor network to estimate the audio information of the target human voice, that is, the speech content of the target human voice.

[0097] To learn more useful information, the input features of the main network model are the real part (Mix_real), the imaginary part (Mix_imag), and the speaker embedding of the mixed signal. The real part of the mixed signal is represented as Mix_real = real(Mixtrue_speech(t,f)), and the imaginary part is represented as Mix_imag = img(Mixtrue_speech(t,f)). Here, real and img represent the extraction of the real and imaginary parts of the complex number, respectively.

[0098] The main network model consists of a complex LSTM network model (i.e., a CLSTM network model) and a sub-band CNN model: the outputs of both the complex LSTM network model and the sub-band CNN model are complex mask values ​​(cmask);

[0099] Optionally, the step of inputting the mixed signal and the voiceprint feature vector into a complex LSTM network model to obtain the first output data includes:

[0100] The mixed signal is divided into a real part and an imaginary part; optionally, the dimension of the mixed signal is 256.

[0101] The real and imaginary parts of the mixed signal and the voiceprint feature vector are input into the first LSTM layer of the complex LSTM network model to obtain the first sub-output data;

[0102] The first sub-output data and the voiceprint feature vector are input into the second LSTM layer of the complex LSTM network model to obtain the first output data.

[0103] Specifically, the output data of the complex LSTM network model can be represented as: clstm_out = fcdnn(fclstm2(fclstm1(Mix_real,Mix_imag,speaker-embedding),speaker-embedding)); where fclstm1 is a two-layer LSTM (i.e., the first LSTM layer, including two LSTMs), and the input data consists of Mix_real, Mix_imag, and speaker-embedding, with the last 256 frequency bands of Mix_real and Mix_imag, so the input nodes have a dimension of 256*3 = 768, and the hidden nodes have a dimension of 512; fclstm2 is a two-layer LSTM (i.e., the second LSTM layer, including two LSTMs), and the input data consists of the output of fclstm1 and speaker-embedding, so the input nodes have a dimension of 512 + 256 = 768; thus, the output of the complex LSTM network model is the estimated complex mask value cmask. The loss function of the complex LSTM network model is expressed as loss1=|S_real*factor-real(cmask)*Mix_real|+|S_imag*factor-imag(cmask)*Mix_imag|. When there is no audio data of the target human voice in the mixed signal, factor=0; when there is audio data of the target human voice in the mixed signal, factor=1; S_real and S_imag represent the real part and imaginary part of the short-time Fourier transform S(t,f) of the audio signal of the target human voice, respectively.

[0104] Optionally, the step of compensating the first output data based on the mixed signal and the voiceprint feature vector to obtain the second output data includes:

[0105] According to the channel dimension, the mixed signal, the voiceprint feature vector and the first output data are concatenated to obtain the concatenated pre-extracted data;

[0106] The pre-extracted data is divided into M sub-bands across the entire frequency band; where M is a positive integer.

[0107] The M sub-band data are respectively input into the CNN model to obtain the second output data.

[0108] To further improve the speech quality of the extracted target human voice audio data, this embodiment of the invention employs a sub-band CNN model. After processing by a complex LSTM network model, the extracted target human voice audio data may exhibit some harmonic loss due to the presence of time-frequency units with low signal-to-interference ratios (SNR = |target sound source energy| / |interfering sound source energy|). This results in distortion. Therefore, a certain degree of harmonic compensation can be achieved through the sub-band CNN model.

[0109] Based on the harmonic information extracted from the audio data of the target human voice in the analysis section, it can be seen that at a certain moment when harmonic loss occurs, the loss phenomenon does not exist in the entire frequency band. The harmonic loss mainly occurs in the sub-band where the harmonics of the target sound source and the interference sound source have a high degree of overlap, and in the time-frequency unit where the interference sound source has strong energy. Moreover, the phenomenon is more obvious in the low-frequency region.

[0110] Based on the above analysis, the input data of the sub-band CNN model in this embodiment of the invention consists of the real and imaginary parts of the last 256 dimensions of the mixed signal, the real and imaginary parts of the last 256 dimensions of the data output by the complex LSTM network model, and the speaker-embedding. The real and imaginary parts of the last 256 dimensions of the mixed signal, the real and imaginary parts of the last 256 dimensions of the data output by the complex LSTM network model, and the speaker-embedding are concatenated according to the channel dimension, such as finally obtaining the pre-extracted data with channel=5.

[0111] Furthermore, the pre-extracted data is divided into M sub-bands across the entire frequency band. For example, the pre_extract_data is divided into 4 sub-bands with 64 frequency points in each sub-band, and these are then input into the sub-band CNN model (subband_Cnn). The weight coefficients of the sub-band_Cnn models can be shared between the sub-bands.

[0112] Optionally, the step of inputting the M sub-band data into the CNN model respectively to obtain the second output data includes:

[0113] For each of the M sub-band data, the sub-band data is input into the first CNN layer of the CNN model to obtain the second sub-output data with a channel dimension of N1 (for ease of distinction, the second sub-output data is referred to as sub-output data 1 here);

[0114] The second sub-output data (i.e., sub-output data 1) corresponding to each sub-band data is sequentially input into the second CNN layer in the CNN model to obtain the third sub-output data with a channel dimension of N2 (for ease of distinction, the third sub-output data is referred to as sub-output data 2 here); wherein, the sub-output data with a channel dimension of N2 is determined by concatenating the output data of the M sub-band data through the second CNN layer according to the frequency band dimension;

[0115] The third sub-output data (i.e., sub-output data 2) is input into the third CNN layer of the CNN model to obtain the second output data with a channel dimension of 2; wherein, one channel dimension is the real part data and the other channel dimension is the imaginary part data;

[0116] Where N1 and N2 are both positive integers.

[0117] For example, when M subband data are input into a subband CNN model, the input of the next subband CNN layer consists of the real and imaginary parts of the corresponding subband data, the real and imaginary parts of the output data of the previous subband CNN layer, and the speaker-embedding.

[0118] Optionally, the step of sequentially inputting the second sub-output data (i.e., sub-output data 1) corresponding to each sub-band data into the second CNN layer of the CNN model to obtain the third sub-output data (i.e., sub-output data 2) with a channel dimension of N2 includes:

[0119] The following steps are executed sequentially according to the descending order of the frequency points corresponding to the M sub-band data:

[0120] Step 1: Input the second sub-output data (i.e., sub-output data 1) corresponding to the i-th sub-band data into the first CNN unit in the second CNN layer to obtain the output data of the first CNN unit with a channel dimension of N2 (for ease of distinction, the output data of the first CNN unit with a channel dimension of N2 is referred to as sub-output data 11); where i is a positive integer and i≤M;

[0121] Step 2: Input the output data of the first CNN unit (i.e., sub-output data 11) into the second CNN unit in the second CNN layer to obtain the output data of the second CNN unit with channel dimension N1 (for ease of distinction, the output data of the second CNN unit with channel dimension N1 is referred to as sub-output data 12); wherein, the output data of the second CNN unit (i.e., sub-output data 12) is used together with the second sub-output data (i.e., sub-output data 1) corresponding to the (i+1)th sub-band data to be input into the second CNN layer;

[0122] Step 3: According to the frequency band dimension, the output data of the first CNN unit corresponding to the M sub-band data (i.e., sub-output data 11) are concatenated to obtain the concatenated output data with a channel dimension of N2 (for easy distinction, the concatenated output data with a channel dimension of N2 is referred to as sub-output data 11M).

[0123] Step 4: Input the spliced ​​output data with channel dimension N2 (i.e., sub-output data 11M) into the third CNN unit in the second CNN layer to obtain the output data of the third CNN unit with channel dimension N1 (for ease of distinction, the output data of the third CNN unit is referred to as sub-output data 13). Use the output data of the third CNN unit (i.e., sub-output data 13) as the input data of the first CNN unit, and return to step 1 to repeat the process multiple times.

[0124] Step 5: Determine the output data with channel dimension N2 obtained from the last splicing (i.e., sub-output data 11M') as the third sub-output data (i.e., sub-output data 2).

[0125] Specifically, since this distortion is more pronounced in the low-frequency region, a sub-band processing order from high to low frequencies can be set. The specific process is as follows: Figure 3 As shown:

[0126] The subband_Cnn model consists of three CNNs. The first CNN (i.e., the first CNN layer) expands the output data dimension of the complex LSTM network model, such as transforming channel=5 into channel=12.

[0127] The second group of CNNs (i.e., the second CNN layer) is implemented by three CNNs in a loop. If the second group of CNNs is defined as cnn_circle, then the first CNN of cnn_circle (i.e., the first CNN unit) can further expand the channel dimension, increasing it from 12 to 32. Its output data is defined as CNN_OUT, with a dimension of channel * subband * t, where channel = 32 and subband = 64. Here, t represents a frame. The second CNN of cnn_circle (i.e., the second CNN unit) provides a reference for the subband data for the next subband_Cnn. Its input data is CNN_OUT, and the channel dimension is restored from 32 to 12. The CNN_OUT of the four subbands are concatenated along the frequency band dimension to obtain CNN_OUT_cat, with a dimension of 32 * 256 * t. The third CNN of cnn_circle (i.e., the third CNN unit) restores the channel dimension of CNN_OUT_cat from 32 to 12, obtaining the input data CNN_IN for the next loop. The CNN_IN is used as the input to cnn_circle again, and the calculation is repeated 6 times (the specific number of repetitions can be set according to actual needs, and this embodiment of the invention is not limited to this).

[0128] The last CNN layer (the third CNN layer) operates on CNN_OUT_cat, reducing the dimension from 32*256*t to 2*256*t, with two channels representing the real and imaginary parts respectively. Finally, two DNNs map the real and imaginary parts of 2*256*t to 257*t.

[0129] Here, the output data of the subband_Cnn model is the complex masking threshold (cmask2). The loss function of the subband_Cnn model can be expressed as loss2=|S_real*factor-real(cmask2)*Mix_real|+|S_imag*factor-imag(cmask2)*Mix_imag|. Therefore, the loss function of the main network model is expressed as loss=loss1+loss2.

[0130] Optionally, obtaining the audio data of the target human voice based on the second output data includes:

[0131] The product of the second output data and the mixed signal is subjected to inverse Fourier transform processing to obtain the audio data of the target human voice.

[0132] Specifically, the second output data is the complex masking threshold cmask2. The product of the second output data and the above-mentioned mixed signal to be processed, i.e. [real(cmask2)*Mix_real, imag(cmask2)*Mix_imag], is subjected to inverse Fourier transform processing to obtain the audio data of the target human voice in the time domain.

[0133] In the above-described solution of the present invention, the method of extracting audio data of the target speaker's voice through a single channel can be applied to any scenario requiring improved auditory perception of the target speaker's voice, such as a meeting scenario, to extract the speech content of the speaker of interest. Furthermore, the above-described solution of the present invention uses a high-performance auxiliary network model, and to ensure that its analysis scope is the same as that of the main network model, the auxiliary network model can use amplitude spectrum. When extracting the target speaker's voice, the main network model estimates the complex mask, meaning that the main network model performs phase estimation while simultaneously estimating amplitude. By adding a subband_Cnn model to the main network model, the quality of the extracted target speaker's voice is further improved, and the loss function calculation can also consider the case where the target speaker's audio does not appear in the mixed signal, thereby improving the accuracy of the extracted target speaker's voice.

[0134] like Figure 4 As shown, an audio extraction device 400 according to an embodiment of the present invention includes:

[0135] The acquisition module 410 is used to acquire the mixed signal to be processed and the voiceprint feature vector of the target human voice to be extracted;

[0136] The first processing module 420 is used to input the mixed signal and the voiceprint feature vector into a complex long short-term memory (LSTM) network model to obtain the first output data.

[0137] The second processing module 430 is used to perform compensation processing on the first output data according to the mixed signal and the voiceprint feature vector to obtain second output data; wherein, the second output data is frequency domain data;

[0138] The acquisition module 440 is used to acquire the audio data of the target human voice using the second output data; wherein the audio data is time-domain data.

[0139] Optionally, the acquisition module 410 includes:

[0140] The acquisition unit is used to acquire the registered voice data of the target human voice to be extracted;

[0141] The first processing unit is used to input the registered voice data into the auxiliary network model to obtain the voiceprint feature vector of the target human voice to be extracted.

[0142] Optionally, the first processing module 420 includes:

[0143] The first partitioning unit is used to divide the mixed signal into a real part and an imaginary part;

[0144] The second processing unit is used to input the real and imaginary parts of the mixed signal and the voiceprint feature vector into the first LSTM layer of the complex LSTM network model to obtain the first sub-output data.

[0145] The third processing unit is used to input the first sub-output data and the voiceprint feature vector into the second LSTM layer of the complex LSTM network model to obtain the first output data.

[0146] Optionally, the second processing module 430 includes:

[0147] The splicing unit is used to splice the mixed signal, the voiceprint feature vector and the first output data according to the channel dimension to obtain the spliced ​​pre-extracted data;

[0148] The second partitioning unit is used to divide the pre-extracted data into M sub-bands across the entire frequency band; where M is a positive integer.

[0149] The fourth processing unit is used to input the M sub-band data into the convolutional neural network (CNN) model respectively to obtain the second output data.

[0150] Optionally, the fourth processing unit is further configured to:

[0151] For each of the M sub-band data, the sub-band data is input into the first CNN layer of the CNN model to obtain the second sub-output data with a channel dimension of N1;

[0152] The second sub-output data corresponding to each sub-band data is sequentially input into the second CNN layer in the CNN model to obtain the third sub-output data with a channel dimension of N2; wherein, the third sub-output data is determined by concatenating the output data of the M sub-band data through the second CNN layer according to the frequency band dimension;

[0153] The third sub-output data is input into the third CNN layer of the CNN model to obtain the second output data with a channel dimension of 2; wherein, one channel dimension is the real part data and the other channel dimension is the imaginary part data;

[0154] Where N1 and N2 are both positive integers.

[0155] Optionally, the fourth processing unit is further configured to:

[0156] The following steps are executed sequentially according to the descending order of the frequency points corresponding to the M sub-band data:

[0157] Step 1: Input the second sub-output data corresponding to the i-th sub-band data into the first CNN unit in the second CNN layer to obtain the output data of the first CNN unit with a channel dimension of N2;

[0158] Step 2: Input the output data of the first CNN unit into the second CNN unit in the second CNN layer to obtain the output data of the second CNN unit with a channel dimension of N1; wherein, the output data of the second CNN unit is used to input the second sub-output data corresponding to the (i+1)th sub-band data into the second CNN layer together;

[0159] Step 3: According to the frequency band dimension, the output data of the first CNN unit corresponding to the M sub-band data are concatenated to obtain the concatenated output data with a channel dimension of N2;

[0160] Step 4: Input the spliced ​​output data with channel dimension N2 into the third CNN unit in the second CNN layer to obtain the output data of the third CNN unit with channel dimension N1, and use the output data of the third CNN unit as the input data of the first CNN unit, and return to step 1 to repeat the process multiple times;

[0161] Step 5: Determine the output data with channel dimension N2 obtained from the last splicing as the third sub-output data.

[0162] Optionally, the acquisition module 440 includes:

[0163] The transformation unit is used to perform inverse Fourier transform processing on the product of the second output data and the mixed signal to obtain the audio data of the target human voice.

[0164] It should be noted that the above-described apparatus in the embodiments of the present invention can implement various embodiments of the above-described method and achieve the same technical effect. To avoid repetition, it will not be described again here.

[0165] This invention also provides an electronic device, such as... Figure 5 As shown, it includes a transceiver 510, a processor 500, a memory 520, and a program or instructions stored in the memory 520 and executable on the processor 500; when the processor 500 executes the program or instructions, it implements the steps described above for the audio extraction method and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0166] The transceiver 510 is used to receive and send data under the control of the processor 500.

[0167] Among them, Figure 5 In this context, the bus architecture may include any number of interconnected buses and bridges, specifically linking various circuits together, represented by one or more processors (processor 500) and memory (memory 520). The bus architecture may also link together various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. The bus interface provides an interface. The transceiver 510 may be multiple elements, including transmitters and receivers, providing a unit for communicating with various other devices over a transmission medium. The processor 500 is responsible for managing the bus architecture and general processing, and the memory 520 may store data used by the processor 500 during operation.

[0168] An embodiment of the present invention provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the steps in the audio extraction method described above and achieve the same technical effect. To avoid repetition, further details are omitted here.

[0169] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0170] It should be further noted that the terminals described in this specification include, but are not limited to, smartphones, tablets, etc., and many of the functional components described are referred to as modules in order to emphasize the independence of their implementation.

[0171] In this embodiment of the invention, the module can be implemented in software so that it can be executed by various types of processors. For example, an identified executable code module may include one or more physical or logical blocks of computer instructions, which may be constructed as objects, procedures, or functions. Nevertheless, the executable code of the identified module does not need to be physically located together, but may include different instructions stored in different bits, which, when logically combined, constitute the module and achieve the module's intended purpose.

[0172] In practice, an executable code module can be a single instruction or many instructions, and can even be distributed across multiple different code segments, different programs, and across multiple memory devices. Similarly, operational data can be identified within the module and can be implemented in any suitable form and organized within any suitable type of data structure. This operational data can be collected as a single dataset or distributed across different locations (including different storage devices), and can exist, at least in part, solely as electronic signals within the system or network.

[0173] When a module can be implemented using software, considering the current level of hardware technology, modules that can be implemented in software can be implemented using hardware circuits by those skilled in the art to achieve the corresponding functions, without considering cost. These hardware circuits include conventional very-large-scale integrated circuits (VLSI) or gate arrays, as well as existing semiconductors such as logic chips and transistors, or other discrete components. Modules can also be implemented using programmable hardware devices, such as field-programmable gate arrays, programmable array logic, and programmable logic devices.

[0174] The exemplary embodiments described above are with reference to the accompanying drawings. Many different forms and embodiments are feasible without departing from the spirit and teachings of the invention. Therefore, the invention should not be construed as limiting the exemplary embodiments set forth herein. Rather, these exemplary embodiments are provided to make the invention complete and convey the scope of the invention to those skilled in the art. In these drawings, component dimensions and relative dimensions may be exaggerated for clarity. The terminology used herein is for the purpose of describing particular exemplary embodiments only and is not intended to be limiting. As used herein, unless clearly indicated otherwise, the singular forms “a,” “an,” and “the” are intended to include all such forms. It will be further understood that the terms “comprising” and / or “including”, when used in this specification, indicate the presence of the stated features, integers, steps, operations, components, and / or elements, but do not exclude the presence or addition of one or more other features, integers, steps, operations, components, and / or groups thereof. Unless otherwise indicated, when stated, a range of values ​​includes the upper and lower limits of the range and any subranges in between.

[0175] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. An audio extraction method, characterized in that, include: Obtain the mixed signal to be processed and the voiceprint feature vector of the target human voice to be extracted; The mixed signal and the voiceprint feature vector are input into a complex long short-term memory (LSTM) network model to obtain the first output data. The following steps are executed sequentially according to the descending order of the frequency points corresponding to the M sub-band data: Step 1: Input the second sub-output data corresponding to the i-th sub-band data into the first CNN unit in the second CNN layer of the convolutional neural network CNN model to obtain the output data of the first CNN unit with a channel dimension of N2. Step 2: Input the output data of the first CNN unit corresponding to the M sub-band data into the third CNN unit in the second CNN layer to obtain the output data of the third CNN unit with a channel dimension of N1. Then, use the output data of the third CNN unit as the input data of the first CNN unit and return to Step 1 to repeat the process multiple times. The M sub-band data are obtained by concatenating the mixed signal, the voiceprint feature vector, and the first output data according to the channel dimension to obtain concatenated pre-extracted data. The pre-extracted data is divided within the full frequency band, where M is a positive integer. The second sub-output data is obtained through the first CNN layer in the CNN model; N2 is a positive integer. The output data with a channel dimension of N2 obtained from the last concatenation is input into the third CNN layer in the CNN model to obtain the second output data with a channel dimension of 2; wherein, the second output data is frequency domain data; The audio data of the target human voice is obtained using the second output data; wherein the audio data is time-domain data.

2. The audio extraction method according to claim 1, characterized in that, Obtain the voiceprint feature vector of the target human voice to be extracted, including: Obtain the registered voice data of the target human voice to be extracted; The registered voice data is input into the auxiliary network model to obtain the voiceprint feature vector of the target human voice to be extracted.

3. The audio extraction method according to claim 1, characterized in that, The process of inputting the mixed signal and the voiceprint feature vector into a complex long short-term memory (LSTM) network model to obtain the first output data includes: The mixed signal is divided into a real part and an imaginary part; The real and imaginary parts of the mixed signal and the voiceprint feature vector are input into the first LSTM layer of the complex LSTM network model to obtain the first sub-output data; The first sub-output data and the voiceprint feature vector are input into the second LSTM layer of the complex LSTM network model to obtain the first output data.

4. The audio extraction method according to claim 1, characterized in that, The method further includes: For each of the M sub-band data, the sub-band data is input into the first CNN layer of the CNN model to obtain the second sub-output data with a channel dimension of N1; where N1 is a positive integer. One channel dimension represents the real data, and the other channel dimension represents the imaginary data.

5. The audio extraction method according to claim 1, characterized in that, After step 1, the following steps are also performed: The output data of the first CNN unit is input into the second CNN unit in the second CNN layer to obtain the output data of the second CNN unit with a channel dimension of N1; wherein, the output data of the second CNN unit is used together with the second sub-output data corresponding to the (i+1)th sub-band data to be input into the second CNN layer; Step 2 includes: concatenating the output data of the first CNN unit corresponding to the M sub-band data according to the frequency band dimension to obtain concatenated output data with a channel dimension of N2; inputting the concatenated output data with a channel dimension of N2 into the third CNN unit in the second CNN layer to obtain the output data of the third CNN unit with a channel dimension of N1; and using the output data of the third CNN unit as the input data of the first CNN unit, returning to step 1 and repeating the process multiple times.

6. The audio extraction method according to claim 1, characterized in that, The step of obtaining the audio data of the target human voice using the second output data includes: The product of the second output data and the mixed signal is subjected to inverse Fourier transform processing to obtain the audio data of the target human voice.

7. An audio extraction device, characterized in that, include: The acquisition module is used to acquire the mixed signal to be processed and the voiceprint feature vector of the target human voice to be extracted; The first processing module is used to input the mixed signal and the voiceprint feature vector into a complex long short-term memory (LSTM) network model to obtain the first output data. The second processing module is used to execute the following steps in descending order of the frequency points corresponding to the M sub-band data: Step 1: Input the second sub-output data corresponding to the i-th sub-band data into the first CNN unit in the second CNN layer of the convolutional neural network CNN model to obtain the output data of the first CNN unit with a channel dimension of N2. Step 2: Input the output data of the first CNN unit corresponding to the M sub-band data into the third CNN unit in the second CNN layer to obtain the output data of the third CNN unit with a channel dimension of N1. Then, use the output data of the third CNN unit as the input data of the first CNN unit and return to Step 1 to repeat the process multiple times. The M sub-band data are obtained by concatenating the mixed signal, the voiceprint feature vector, and the first output data according to the channel dimension to obtain concatenated pre-extracted data. The pre-extracted data is divided within the full frequency band, where M is a positive integer. The second sub-output data is obtained through the first CNN layer in the CNN model; N2 is a positive integer. The output data with a channel dimension of N2 obtained from the last concatenation is input into the third CNN layer in the CNN model to obtain the second output data with a channel dimension of 2; wherein, the second output data is frequency domain data; The acquisition module is used to acquire the audio data of the target human voice using the second output data; wherein the audio data is time-domain data.

8. An electronic device, comprising: A transceiver, a processor, a memory, and a program or instructions stored in the memory and executable on the processor; characterized in that, when the processor executes the program or instructions, it implements the steps of the audio extraction method as described in any one of claims 1 to 6.

9. A readable storage medium having a program or instructions stored thereon, characterized in that, When the program or instructions are executed by the processor, they implement the steps of the audio extraction method as described in any one of claims 1 to 6.

Images