Speech separation method and device

Abstract

Embodiments of the present application provide a speech separation method and apparatus, applicable to electronic devices, that collects audio containing mixed speech from multiple speakers and obtains information about the speakers to be separated, including at least one speaker whose voiceprint has been pre-registered. The speech of the speaker to be separated is separated from the audio based on the information about the speaker to be separated, a strongly labeled mixed speech sample, and a collected weakly labeled mixed speech sample. As can be seen, the basis for speech separation includes the collected weakly labeled mixed speech sample. Therefore, not only can the accuracy of separating the speech of the speaker to be separated be improved, but the collected real mixed speech sample can also be fully utilized to fully utilize the data.

Description

Technical Field

[0001] The present application relates to the field of audio technology, and in particular to a speech separation method and device. Background Art

[0002] Voiceprint-based speech separation technology is applicable to a variety of scenarios. Voiceprint-based speech separation can be understood as separating the voice of a specific speaker from mixed audio of multiple voices based on the voiceprint of the specific speaker.

[0003] Currently, there is still room for improvement in the speech separation function of electronic devices. Summary of the Invention

[0004] The present application provides a speech separation method and apparatus, the purpose of which is to improve the speech separation function of an electronic device.

[0005] In order to achieve the above objectives, this application provides the following technical solutions:

[0006] The first aspect of the present application provides a speech separation method for use in an electronic device. The method comprises: collecting audio comprising mixed speech from multiple speakers, obtaining information about the speakers to be separated, wherein the speakers to be separated include at least one speaker whose voiceprint has been pre-registered, and separating the speech of the speakers to be separated from the audio based on the information about the speakers to be separated, a strongly annotated mixed speech sample, and a collected weakly annotated mixed speech sample. As can be seen, the basis for speech separation includes the collected weakly annotated mixed speech sample. Therefore, the method not only improves the accuracy of separating the speech of the speakers to be separated, but also fully utilizes the collected real mixed speech samples to fully utilize the data.

[0007] In some implementations, based on the information of the speaker to be separated, the strongly labeled mixed speech samples, and the collected weakly labeled mixed speech samples, the speech of the speaker to be separated is separated from the audio, including: based on the information of the speaker to be separated, calling a speech separation model to separate the speech of the speaker to be separated from the audio, the speech separation model is trained based on the strongly labeled mixed speech samples and the collected weakly labeled mixed speech samples, so that it not only has higher accuracy, but also is conducive to the full utilization of real speech samples.

[0008] In some implementations, the speech separation model is trained based on the strongly labeled mixed speech samples and the collected weakly labeled mixed speech samples, including: training a pre-built model based on the strongly labeled mixed speech samples to obtain a pre-trained model; training the pre-trained model based on the weakly labeled mixed speech samples to obtain the speech separation model. The model is trained in two stages. The first stage is based on the training of strongly labeled mixed speech samples, which can lay the foundation for the training of the second stage. The second stage is based on the training of weakly labeled mixed speech samples, which can improve the accuracy of the model in real mixed speech scenarios. In addition to the training based on the weakly labeled mixed speech samples, the second stage can also repeat the training of the first stage.

[0009] In some implementations, training the pre-trained model based on the weakly annotated mixed speech sample includes: obtaining information of each speaker contained in the first mixed speech sample as first annotated data; obtaining a spectrogram of the target speaker in the first mixed speech sample based on the spectrogram of the first mixed speech sample, the voiceprint of the target speaker indicated by the first annotated data, and the pre-trained model; obtaining a speech vector of the target speaker based on the spectrogram of the target speaker; and training the pre-trained model based on the speech vector of the target speaker and the voiceprint of the target speaker.

[0010] In some implementations, training the pre-trained model based on the target speaker's speech vector and the target speaker's voiceprint includes training the pre-trained model based on a comparative error between the target speaker's speech vector and the target speaker's voiceprint. The comparative error can eliminate the influence of speech content on the model, making the trained model applicable to speech of various contents.

[0011] In some implementations, the pre-constructed model is trained based on the strongly labeled mixed speech sample to obtain the pre-trained model, including: obtaining the spectrogram of the speech of each speaker in the second mixed speech sample as the second labeled data; obtaining the spectrogram of the speech of the target speaker in the second mixed speech sample based on the spectrogram of the second mixed speech sample, the voiceprint of the target speaker and the speech separation model; and training the pre-constructed model based on the spectrogram of the speech of the target speaker and the second labeled data.

[0012] In some implementations, separating the speech of the speaker to be separated from the audio based on information about the speaker to be separated, strongly labeled mixed speech samples, and collected weakly labeled mixed speech samples includes: extracting multi-scale features from the audio; and separating the speech of the speaker to be separated from the audio based on the multi-scale features, information about the speaker to be separated, strongly labeled mixed speech samples, and collected weakly labeled mixed speech samples. Because the multi-scale features correspond to the scale on the time axis of the spectrogram, the multi-scale features can eliminate the impact of speech rate on the model, making the trained speech separation model adaptable to audio of various speaking rates.

[0013] In some implementations, before separating the speech of the speaker to be separated from the audio, the method further includes: based on the information of the speaker to be separated, increasing the weight of the speech features of the speaker to be separated in the multi-scale features to facilitate subsequent extraction of the time-frequency mask and improve the accuracy of the time-frequency mask.

[0014] In some implementations, before obtaining the information of the speaker to be separated, the method further includes: collecting voices of multiple speakers for voiceprint registration to provide a priori features for the speech separation model to separate the speech, thereby improving the accuracy of speech separation.

[0015] The second aspect of the present application provides an electronic device, comprising: a processor and a memory, the memory being used to store an application, the processor being used to run the application, and the speech separation method provided in the first aspect of the present application.

[0016] The third aspect of the present application provides a computer-readable storage medium having a program stored thereon, which implements the speech separation method provided in the first aspect of the present application when the program is executed by a computer device.

[0017] The fourth aspect of the present application provides a computer program product. When the computer program product is run on a computer, it enables the computer to execute the noise elimination method provided in the first aspect of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 An example diagram of a scenario of the speech separation method provided in an embodiment of the present application;

[0019] Figure 2 This is a structural example diagram of a speech separation system;

[0020] Figure 3 This is a diagram illustrating an example of the structure of a speech separation system provided in an embodiment of the present application;

[0021] Figure 4 An example diagram of the first stage training of the speech separation system provided in an embodiment of the present application;

[0022] Figure 5 An example diagram of the second stage training of the speech separation system provided in an embodiment of the present application;

[0023] Figure 6 A flowchart of the speech separation method provided in an embodiment of the present application;

[0024] Figure 7 An example diagram of the structure of an electronic device provided in an embodiment of the present application;

[0025] Figure 8 This is an example diagram of a software framework running on an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0026] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. The terms used in the following embodiments are only for the purpose of describing specific embodiments and are not intended to be limiting of the present application. As used in the specification and appended claims of the present application, the singular expressions "one", "a kind of", "said", "above", "the" and "this" are intended to also include expressions such as "one or more", unless there is a clear contrary indication in the context. It should also be understood that in the embodiments of the present application, "one or more" refers to one, two or more; "and / or" describes the association relationship of associated objects, indicating that three relationships may exist; for example, A and / or B can represent: the existence of A alone, the existence of A and B at the same time, and the existence of B alone, where A and B can be singular or plural. The character " / " generally indicates that the related objects before and after are in an "or" relationship.

[0027] References to "one embodiment" or "some embodiments" in this specification mean that a particular feature, structure, or characteristic described in conjunction with that embodiment is included in one or more embodiments of the present application. Thus, phrases such as "in one embodiment," "in some embodiments," "in other embodiments," and "in yet other embodiments" appearing in various places in this specification do not necessarily refer to the same embodiment, but rather mean "one or more but not all embodiments," unless otherwise specifically emphasized. The terms "including," "comprising," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0028] The "multiple" involved in the embodiments of the present application means greater than or equal to two. It should be noted that in the description of the embodiments of the present application, the words "first" and "second" are only used for the purpose of distinguishing the description and cannot be understood as indicating or implying relative importance or order.

[0029] Figure 1 This is an example of an application scenario of voiceprint-based speech separation technology. Figure 1 In the example, the "Karaoke" function of the large screen 1 is turned on, and the large screen 1 collects the voices of multiple people singing along with the accompaniment to obtain a mixed audio. It can be understood that the mixed audio includes the voices of multiple people and the sound of the accompaniment (or may not include the sound of the accompaniment).

[0030] After the "Karaoke" is over, there is a need to obtain the audio of a single person's singing. In this case, the big screen separates the singing audio of a certain person participating in the singing from the mixed audio (the singing audio may or may not include the accompaniment).

[0031] It is understandable that voiceprint-based speech separation is not limited to Figure 1 The scenario shown can also be applied to scenarios such as conference audio transcription.

[0032] Figure 2 Shown is an example of a system implementing voiceprint-based speech separation. Figure 2 In [1], it is assumed that speaker i is any speaker. The speech of speaker i refers to the audio signal that only includes the speech of speaker i and does not include the speech of other speakers and the noise can be ignored.

[0033] The speech of speaker i is input into Time Delay Neural Network (TDNN) 1, and after being converted by TDNN1, the speech vector of speaker i (i.e., the voiceprint vector of speaker i) is output.

[0034] The real mixed speech of multiple speakers refers to an audio signal that is a mixture of the real speech of multiple speakers. The real mixed speech of multiple speakers is input into the Long Short-Term Memory (LSTM) 2. Based on the speech vector of speaker i output by TDNN 1, LSTM 2 extracts and outputs features from the real mixed speech of multiple speakers. These features are input into the time-frequency mask generation module 3, which estimates and outputs the location of the time-frequency points of speaker i's speech in the spectrogram (including both time and frequency dimensions) of the real mixed speech of multiple speakers (referred to as the time-frequency mask). The real mixed speech of multiple speakers and the time-frequency mask output by the time-frequency mask generation module 3 are respectively input into the calculation module 4, which performs a dot product operation and outputs the time-frequency vector of speaker i's speech. The time-frequency vector of speaker i's speech is then input into the inverse Fourier transform (IFFT) module 5. After inverse Fourier transform, IFFT module 5 outputs the speech of speaker i, thereby separating the speech of speaker i from the real mixed speech of multiple speakers.

[0035] Combine Figure 1As shown, by inputting the voice of speaker A into the large screen as a priori feature, the voice of speaker A can be separated from the chorus audio of multiple people.

[0036] During the research process, the inventors found that one of the keys to voiceprint-based speech separation lies in the generation of time-frequency masks. However, at present, the training samples of the time-frequency mask generation module are only artificially synthesized mixed audio, because only artificially synthesized mixed audio can obtain the position of the speech time-frequency points of each speaker in the spectrogram of the mixed audio (that is, the spectrogram of each speaker), and then obtain labeled data for training the time-frequency mask generation module.

[0037] However, real mixed speech (real speech collected from multiple speakers rather than artificial synthesis) cannot be used to train the time-frequency mask generation module because labeled data (i.e., the spectrogram of each speaker) is not available.

[0038] Using artificially synthesized mixed audio to train the time-frequency mask generation module will lead to the following problems:

[0039] During the testing process, the accuracy of extracting the speech of a specific speaker from real mixed speech based on the trained time-frequency mask generation module needs to be improved. In addition, a large amount of real speech cannot be used for training, resulting in a waste of data resources.

[0040] In order to solve the problem that real speech cannot be used to train the time-frequency mask generation module, the present application provides a new training method, which jointly trains the time-frequency mask generation module based on artificially synthesized mixed audio and real mixed audio.

[0041] Furthermore, in some implementations, in order to enhance the adaptability and accuracy of the model, the model structure used in the embodiments of the present application is Figure 1 Improvements are made on this basis.

[0042] The improved model and the training process of the improved model are described in detail below.

[0043] An example of the structure of the improved model is Figure 3 As shown, a speech separation model provided in an embodiment of the present application includes: a multi-scale feature extraction module 101, a self-attention module 102, a time-frequency mask generation module 103, an operation module 104, a TDNN module 105 and an IFFT module 106.

[0044] Multi-scale feature extraction module 101 is used to extract features from mixed speech from multiple speakers using a multi-scale convolution kernel, generating multi-scale features. This multi-scale feature extraction adapts to various speaking rates. This means that regardless of whether the mixed speech from multiple speakers is fast or slow, sufficient features can be extracted, laying a solid foundation for subsequent time-frequency mask generation.

[0045] TDNN105 is used to extract (voiceprint) features from the speech of speaker i as prior information as prior features. It can be understood that the speech of speaker i is any one of the speech of speaker 1, the speech of speaker 2, ... the speech of speaker n, with Figure 3 For example, the voice input to TDNN105 is selected through switch K.

[0046] Self-attention module 102 is used to strengthen the weight of speaker i's speech features in the multi-scale features output by the multi-scale feature extraction module based on the prior features, laying a good foundation for the subsequent generation of the time-frequency mask. In some implementations, self-attention module 102 concatenates the prior features with the multi-scale features to generate concatenated features. It is understandable that the prior features are more heavily weighted in the concatenated features than in the multi-scale features.

[0047] The time-frequency mask generation module 103 is used to generate and output the time-frequency mask of the speech of speaker i based on the multi-scale features output by the self-attention module 102.

[0048] The operation module 104 is configured to perform a dot product operation on the time-frequency mask of the speech of speaker i and the real mixed speech of multiple speakers, and output the time-frequency vector of the speech of speaker i (ie, the spectrogram of speaker i).

[0049] The IFFT module 106 is used to perform an IFFT operation on the time-frequency vector of the speech of speaker i to obtain the speech of speaker i.

[0050] It is understandable that Figure 3 In the model shown, the modules that need to be trained are the multi-scale feature extraction module 101, the self-attention module 102, and the time-frequency mask generation module 103.

[0051] Figure 4 and Figure 5 An example diagram for training the multi-scale feature extraction module 101, the self-attention module 102, and the time-frequency mask generation module 103.

[0052] The training process provided in this embodiment is divided into two stages: Figure 4 The first stage is shown. Figure 5 Shown is the second stage.

[0053] Combine first Figure 4 Explain the first stage of training:

[0054] Figure 4 The sample data used for training shown in is a strongly labeled artificial mixed speech. It is understandable that Figure 4 In this context, strongly annotated artificially synthesized mixed speech refers to audio that is artificially synthesized rather than actually collected and includes multiple speakers. Strong annotation can be understood as the use of the known time-frequency spectrograms (i.e., spectrograms) of each speaker's speech in the audio as annotated data. Furthermore, it is understood that the spectrograms of artificially synthesized mixed speech can be obtained based on the artificially synthesized mixed speech.

[0055] It is understandable that in Figure 4 Before the training process shown, TDNN105 has been trained, that is, TDNN105 is a pre-trained module, and the specific training process is not repeated here.

[0056] Combine Figure 4 The training process is as follows:

[0057] One of the speech of speaker 1, speaker 2, ..., speaker n is selected as the speech of speaker i and input into TDNN 105. TDNN 105 outputs the voiceprint of speaker i as a priori features. The strongly annotated artificial mixed speech is processed by multi-scale feature extraction module 101 to extract multi-scale features. Based on the multi-scale features and the priori features output by TDNN 105, self-attention module 102 outputs a concatenated feature, which is a combination of the priori features and the multi-scale features. Time-frequency mask generation module 103 generates a time-frequency mask for speaker i's speech based on the concatenated features.

[0058] The operation module 104 performs a dot product operation on the time-frequency mask of the speech of speaker i and the spectrogram of the artificial mixed speech to obtain the spectrogram of the speech of speaker i.

[0059] After obtaining the spectrogram of speaker i’s speech, the parameters of each module are trained using the annotated data: the mean square error L between the spectrogram of speaker i’s speech output by the operation module 104 and the spectrogram of speaker i’s speech in the annotated data is calculated. mse , and based on L mse Optimize the parameters of the multi-scale feature extraction module 101, the self-attention module 102, and the time-frequency mask generation module 103.

[0060] It is understandable that the first stage of training can obtain a more accurate model (which can be called a pre-training model) for use in the second stage of training.

[0061] Combine Figure 5 Explain the second stage of training:

[0062] Figure 5 The sample data used for training shown in Figure 1 is weakly labeled real mixed speech. Weakly labeled data can be understood as the information about each speaker in the real mixed speech is known, that is, the speech of each speaker is included in the real mixed speech, but the spectrogram of each speaker's speech is unknown.

[0063] Figure 5 and Figure 3 The difference is that a TDNN107 is added, and because IFFT106 does not need to be trained and does not contribute to the training process, Figure 4 and Figure 5 The IFFT module 106 is not included in the training process, that is, the IFFT module 106 does not participate in the training process.

[0064] Combine Figure 5 The training process is as follows:

[0065] From the speech of speaker 1, speaker 2, ..., speaker n, the speech of any speaker indicated by the weak annotation is selected as the speech of speaker i and input into TDNN 105. TDNN 105 outputs the voiceprint of speaker i as the prior feature. The multi-scale feature extraction module 101 extracts multi-scale features from the weakly annotated real mixed speech. The self-attention module 102 outputs a concatenated feature based on the multi-scale features and the prior features output by TDNN 105. The time-frequency mask generation module 103 generates a time-frequency mask for the speech of speaker i based on the concatenated features.

[0066] The operation module 104 performs a dot product operation on the time-frequency mask of the speech of speaker i and the spectrogram of the real mixed speech to obtain the spectrogram of the speech of speaker i.

[0067] TDNN 107 converts the spectrogram of speaker i's speech into a time-frequency vector (voiceprint) of speaker i's speech.

[0068] Then further train the parameters of each module:

[0069] In this embodiment, the comparison error L between the time-frequency vector (voiceprint) of the speech of speaker i and the prior feature (i.e., the pre-obtained voiceprint of speaker i) is calculated. contrastive , and based on L contrastive Optimize the parameters of the multi-scale feature extraction module 101, the self-attention module 102, and the time-frequency mask generation module 103.

[0070] Contrast error L contrastive The calculation method of is shown in formula (1):

[0071]

[0072] Among them, D w =||vec1-vec2||2, where vec1 represents the voiceprint of speaker i separated from the real mixed speech, and vec2 represents the prior voiceprint vector of speaker i. N represents the number of sample pairs in the minibatch; Y represents the label of the sample pair. If the speaker i separated from the real mixed speech is the same as the prior speaker i, the value is 1; otherwise, the value is 0. m represents a preset threshold, the purpose of which is to keep the distance between different speaker vectors at least m.

[0073] Understandably, L contrastive It means that different voices of the same speaker are mapped to similar positions in the feature space, and voices of different speakers are mapped to farther positions in the feature space.

[0074] It can be seen that the contrast error can eliminate the impact of content differences (i.e., the text corresponding to the audio) on the model.

[0075] It is understandable that in the second phase, in addition to Figure 5 In addition to the training shown, you can also Figure 4 The training shown here does not limit the order of the two.

[0076] from Figure 4 as well as Figure 5 As can be seen from the training process shown, in the training process of the model in this embodiment, not only strongly labeled artificial mixed speech is used, but also weakly labeled real mixed speech is used to train the model. Under the premise of being able to use labeled data for training, the model can learn the characteristics of real mixed speech. Therefore, it is beneficial to improve the accuracy of speech separation in the test process for real mixed speech conditions, and it can also make full use of real mixed speech to avoid the problem of data waste.

[0077] It is understandable that Figure 3 The structure of the speech separation model is only an example, and modules, networks and models that can achieve the same function are all within the scope of protection of this application.

[0078] It is understandable that Figure 3 The multi-scale feature extraction module 101, the self-attention module 102, and the time-frequency mask generation module 103 shown in FIG are Figure 4 and Figure 5 After the training process shown, it can be used to separate the speech of a certain speaker from the collected real mixed audio.

[0079] It is understandable that Figure 3 Application scenarios of the model shown in this paper include but are not limited to speech separation Figure 1The scenario shown, that is, except for Figure 1 In addition to the scenario shown, another scenario example is a multi-person conference scenario, in which speeches of multiple people in the meeting are collected, and the voice of a certain speaker is separated from the audio of the speeches of multiple people. In some implementations, the separated speaker's voice can also be transcribed to obtain meeting minutes. It can be seen that the speech separation method provided in the embodiment of the present application is applicable to scenarios where audio can be collected, and is used to separate the voice of a specific speaker from the collected audio. It can be understood that the "speaker" is the object that makes the speech, and can also be called the "speaker" and "sound source", "sound source", etc. "Speech" can be understood as the voice of the speaker, and can also be called "sound", "audio", etc.

[0080] The execution subject of the speech separation method provided in the embodiments of the present application is an electronic device, including but not limited to tablet computers, PCs, ultra-mobile personal computers (UMPCs), vehicle-mounted devices, netbooks, personal digital assistants (PDAs), large screens and other electronic devices with sound collection functions.

[0081] Below is Figure 1 Taking the scenario shown in the figure as an example, the process of separating the voice of speaker A from the audio of multiple people singing karaoke is as follows: Figure 6 As shown, the following steps are included:

[0082] S601: Collect the voices of multiple speakers for voiceprint registration.

[0083] Registration can be understood as storing the speaker's voiceprint. Figure 1 In the scenario shown, a family member, such as A, speaks to the microphone connected to the large screen. The microphone collects the speaker's voice and combines it with the Figure 3 The model shown uses TDNN 105 to extract the time-frequency vector (i.e., voiceprint) of the speaker's speech, and the large screen stores the extracted time-frequency vector. In some implementations, the user enters the registered speaker's information into the large screen, and after receiving the registered speaker's information, the large screen stores it in correspondence with the speaker's voiceprint.

[0084] It is understandable that when a speaker is collecting voice for registration, other speakers remain silent. That is, the registration process needs to collect the pure voice of a speaker and save the voiceprint extracted from it to provide more accurate prior features for the subsequent generation of time-frequency masks.

[0085] After multiple speakers are registered in sequence, the electronic device obtains multiple prior features.

[0086] S602: Collect real mixed speech from multiple speakers.

[0087] Combine Figure 1 In the scenario shown, after the large screen has the "Karaoke" function turned on, a microphone connected to the large screen collects audio of multiple people singing. In some implementations, the real mixed voice is audio with accompaniment music, while in other implementations, the real mixed voice is audio without accompaniment music, including only the voices of the multiple people singing.

[0088] After the performance, the big screen captured the real mixed voices of multiple speakers.

[0089] S603: Acquire information of the speaker to be separated.

[0090] Combine Figure 1 In the scenario shown, in response to the end of the performance or the triggering of the voice separation function, the large screen displays an interface for obtaining the information of the speaker to be separated. The user can enter the information of the speaker to be separated in the interface. For example, the user can enter the name of speaker A, "A". Alternatively, a microphone connected to the large screen can be used to capture the voice of the speaker to be separated, such as speaker A, as the information of the speaker to be separated.

[0091] In some implementations, information of one speaker to be separated is obtained, and in other implementations, information of multiple speakers to be separated is obtained.

[0092] S604: Separate the speech of the speaker to be separated from the real mixed speech of multiple speakers based on the information of the speaker to be separated.

[0093] Combine Figure 3 As shown, an example of a specific implementation process of S604 is:

[0094] TDNN 105 searches for the voiceprint corresponding to the information of the speaker to be separated in the pre-stored correspondence, or searches for the voiceprint matching the voice of the speaker to be separated in the pre-stored voiceprints, and inputs the voiceprint as a priori feature into the self-attention module 102 .

[0095] The multi-scale feature extraction module 101 extracts multi-scale features from the real mixed speech of multiple speakers (such as audio of multiple people singing) collected in S602, and inputs the multi-scale features into the self-attention module 102.

[0096] The self-attention module 102 concatenates the multi-scale features with the prior features to obtain concatenated features, and inputs the concatenated features into the time-frequency mask generation module 103. It is understandable that the concatenated features have a greater weight on the time-frequency components of the speaker's speech to be separated than the multi-scale features.

[0097] The time-frequency mask generation module 103 generates and outputs a time-frequency mask of the speech of the speaker to be separated based on the concatenated features.

[0098] The operation module 104 performs a dot product operation on the time-frequency mask of the speaker to be separated and the real mixed speech of multiple speakers, and outputs the time-frequency vector of the speech of the speaker to be separated.

[0099] The IFFT module 106 performs IFFT transformation on the time-frequency vector of the speech of the speaker to be separated, and outputs the speech of the speaker to be separated (ie, speaker i).

[0100] It is understood that when there are multiple speakers to be separated, in some implementations, TDNN 105 inputs the voiceprint of one speaker to be separated (speaker i) at a time as a prior feature. After IFFT 106 outputs the speech of speaker i, TDNN 105 then inputs the voiceprint of speaker i+1 as a prior feature. After IFFT 106 outputs the speech of speaker i+1, TDNN 105 then inputs the voiceprint of speaker i+2 as a prior feature, and so on, until the speech of all speakers to be separated is obtained. In other implementations, assuming that the self-attention module 102, the time-frequency mask generation module 103, the operation module 104, and the IFFT module 106 constitute a branch, multiple branches can be provided, any of which is used to receive the voiceprint of a speaker as a prior feature and output the speech of that speaker separated from the real mixed speech of multiple speakers.

[0101] It is understandable that Figure 6 In the process shown, the multi-scale feature extraction module 101, the self-attention module 102, and the time-frequency mask generation module 103 are trained using real mixed speech, so the speech of a specific speaker can be separated from the real mixed speech more accurately.

[0102] Moreover, due to the multi-scale feature extraction module in the model, Figure 6 The process shown can be applied to mixed speech of various speech rates, and because contrast error is used in the training process, Figure 6 The process shown is applicable to mixed speech with various contents.

[0103] based on Figure 4 and Figure 5 The training process shown, Figure 6 The process shown is applicable to audio that does not contain the voice of the speaker being separated.

[0104] implement Figure 6 An example of the structure of an electronic device of the process shown is as follows Figure 7As shown, it includes: a processor 110, an antenna 1, an antenna 2, a mobile communication module 120, a wireless communication module 130, an audio module 140, a microphone 140A, etc.

[0105] It should be understood that the structure illustrated in this embodiment does not constitute a specific limitation on the electronic device. In other embodiments, the electronic device may include more or fewer components than shown, or may combine or separate certain components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0106] The processor 110 may include one or more processing units, for example: the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processor (NPU). The NPU is a neural network (NN) computing processor that quickly processes input information by drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, and can also continuously self-learn. The NPU can realize applications such as intelligent cognition of electronic devices, such as image recognition, face recognition, speech recognition, text understanding, etc. In the embodiment of the present application, the DSP is used to support a DSP-based noise elimination method, and the NPU is used to support an NN-based noise elimination method.

[0107] Among them, different processing units can be independent devices or integrated into one or more processors.

[0108] In some embodiments, the processor 110 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a general-purpose input / output (GPIO) interface, etc.

[0109] In some embodiments, the GPIO interface can be used to connect the processor 110 with the wireless communication module 130, the audio module 140, etc.

[0110] The I2S interface can be used for audio communication. In some embodiments, the processor 110 can include multiple I2S buses. The processor 110 can be coupled to the audio module 140 via the I2S bus to enable communication between the processor 110 and the audio module 140. In some embodiments, the audio module 140 can transmit audio signals to the wireless communication module 130 via the I2S interface, enabling the function of answering calls through a Bluetooth headset.

[0111] The PCM interface can also be used for audio communication, sampling, quantizing, and encoding analog signals. In some embodiments, the audio module 140 and the wireless communication module 130 can be coupled via a PCM bus interface. In some embodiments, the audio module 140 can also transmit audio signals to the wireless communication module 130 via the PCM interface, enabling the function of answering calls via a Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.

[0112] It is understood that the interface connection relationship between the modules illustrated in this embodiment is only a schematic illustration and does not constitute a structural limitation on the electronic device. In other embodiments of the present application, the electronic device may also adopt different interface connection methods from the above embodiments, or a combination of multiple interface connection methods. The mobile communication module 120 can provide solutions for wireless communications including 2G / 3G / 4G / 5G applied to electronic devices.

[0113] The wireless communication module 130 can provide wireless communication solutions for electronic devices, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared technology (IR), etc.

[0114] In some implementations, the electronic device receives the sound signal via the mobile communication module 120 or the wireless communication module 130 .

[0115] The audio module 140 is used to convert analog audio input into digital audio signals. The audio module 140 can also be used to encode and decode audio signals. In some embodiments, the audio module 140 can be provided in the processor 110, or some functional modules of the audio module 140 can be provided in the processor 110.

[0116] Microphone 140A, also known as "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to the microphone 140A to input the sound signal into the microphone 140A. The electronic device can be provided with at least one microphone 140A. In other embodiments, the electronic device can be provided with two microphones 140A, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the electronic device can also be provided with three, four or more microphones 140A to realize sound signal collection, noise reduction, and identification of sound sources, and realize directional recording function, etc.

[0117] In some other implementations, the electronic device can implement functions such as recording through the audio module 140 and the microphone 140A.

[0118] After the electronic device acquires the sound signal in the above manner, the sound signal is transmitted to the processor 110 through the above interface of the processor 110 , and the processor 110 performs noise cancellation processing as described in the following embodiments on the sound signal.

[0119] The processor 110 can implement an operating system such as iOS operating system, Android open source operating system, Windows operating system, etc. by running program codes.

[0120] Take the Android open source operating system as an example. Figure 8 As shown, in some embodiments, the Android system is divided into four layers, which are, from top to bottom, the application layer, the application framework layer, the hardware abstraction layer, and the kernel layer.

[0121] The application layer can include a series of applications. Figure 8 As shown, in the embodiment of the present application, examples of applications related to sound signal collection include calling and recording.

[0122] The application framework layer provides application programming interface (API) and programming framework for the applications in the application layer. The application framework layer includes some predefined functions. Figure 8 As shown, the application framework layer may include call services and recording services related to the embodiments of the present application.

[0123] The hardware abstraction layer (HAL), also known as the Android runtime, is responsible for scheduling and management of the Android system. The noise cancellation device runs in the hardware abstraction layer to implement the noise cancellation function described in this embodiment.

[0124] The kernel layer is the layer between hardware and software. In the embodiment of the present application, the kernel layer includes at least a microphone driver and a sound card driver. Each driver is used to process the acquired hardware data and report the processing results to the corresponding module of the hardware abstraction layer.

Claims

1. A speech separation method, characterized in that: Applied to electronic equipment, the method includes: Collecting audio, where the audio includes mixed speech of multiple speakers; Obtaining information of a speaker to be separated, wherein the speaker to be separated includes at least one speaker whose voiceprint has been pre-registered; Based on the information of the speaker to be separated, a speech separation model is called to separate the speech of the speaker to be separated from the audio. The speech separation model is trained based on strongly labeled mixed speech samples and collected weakly labeled mixed speech samples. The weak labeling includes information of each speaker in the real mixed speech sample. The spectrogram of the speech of each speaker in the real mixed speech sample is unknown. The strong labeling includes the spectrogram of the speech of each speaker in the strongly labeled mixed speech sample. The speech separation model is trained based on the strongly labeled mixed speech sample and the collected weakly labeled mixed speech sample, including: training a pre-constructed model based on the strongly labeled mixed speech sample to obtain a pre-trained model; training the pre-trained model based on the weakly labeled mixed speech sample to obtain the speech separation model.

2. The method according to claim 1, characterized in that The training of the pre-trained model based on the weakly labeled mixed speech sample includes: Obtaining information of each speaker included in the first mixed speech sample as first labeled data; Obtaining a spectrogram of the target speaker in the first mixed speech sample based on the spectrogram of the first mixed speech sample, the voiceprint of the target speaker indicated by the first annotated data, and the pre-trained model; Obtaining a speech vector of the target speaker based on the spectrogram of the target speaker; The pre-trained model is trained based on the speech vector of the target speaker and the voiceprint of the target speaker.

3. The method according to claim 2, characterized in that The training of the pre-trained model based on the speech vector of the target speaker and the voiceprint of the target speaker includes: The pre-training model is trained based on a comparison error between the speech vector of the target speaker and the voiceprint of the target speaker.

4. The method according to claim 1, wherein The pre-built model is trained based on the strongly labeled mixed speech sample to obtain a pre-trained model, comprising: Obtaining a spectrogram of the speech of each speaker in the second mixed speech sample as second labeled data; Obtaining a spectrogram of the target speaker's speech in the second mixed speech sample based on the spectrogram of the second mixed speech sample, the target speaker's voiceprint, and the speech separation model; The pre-built model is trained based on the spectrogram of the target speaker's speech and the second annotated data.

5. The method according to any one of claims 1 to 4, characterized in that The method of separating the speech of the speaker to be separated from the audio based on the information of the speaker to be separated, the strongly labeled mixed speech sample, and the collected weakly labeled mixed speech sample includes: extracting multi-scale features from the audio; The speech of the speaker to be separated is separated from the audio based on the multi-scale features, the information of the speaker to be separated, the strongly labeled mixed speech sample and the collected weakly labeled mixed speech sample.

6. The method according to claim 5, characterized in that Before separating the speech of the speaker to be separated from the audio, the method further includes: Based on the information of the speaker to be separated, the weight of the speech feature of the speaker to be separated in the multi-scale features is increased.

7. The method according to claim 1, characterized in that Before obtaining the information of the speaker to be separated, the method further includes: The voices of the multiple speakers are collected for voiceprint registration.

8. An electronic device, characterized in that: include: processor and memory; The memory is used to store an application program, and the processor is used to run the application program to implement the speech separation method according to any one of claims 1 to 7.

9. A computer-readable storage medium having a program stored thereon, characterized in that: When the computer device runs the program, the speech separation method according to any one of claims 1 to 7 is implemented.

Images