Speaker separation model training method and speaker separation method and related device

Abstract

The present application discloses a speaker separation model training method and a speaker separation method and related devices, wherein the speaker separation model training method includes: obtaining sample data of a sample environment, and the sample data includes sample speech data collected in the same time period and a sample image sequence of several speakers in the sample environment; using the speaker separation model, processing the sample data to obtain a speech separation mask of each speaker, and processing the speech separation mask and the sample data to obtain a speaker separation result, wherein the speaker separation result is used to determine the speakers in different time segments in the sample speech data; using the speech separation mask and the speaker separation result, adjusting the network parameters of the speaker separation model. The above scheme can improve the speaker separation effect.

Description

Technical Field

[0001] The present application relates to the technical field of image and speech processing, and in particular to a training method for a speaker separation model, a speaker separation method and related devices. Background Art

[0002] The application of speaker separation technology is particularly important in many scenarios. For example, in a multi-person meeting, speaker separation technology can be used to confirm the opinions of each participant; or in a smart classroom, speaker separation technology can help to make the teaching effect more prominent when the course is repeated, and so on.

[0003] At present, speaker separation generally uses pure speech speaker separation technology, which has poor effects in actual complex scenarios and cannot effectively solve practical problems such as richer input data leading to better speaker separation accuracy and robustness. With the improvement of hardware performance and the development of artificial intelligence algorithms, speaker separation solutions for multimodal data are increasingly being used in real life. In view of this, how to improve the speaker separation effect has become an urgent problem to be solved. Summary of the invention

[0004] The main technical problem solved by the present application is to provide a speaker separation model training method and a speaker separation method and related devices, which can improve the speaker separation effect.

[0005] In order to solve the above technical problems, the first aspect of the present application provides a training method for a speaker separation model, comprising: obtaining sample data of a sample environment, and the sample data includes sample speech data collected in the same time period and a sample image sequence of several speakers in the sample environment; using the speaker separation model, processing the sample data to obtain a speech separation mask for each speaker, and processing the speech separation mask and the sample data to obtain a speaker separation result, wherein the speaker separation result is used to determine speakers in different time segments in the sample speech data; using the speech separation mask and the speaker separation result, adjusting the network parameters of the speaker separation model.

[0006] In order to solve the above technical problems, the second aspect of the present application provides a speaker separation method, including: acquiring target data of a target environment, and the target data includes target speech data collected in the same time period and a target image sequence of several speakers in the target environment; processing the target data using a speaker separation model to obtain a speech separation mask for each speaker; processing the speech separation mask and the target data using the speaker separation model to obtain a speaker separation result, wherein the speaker separation result is used to determine the speakers in different time segments in the target speech data.

[0007] In order to solve the above technical problems, the third aspect of the present application provides a training device for a speaker separation model, including an acquisition module, a processing module and an adjustment module; the acquisition module is used to acquire sample data of a sample environment, wherein the sample data includes sample speech data collected in the same time period and a sample image sequence of several speakers in the sample environment; the processing module is used to use the speaker separation model to process the sample data to obtain a speech separation mask for each speaker, and to process the speech separation mask and the sample data to obtain a speaker separation result, wherein the speaker separation result is used to determine speakers in different time segments in the sample speech data; the adjustment module is used to adjust the network parameters of the speaker separation model using the speech separation mask and the speaker separation result.

[0008] In order to solve the above technical problems, the fourth aspect of the present application provides a speaker separation device, including an acquisition module, a processing module and a determination module; the acquisition module is used to acquire target data of a target environment, wherein the target data includes target speech data collected in the same time period and a target image sequence of several speakers in the target environment; the processing module is used to process the target data using a speaker separation model to obtain a speech separation mask for each speaker; the determination module is used to process the speech separation mask and the target data using the speaker separation model to obtain a speaker separation result, wherein the speaker separation result is used to determine speakers in different time segments in the target speech data.

[0009] In order to solve the above-mentioned technical problems, the fifth aspect of the present application provides an electronic device, including a memory and a processor coupled to each other, the memory storing program instructions, and the processor being used to execute the program instructions to implement the training method of the speaker separation model in the first aspect or the speaker separation method in the second aspect.

[0010] In order to solve the above technical problems, the sixth aspect of the present application provides a computer-readable storage medium storing program instructions that can be executed by a processor, wherein the program instructions are used to implement the training method of the speaker separation model in the first aspect or the speaker separation method in the second aspect.

[0011] The above scheme obtains the target speech data of the target environment and the target image sequence of several speakers, and processes the target data using the speaker separation model to obtain the speech separation mask of each speaker; and processes the speech separation mask and the target data using the speaker separation model to obtain the speaker separation result. Since the speaker separation model first processes to obtain the speech separation mask, the speech separation mask can filter out the sounds other than the corresponding speaker, and thus the speech separation mask is used to process the target data, and speakers in complex scenes such as high noise and aliasing can be separated, thereby improving the speaker separation effect, and also improving the application effect of speaker separation in high noise scenes.

[0012] In addition, during the training process, the speaker separation model also uses the speech separation mask and the speaker separation results to train the speaker separation model. In other words, in addition to supervising the model output results, the model's intermediate processing data, namely the speech separation mask, is also supervised, which can ensure that the intermediate processing data and output results of the trained model achieve the ideal effect, thereby improving the accuracy of speaker separation of the speaker separation model. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 It is a flowchart of an embodiment of a training method for a speaker separation model of the present application;

[0014] Figure 2 yes Figure 1 A schematic diagram of a flow chart of an embodiment of step S13;

[0015] Figure 3 It is a flowchart of an embodiment of the speaker separation method of the present application;

[0016] Figure 4 It is a schematic diagram of a framework of an embodiment of a training device for a speaker separation model of the present application;

[0017] Figure 5 It is a schematic diagram of the framework of an embodiment of the speaker separation device of the present application;

[0018] Figure 6 It is a schematic diagram of the framework of an embodiment of the electronic device of the present application;

[0019] Figure 7 It is a schematic diagram of a framework of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0020] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.

[0021] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0022] The terms "system" and "network" are often used interchangeably in this article. The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship. In addition, "many" in this article means two or more than two.

[0023] See also Figure 1 , Figure 1 This is a flowchart of an embodiment of a training method for a speaker separation model of the present application. Specifically, the following steps may be included:

[0024] Step S11: Acquire sample data of a sample environment.

[0025] In an implementation scenario, the sample environment may be a multi-person conference room or a noisy supermarket. The specific sample environment may be determined based on actual conditions and is not specifically limited here.

[0026] In the embodiment of the present disclosure, the sample data includes sample voice data collected in the same time period and sample image sequences of several speakers in the sample environment. The sample voice data collected in the same time period can be audio data within a time interval. The sample image sequences of several speakers collected in the same time period can be facial images of the speakers, lip images of the speakers, sample environment images, etc. The sample image sequences can be selected according to actual conditions and are not specifically limited here.

[0027] In a specific implementation scenario, the sample image in the sample image sequence is a lip image of a speaker, and the lip image of the speaker can be represented by a data dimension. The representation method of the sample image sequence can be determined according to actual conditions and is not specifically limited here.

[0028] Step S12: using the speaker separation model, the sample data is processed to obtain the speech separation mask of each speaker, and the speech separation mask and the sample data are processed to obtain the speaker separation result.

[0029] In one implementation scenario, the speaker separation model includes a speech separation module and a speaker separation module of the speaker separation model; the speech separation module of the speaker separation model is used to process sample data to obtain a speech separation mask for each speaker, and to process the speech separation mask and sample data to obtain a speaker separation result; the speaker separation module of the speaker separation model is used to process sample features using the speech separation module to obtain a speech separation mask for each speaker, wherein the speaker's speech separation mask is used to filter out speech features that do not belong to the speaker.

[0030] In an implementation scenario, the speech separation module of the speaker separation model uses the speaker separation model to extract features from the sample data to obtain sample features, wherein the sample features include sample speech features of the sample speech data and sample image features of the sample image sequence. The processing of the sample speech data in the sample data can be selected by performing a short-time Fourier transform (STFT) on the sample speech data to obtain the sample speech features, or by extracting the sample speech features through a speech feature extraction module, or by marking the sample speech data to obtain the sample speech features. The processing method of the sample speech data in the sample data can be selected according to the actual situation, and no specific limitation is made here. If the sample speech data in the sample data is subjected to a short-time Fourier transform (STFT) to obtain the sample speech features, at this time, the sample speech features contain the speech features of the current speaker (possibly more than one) and the interference of environmental noise, and the obtained sample speech features are composed of phase and amplitude, and both the phase and amplitude can be represented by data dimensions. The representation method of the sample speech features can be determined according to the actual situation, and no specific limitation is made here.

[0031] In one implementation scenario, the processing of the sample image sequence in the sample data can be performed by extracting the sample image features through an image feature extraction module, and the sample image sequence can also be marked to obtain the sample image features. The processing method of the sample image sequence in the sample data can be selected according to the actual situation and is not specifically limited here. If the sample image sequence is a speaker's lip image, the sample image features are extracted from the sample image sequence through an image feature extraction module, and the sample image features can be represented by data dimensions. In the above method, if only a single modal information is used, such as pure speech or pure vision, it is very difficult to separate effective features from these features, and the final effect is poor. After comprehensive analysis of the speech data and image sequence, more information with reference significance can be obtained.

[0032] In one implementation scenario, the sample features are processed by a speech separation module to obtain a speech separation mask for each speaker, wherein the speech separation mask for the speaker is used to filter out speech features that do not belong to the speaker. Since there must be some connection between the sample speech features and the sample image features, the sample speech features are fused with the sample image features, that is, the sample speech features and the sample image features are fused in time sequence. The specific method of fusion can be selected according to the actual situation and is not specifically limited here. The fused features are calculated by a neural network to obtain a speech separation mask for each speaker. The speech separation mask can be represented by a data dimension. The speech mask for the speaker can effectively filter out speech features that do not belong to the speaker, including speech noise, etc., thereby alleviating the adverse effects of environmental noise and multiple people speaking at the same time on the speaker separation task.

[0033] In an implementation scenario, after obtaining the speech separation mask of each speaker, only the result of speech data separation can be obtained, and the judgment result of whether N speakers speak in each time interval cannot be known. Therefore, the speech separation mask and sample features are processed by the speaker separation module to obtain the speaker separation result. Before the speaker separation module processes the speech separation mask and sample features, the previously obtained sample feature data can be directly obtained for reuse to achieve one-time extraction and multiple reuse, reduce repeated calculations, and improve operation efficiency. In the speaker separation module, the sample speech features, sample image features and speech separation masks of each speaker are first fused to obtain a new N-channel feature (N matches the number of speakers), and then the N-channel feature is processed using the Transformer structure to fuse the multi-feature information between each channel of the multi-channel feature, so that each feature channel not only has the multi-feature information of a certain speaker, but also has the multi-feature information of other speakers, so that the subsequent speaker judgment result is more global. The multi-feature information obtained by the Transformer structure can be used for subsequent neural network calculation to obtain the speaker separation results. For example, if N is 3 persons, the sample speech features, sample image features and speech separation masks of the 3 persons are fused to obtain a new 3-channel feature. Then the feature information between the 3-channel features is fused, that is, each channel feature contains the features of the other two channels. Then, the separation results of each speaker are calculated to ensure the globality of the judgment of the speaker separation results.

[0034] In the disclosed embodiment, the speaker separation result is used to determine the speakers of different time segments in the sample speech data. The speaker separation result can be expressed by confidence, and the confidence interval of a probability sample is an interval estimate of a certain population parameter of the sample. The confidence interval shows the degree to which the true value of the parameter has a certain probability of falling around the measurement result. That is, the speaker separation result can be expressed by the speaking confidence of the participants in each time interval.

[0035] Step S13: Using the speech separation mask and the speaker separation result, adjust the network parameters of the speaker separation model.

[0036] In an implementation scenario, the speech separation mask and speaker separation results are used to calculate the loss of the speaker separation model. The loss calculation will be back-propagated, that is, back-propagated layer by layer, to update the network parameters, that is, continuously adjust the parameters in the network.

[0037] In a specific implementation scenario, a speech separation mask can be used to obtain a first loss, specifically, the speech separation mask is compared with the annotated real separation mask to obtain a corresponding first loss; the speech separation mask can also be further used to process the sample speech data, and the voice features of the speaker are obtained by using the speech separation mask, and the obtained voice features are compared with the actual voice features to obtain the corresponding first loss. The method for determining the first loss obtained by the speech separation mask can be selected according to actual conditions, and is not specifically limited here. The first loss can be minimizing the square error (L2 loss), the first loss can also minimize the absolute error, etc. The representation method of the first loss can be selected according to actual applications, and is not specifically limited here.

[0038] In a specific implementation scenario, the speaker separation result is used to obtain a second loss, and the speaker separation result is used to determine the speakers of different time segments in the sample speech data. The second loss can be a cross entropy loss, or the second loss can be a minimization of square error, etc. The representation of the second loss can be selected according to the actual application and is not specifically limited here.

[0039] In one implementation scenario, the first loss and the second loss are weighted and summed to obtain a third loss, and the network parameters of the speaker separation model are adjusted according to the third loss. In the speaker separation model, the first loss and the second loss have different effects on the speaker separation model. If the first loss is L2 loss and the second loss is cross-entropy loss (CrossEntry Loss), the comprehensive loss function of the entire speaker separation model is Loss = a*L2 Loss + b*CrossEntry Loss (a>0, b>0, a+b=1). If the value of a is set too large, the network effect of the speaker separation model tends to be a better speech separation module, otherwise it tends to be a better speaker separation module. In this embodiment, a=0.3 and b=0.7 are taken, but are not limited to this combination.

[0040] See also Figure 2 , Figure 2 yes Figure 1 The flowchart of step S13 in the embodiment is as follows: Figure 2As shown, sample data is obtained, the sample data includes sample speech data 20 and sample image sequence 21, the sample data is processed to obtain sample speech features 22 and sample image features 23, the sample speech features 22 and sample image features 23 are fused in a speech separation module 24, the fused features are calculated by a neural network to obtain a speech separation mask 25 of each speaker, and a first loss 26 is obtained through the speech separation mask 25 of each speaker; in a speaker separation module 27, the sample speech features 22, the sample image features 23 and the speech separation masks 25 of each speaker are fused to obtain new channel features of each speaker, the channel features of each speaker are processed to obtain a speaker separation result 28, a second loss 29 is obtained through the speaker separation result 28, the first loss 26 and the second loss 29 are weightedly calculated, and the network parameters of the speaker separation model are adjusted to make the speaker separation model have a better effect on speaker separation.

[0041] The above scheme obtains the target speech data of the target environment and the target image sequence of several speakers, and processes the target data using the speaker separation model to obtain the speech separation mask of each speaker; and processes the speech separation mask and the target data using the speaker separation model to obtain the speaker separation result. Since the speaker separation model first processes to obtain the speech separation mask, the speech separation mask can filter out the sounds other than the corresponding speaker, and thus the speech separation mask is used to process the target data, and speakers in complex scenes such as high noise and aliasing can be separated, thereby improving the speaker separation effect, and also improving the application effect of speaker separation in high noise scenes.

[0042] In addition, during the training process, the speaker separation model also uses the speech separation mask and the speaker separation results to train the speaker separation model. In other words, in addition to supervising the model output results, the model's intermediate processing data, namely the speech separation mask, is also supervised, which can ensure that the intermediate processing data and output results of the trained model achieve the ideal effect, thereby improving the accuracy of speaker separation of the speaker separation model.

[0043] See also Figure 3 , Figure 3 This is a flow chart of an embodiment of the speaker separation method of the present application. Specifically, it may include the following steps:

[0044] Step S31: Acquire target data of the target environment.

[0045] In the disclosed embodiment, the target data includes target voice data collected in the same time period and a target image sequence of several speakers in the target environment. The target voice data collected in the same time period may be audio data within a time interval. For example, the audio data may be represented by a data dimension [S]. If the time interval length is t, then S is related to t and the voice sampling rate. If the time interval length t is 1 second and the voice sampling frequency is 16kHz, then S may be 16k, and the data dimension [S] represents 16k audio data points taken within 1 second. The representation method of the target voice data may be determined based on actual conditions and is not specifically limited here. The target image sequence of several speakers collected in the same time period may be selected based on actual conditions and is not specifically limited here.

[0046] In a specific implementation scenario, the target image in the target image sequence is a lip image of a speaker, and the lip image of the speaker can be represented by a data dimension. For example, the lip image sequence of the speaker can be represented by a data dimension [N, T V , H, W], N represents the number of speakers, the time interval length is t, T V Related to the time interval length t and the video frame rate, T V V represents the time progress, H and W represent the height and width of the video screen. The representation method of the target image sequence can be determined according to the actual situation and is not specifically limited here.

[0047] Step S32: Process the target data using the speaker separation model to obtain a speech separation mask for each speaker.

[0048] In an implementation scenario, the speaker separation model includes a speech separation module and a speaker separation module of the speaker separation model; the speech separation module of the speaker separation model is used to process the target data to obtain the speech separation mask of each speaker, wherein the speech separation mask of the speaker is used to filter the speech features that do not belong to the speaker. The target speech data in the target data can be selected to be subjected to a short-time Fourier transform (STFT) to obtain the target speech feature. At this time, the target speech feature contains the speech features of the current speaker (may be more than one) and the interference of environmental noise. The obtained target speech feature is composed of phase and amplitude, and both the phase and amplitude can be represented by data dimensions. For example, the amplitude in the target speech feature can be represented by the data dimension [T, F], wherein T represents that the time in the time interval is divided into multiple subintervals, and the value of T is positively correlated with the sampling rate, and F represents multiple sampling feature points after the sound is converted from the time domain to the frequency domain. The representation method of the target speech feature can be determined according to the actual situation, and is not specifically limited here.

[0049] In an implementation scenario, if the target image sequence is a speaker's lip image, the target image features can be extracted from the target image sequence by an image feature extraction module, and the target image features can be represented by data dimensions. For example, the target image features are represented by data dimensions [N, T V , M] represents, N represents the number of speakers, T V Related to the time interval length t and the video frame rate, T V V represents the time progress, and M represents the lip feature information. At this time, the lip image features include the lip movement features of each participant and possible interference (for example, yawning, chewing and other non-speaking lip movements). In the above method, after comprehensive analysis of speech data and image sequences, more information with reference significance can be obtained.

[0050] In one implementation scenario, the target image features of each speaker include sub-image features of multiple time segments, the target speech features include sub-speech features of multiple time segments, the target speech features and the target image features are fused to obtain a first fused feature, and for each speaker, the sub-speech features of each time segment are fused with the sub-image features of the corresponding time segment of the speaker to obtain the sub-fused features of the speaker, wherein the sub-fused features of several speakers constitute the first fused feature; the specific method of fusion can be selected according to actual conditions, and is not specifically limited here. The first fused feature is calculated by a neural network to obtain a speech separation mask for each speaker, and the speech separation mask can be represented by a data dimension, for example, the speech separation mask can be represented as [N, T, F], N represents the number of speakers, T is related to the time interval, and F represents multiple feature points. The speech mask of N speakers can effectively filter out speech features that do not belong to the speaker, and the speech features that do not belong to the speaker include speech noise, etc., thereby alleviating the adverse effects of environmental noise and multiple people speaking at the same time on the speaker separation task.

[0051] Step S33: Process the speech separation mask and target data using the speaker separation model to obtain a speaker separation result.

[0052] In an implementation scenario, after obtaining the speech separation mask of each speaker, only the result of speech data separation can be obtained, and the judgment result of whether N speakers speak in each time interval cannot be known. Therefore, the speech separation mask and target data are processed by the speaker separation module to obtain the speaker separation result. That is, the speech separation mask, target speech feature and target image feature are fused to obtain the second fusion feature, that is, for each speaker, the sub-speech features of different time segments are fused with the sub-image features and sub-masks of the corresponding time segments of the speaker, respectively, to obtain the first channel feature corresponding to the speaker, and the second fusion feature includes a plurality of first channel features, each of which corresponds to a speaker, and the speech separation mask of each speaker includes sub-masks of multiple time segments.

[0053] In one implementation scenario, the speaker separation result is used to determine the speakers in different time segments in the target speech data. For each speaker, the first channel feature corresponding to the speaker is fused with the remaining first channel features to obtain the second channel feature corresponding to the speaker. That is, after fusing the speech separation mask, the target speech feature and the target image feature, the first channel feature corresponding to the speaker is obtained. The first channel feature includes the feature information of each speaker. The first channel feature corresponding to the speaker is fused with the remaining first channel features, that is, the feature information of each speaker is fused to obtain the corresponding second channel feature. For example, if there are three speakers, the target speech features, target image features and speech separation mask of the three speakers are fused to obtain three first channel features. The first channel features include the feature information of each speaker. Then, the first channel feature corresponding to the speaker is fused with the remaining two first channel features to obtain three new second channel features, that is, each second channel feature contains the other two second channel features. For each speaker, the second channel feature corresponding to the speaker is used to predict the speaker's speaking confidence in each time segment corresponding to the target speech data. For example, if the speaking confidence is represented by the data dimension [N, T], T is the time dimension, such as the input speech length t is 1 second, T is 100, then each confidence in the T dimension represents the speaking confidence of the participant in each 10 millisecond interval. By fusing channel features, the globality of the judgment of speaker separation results is ensured.

[0054] The above scheme obtains the target speech data of the target environment and the target image sequence of several speakers, and processes the target data using the speaker separation model to obtain the speech separation mask of each speaker; and processes the speech separation mask and the target data using the speaker separation model to obtain the speaker separation result. Since the speaker separation model first processes to obtain the speech separation mask, the speech separation mask can filter out the sounds other than the corresponding speaker, and thus the speech separation mask is used to process the target data, and speakers in complex scenes such as high noise and aliasing can be separated, thereby improving the speaker separation effect, and also improving the application effect of speaker separation in high noise scenes.

[0055] In addition, during the training process, the speaker separation model also uses the speech separation mask and the speaker separation results to train the speaker separation model. In other words, in addition to supervising the model output results, the model's intermediate processing data, namely the speech separation mask, is also supervised, which can ensure that the intermediate processing data and output results of the trained model achieve the ideal effect, thereby improving the accuracy of speaker separation of the speaker separation model.

[0056] See also Figure 4 , Figure 4 It is a schematic diagram of a framework of an embodiment of a training device for a speaker separation model of the present application; the training device 40 for a speaker separation model includes an acquisition module 41, a processing module 42 and an adjustment module 43. The acquisition module 41 is used to acquire sample data of a sample environment, and the sample data includes sample speech data collected in the same time period and a sample image sequence of several speakers in the sample environment; the processing module 42 is used to use a speaker separation model to process the sample data to obtain a speech separation mask for each speaker, and to process the speech separation mask and the sample data to obtain a speaker separation result, wherein the speaker separation result is used to determine speakers in different time segments in the sample speech data; the adjustment module 43 is used to use the speech separation mask and the speaker separation result to adjust the network parameters of the speaker separation model.

[0057] In the above scheme, since the speaker separation model is first processed to obtain a speech separation mask, and the speech separation mask can filter out sounds other than the corresponding speaker, the speech separation mask is used to process the target data again, and the speakers in complex scenes such as high noise and aliasing can be separated, thereby improving the speaker separation effect and also improving the application effect of speaker separation in high noise scenes.

[0058] In addition, during the training process, the speaker separation model also uses the speech separation mask and the speaker separation results to train the speaker separation model. In other words, in addition to supervising the model output results, the model's intermediate processing data, namely the speech separation mask, is also supervised, which can ensure that the intermediate processing data and output results of the trained model achieve the ideal effect, thereby improving the accuracy of speaker separation of the speaker separation model.

[0059] In some disclosed embodiments, the adjustment module 43 includes a separation module, which is used to use a speech separation mask to obtain a first loss, and use a speaker separation result to obtain a second loss; the adjustment module 43 also includes an adjustment submodule, which is used to adjust the network parameters of the speaker separation model according to the first loss and the second loss.

[0060] In some disclosed embodiments, the first loss is L2 loss, and the second loss is cross entropy loss; and / or, the adjustment submodule can perform a weighted summation of the first loss and the second loss to obtain a third loss; and then adjust the network parameters of the speaker separation model based on the third loss.

[0061] Therefore, by adjusting the network parameters of the speaker separation model, the accuracy of speaker separation by the speaker separation model is improved.

[0062] In some disclosed embodiments, the speaker separation model includes a speech separation module and a speaker separation module; the processing module 42 includes an extraction module, the extraction module is used to use the speaker separation model to perform feature extraction on the sample data to obtain sample features, and the sample features include sample speech features of the sample speech data and sample image features of the sample image sequence; the processing module 42 includes a processing submodule, the processing submodule is used to use the speech separation module to process the sample features to obtain a speech separation mask for each speaker, wherein the speaker's speech separation mask is used to filter out speech features that do not belong to the speaker; the processing module 42 includes a separation module, the separation module is used to use the speaker separation module to process the speech separation mask and the sample features to obtain a speaker separation result.

[0063] Therefore, by processing the speech separation mask and sample features, the speaker separation result is obtained, thereby improving the accuracy of the speaker separation model in speaker separation.

[0064] In some disclosed embodiments, the sample images in the sample image sequence are lip images of speakers; and / or the speaker separation result includes the speaking confidence of each speaker in each time segment corresponding to the sample speech data.

[0065] Therefore, by combining the speaker's lip sequence and the speaker's voice data, the accuracy of speaker separation by the speaker separation model is improved.

[0066] See also Figure 5 , Figure 5 1 is a schematic diagram of a framework of an embodiment of a speaker separation device of the present application; the speaker separation device 50 includes an acquisition module 51, a processing module 52 and a determination module 53. The acquisition module 51 is used to acquire target data of a target environment, and the target data includes target speech data collected in the same time period and a target image sequence of several speakers in the target environment; the processing module 52 is used to process the target data using a speaker separation model to obtain a speech separation mask of each speaker; the determination module 53 is used to process the speech separation mask and the target data using a speaker separation model to obtain a speaker separation result, wherein the speaker separation result is used to determine the speakers of different time segments in the target speech data.

[0067] In the above scheme, since the speaker separation model is first processed to obtain a speech separation mask, and the speech separation mask can filter out sounds other than the corresponding speaker, the speech separation mask is used to process the target data again, and the speakers in complex scenes such as high noise and aliasing can be separated, thereby improving the speaker separation effect and also improving the application effect of speaker separation in high noise scenes.

[0068] In addition, during the training process, the speaker separation model also uses the speech separation mask and the speaker separation results to train the speaker separation model. In other words, in addition to supervising the model output results, the model's intermediate processing data, namely the speech separation mask, is also supervised, which can ensure that the intermediate processing data and output results of the trained model achieve the ideal effect, thereby improving the accuracy of speaker separation of the speaker separation model.

[0069] In some disclosed embodiments, the acquisition module 51 includes an image module, and the target image in the target image sequence in the image module is a lip image of the speaker; and / or, before the acquisition module 51 acquires the target data of the target environment, it is also necessary to train the speaker separation model using the steps in any of the above-mentioned speaker separation model training method embodiments.

[0070] Therefore, by training the speaker separation model, better speaker separation accuracy and robustness can be achieved.

[0071] In some disclosed embodiments, the speech separation module and the speaker separation module of the speaker separation model; the processing module 52 includes an extraction submodule, the extraction submodule is used to use the speaker separation model to extract features of the target data to obtain target features, and the target features include target speech features of the target speech data and target image features of the target image sequence; the processing module 52 includes a processing submodule, the processing submodule is used to use the speech separation module to process the target features to obtain a speech separation mask for each speaker, wherein the speaker's speech separation mask is used to filter out speech features that do not belong to the speaker; the determination module 53 includes a fusion module, the fusion module is used to use the speaker separation module to process the speech separation mask and the target features to obtain a speaker separation result.

[0072] Therefore, the speech separation mask of each speaker is obtained through the processing submodule, and the speech separation mask of each speaker can process the interference data in the speech data, thereby improving the effect of speech separation.

[0073] In some disclosed embodiments, the extraction submodule performs short-time Fourier transform on the target speech data to obtain frequency domain features as target speech features, and performs feature extraction on the target image sequence of each speaker to obtain the target image features of each speaker; the processing submodule is used to fuse the target speech features and the target image features to obtain a first fused feature; based on the first fused feature, a speech separation mask for each speaker is obtained; the fusion module is used to fuse the speech separation mask, the target speech features and the target image features to obtain a second fused feature, wherein the second fused feature includes a plurality of first channel features, each first channel feature corresponding to a speaker; and based on the second fused feature, a speaker separation result is obtained.

[0074] In some disclosed embodiments, the target image features of each speaker include sub-image features of multiple time segments, the target speech features include sub-speech features of multiple time segments, the target speech features and the target image features are fused to obtain a first fused feature, and for each speaker, the sub-speech features of each time segment can be fused with the sub-image features of the speaker's corresponding time segment to obtain the speaker's sub-fused features, wherein the sub-fused features of several speakers constitute the first fused feature; the speech separation mask of each speaker includes sub-masks of multiple time segments, and the speech separation mask, the target speech features and the target image features are fused to obtain a second fused feature, and for each speaker, the sub-speech features of different time segments can be fused with the sub-image features and sub-masks of the speaker's corresponding time segment to obtain the first channel feature corresponding to the speaker.

[0075] Therefore, by fusing the sub-speech features of different time segments with the sub-image features and sub-masks of the corresponding time segments of the speaker, the first channel features corresponding to the speaker are obtained, thereby improving the accuracy of determining the speaker.

[0076] In some disclosed embodiments, based on the second fusion feature, the speaker separation result is obtained, and for each speaker, the first channel feature corresponding to the speaker can be fused with the remaining first channel features to obtain the second channel feature corresponding to the speaker; for each speaker, the second channel feature corresponding to the speaker is used to predict the speaker's speaking confidence in each time segment corresponding to the target speech data.

[0077] Therefore, the speaker in each time segment can be determined through the speaking confidence of each time segment, thereby achieving speaker separation in complex scenarios such as high noise and aliasing.

[0078] See also Figure 6 , Figure 6 6 is a schematic diagram of a framework of an embodiment of an electronic device of the present application. The electronic device 60 includes a memory 61 and a processor 62 coupled to each other, the memory 61 stores program instructions, and the processor 62 is used to execute the program instructions to implement the training method of any speaker separation model or the steps in the embodiment of the speaker separation method. Specifically, the electronic device 60 may include but is not limited to: a desktop computer, a laptop computer, a server, a mobile phone, a tablet computer, etc., which are not limited here.

[0079] Specifically, the processor 62 is used to control itself and the memory 61 to implement the training method of any speaker separation model or the steps in the speaker separation method embodiment. The processor 62 can also be called a CPU (Central Processing Unit). The processor 62 may be an integrated circuit chip with signal processing capabilities. The processor 62 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 62 can be implemented by an integrated circuit chip.

[0080] In the above scheme, since the speaker separation model is first processed to obtain a speech separation mask, and the speech separation mask can filter out sounds other than the corresponding speaker, the speech separation mask is used to process the target data again, and the speakers in complex scenes such as high noise and aliasing can be separated, thereby improving the speaker separation effect and also improving the application effect of speaker separation in high noise scenes.

[0081] In addition, during the training process, the speaker separation model also uses the speech separation mask and the speaker separation results to train the speaker separation model. In other words, in addition to supervising the model output results, the model's intermediate processing data, namely the speech separation mask, is also supervised, which can ensure that the intermediate processing data and output results of the trained model achieve the ideal effect, thereby improving the accuracy of speaker separation of the speaker separation model.

[0082] See also Figure 7 , Figure 7 The computer-readable storage medium 70 stores program instructions 71 that can be executed by a processor, and the program instructions 71 are used to implement the steps in any of the above-mentioned speaker separation model training methods or speaker separation method embodiments.

[0083] In the above scheme, since the speaker separation model is first processed to obtain a speech separation mask, and the speech separation mask can filter out sounds other than the corresponding speaker, the speech separation mask is used to process the target data again, and the speakers in complex scenes such as high noise and aliasing can be separated, thereby improving the speaker separation effect and also improving the application effect of speaker separation in high noise scenes.

[0084] In addition, during the training process, the speaker separation model also uses the speech separation mask and the speaker separation results to train the speaker separation model. In other words, in addition to supervising the model output results, the model's intermediate processing data, namely the speech separation mask, is also supervised, which can ensure that the intermediate processing data and output results of the trained model achieve the ideal effect, thereby improving the accuracy of speaker separation of the speaker separation model.

[0085] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0086] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0087] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0088] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0089] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0090] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

Claims

1. A method for training a speaker separation model, characterized in that: include: Acquire sample data of a sample environment, wherein the sample data includes sample speech data collected in the same time period and a sample image sequence of several speakers in the sample environment; Using a speaker separation model, the sample data is processed to obtain a speech separation mask of each speaker, and the speech separation mask and the sample data are processed to obtain a speaker separation result, wherein the speaker separation result is used to determine speakers of different time segments in the sample speech data; The network parameters of the speaker separation model are adjusted using the speech separation mask and the speaker separation result.

2. The method according to claim 1, characterized in that The adjusting the network parameters of the speaker separation model by using the speech separation mask and the speaker separation result comprises: Using the speech separation mask, a first loss is obtained, and using the speaker separation result, a second loss is obtained; Based on the first loss and the second loss, network parameters of the speaker separation model are adjusted.

3. The method according to claim 2, characterized in that The first loss is L2 loss, and the second loss is cross entropy loss; And / or, adjusting the network parameters of the speaker separation model based on the first loss and the second loss includes: Perform a weighted summation of the first loss and the second loss to obtain a third loss; Based on the third loss, network parameters of the speaker separation model are adjusted.

4. The method according to claim 1, characterized in that: a speech separation module and a speaker separation module of the speaker separation model; The using of the speaker separation model to process the sample data to obtain a speech separation mask of each speaker, and processing the speech separation mask and the sample data to obtain a speaker separation result, comprises: Using the speaker separation model to perform feature extraction on the sample data to obtain sample features, wherein the sample features include sample speech features of the sample speech data and sample image features of the sample image sequence; Processing the sample features using the speech separation module to obtain a speech separation mask for each of the speakers, wherein the speech separation mask for the speaker is used to filter speech features that do not belong to the speaker; The speaker separation module is used to process the speech separation mask and the sample features to obtain the speaker separation result.

5. The method according to claim 1, characterized in that The sample images in the sample image sequence are lip images of the speaker; And / or, the speaker separation result includes the speaking confidence of each speaker in each time segment corresponding to the sample speech data.

6. A method for speaker separation, characterized in that: include: Acquire target data of a target environment, wherein the target data includes target speech data collected in the same time period and a target image sequence of a plurality of speakers in the target environment, wherein the target image in the target image sequence is a lip image of the speaker; Processing the target data using a speaker separation model to obtain a speech separation mask for each of the speakers; The speech separation mask and the target data are processed using the speaker separation model to obtain a speaker separation result, wherein the speaker separation result is used to determine speakers of different time segments in the target speech data.

7. The method according to claim 6, characterized in that Before acquiring the target data of the target environment, the method further includes: The speaker separation model is trained using the method according to any one of claims 1 to 5.

8. The method according to claim 6, characterized in that a speech separation module and a speaker separation module of the speaker separation model; The method of processing the target data using the speaker separation model to obtain a speech separation mask for each speaker includes: Using the speaker separation model to extract features from the target data to obtain target features, wherein the target features include target speech features of the target speech data and target image features of the target image sequence; Processing the target feature using the speech separation module to obtain a speech separation mask for each of the speakers, wherein the speech separation mask for the speaker is used to filter speech features that do not belong to the speaker; The using the speaker separation model to process the speech separation mask and the target data to obtain a speaker separation result includes: The speaker separation module is used to process the speech separation mask and the target feature to obtain the speaker separation result.

9. The method according to claim 8, characterized in that The step of extracting features from the target data to obtain target features includes: Performing short-time Fourier transform on the target speech data to obtain frequency domain features as target speech features, and performing feature extraction on the target image sequence of each speaker to obtain target image features of each speaker; The processing of the target features to obtain a speech separation mask for each of the speakers includes: Fusing the target speech feature and the target image feature to obtain a first fused feature; Based on the first fusion feature, obtaining a speech separation mask for each of the speakers; The processing of the speech separation mask and the target feature to obtain the speaker separation result includes: fusing the speech separation mask, the target speech feature, and the target image feature to obtain a second fused feature, wherein the second fused feature includes a plurality of first channel features, each of which corresponds to a speaker; The speaker separation result is obtained based on the second fusion feature.

10. The method according to claim 9, characterized in that The target image features of each of the utterances include sub-image features of multiple time segments, the target voice features include sub-voice features of multiple time segments, and the fusion of the target voice features and the target image features to obtain a first fusion feature includes: For each of the speakers, the sub-speech features of each time segment are respectively fused with the sub-image features of the corresponding time segment of the speaker to obtain the sub-fusion features of the speaker, wherein the sub-fusion features of the plurality of speakers constitute the first fusion feature; The speech separation mask of each speaker includes a plurality of sub-masks of time segments, and the fusion of the speech separation mask, the target speech feature and the target image feature to obtain the second fusion feature includes: For each of the speakers, the sub-speech features of different time segments are fused with the sub-image features and sub-masks of the corresponding time segments of the speaker, so as to obtain the first channel features corresponding to the speaker.

11. The method according to claim 9, characterized in that The obtaining the speaker separation result based on the second fusion feature includes: For each of the speakers, the first channel feature corresponding to the speaker is fused with the remaining first channel features to obtain a second channel feature corresponding to the speaker; For each of the speakers, the second channel features corresponding to the speaker are used to predict the speaking confidence of the speaker in each time segment corresponding to the target speech data.

12. A training device for a speaker separation model, characterized in that: include: An acquisition module, used to acquire sample data of a sample environment, wherein the sample data includes sample speech data collected in the same time period and a sample image sequence of several speakers in the sample environment; A processing module, configured to process the sample data using a speaker separation model to obtain a speech separation mask of each speaker, and to process the speech separation mask and the sample data to obtain a speaker separation result, wherein the speaker separation result is used to determine speakers of different time segments in the sample speech data; An adjustment module is used to adjust the network parameters of the speaker separation model using the speech separation mask and the speaker separation result.

13. A speaker separation device, characterized in that: include: An acquisition module, used to acquire target data of a target environment, wherein the target data includes target speech data collected in the same time period and a target image sequence of several speakers in the target environment, wherein the target image in the target image sequence is a lip image of the speaker; A processing module, used for processing the target data using a speaker separation model to obtain a speech separation mask for each speaker; A determination module is used to process the speech separation mask and the target data using the speaker separation model to obtain a speaker separation result, wherein the speaker separation result is used to determine the speakers of different time segments in the target speech data.

14. An electronic device, characterized in that: It comprises a memory and a processor coupled to each other, wherein the memory stores program instructions, and the processor is used to execute the program instructions to implement the training method of the speaker separation model described in any one of claims 1 to 5 or the speaker separation method described in any one of claims 6 to 11.

15. A computer-readable storage medium, characterized in that: Program instructions that can be executed by a processor are stored, and the program instructions are used to implement the training method of the speaker separation model described in any one of claims 1 to 5 or the speaker separation method described in any one of claims 6 to 11.

Images