Method, device and equipment for multi-speaker speech separation and storage medium

Abstract

The application discloses a multi-speaker speech separation method and device, equipment and storage medium, and relates to the field of artificial intelligence. The method comprises the following steps: acquiring video frame data and audio frame data in a video; acquiring face region data from the video frame data; acquiring mouth shape change data from the face region data, and extracting semantic embedding representation based on the mouth shape change data; extracting speech embedding representation based on the audio frame data; inputting the semantic embedding representation and the speech embedding representation into a multi-modal speech separation model to obtain speech data of a target speaker. The method extracts semantic embedding representation and speech embedding representation respectively, inputs the semantic embedding representation and the speech embedding representation into the multi-modal speech separation model, and then obtains the speech data of the target speaker, thereby realizing speech enhancement effect of the target speaker.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence, in particular to a method and device for speech separation of multiple speakers, equipment and storage medium. BACKGROUND

[0002] Speech separation refers to extracting one or more target speech signals from mixed speech generated by multiple speakers.

[0003] With the development of deep learning, neural networks have also been applied in the field of speech separation. After training a speech separation model using audio samples of target speakers, the mixed speech of multiple speakers is input into the speech separation model, and the speech separation model outputs the speech signal of the target speaker.

[0004] However, the speech signal of the target speaker extracted by the above method is still not pure enough. SUMMARY

[0005] The embodiments of the present application provide a method and device for speech separation of multiple speakers, equipment and storage medium. The technical solutions are as follows:

[0006] According to an aspect of the present application, a method for speech separation of multiple speakers is provided, video frame data and audio frame data in a video are obtained;

[0007] Face region data is obtained from the video frame data; lip movement change data is obtained from the face region data, and semantic embedding representation is extracted based on the lip movement change data;

[0008] Speech embedding representation is extracted based on the audio frame data;

[0009] The semantic embedding representation and the speech embedding representation are input into a multi-modal speech separation model, and target speaker speech data is separated.

[0010] According to another aspect of the present application, a device for speech separation of multiple speakers is provided, the device comprises:

[0011] An acquisition module is configured to obtain video frame data and audio frame data in a video;

[0012] The acquisition module is further configured to obtain face region data from the video frame data; obtain lip movement change data from the face region data, and extract semantic embedding representation based on the lip movement change data;

[0013] The acquisition module is further configured to extract speech embedding representation based on the audio frame data;

[0014] A separation module is configured to input the semantic embedding representation and the speech embedding representation into a multi-modal speech separation model to obtain speech data of a target speaker.

[0015] According to another aspect of the present application, a computer device is provided, which includes a processor and a memory, the memory storing at least one program, the at least one program being loaded and executed by the processor to enable the wireless device to implement the method for multi-speaker speech separation according to the above aspect.

[0016] According to another aspect of the present application, a computer readable storage medium is provided, which stores at least one program, the at least one program being loaded and executed by a processor to enable the wireless device to implement the method for multi-speaker speech separation according to the above aspect.

[0017] According to another aspect of the present application, a computer program product is provided, which includes at least one program stored in a computer readable storage medium; a processor of a communication device reads the at least one program from the computer readable storage medium, and executes the at least one program to enable the communication device to implement the method for multi-speaker speech separation according to the above aspect.

[0018] According to another aspect of the present application, a computer program is provided, which includes at least one program stored in a computer readable storage medium; a processor of a communication device reads the at least one program from the computer readable storage medium, and executes the at least one program to enable the communication device to implement the method for multi-speaker speech separation according to the above aspect.

[0019] The technical scheme provided by the embodiments of the present application has at least the following beneficial effects:

[0020] By respectively acquiring video frame data and audio frame data in the video, obtaining face region data from the video frame data, and further obtaining lip shape change data, semantic embedding representation is extracted; speech embedding representation is extracted from the audio frame data; and speech data of the target speaker is obtained based on the semantic embedding representation and the speech embedding representation, so as to improve the accuracy of the target speaker recognition and achieve the effect of speech enhancement of the target speaker. BRIEF DESCRIPTION OF DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the description of the embodiments will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without any creative effort based on these drawings.

[0022] Figure 1 is a flowchart of a method for speech separation of multiple speakers provided by another exemplary embodiment of the present application;

[0023] Figure 2 is a flowchart of a method for speech separation of multiple speakers provided by another exemplary embodiment of the present application;

[0024] Figure 3 is a schematic diagram of data processing provided by another exemplary embodiment of the present application;

[0025] Figure 4 is a schematic diagram of a method for speech separation of multiple speakers provided by another exemplary embodiment of the present application;

[0026] Figure 5 is a flowchart of a method for speech separation of multiple speakers provided by another exemplary embodiment of the present application;

[0027] Figure 6 is a flowchart of a method for speech separation of multiple speakers provided by another exemplary embodiment of the present application;

[0028] Figure 7 is a flowchart of a method for speech separation of multiple speakers provided by another exemplary embodiment of the present application;

[0029] Figure 8 is a schematic diagram of speech processing provided by another exemplary embodiment of the present application;

[0030] Figure 9 is a flowchart of a method for speech separation of multiple speakers provided by another exemplary embodiment of the present application;

[0031] Figure 10 is a block diagram of an apparatus for speech separation of multiple speakers provided by another exemplary embodiment of the present application;

[0032] Figure 11 is a block diagram of a computer device provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0033] In order to make the purposes, technical solutions and advantages of the present application clearer, the embodiments of the present application will be further described in detail below with reference to the drawings.

[0034] First, the terms involved in the embodiments of the present application are briefly introduced:

[0035] Neural network model: It is a mathematical model that simulates the behavior characteristics of animal neural networks for distributed parallel information processing. This network relies on the complexity of the system to adjust the relationship between a large number of nodes connected to each other, thereby achieving the purpose of processing information.

[0036] Embedded representation: It refers to a distributed representation of an input object generated based on a neural network model. The main role is to convert the original object's high-dimensional sparse vector into a low-dimensional, dense vector, so that these low-dimensional, dense vectors can express one or more characteristics of the corresponding object. At the same time, the distance between different vectors can reflect the similarity between objects.

[0037] Fourier transform (FT): It refers to a mathematical method that can convert signals in the time domain into signals in the frequency domain.

[0038] In the related art, in a scene containing multiple speakers, the speech data of multiple speakers needs to be separated to distinguish the voice of the target speaker from that of others. Therefore, the present application proposes a method for speech separation of multiple speakers, and the specific implementation is as follows:

[0039] Figure 1 It is a flowchart of a method for speech separation of multiple speakers provided by an exemplary embodiment of the present application. The method comprises:

[0040] Step 120: Obtain video frame data and audio frame data in the video;

[0041] In some embodiments, in the case where the video contains only one speaker, the speech data of the speaker can be obtained directly by speech extraction.

[0042] In the embodiments of the present application, the case where the video contains multiple speakers is mainly introduced, that is, the video contains at least two speakers. Exemplarily, taking the case where the video contains m speakers as an example, m is a positive integer greater than or equal to 2.

[0043] Obtain video frame data and audio frame data in the video respectively. The video frame data specifically refers to image frame data obtained according to the time sequence of the video. Exemplarily, the video includes N image frames according to the time sequence, that is, the video is composed of N image frames, and N is a positive integer greater than or equal to 2.

[0044] In one example, all N image frame data in the video is obtained.

[0045] In one example, the N image frames are extracted at intervals due to large amount of calculation data of the N image frame data. The extraction can be performed at fixed intervals, for example, one or more image frames are extracted every 5 seconds. The extraction can also be performed at random time, for example, the extraction is performed at random intervals, and the intervals can be the same or different. In one example, the N image frames are extracted randomly according to the recognition of the N image frames, for example, when the i-th image frame is recognized to contain a speaker, the i-th image frame is extracted.

[0046] The audio frame data specifically refers to audio frame information in the video, that is, information generated by sound signals emitted by multiple speakers when speaking.

[0047] Step 142: obtaining face region data from the video frame data;

[0048] The face region data is obtained from the video frame data, and one or more face regions can be included in the same image frame, that is, the face region data is all or part of the video frame data. The face region data is the face data of the speaker in the i-th image frame in the video frame data.

[0049] For example, one artificial intelligence (AI) model for face recognition is used to perform face recognition on the image in the i-th image frame, and one or more faces are recognized from the i-th image frame. The region data where the recognized one or more faces are located is the face region data described above. The AI model for face recognition is trained with face data of multiple speakers as training samples before the embodiment of the present application is implemented.

[0050] In one example, the video frame data only contains face region data, and the face region data is all of the video frame data.

[0051] In one example, the video frame data contains at least two feature regions of the speaker, such as a face region and a hand region, and the face region data is part of the video frame data.

[0052] In some embodiments, the video contains m speakers, and the face region data obtained from the video frame data includes face region data of the first speaker to face region data of the m-th speaker, and m is a positive integer greater than or equal to 2.

[0053] Step 144: obtaining lip shape change data from the face region data;

[0054] In some embodiments, the face region data at least includes lip region data, and the lip movement change data is obtained based on a change in the lip region data. The lip region data is one of the face region data. For example, according to different features (facial features) in the face, the face region data can include forehead region data, eye region data, nose region data, ear region data, and lip region data.

[0055] For example, the AI model for lip recognition is used to recognize the lips of each face region to obtain the lip movement change of each face region. For example, it is recognized that the lip state of the speaker changes from an open state to a closed state. The AI model for lip recognition is obtained by pre-training multiple speaker lip data as training samples before implementing the embodiments of the present application.

[0056] In some embodiments, the lip movement change data is one-to-one corresponding to the speaker. In the case where the video includes m speakers, the lip movement change data corresponding to the first speaker is obtained from the face region data of the first speaker, and the lip movement change data corresponding to the mth speaker is obtained from the face region data of the mth speaker. The value of m is a positive integer greater than or equal to 2. The m speakers include the target speaker.

[0057] Step 146: extracting semantic embedding representation based on the lip movement change data;

[0058] The semantic embedding representation is semantic information corresponding to the speech information in the audio frame data obtained based on the video frame data. The semantic embedding representation is obtained based on the lip movement change, such as by lip reading, to identify that the speaker is saying "hello". In the case where the video includes m speakers, the semantic embedding representation corresponding to the first speaker is obtained based on the lip movement change of the first speaker, and the semantic embedding representation corresponding to the mth speaker is obtained based on the lip movement change of the mth speaker. The m speakers include the target speaker.

[0059] Step 160: extracting speech embedding representation based on the audio frame data;

[0060] The speech embedding representation is obtained based on the speech information in the audio frame data, such as by voiceprint extraction, speech recognition, etc. The audio frame data mainly includes audio data of the speaking sound of the multiple speakers in the video.

[0061] Step 180: inputting the semantic embedding representation and the speech embedding representation into a multi-modal speech separation model to separate and obtain the speech data of the target speaker.

[0062] The semantic embedding representation extracted based on the video frame data and the speech embedding representation extracted based on the audio frame data are input into a multi-modal speech separation model. The multi-modal speech separation model separates speech data of a target speaker by combining the semantic embedding representation and the speech embedding representation to obtain the speech data of the target speaker.

[0063] In summary, the method provided in the embodiment extracts semantic embedding representation by respectively obtaining video frame data and audio frame data in a video, obtaining face region data from the video frame data, and further obtaining mouth shape change data, thereby extracting semantic embedding representation; extracts speech embedding representation from audio frame data; and obtains speech data of a target speaker based on the semantic embedding representation and the speech embedding representation, thereby improving the accuracy of target speaker recognition and achieving the effect of speech enhancement of the target speaker.

[0064] Figure 2 is a flowchart of a method for multi-speaker speech separation provided in an example embodiment of the present application. The method mainly includes a face recognition network 21 for recognizing face region data in video frame data, a network 22 for mouth shape detection, a speech extraction network 23 for speech separation, and a speech evaluation network 24 for speech evaluation.

[0065] In some embodiments, each of the N image frames corresponds to time information (e.g. timestamp) of the image frame, and each audio frame also corresponds to time information of the audio frame. According to the time information of the image frame and the time information of the audio frame, the corresponding image frame and audio frame can be aligned. However, it should be understood that the timestamp of the image frame and the timestamp of the audio frame are not one-to-one corresponding. For example, 1 second corresponds to 60 frames of image frame; and every 10 ms of sound is encoded as an audio frame.

[0066] In some embodiments, in the case where the mouth shape change data is obtained in the face region data, the speech embedding representation is extracted based on the audio frame data corresponding to the mouth shape change time period. The mouth shape change time period is a time period corresponding to the mouth shape change data. For example, as shown in Figure 3 the time length of a video is 150s, the mouth shape change data in the face region data in the video frame data is obtained from the 50th second to the 100th second, and the speech embedding representation is extracted from the audio frame data corresponding to the 50th second to the 100th second.

[0067] In some embodiments, as shown in Figure 4 the above step 160 can optionally include the following sub-steps:

[0068] Step 162: Fourier transforming the audio frame data to obtain multi-speaker spectral graph frequency data;

[0069] The audio frame data of multiple speakers is subjected to Fourier transform to convert into frequency data of a frequency spectrum of multiple speakers. For example, as shown in Figure 2 The audio frame data is subjected to short-time Fourier transform (STFT) to convert into frequency data of a frequency spectrum of multiple speakers. The frequency spectrum is a frequency distribution diagram of sound signals of multiple speakers, and is used to represent energy values of sound signals of multiple speakers at corresponding frequencies.

[0070] Step 164: inputting the frequency data of the frequency spectrum of multiple speakers into a feature extraction network to extract a speech embedding representation.

[0071] In some embodiments, the feature extraction network mainly extracts deeper data features by performing convolution processing on input data.

[0072] The frequency data of the frequency spectrum of multiple speakers is inputted into a feature extraction network to obtain a speech embedding representation by performing convolution processing and feature extraction on the frequency data of the frequency spectrum of multiple speakers.

[0073] In summary, the method provided in the embodiment converts speech signals in an audio frame into frequency domain data by Fourier transform, thereby simplifying subsequent processing of the speech signals, so as to obtain a speech embedding representation by processing the speech signals.

[0074] In some embodiments, as shown in Figure 5 The step 180 can optionally include the following sub-steps:

[0075] Step 182: obtaining a first speaker embedding representation of a target speaker;

[0076] The first speaker embedding representation is an embedding representation corresponding to speech data of the target speaker, or is understood as a speech embedding representation of the real target speaker. The first speaker embedding representation is also called a speaker identity (Speaker ID), that is, information capable of representing the speaker identity. Each speaker corresponds to a Speaker ID, and the Speaker IDs can be similar, but at least have slight differences, so different speakers correspond to different Speaker IDs.

[0077] In some embodiments, the first speaker embedding representation is obtained according to a pre-trained neural network model. The neural network model is trained by using clean (noise-free) speech data of the target speaker, and the output of the neural network model is taken as the first speaker embedding representation.

[0078] Step 184: input the first speaker embedding representation, the semantic embedding representation, and the speech embedding representation into a multi-modal speech separation model to separate speech data of the target speaker.

[0079] The first speaker embedding representation of the target speaker, the semantic embedding representation extracted based on the video frame data, and the speech embedding representation extracted based on the audio frame data are input into the multi-modal speech separation model. By combining the first speaker embedding representation, the semantic embedding representation, and the speech embedding representation, the speech data of the target speaker is separated to obtain the speech data of the target speaker.

[0080] The first speaker embedding representation is the real speech embedding representation of the target speaker, which is used as a reference for the speech data of the target speaker during separation. The semantic embedding representation is mainly used to assist the speech embedding representation in separating the speech data of the target speaker, such as assisting in separating the speech data of the target speaker expressing "hello" through the semantics of the word "hello".

[0081] In summary, the method provided in the embodiment can effectively improve the accuracy of identifying the target speaker by combining the first speaker embedding representation, the semantic embedding representation, and the speech embedding representation.

[0082] In some embodiments, the multi-modal speech separation model includes a Long Short-Term Memory (LSTM) network and a fully connected layer. As shown in Figure 6 The above step 184 can optionally include the following sub-steps:

[0083] Step 1842: input the first speaker embedding representation, the semantic embedding representation, and the speech embedding representation into the LSTM network to obtain a fused high-dimensional output;

[0084] The LSTM network is a time recurrent neural network, which mainly includes three stages: 1. forgetting stage: selectively forgetting the input of the LSTM network, such as forgetting the environmental noise data in the speech embedding representation. 2. selective memory stage: selectively memorizing the input of the LSTM network, such as selectively memorizing the data corresponding to the first speaker embedding representation in the speech embedding representation. 3. output stage: fusing the results of selective forgetting and selective memory, and outputting the results.

[0085] In some embodiments, the first speaker embedding representation, the semantic embedding representation, and the speech embedding representation are input into the LSTM network, and after the selection forgetting stage and the selection memory stage, a fused high-dimensional output is obtained. The fused high-dimensional output is an output result obtained by fusing the first speaker embedding representation, the semantic embedding representation, and the speech embedding representation after forgetting and memory in the LSTM network.

[0086] Step 1844: inputting the fused high-dimensional output into a full connection layer to obtain spectral graph frequency data of the target speaker;

[0087] The fused high-dimensional output is input into a full connection layer for classification, and the full connection layer classifies the spectral graph frequency data of the target speaker by performing feature extraction on the fused high-dimensional output. The spectral graph frequency data is dimension data of the spectral graph frequency to which the fused high-dimensional output is mapped by the full connection layer.

[0088] Step 1846: performing inverse Fourier transform on the spectral graph frequency data of the target speaker to obtain speech data of the target speaker.

[0089] The spectral graph frequency data of the target speaker is subjected to inverse Fourier transform to convert the spectral graph frequency data of the target speaker into speech data of the target speaker.

[0090] In summary, the method provided in the embodiment can effectively improve the accuracy of identifying the target speaker by combining the first speaker embedding representation, the semantic embedding representation, and the speech embedding representation.

[0091] In some embodiments, the speech features of the target speaker are not always constant. For example, as the target speaker grows older, the voice will become mature from childish, and this change is usually irreversible. For example, the target speaker may have a temporary voice change due to physical reasons (cold), and this change is usually reversible, i.e., it has a time property, and in the case that the physical condition of the target speaker recovers, the speech features of the target speaker will also recover.

[0092] In view of the above situation that the speech features of the target speaker may change, the method for multi-speaker speech separation provided in the embodiment can also adaptively update the embedding representation of the target speaker.

[0093] Figure 7 is a flowchart of a method for multi-speaker speech separation provided in an exemplary embodiment of the present application. The method comprises:

[0094] Step 220: inputting the speech data of the target speaker into a speech evaluation model to obtain a speech quality evaluation;

[0095] The speech data of the target speaker is obtained from speech data separated from a video containing multiple speakers, and is speech data corresponding to the target speaker in the current video.

[0096] The speech evaluation model is a model for speech quality detection of speech data. The speech data of the target speaker separated from the multi-modal speech separation model is input into the speech evaluation model to perform speech quality detection on the speech data of the target speaker, and obtain a speech quality evaluation.

[0097] Step 240: generating a second speaker embedding representation of the target speaker when the speech quality evaluation is greater than an evaluation threshold.

[0098] In some embodiments, the speech quality evaluation is provided with an evaluation threshold. The evaluation threshold is a threshold for reflecting the output signal-to-noise ratio (proportion of signal and noise) of the speech evaluation model. When the speech quality evaluation is greater than the evaluation threshold, the output signal-to-noise ratio of the speech evaluation model is greater than the threshold, the noise of the speech data of the target speaker is larger, and the second speaker embedding representation of the target speaker is generated at this time.

[0099] In some embodiments, the second speaker embedding representation is obtained in a similar manner to the first speaker embedding representation described above. The speech data of the target speaker separated from the video containing multiple speakers is input into the neural network model for extracting the speech embedding representation, and the second speaker embedding representation is output.

[0100] Step 260: updating the first speaker embedding representation based on the second speaker embedding representation.

[0101] In some embodiments, the second speaker embedding representation is a speaker embedding representation obtained based on noisy speech data of the target speaker, and the first speaker embedding representation is a speaker embedding representation obtained based on clean (noise-free) speech data of the target speaker. In some embodiments, the first speaker embedding representation is updated based on the second speaker embedding representation.

[0102] The first speaker embedding representation is obtained by sampling clean (noise-free) speech data of the target speaker in advance. The second speaker embedding representation is obtained by first separating noisy speech data of the target speaker from a video containing multiple speakers based on a multi-speaker speech separation method, and then obtaining the noisy speech data of the target speaker.

[0103] In some embodiments, a distance between the first speaker embedding representation and the second speaker embedding representation is calculated, which is or understood as a difference value for describing the similarity of the first speaker embedding representation and the second speaker embedding representation. The distance includes a first threshold value and a second threshold value, the first threshold value is used to distinguish that the similarity of the first speaker embedding representation and the second speaker embedding representation is greater and the similarity of the first speaker embedding representation and the second speaker embedding representation is general; the second threshold value is used to distinguish that the similarity of the first speaker embedding representation and the second speaker embedding representation is general and the similarity of the first speaker embedding representation and the second speaker embedding representation is smaller. The first threshold value and the second threshold value can be preset in advance, or can be dynamically adjusted according to actual conditions.

[0104] In summary, the method provided by the embodiment can make the embedding representation of the target speaker closer to the current state by combining the speech quality evaluation and adaptively updating the embedding representation of the target speaker, and avoid using the original migration representation of the target speaker when the embedding representation of the target speaker has changed, so as to cause the output of the target speaker's voice data to be inaccurate.

[0105] In some embodiments, based on the first threshold value and the second threshold value of the distance, the distance between the first speaker embedding representation and the second speaker embedding representation can be divided into the following three cases:

[0106] The first case: the distance between the first speaker embedding representation and the second speaker embedding representation is less than or equal to the first threshold value, or understood as the similarity between the first speaker embedding representation and the second speaker embedding representation is greater.

[0107] In some embodiments, in the case that the distance between the first speaker embedding representation and the second speaker embedding representation is less than or equal to the first threshold value, the first speaker embedding representation is not updated, or the first speaker embedding representation is updated using the second speaker embedding representation. At this time, it is understood that the distance between the first speaker embedding representation and the second speaker embedding representation is infinitely close or negligible. Therefore, the effect of updating the first speaker embedding representation using the second speaker embedding representation or not updating the first speaker embedding representation using the second speaker embedding representation is consistent.

[0108] It is worth noting that in the case that the distance between the first speaker embedding representation and the second speaker embedding representation is less than or equal to the first threshold value, it is preferred that the first speaker embedding representation is not updated. Compared with the second speaker embedding representation, the first speaker embedding representation is less affected by noise, so it is generally considered that the authenticity of the first speaker embedding representation is higher.

[0109] The second case: the distance between the first speaker embedding representation and the second speaker embedding representation is greater than the first threshold and less than the second threshold, or understood as the similarity between the first speaker embedding representation and the second speaker embedding representation is general.

[0110] In some embodiments, the weighted sum of the first speaker embedding representation and the second speaker embedding representation is calculated in the case that the distance between the first speaker embedding representation and the second speaker embedding representation is greater than the first threshold and less than the second threshold, to obtain a third speaker embedding representation.

[0111] In some embodiments, the weight of the first speaker embedding representation and the weight of the second speaker embedding representation are preset in advance. For example, assuming that the first speaker embedding representation is A and the second speaker embedding representation is B, the weight of the first speaker embedding representation is preset as 80% and the weight of the second speaker embedding representation is preset as 20%, then the third speaker embedding representation = 80% A + 20% B. It should be noted that the 80% and 20% are only reference values, and the actual weight values can be adjusted according to the cognition of those skilled in the art. In some embodiments, the first speaker embedding representation is updated using the third speaker embedding representation. For example, the first speaker embedding representation A is replaced by the calculated third speaker embedding representation 80% A + 20% B.

[0112] In some embodiments, the above weights are dynamically adjusted. In the case that the voice state of the target speaker changes, the weight of the first speaker embedding representation and the weight of the second speaker embedding representation are dynamically adjusted; and the weighted sum of the first speaker embedding representation and the second speaker embedding representation is calculated based on the adjusted weight of the first speaker embedding representation and the weight of the second speaker embedding representation. The weight is dynamically adjusted based on the actual situation of different target speakers in different states, which is related to different factors, such as adjustment based on the age change of the target speaker, adjustment based on the temporary physical change (such as cold) of the target speaker, and adjustment based on the long-term physical change of the target speaker.

[0113] In some embodiments, the type of change of the voice state of the target speaker is identified by a voice state identification model, which is pre-trained by taking the voice state of the target speaker as a training sample before the implementation of the embodiments of the present application. The type of change of the voice state of the target speaker includes at least one of a long-term change and a short-term change, the long-term change is, for example, the voice change of the target speaker from childhood to adulthood, and the short-term change is, for example, the voice change of the target speaker due to cold.

[0114] In some embodiments, in a case where the change type of the voice state of the target speaker is a long-term change, the similarity between the first speaker embedding representation and the target speaker embedding representation is less than the similarity between the second speaker embedding representation and the target speaker embedding representation, and the weight of the first speaker embedding representation is adjusted to be less than the weight of the second speaker embedding representation.

[0115] In some embodiments, in a case where the change type of the voice state of the target speaker is a short-term change, the similarity between the first speaker embedding representation and the target speaker embedding representation is greater than the similarity between the second speaker embedding representation and the target speaker embedding representation, and the weight of the first speaker embedding representation is adjusted to be greater than the weight of the second speaker embedding representation.

[0116] Based on the adjusted weights of the first speaker embedding representation and the second speaker embedding representation, a weighted sum of the first speaker embedding representation and the second speaker embedding representation is calculated.

[0117] In one example, the weight is dynamically adjusted based on the age of the target speaker. Generally, the age change of the target speaker is irreversible, so when the target speaker is in the voice change period, the reference value of the second speaker embedding representation is greater than that of the first speaker embedding representation, and at this time, the weight of the second speaker embedding representation is dynamically adjusted to be greater than that of the first speaker embedding representation.

[0118] In one example, the weight is dynamically adjusted based on the temporary physical change (such as a cold) of the target speaker. Generally, the temporary physical change (such as a cold) of the target speaker will be reversed as the body recovers, but when the target speaker has a temporary physical change, the voice features of the target speaker will have a temporary change, and at this time, the embedding representation of the first speaker and the embedding representation of the second speaker need to be dynamically adjusted during the period when the target speaker is ill. In a case where the target speaker is ill, the weight of the second speaker embedding representation is dynamically adjusted to be greater than that of the first speaker embedding representation; in a case where the target speaker recovers, the weight of the first speaker embedding representation is dynamically adjusted to be greater than that of the second speaker embedding representation. In some embodiments, the weight is also adjusted according to the degree of illness of the target speaker, such as in a case where the target speaker is in a poor physical condition (with a significant voice change), the weight of the first speaker embedding representation is adjusted to be 80%, and the weight of the second speaker embedding representation is adjusted to be 20%; in a case where the target speaker recovers (with a smaller voice change), the weight of the first speaker embedding representation is adjusted to be 60%, and the weight of the second speaker embedding representation is adjusted to be 40%.

[0119] In some embodiments, the weight is dynamically adjusted periodically according to a period. In one example, the weight is dynamically adjusted with a period of one week (7 days), considering that the target speaker can have a short-term illness (short-term voice change) in a week, such as from getting a cold to recovery. In one example, the weight is dynamically adjusted with a period of one month (30 days). In one example, the weight is dynamically adjusted with a period of half a year (180 days). In some embodiments, the above period is pre-set or dynamically adjusted according to actual conditions.

[0120] The third case: the distance between the first speaker embedding representation and the second speaker embedding representation is greater than or equal to the second threshold, or understood as the similarity between the first speaker embedding representation and the second speaker embedding representation is small.

[0121] In some embodiments, in the case where the distance between the first speaker embedding representation and the second speaker embedding representation is greater than or equal to the second threshold, the output prompt information that the separated speech data is not the speech data of the target speaker. At this time, it is understood that the distance between the first speaker embedding representation and the second speaker embedding representation is large, and the separated speech data (corresponding to the second speaker embedding representation) and the speech data of the target speaker (corresponding to the first speaker embedding representation) have a large gap.

[0122] In summary, the method provided by the embodiment realizes adaptive updating of the embedding representation of the target speaker based on the distance between the first speaker embedding representation and the second speaker embedding representation, by inputting the speech data of the target speaker into the speech evaluation model, thereby further improving the speech recognition accuracy of the target speaker in multiple scenarios.

[0123] The embodiment of the present application provides a method for multi-speaker speech separation. Generally, the method for speaker separation is to input mixed speech data, and different speaker embedding representations are used. After neural network training, multiple speakers can be separated. If the information of the target speaker is saved in the neural network in advance, the speech information of the target speaker can be obtained through multi-speaker speech extraction. The method provided by the embodiment of the present application combines a single speech data separation method and a face recognition method based on images, uses multi-modal joint biometric recognition to achieve higher recognition rate. Meanwhile, the method provided by the embodiment of the present application considers that the features of the speech data of the target speaker will also change over time and environment, for example, voice change of a child, poor state of a cold, etc. At this time, the speech embedding representation of the target speaker is different from the speech embedding representation in normal conditions. The method provided by the embodiment of the present application updates the speech embedding representation of the target speaker adaptively, thereby realizing speech enhancement effect.

[0124] The method provided in the embodiments of the present application processes the obtained video by dividing the main data of the video into video frame data and audio frame data. The pictures in the video frame data are input into a face recognition network for processing, and the output of the face recognition network is subjected to lip shape detection. When it is detected that the speaking lip shape changes in multiple pictures, the semantic embedding representation of the speaker is obtained. The audio frame data is subjected to STFT to obtain the speech embedding representation of the speaker. At this time, the processing of the audio signal is started, and the semantic embedding representation, the speech embedding representation and the first speaker embedding representation obtained by pre-registration are output to a speech extraction network. As shown in the example of Figure 8 , the mixed speech data is subjected to speech separation to obtain the speech data of the target speaker. At this time, the output speech data is subjected to quality evaluation by a speech evaluation model. When the speech quality is high, the second speaker embedding representation is generated according to the speech data at this time, the distance between the second speaker embedding representation and the first speaker embedding representation is calculated, and the embedding representation of the target speaker is dynamically updated according to the updating strategy. Through the adaptive updating strategy, the voiceprint features of the target speaker are dynamically updated with the state of the speaker, so as to improve the speech recognition rate and separation effect of the target speaker. In this process, the video frame data is mainly used to detect whether a person is speaking, and the semantic information implied by the lip shape change is used to assist the multi-speaker speech separation.

[0125] As shown in Figure 9 , the method provided in the embodiments of the present application mainly includes the following modules:

[0126] 1. Face extraction module 31. Based on a convolutional neural network (CNN) and a face recognition network. Before the original face image is input into the face recognition network, the CNN is used for face detection, because the CNN performs better in some scenes, such as partial occlusion or unclear outline, etc. At the same time, the face region is cropped and all the faces are resized to a fixed size for face embedding extraction. The extracted face features are analyzed for lip shape changes. If it is judged that a person is speaking in the scene, the semantic information implied by the lip shape change is formed into a semantic embedding representation, and the work of the speech extraction module is started at the same time.

[0127] 2. Speech extraction module 32. First, a voiceprint library of a target speaker to be obtained is established in advance, and a clean speech signal of the target speaker is input as a reference. One method of establishing the voiceprint library of the target speaker is to input the clean speech signal of the target speaker into a neural network, label the speaker, and train the network output as a deep network feature vector (d-vector), i.e., the speaker ID.

[0128] 3. Mask estimation module 33. Predict the target speaker mask in the time-frequency domain, the network starts from a multi-layer CNN, which captures the changes in time and frequency, the output of the CNN layer and the output of the speech embedding module and the face extraction module are jointly input into the LSTM layer, and then pass through the fully connected (Fully Connected layers, FC) layer to map the high-dimensional output of the LSTM to the dimension of the frequency of the spectrogram.

[0129] 4. Adaptive update module 34. Since the speech characteristics of the target speaker do not always remain unchanged, as the environment and time change, such as the target speaker's voice changes as he or she grows up from childhood, and the speaker is in a sick state. If the original speaker ID is used as the identity for recognition in these scenarios, it will result in poor performance. The method provided in the embodiment of the application separates the speech signal through the speech evaluation system for speech quality detection, and generates a new speaker ID for the signal with a high signal-to-noise ratio through a decision criterion. Then, the distance between the original speaker ID and the new speaker ID is calculated. One update strategy is that when the distance between the original speaker ID and the new speaker ID is less than a first threshold value (minimum distance), no update is performed; when the distance is greater than the first threshold value (minimum distance) and less than a second threshold value (maximum distance), the original speaker ID and the new speaker ID are weighted; when the distance is greater than the second threshold value (maximum distance), the user is prompted that the recorded voice is not the same person, and the user is prompted to re-record. Through this update strategy, the speech recognition accuracy of the target speaker in multiple scenarios is improved.

[0130] 5. Multi-modal target speaker separation and speech enhancement module. First, record the clean speech signal of the target speaker into the speaker ID embedded into the neural network; then in the video mixed with multiple speakers, process the mixed speech signal according to the lip movement change, perform short-time Fourier transform on the speech signal to obtain the frequency spectrum of the mixed signal, and input the frequency spectrum of the mixed signal into the convolutional neural network, and combine the output of the speech embedding module and the face extraction module as the input of the LSTM layer, and finally output the estimated speaker's speech signal through the fully connected layer, i.e. the frequency spectrum of the separated target speaker, and finally obtain the separated target speaker's speech signal through inverse Fourier transform.

[0131] The embodiment of the application provides a multi-modal target speaker separation and adaptive speech enhancement method, which has at least the following beneficial effects:

[0132] 1. The method can extract the voice data of the target speaker in the video, and use multi-modal joint processing to significantly improve the recognition accuracy of the target speaker from the aspects of voice data extraction and image extraction, and improve the output signal distortion ratio of the voice data.

[0133] 2. When watching or listening to audio and video, the user can hear the sound he wants to hear clearly, thereby obtaining excellent audio and video experience.

[0134] 3. The voiceprint features of the target speaker can be dynamically adjusted according to the voice data of the target speaker in different states, thereby significantly improving the recognition rate of the voice of the target speaker.

[0135] Figure 10 is a block diagram of a multi-speaker voice separation device provided by another exemplary embodiment of the present application. The device comprises:

[0136] The acquisition module 1020 is configured to acquire video frame data and audio frame data in a video.

[0137] The acquisition module 1020 is further configured to acquire face region data from the video frame data, acquire lip shape change data from the face region data, and extract semantic embedding representation based on the lip shape change data.

[0138] The acquisition module 1020 is further configured to extract voice embedding representation based on the audio frame data.

[0139] The separation module 1040 is configured to input the semantic embedding representation and the voice embedding representation into a multi-modal voice separation model to separate the voice data of the target speaker.

[0140] The acquisition module 1020 is further configured to acquire a first speaker embedding representation of the target speaker.

[0141] The separation module 1040 is further configured to input the first speaker embedding representation, the semantic embedding representation, and the voice embedding representation into the multi-modal voice separation model to separate the voice data of the target speaker.

[0142] The multi-modal voice separation model comprises a long short-term memory (LSTM) network and a fully connected layer.

[0143] The separation module 1040 is further configured to input the first speaker embedding representation, the semantic embedding representation, and the voice embedding representation into the LSTM network to obtain a fused high-dimensional output.

[0144] The separation module 1040 is further configured to input the fused high-dimensional output into the fully connected layer to obtain the frequency data of the frequency spectrum of the target speaker.

[0145] The separation module 1040 is further configured to perform inverse Fourier transform on the spectrogram frequency data of the target speaker to obtain voice data of the target speaker.

[0146] The separation module 1040 is further configured to, in a case where the mouth shape change data is obtained from the face region data, extract the speech embedding representation based on the audio frame data corresponding to the mouth shape change time period.

[0147] The mouth shape change time period is a time period corresponding to the mouth shape change data.

[0148] The acquisition module 1020 is further configured to perform Fourier transform on the audio frame data to obtain spectrogram frequency data of multiple speakers.

[0149] The acquisition module 1020 is further configured to input the spectrogram frequency data of the multiple speakers into a feature extraction network to extract the speech embedding representation.

[0150] The apparatus further includes:

[0151] The evaluation module 1060 is configured to input the voice data of the target speaker into a voice evaluation model to obtain a voice quality evaluation.

[0152] The evaluation module 1060 is further configured to, in a case where the voice quality evaluation is greater than an evaluation threshold, generate a second speaker embedding representation of the target speaker.

[0153] The evaluation module 1060 is further configured to update the first speaker embedding representation based on the second speaker embedding representation.

[0154] The evaluation module 1060 is further configured to calculate a distance between the first speaker embedding representation and the second speaker embedding representation.

[0155] The evaluation module 1060 is further configured to, in a case where the distance is greater than a first threshold and less than a second threshold, calculate a weighted sum of the first speaker embedding representation and the second speaker embedding representation to obtain a third speaker embedding representation.

[0156] The evaluation module 1060 is further configured to update the first speaker embedding representation using the third speaker embedding representation.

[0157] The evaluation module 1060 is further configured to identify a change type of the voice state of the target speaker, the change type including at least one of a long-term change and a short-term change.

[0158] The evaluation module 1060 is further configured to, in a case where the change type is the long-term change, adjust a weight of the first speaker embedding representation to be less than a weight of the second speaker embedding representation.

[0159] The evaluation module 1060 is further configured to, in a case where the change type is a short-term change, adjust a weight of the first speaker embedding representation to be greater than a weight of the second speaker embedding representation.

[0160] The evaluation module 1060 is further configured to, based on the adjusted weight of the first speaker embedding representation and the weight of the second speaker embedding representation, calculate a weighted sum of the first speaker embedding representation and the second speaker embedding representation.

[0161] The evaluation module 1060 is further configured to, in a case where the distance is less than or equal to the first threshold, not update the first speaker embedding representation, or update the first speaker embedding representation using the second speaker embedding representation.

[0162] The evaluation module 1060 is further configured to, in a case where the distance is greater than or equal to the second threshold, output prompt information that the separated speech data is not speech data of the target speaker.

[0163] Figure 11 A structural schematic diagram of a computer device provided by an example embodiment of the present application is shown. Illustratively, the computer device 1100 includes a central processing unit (CPU) 1101, a system memory 1104 including a random access memory (RAM) 1102 and a read-only memory (ROM) 1103, and a system bus 1105 that couples the system memory 1104 to the central processing unit 1101. The computer device 1100 further includes a basic input / output system (BIOS) 1106 that helps to transfer information between the various devices within the computer, and a mass storage device 1107 for storing an operating system 1113, a client 1114, and other program modules 1115.

[0164] The basic input / output system 1106 includes a display 1108 for displaying information and an input device 1109, such as a mouse, keyboard, or the like, for inputting information. The display 1108 and the input device 1109 are both connected to the central processing unit 1101 through an input / output controller 1110 that is connected to the system bus 1105. The basic input / output system 1106 can also include the input / output controller 1110 for receiving and processing input from a number of other devices, such as a keyboard, a mouse, or an electronic stylus. Similarly, the input / output controller 1110 also provides output to a display screen, a printer, or other types of output devices.

[0165] The mass storage device 1107 is connected to the central processing unit 1101 through a mass storage controller (not shown) connected to the system bus 1105. The mass storage device 1107 and its associated computer readable media provide nonvolatile storage for the computer device 1100, that is, storage of information

[0166] The computer readable media can include computer storage media and communication media. Computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media includes RAM, ROM, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other solid state memory technology, CD-ROM, digital versatile disks (DVD), or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices. Of course, the computer storage media is not limited to the above. The system memory 1104 and the mass storage device 1107 described above can be collectively referred to as memory.

[0167] According to various embodiments of the present application, the computer device 1100 can also operate in connection with a remote computer through a network such as the Internet. That is, the computer device 1100 can be connected to the network 1112 through the network interface unit 1111 connected to the system bus 1105, or can be connected to other types of networks or remote computer systems (not shown) using the network interface unit 1111.

[0168] The exemplary embodiments of the present application also provide a computer readable storage medium having stored therein at least one program, the at least one program being loaded and executed by a processor to implement the method for multi-speaker speech separation provided by each of the above-mentioned method embodiments.

[0169] An example embodiment of the present application further provides a computer program product, which comprises at least one program, and the at least one program is stored in a readable storage medium; a processor of a communication device reads signaling from the readable storage medium, and the processor executes the signaling, so that the communication device performs the method for multi-speaker speech separation provided in each method embodiment.

[0170] An example embodiment of the present application further provides a computer program, which comprises at least one program, and the at least one program is stored in a readable storage medium; a processor of a communication device reads signaling from the readable storage medium, and the processor executes the signaling, so that the communication device performs the method for multi-speaker speech separation provided in each method embodiment.

[0171] It should be understood that "multiple" mentioned herein refers to two or more than two. "And / or" describes the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B can represent the following three cases: A exists alone, A and B exist simultaneously, and B exists alone. The character " / " generally represents that the associated objects before and after it are in an "or" relationship.

[0172] A person of ordinary skill in the art can understand that all or part of the steps of the above-mentioned embodiments can be completed by hardware, or by a program instructing relevant hardware, and the program can be stored in a computer readable storage medium. The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk.

[0173] The above is only an optional embodiment of the present application, and is not used to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for multi-speaker speech separation, characterized by, The method comprises: obtaining video frame data and audio frame data in a video; obtaining face region data from the video frame data; obtaining mouth shape change data from the face region data, and extracting semantic embedding representation based on the mouth shape change data; extracting speech embedding representation based on the audio frame data; obtaining a first speaker embedding representation of a target speaker, the first speaker embedding representation being a speaker embedding representation obtained based on noise-free speech data of the target speaker; inputting the first speaker embedding representation, the semantic embedding representation and the speech embedding representation into a multi-modal speech separation model to separate the speech data of the target speaker; inputting the speech data of the target speaker into a speech evaluation model to obtain a speech quality evaluation; in a case where the speech quality evaluation is greater than an evaluation threshold, generating a second speaker embedding representation of the target speaker, the second speaker embedding representation being a speaker embedding representation obtained based on speech data of the target speaker separated from a video containing multiple speakers; calculating a distance between the first speaker embedding representation and the second speaker embedding representation; in a case where the distance is greater than a first threshold and less than a second threshold, calculating a weighted sum of the first speaker embedding representation and the second speaker embedding representation to obtain a third speaker embedding representation, a weight used for the weighted sum being dynamically adjusted based on a change type of a voice state of the target speaker, the change type of the voice state including at least one of a long-term change and a short-term change; updating the first speaker embedding representation using the third speaker embedding representation.

2. The method of claim 1, wherein, The multi-modal speech separation model comprises a long short-term memory (LSTM) network and a fully connected layer. The inputting of the first speaker embedding representation, the semantic embedding representation and the speech embedding representation into the multi-modal speech separation model to separate the speech data of the target speaker comprises: inputting the first speaker embedding representation, the semantic embedding representation and the speech embedding representation into the LSTM network to obtain a fused high-dimensional output; inputting the fused high-dimensional output into the fully connected layer to obtain spectral graph frequency data of the target speaker; performing inverse Fourier transform on the spectral graph frequency data of the target speaker to obtain the speech data of the target speaker.

3. The method according to claim 1 or 2, characterized in that, The extracting of the speech embedding representation based on the audio frame data comprises: in a case where the mouth shape change data is obtained in the face region data, extracting the speech embedding representation based on audio frame data corresponding to a mouth shape change time period; wherein the mouth shape change time period is a time period corresponding to the mouth shape change data.

4. The method according to claim 1 or 2, characterized in that, The extracting of the speech embedding representation based on the audio frame data comprises: performing Fourier transform on the audio frame data to obtain spectral graph frequency data of multiple speakers; inputting the spectral graph frequency data of the multiple speakers into a feature extraction network to extract the speech embedding representation.

5. The method of claim 1, wherein, The calculating of the weighted sum of the first speaker embedding representation and the second speaker embedding representation comprises: identify a type of change in a voice state of the target speaker, the type of change comprising at least one of a long-term change and a short-term change; in a case where the type of change is the long-term change, adjust a weight of the first speaker embedding representation to be less than a weight of the second speaker embedding representation; in a case where the type of change is the short-term change, adjust the weight of the first speaker embedding representation to be greater than the weight of the second speaker embedding representation; calculate a weighted sum of the first speaker embedding representation and the second speaker embedding representation based on the adjusted weights of the first speaker embedding representation and the second speaker embedding representation.

6. The method of claim 1, wherein, The method further comprises: in a case where the distance is less than or equal to the first threshold, not updating the first speaker embedding representation, or updating the first speaker embedding representation using the second speaker embedding representation.

7. The method of claim 1, wherein, The method further comprises: in a case where the distance is greater than or equal to the second threshold, outputting prompt information that the separated voice data is not voice data of the target speaker.

8. An apparatus for multi-speaker speech separation, comprising: The apparatus comprises: an obtaining module configured to obtain video frame data and audio frame data in a video; the obtaining module is further configured to obtain face region data from the video frame data, obtain mouth shape change data from the face region data, and extract a semantic embedding representation based on the mouth shape change data; the obtaining module is further configured to extract a voice embedding representation based on the audio frame data; the obtaining module is further configured to obtain a first speaker embedding representation of a target speaker, the first speaker embedding representation being a speaker embedding representation obtained based on noise-free voice data of the target speaker; a separating module configured to input the first speaker embedding representation, the semantic embedding representation, and the voice embedding representation into a multi-modal voice separation model to obtain voice data of the target speaker; an evaluating module configured to input the voice data of the target speaker into a voice evaluation model to obtain a voice quality evaluation; the evaluating module is further configured to, in a case where the voice quality evaluation is greater than an evaluation threshold, generate a second speaker embedding representation of the target speaker, the second speaker embedding representation being a speaker embedding representation obtained based on voice data of the target speaker separated from a video containing multiple speakers; the evaluating module is further configured to calculate a distance between the first speaker embedding representation and the second speaker embedding representation; the evaluating module is further configured to, in a case where the distance is greater than a first threshold and less than a second threshold, calculate a weighted sum of the first speaker embedding representation and the second speaker embedding representation to obtain a third speaker embedding representation, a weight for the weighted sum being dynamically adjusted based on a type of change in a voice state of the target speaker, the type of change comprising at least one of a long-term change and a short-term change; the evaluating module is further configured to update the first speaker embedding representation using the third speaker embedding representation.

9. A computer device, comprising: The computer device comprises a processor and a memory, and the memory stores at least one program, which is loaded and executed by the processor to implement the method for speech separation of multiple speakers according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The storage medium stores at least one program, which is loaded and executed by the processor to enable the wireless device to implement the method for speech separation of multiple speakers according to any one of claims 1 to 7.

11. A computer program product, characterised in that, The computer program product comprises at least one program stored in a computer readable storage medium, and the processor of the communication device reads the at least one program from the computer readable storage medium, and executes the at least one program to enable the communication device to implement the method for speech separation of multiple speakers according to any one of claims 1 to 7.

Images