Voice separation method and device, storage medium and equipment

Abstract

The invention discloses a voice separation method and device, a storage medium and equipment. The method comprises the following steps: firstly, obtaining a to-be-separated mixed voice and a to-be-separated target video; acoustic features of the mixed voice are extracted; inputting the acoustic features of the mixed voice and an image of the lip shape of the target user contained in the target video into a multi-mode alignment model to obtain lip shape features of the target user; inputting the acoustic features of the mixed voice and the lip features of the target user into a multi-mode VAD model to obtain the separation probability of the mixed voice; and according to the separation probability of the mixed voice, in combination with a beam forming algorithm, multi-mode beam features corresponding to the mixed voice are calculated, then the acoustic features of the mixed voice, the lip features of the target user and the multi-mode beam features can be input into a multi-mode voice separation model, and according to a masking signal output by the model, the multi-mode voice separation is carried out. The target voice is separated from the mixed voice more accurately, so that the separation effect of the target voice is improved.

Description

Technical Field

[0001] The present application relates to the field of speech processing technology, and in particular to a speech separation method, apparatus, storage medium and device. Background Art

[0002] With the rapid development of speech recognition technology, there is an urgent need for speech separation technology in multi-channel speech recognition scenarios. Speech separation is an important step in speech recognition processing, which is used to separate the target speech from the interfering background speech, thereby solving the problem of speech recognition difficulties caused by complex speech signals in noisy environments or environments with multiple sound sources.

[0003] At present, in order to achieve better speech separation performance, visual information has been incorporated into speech separation applications. Specifically, the existing method for achieving speech separation is usually to first extract the speaker's lip movement features from the video signal, and generate visual information to assist audio separation through a lip reading pre-training module. At the same time, the audio signal and the speaker's orientation information captured by the multi-channel microphone array and video information are used to calculate the orientation features related to the speaker's position. These information from different modes are integrated in the audio branch module to optimize the representation of the audio signal. Subsequently, after processing by beamforming technology, multi-channel audio and orientation features are used to enhance the target speech while suppressing interference signals from other directions. After further optimizing the sound quality through the post-processing module, the final audio signal is input into the single-mode recognition module for speech recognition, etc. However, the extraction of lip shape features in this method requires lip reading pre-training, which will consume a lot of manpower costs to annotate the massive text data required for training large models, which is costly, and this method requires knowing the specific location information of the target speaker, and has poor robustness for moving speakers, thereby reducing the speech separation effect. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to provide a speech separation method, apparatus, storage medium and device, which can solve the problem in the prior art that the speaker's position is unknown and movement leads to decreased separation performance, and effectively improve the speech separation effect.

[0005] The present application provides a speech separation method, including:

[0006] Acquire a mixed speech to be separated and a target video; and extract the acoustic features of the mixed speech; the mixed speech contains the target speech emitted by the target user; the target video is a video containing the lip shape of the target user;

[0007] Inputting the acoustic features of the mixed speech and the image of the target user's lip shape contained in the target video into a pre-built multi-modal alignment model to obtain the lip shape features of the target user;

[0008] Inputting the acoustic features of the mixed speech and the lip shape features of the target user into a pre-built multi-modal voice activity detection VAD model to obtain a separation probability of the mixed speech;

[0009] Calculate the multi-mode beam characteristics corresponding to the mixed speech according to the separation probability of the mixed speech and in combination with the beamforming algorithm;

[0010] The acoustic features of the mixed speech, the lip shape features of the target user, and the multi-mode beam features are input into a pre-constructed multi-mode speech separation model, and the target speech is separated from the mixed speech according to the masking signal output by the model.

[0011] In a possible implementation, the multimodal alignment model includes an audio branch, a video branch, and a cross-modal transformer Transformer; the step of inputting the acoustic features of the mixed speech and the image of the target user's lip shape contained in the target video into the pre-built multimodal alignment model to obtain the lip shape features of the target user includes:

[0012] Inputting the acoustic features of the mixed speech into the audio branch of the multi-mode alignment model to obtain an audio coding feature vector;

[0013] Inputting the image of the target user's lip shape contained in the target video into the video branch of the multi-mode alignment model to obtain an image coding feature vector;

[0014] The audio coding feature vector and the image coding feature vector are input into the cross-modal Transformer of the multi-modal alignment model to obtain the lip shape features of the target user.

[0015] In a possible implementation, the multi-mode alignment model is constructed as follows:

[0016] Acquire a synchronized pair of sample speech and sample video, and acquire an unsynchronized pair of sample speech and sample video; and extract acoustic features of the sample speech; the target video is a video containing the sample user's lip shape;

[0017] Using the acoustic features of the sample speech, the image of the sample user's lip shape contained in the sample video, and the contrast loss function, an initial multimodal alignment model is trained to obtain the multimodal alignment model;

[0018] The initial multimodal alignment model includes an initial audio branch, an initial video branch, an initial cross-modal Transformer and a binary classification layer.

[0019] In a possible implementation, the initial multimodal alignment model is trained using the acoustic features of the sample speech, the image of the sample user's lip shape contained in the sample video, and the contrast loss function to obtain the multimodal alignment model, including:

[0020] Inputting the acoustic features of the sample speech into the initial audio branch of the initial multi-mode alignment model to obtain a sample audio coding feature vector;

[0021] Inputting the image of the sample user's lip shape contained in the sample video into the initial video branch of the initial multimodal alignment model to obtain a sample image encoding feature vector;

[0022] Inputting the sample audio coding feature vector and the sample image coding feature vector into the initial cross-modal Transformer of the initial multimodal alignment model to obtain intermediate sample features output by the cross-modal Transformer;

[0023] The intermediate sample features are input into the binary classification layer of the initial multimodal alignment model to obtain a synchronization score, and the initial multimodal alignment model is trained using the synchronization score and the contrast loss function until a preset condition is met, then the updating of the model parameters is stopped to obtain a trained multimodal alignment model.

[0024] In a possible implementation, the multi-mode VAD model is constructed as follows:

[0025] The initial multimodal VAD model is trained using the acoustic features of the sample speech, the intermediate features output by the cross-modal Transformer, and the cross-entropy loss function to obtain the multimodal VAD model; wherein the initial multimodal VAD model includes four convolutional layers and one fully connected layer.

[0026] In a possible implementation, the multi-mode speech separation model is constructed as follows:

[0027] Inputting the acoustic features of the sample speech and the intermediate features output by the cross-modal Transformer into a multimodal VAD model to obtain a speech existence probability of the sample speech;

[0028] Calculating the multi-mode beam characteristics corresponding to the sample speech according to the speech existence probability of the sample speech in combination with a beamforming algorithm;

[0029] The initial multimodal speech separation model is trained using the acoustic features of the sample speech, the intermediate features output by the cross-modal Transformer, the multimodal beam features corresponding to the sample speech, and the minimum mean square error loss function to obtain the multimodal speech separation model; wherein the initial multimodal speech separation model includes a five-layer convolutional neural network CNN and a two-layer long short-term memory network LSTM.

[0030] In a possible implementation, the acoustic feature of the mixed speech is a logarithmic power spectrum LPS.

[0031] The present application also provides a speech separation device, the device comprising:

[0032] The first acquisition unit is used to acquire the mixed speech to be separated and the target video; and extract the acoustic features of the mixed speech; the mixed speech contains the target speech emitted by the target user; the target video is a video containing the lip shape of the target user;

[0033] A first input unit is used to input the acoustic features of the mixed speech and the image of the lip shape of the target user contained in the target video into a pre-built multi-modal alignment model to obtain the lip shape features of the target user;

[0034] A second input unit is used to input the acoustic features of the mixed speech and the lip shape features of the target user into a pre-built multi-modal voice activity detection VAD model to obtain a separation probability of the mixed speech;

[0035] A first calculation unit, configured to calculate a multimode beam feature corresponding to the mixed speech according to the separation probability of the mixed speech and in combination with a beamforming algorithm;

[0036] The separation unit is used to input the acoustic features of the mixed speech, the lip shape features of the target user, and the multi-mode beam features into a pre-constructed multi-mode speech separation model, and separate the target speech from the mixed speech according to the masking signal output by the model.

[0037] In a possible implementation, the multi-modal alignment model includes an audio branch, a video branch, and a cross-modal converter Transformer; the first input unit includes:

[0038] A first input subunit, configured to input the acoustic features of the mixed speech into the audio branch of the multi-mode alignment model to obtain an audio coding feature vector;

[0039] A second input subunit is used to input the image of the target user's lip shape contained in the target video into the video branch of the multi-mode alignment model to obtain an image coding feature vector;

[0040] The third input subunit is used to input the audio coding feature vector and the image coding feature vector into the cross-modal Transformer of the multi-modal alignment model to obtain the lip shape features of the target user.

[0041] In a possible implementation manner, the device further includes:

[0042] The second acquisition unit is used to acquire a synchronized sample speech and sample video pair, and an unsynchronized sample speech and sample video pair; and extract the acoustic features of the sample speech; the target video is a video containing the sample user's lip shape;

[0043] A first training unit is used to train an initial multimodal alignment model using the acoustic features of the sample speech, the image of the sample user's lip shape contained in the sample video, and a contrast loss function to obtain the multimodal alignment model;

[0044] The initial multimodal alignment model includes an initial audio branch, an initial video branch, an initial cross-modal Transformer and a binary classification layer.

[0045] In a possible implementation, the first training unit includes:

[0046] A fourth input subunit, used for inputting the acoustic features of the sample speech into the initial audio branch of the initial multi-mode alignment model to obtain a sample audio coding feature vector;

[0047] A fifth input subunit, configured to input the image of the sample user's lip shape contained in the sample video into the initial video branch of the initial multi-modal alignment model to obtain a sample image encoding feature vector;

[0048] a sixth input subunit, configured to input the sample audio coding feature vector and the sample image coding feature vector into an initial cross-modal Transformer of the initial multimodal alignment model to obtain intermediate sample features output by the cross-modal Transformer;

[0049] The training subunit is used to input the intermediate sample features into the binary classification layer of the initial multimodal alignment model to obtain a synchronization score, and use the synchronization score and the contrast loss function to train the initial multimodal alignment model until a preset condition is met, then stop updating the model parameters to obtain a trained multimodal alignment model.

[0050] In a possible implementation manner, the device further includes:

[0051] The second training unit is used to train the initial multimodal VAD model using the acoustic features of the sample speech, the intermediate features output by the cross-modal Transformer and the cross-entropy loss function to obtain the multimodal VAD model; wherein the initial multimodal VAD model includes four convolutional layers and one fully connected layer.

[0052] In a possible implementation manner, the device further includes:

[0053] A third input unit is used to input the acoustic features of the sample speech and the intermediate features output by the cross-modal Transformer into a multimodal VAD model to obtain a speech existence probability of the sample speech;

[0054] A second calculation unit, configured to calculate a multimode beam feature corresponding to the sample speech according to the speech existence probability of the sample speech and in combination with a beamforming algorithm;

[0055] The third training unit is used to train the initial multimodal speech separation model by using the acoustic features of the sample speech, the intermediate features output by the cross-modal Transformer, the multimodal beam features corresponding to the sample speech, and the minimum mean square error loss function to obtain the multimodal speech separation model; wherein the initial multimodal speech separation model includes a five-layer convolutional neural network CNN and a two-layer long short-term memory network LSTM.

[0056] In a possible implementation, the acoustic feature of the mixed speech is a logarithmic power spectrum LPS.

[0057] The embodiment of the present application also provides a speech separation device, including: a processor, a memory, and a system bus;

[0058] The processor and the memory are connected via the system bus;

[0059] The memory is used to store one or more programs, and the one or more programs include instructions, and when the instructions are executed by the processor, the processor executes any one of the implementations of the above-mentioned speech separation method.

[0060] An embodiment of the present application also provides a computer-readable storage medium, in which instructions are stored. When the instructions are executed on a terminal device, the terminal device executes any one of the implementations of the above-mentioned speech separation method.

[0061] The embodiment of the present application also provides a computer program product. When the computer program product is run on a terminal device, the terminal device executes any one of the implementations of the above-mentioned speech separation method.

[0062] The embodiments of the present application provide a speech separation method, apparatus, storage medium and device, which first obtain a mixed speech and a target video to be separated; and extract the acoustic features of the mixed speech; wherein the mixed speech contains a target speech emitted by a target user; the target video is a video containing the lip shape of the target user, and then the acoustic features of the mixed speech and the image of the lip shape of the target user contained in the target video are input into a pre-constructed multi-mode alignment model to obtain the lip shape features of the target user; then, the acoustic features of the mixed speech and the lip shape features of the target user are input into a pre-constructed multi-mode VAD model to obtain the separation probability of the mixed speech; then, based on the separation probability of the mixed speech and in combination with a beamforming algorithm, the multi-mode beam features corresponding to the mixed speech are calculated, and then the acoustic features of the mixed speech, the lip shape features of the target user, and the multi-mode beam features can be input into a pre-constructed multi-mode speech separation model, and the target speech can be separated from the mixed speech based on the masking signal output by the model.

[0063] It can be seen that since the present application utilizes a pre-built multi-mode alignment model, a multi-mode VAD model and a multi-mode speech separation model, it realizes an audio and video alignment based on the acoustic features of the mixed speech and the lip shape features of the target user (ie, the speaker) and a multi-mode separation method based on multi-mode directional features. This not only solves the problem in the prior art that the separation performance is reduced due to the unknown position and movement of the speaker, but also can more accurately separate the target speech from the mixed speech, thereby improving the separation effect of the target speech. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0065] Figure 1 A schematic diagram of an existing speech separation system framework provided in an embodiment of the present application;

[0066] Figure 2 A flowchart of a speech separation method provided in an embodiment of the present application;

[0067] Figure 3 A schematic diagram of the structure of a multi-mode alignment model provided in an embodiment of the present application;

[0068] Figure 4 A schematic diagram of a process for calculating multimode beam characteristics corresponding to mixed speech provided in an embodiment of the present application;

[0069] Figure 5 A schematic diagram of the composition of a speech separation device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0070] With the rapid development of computer technology and intelligent speech processing technology, speech separation technology has been widely used in various fields. Among them, speech separation refers to a technology that separates the target speech from the interfering background speech. In addition, long-distance automatic speech recognition (ASR) based on microphone arrays has become an important part of daily life. This is because the emergence of hands-free, voice-enabled applications on multi-microphone portable devices has made it convenient and flexible. However, the speech signals recorded by long-distance microphones are often contaminated by reverberation and background noise, resulting in a significant decrease in ASR performance at low signal-to-noise ratio (SNR).

[0071] Therefore, in order to improve speech processing performance, in addition to time domain, frequency domain information and spatial information (such as the speaker's position, etc.), visual information is also used in multi-channel speech enhancement and separation. This is because the advantage of video information over speech is its immunity to background noise and interfering speakers. Specifically, the existing process framework for speech separation by combining visual information is as follows: Figure 1 As shown in the figure, it shows the process of a multimodal speech separation and recognition system. The system first extracts the speaker's lip movement features from the video signal, and generates visual information (i.e., Figure 1 At the same time, the system uses a multi-channel microphone array (device that collects signals) and video information to capture audio signals (i.e. Figure 1 The system uses the multi-channel audio in the audio branch module) and the speaker's position information to calculate the positional features related to the speaker's position (for example, after estimating the angle of the speaker relative to the camera through the camera's parameters and placement, the positional features can be calculated based on the estimated angle.). These information from different modes are integrated in the audio branch module to optimize the representation of the audio signal. Subsequently, after processing with beamforming technology, the system uses multi-channel audio and directional features to enhance the target speech while suppressing interference signals from other directions. After further optimizing the sound quality through the post-processing module, the final audio signal is input into the single-mode recognition module for speech separation or speaker recognition.

[0072] Although the above method can combine the multimodal information of audio and video, and improve the accuracy of separation and recognition of target speech in complex environments to a certain extent, it also has two defects: on the one hand, the extraction of lip shape features in this method requires pre-training of lip reading, and the training method adopted is to learn the relationship between lip movement features and their corresponding texts. Therefore, it is necessary to manually annotate the massive text data information required for training large models, which requires a lot of human resources and is costly. On the other hand, this method requires knowing the specific location information of the target speaker. When the speaker moves, it is necessary to re-estimate the angle of the speaker relative to the camera based on the parameters and placement of the camera, and then recalculate the orientation features based on the estimated angle, etc., resulting in poor robustness to moving speakers, which greatly increases the difficulty of implementing the algorithm in actual environments and reduces the speech separation effect.

[0073] To solve the above-mentioned defects, the present application provides a speech separation method, which first obtains a mixed speech and a target video to be separated; and extracts the acoustic features of the mixed speech; wherein the mixed speech contains the target speech emitted by the target user; the target video is a video containing the lip shape of the target user, and then the acoustic features of the mixed speech and the image of the lip shape of the target user contained in the target video are input into a pre-constructed multimodal alignment model to obtain the lip shape features of the target user; then, the acoustic features of the mixed speech and the lip shape features of the target user are input into a pre-constructed multimodal voice activity detection (VAD) model to obtain the separation probability of the mixed speech; then, based on the separation probability of the mixed speech and in combination with a beamforming algorithm, the multimodal beam features corresponding to the mixed speech are calculated, and then the acoustic features of the mixed speech, the lip shape features of the target user, and the multimodal beam features can be input into a pre-constructed multimodal speech separation model, and the target speech can be separated from the mixed speech based on the masking signal output by the model.

[0074] It can be seen that since the present application utilizes a pre-built multi-mode alignment model, a multi-mode VAD model and a multi-mode speech separation model, it realizes an audio and video alignment based on the acoustic features of the mixed speech and the lip shape features of the target user (ie, the speaker) and a multi-mode separation method based on multi-mode directional features. This not only solves the problem in the prior art that the separation performance is reduced due to the unknown position and movement of the speaker, but also can more accurately separate the target speech from the mixed speech, thereby improving the separation effect of the target speech.

[0075] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0076] First embodiment

[0077] See also Figure 2 , is a flow chart of a speech separation method provided in this embodiment, the method comprising the following steps:

[0078] S201: Acquire a mixed speech and a target video to be separated; and extract acoustic features of the mixed speech; wherein the mixed speech includes a target speech emitted by a target user; and the target video is a video including the lip shape of the target user.

[0079] In this embodiment, any speech that is processed by speech separation using this embodiment is defined as a mixed speech to be separated, and the video recorded while recording the mixed speech is defined as a target video, and the user who issues the target speech is defined as a target user. The purpose of this application is to combine the target video to quickly and accurately separate the target speech from the mixed speech. It can be understood that the mixed speech not only contains the target speech, but also contains noise or interference speech such as speech from other users. It should be noted that this embodiment does not limit the language type of the target speech in the mixed speech. For example, the target speech can be a speech composed of Chinese or English, etc. At the same time, this embodiment does not limit the length of the mixed speech and the target speech. For example, the mixed speech can be a speech of 1 hour, and the target speech can be a sentence or a paragraph, etc.

[0080] It should also be noted that the target video refers to a video containing the lip shape of the target user and is composed of frame-by-frame images. In addition, this embodiment does not limit the type of the target video. For example, the target video can be a color video stream (i.e., RGB video stream) composed of the three primary colors of red (R), green (G), and blue (B), or it can be grayscale video stream data composed of grayscale images.

[0081] It is understandable that the mixed voice can be obtained by recording or other methods according to actual needs, and accordingly, the target video can be obtained by recording or other methods according to actual needs. For example, the recorded voice of a teacher giving a lesson to a student in a classroom, or the recorded chat between A and B on the road, etc. can be used as the mixed voice. At the same time, the video of a teacher giving a lesson to a student in a classroom shot by a mobile phone, or the video of A and B chatting on the road shot by a camera, etc. can be the target video corresponding to the mixed voice. Among them, the voice of the teacher can also be used as the target voice, and the fan sound in the classroom can be used as the interference sound; or the voice of A on the road can be used as the target voice, and the horn sound of the vehicle on the road can be used as the interference voice.

[0082] It should be noted that, since the target speech is extremely susceptible to background noise and interfering human voices in the real environment of mixed speech, the estimation of the speech masking signal is not accurate enough, which will reduce the speech quality and intelligibility after multi-channel beam processing, affecting the accuracy of the speech separation result. Audio and video signals provide different perceptual modalities. Among them, the audio signal captures the frequency and intensity information of the sound, while the video signal can not only capture the spatial and temporal information in the visual scene, but also capture the visual information (such as lip movement and facial expression) of the speaker (such as the target user) to provide supplementary information about the speech signal. And the video signal has natural immunity to background noise and other interfering speakers and is not interfered with. These visual information can be used to improve the speech separation method so as to more accurately identify and separate the target speech from the mixed speech. Therefore, the speech separation method adopted in the present application is to use the multimodal information of mixed speech and target video to provide richer and more accurate features, so as to improve the performance and robustness of speech separation, and then provide speech separation effect.

[0083] Specifically, after obtaining the mixed speech to be separated, in order to accurately separate the target speech from the mixed speech, the acoustic features of the mixed speech can be extracted using existing or future feature extraction methods as part of the separation basis to execute the subsequent step S202. For example, the mixed speech can first be framed to obtain a corresponding speech frame sequence, and then the framed speech frame sequence is pre-emphasized; and then the acoustic features of each speech frame are extracted in turn, wherein the acoustic features refer to feature data used to characterize the acoustic information of the corresponding speech frame, such as, but not limited to, Log-Power Spectrum (LPS) and Inter-Channel Phase Difference (IPD) features, etc., which constitute the acoustic features of the mixed speech.

[0084] It should be noted that the embodiment of the present application does not limit the method for extracting the acoustic features of the mixed speech, nor does it limit the specific extraction process. An appropriate extraction method can be selected according to the actual situation, and corresponding feature extraction operations can be performed. For ease of understanding, this embodiment will be described later using the acoustic features of the mixed speech as the log power spectrum (LPS) feature as an example.

[0085] S202: Input the acoustic features of the mixed speech and the image of the target user's lip shape contained in the target video into a pre-built multi-modal alignment model to obtain the lip shape features of the target user.

[0086] In this embodiment, after the mixed speech and target video to be separated are obtained through step S201 and the acoustic features of the mixed speech are extracted, in order to accurately separate the target speech from the mixed speech, the acoustic features of the mixed speech and the image of the target user's lip shape contained in the target video can be further input into a pre-built multimodal alignment model to identify the lip shape features of the target user for executing the subsequent step S203.

[0087] It should be noted that in order to improve the accuracy of lip shape feature extraction of the target user, the present application pre-constructs a multi-modal alignment model, such as Figure 3 As shown, the model may include but is not limited to an audio branch, a video branch and a cross-modal transformer. The audio branch may be composed of but not limited to a temporal convolutional neural network (TCN) for extracting interdependent information between long-term audio frames; the video branch may be composed of but not limited to a 3D convolutional layer and a residual network (ResNet) for encoding the image of the user's lip shape into a 2D feature to characterize the user's lip movement information; the cross-modal transformer uses the features of the audio and video branch output learned in its cross-modal attention module to infer the audio-video correspondence in the synchronized speech and lip movement pair.

[0088] On this basis, an optional implementation method is that the specific implementation process of this step S202 may include: first, the acoustic features of the mixed speech may be input into the audio branch of the multi-mode alignment model to obtain an audio coding feature vector. Then, the image of the target user's lip shape contained in the target video is input into the video branch of the multi-mode alignment model to obtain an image coding feature vector. Next, the audio coding feature vector and the image coding feature vector are input into the cross-modal Transformer of the multi-modal alignment model to obtain the intermediate features output by the cross-modal Transformer as the lip shape features of the target user, so as to execute the subsequent step S203.

[0089] It should be noted that the reason why the present application pre-constructs a multi-mode alignment model is that when acoustic and visual signals are recorded using different devices, audio and video synchronization problems may occur, that is, the audio and video may not be aligned in time. At present, people can detect the lack of synchronization when the audio leads the video by more than 45 milliseconds or the video exceeds the audio by more than 125 milliseconds. Audio and video synchronization also affects audio and video separation. Since in most existing audio and video data processing, the data sets used to train and evaluate audio and video separation methods are properly synchronized, the time alignment problem of audio and video signals is usually not solved. But in fact, when the audio leads the video or vice versa, it is difficult to achieve the ideal audio and video separation effect under low signal-to-noise ratio conditions. Therefore, the present application pre-constructs a multi-mode alignment model that can learn the correlation between audio features and video features, which is used to achieve more accurate speech separation, and the entire model construction process does not require manual annotation of text data, but instead generates counter-example data by artificially offsetting the alignment data of the positive example, and then trains the model through the positive and negative example data.

[0090] Next, this embodiment will introduce the construction process of the multi-mode alignment model. In one optional implementation, the construction process of the multi-mode alignment model may specifically include the following steps AB:

[0091] Step A: obtaining a synchronized pair of sample speech and sample video, and obtaining an unsynchronized pair of sample speech and sample video; and extracting acoustic features of the sample speech; wherein the target video is a video containing the sample user's lip shape.

[0092] In this embodiment, in order to build a multi-mode alignment model, a lot of preparation work needs to be done in advance. First, it is necessary to collect a large amount of voice data emitted by different sample users and video data containing user lip shapes. For example, a microphone array can be used to pick up sound. The sound pickup device can be a tablet computer or an intelligent hardware device, such as a smart speaker, a TV, and an air conditioner. A large amount of voice data emitted by 100 users is collected, and videos of these users emitting these voices (including the user's lip shape) are recorded at the same time as sample voices and sample videos, respectively. Then, these sample voices and sample videos are used to construct synchronized sample voice and sample video pairs and asynchronous sample voice and sample video pairs manually or by machine.

[0093] Furthermore, a method similar to that used in step S201 to extract acoustic features of mixed speech is used to replace the mixed speech with sample speech, so as to extract acoustic features (such as LPS features) of each sample speech. For relevant details, please refer to the introduction of step S201, which will not be repeated here to execute the subsequent step B.

[0094] Step B: using the acoustic features of the sample speech, the image of the sample user's lip shape contained in the sample video and the contrast loss function, the initial multimodal alignment model is trained to obtain the multimodal alignment model.

[0095] It should be noted that the initial multimodal alignment model used in this application may include but is not limited to an initial audio branch (which may be composed of a temporal convolutional neural network (TCN)), an initial video branch (which may include a 3D convolutional layer and a ResNet), an initial cross-modal Transformer, and a binary classification layer (which may include a fully connected layer for outputting a score to indicate whether the sound and lip movement are correctly synchronized (i.e., whether the audio and video are aligned), such as Figure 3 shown.

[0096] Specifically, in this embodiment, after obtaining a synchronized sample speech and sample video pair, as well as an asynchronous sample speech and sample video pair through step A, and extracting the acoustic features of each sample speech (such as LPS features), further, first, any pair of synchronized sample speech and sample video pairs, or any pair of asynchronous sample speech and sample video pairs can be extracted, and then the acoustic features of the sample speech (such as LPS features) can be input into the initial audio branch of the initial multi-mode alignment model to obtain a sample audio coding feature vector to represent the information of mutual dependence between the sample speech audio frames.

[0097] Then, the image of the sample user's lip shape contained in the sample video can be input into the initial video branch of the initial multi-modal alignment model to obtain a sample image encoding feature vector for characterizing the sample user's lip movement information.

[0098] Next, the sample audio coding feature vector and the sample image coding feature vector can be input into the initial cross-modal Transformer of the initial multimodal alignment model to obtain the intermediate sample features output by the cross-modal Transformer, and then the intermediate sample features can be input into the binary classification layer of the initial multimodal alignment model to obtain a synchronization score (used to indicate whether the sound and lip movement are correctly synchronized, such as a score of 0 can be used to indicate alignment or synchronization, and a score of 1 can be used to indicate no alignment or synchronization, etc.), and the initial multimodal alignment model can be trained using the synchronization score and the contrast loss function until the preset conditions are met (the specific content is not limited and can be set according to actual conditions and experience values, such as the value of the contrast loss function is very small and basically unchanged, or the preset maximum number of iterations (such as 100 times) is reached, etc.), then the model parameter update is stopped to obtain the trained multimodal alignment model, and the model is enabled to learn the matching relationship between the target speaker's voice and the target lip shape in the mixed speech with the interfering speaker.

[0099] S203: Input the acoustic features of the mixed speech and the lip shape features of the target user into a pre-built multi-modal voice activity detection VAD model to obtain a separation probability of the mixed speech.

[0100] In this embodiment, after the lip shape features of the target user (i.e., the intermediate features output by the cross-modal Transformer of the multi-modal alignment model) are obtained in step S202, in order to accurately separate the target speech from the mixed speech, the acoustic features of the mixed speech and the lip shape features of the target user (i.e., the intermediate features output by the cross-modal Transformer of the multi-modal alignment model) can be further input into a pre-constructed multi-modal VAD model, thereby predicting the separation probability of the mixed speech (which can also be understood as Figure 4 The “VAD-based masking signal” shown in FIG. 1 is used to perform the subsequent step S204.

[0101] S204: Calculate the multi-mode beam characteristics corresponding to the mixed speech according to the separation probability of the mixed speech in combination with the beamforming algorithm.

[0102] In this embodiment, after the separation probability of the mixed speech is obtained through step S203, in order to accurately separate the target speech from the mixed speech, the multi-mode beam characteristics (which can be expressed as MMDF) corresponding to the mixed speech can be further calculated based on the separation probability of the mixed speech in combination with the beamforming algorithm to execute the subsequent step S205.

[0103] Among them, it should be noted that in order to accurately separate the target speech from the mixed speech, the present application pre-builds a multi-mode VAD model, because the quality and intelligibility of speech are usually destroyed by noise and interference. The voice activity detector (VAD) is used to mark the frame as containing only noise or speech, which can be regarded as a rough mask at the frame level. When the voice activity detector (VAD) estimates the current frame as noise, the matrix of the noise vector in the subsequent beamforming algorithm (or beamformer) can be updated. However, if the voice activity detector (VAD) estimates that the current frame is noise, the estimate of the matrix remains unchanged; it is the same as that of the previous frame.

[0104] Specifically, VAD becomes more challenging when transients are present, because transients are usually more dominant than speech, with short duration, high amplitude, and fast spectral changes. Specifically, the audio features extracted from frames containing both speech and transients are usually similar to the features extracted from frames containing only transients, so they are often misclassified as non-speech frames. To solve this problem, this application constructs a multimodal VAD model based on deep learning for speech presence / absence, and the specific implementation block diagram is shown in the left figure in Figure 4.

[0105] An optional implementation method is that the multimodal VAD model specifically utilizes the acoustic features of the sample speech, the intermediate features of the cross-modal Transformer output, and the cross-entropy loss function to train the initial multimodal VAD model to obtain the multimodal VAD model; wherein the initial multimodal VAD model includes four convolutional layers and one fully connected layer.

[0106] In this implementation, when training the model, the acoustic features of a sample speech (such as LPS features) and the intermediate features of the cross-modal Transformer output of the multi-modal alignment model can be extracted from the training data in turn as model inputs, so as to output the probability of speech existence through the model (such as the presence / absence of speech can be used as a 1 / 0 label, etc.), and combined with the cross-entropy loss function (Cross-Entropy Loss, CE loss), the initial multi-modal VAD model is trained until the preset conditions are met (the specific content is not limited and can be set according to actual conditions and experience values, such as the value of the cross-entropy loss function is very small and basically unchanged, or the preset maximum number of iterations (such as 100 times) is reached, etc.), then the update of the model parameters is stopped to obtain the trained multi-modal VAD model. In addition, the separation probability output by the model can finally be converted into a hard decision of 0 or 1, as shown in the following formula:

[0107]

[0108] Among them, f θ (·) represents the multi-mode VAD model parameters, that is, the nonlinear mapping function from input features to actual labels; θ represents the MM-VAD model parameters; y(l) and v(l) represent the input features of the log power spectrum (LPS) feature and the lip feature, respectively; ξ represents a decision threshold, the specific value of which is not limited and can be set according to the situation and experience, for example, it can be set to 0.6.

[0109] Furthermore, it should be noted that voice activity detection (VAD) is also critical to the performance of the beamforming algorithm. If the noise matrix is ​​updated during speech instead of noise instances, speech will be suppressed, and the noise will also be suppressed. Therefore, the voice activity detection (VAD) model needs to be turned on quickly when speech occurs and turned off during noise / interference periods.

[0110] M estimated by the above formula (1) MM-VAD The value of (l) can be obtained by combining the beamforming algorithm as follows: Figure 4 The target beam speech and interference beam speech are shown in FIG. The calculation formula of the target beam speech s(k, l) is as follows:

[0111]

[0112] s(k, l) = w s (k)y(k,l)

[0113] Among them, w s (k) represents the filter of the target speech, which aims to enhance the sound source from the target direction; g ss (k) represents the covariance matrix R of the target speech space ss (k) The main eigenvector obtained by eigendecomposition; R xx (k) represents the spatial covariance matrix of noisy speech, and the specific calculation formula is as follows:

[0114]

[0115] Similarly, the calculation formula of the interference beam speech n(k, l) is as follows:

[0116]

[0117] m(k, l) = w n (k)y(k,l) (7)

[0118] Among them, w n (k) represents the filter of the interfering speech, which aims to weaken the sound source from the target direction; g nn (k) represents the spatial covariance matrix R of the interfering speech nn (k) The main eigenvector obtained by eigendecomposition; where R nn The specific calculation formula of (k) is as follows:

[0119]

[0120] On this basis, the calculation formula of the multimode beam characteristic (MMDF) can be obtained as follows:

[0121]

[0122] S205: Input the acoustic features of the mixed speech, the lip shape features of the target user, and the multi-mode beam features into a pre-built multi-mode speech separation model, and separate the target speech from the mixed speech according to the masking signal output by the model.

[0123] In this embodiment, after obtaining the acoustic features of the mixed speech (such as LPS features) and the lip shape features of the target user (i.e., the intermediate features output by the cross-modal Transformer of the multi-modal alignment model) through steps S201 and S202 respectively, and obtaining the multi-mode beam features (MMDF) corresponding to the mixed speech through step S204, in order to accurately separate the target speech from the mixed speech, the acoustic features of the mixed speech (such as LPS features), the lip shape features of the target user (i.e., the intermediate features output by the cross-modal Transformer of the multi-modal alignment model), and the multi-mode beam features (MMDF) can be further input into a pre-constructed multi-modal speech separation model to obtain the masking signal output by the model, and then based on the masking signal output by the model in combination with the beamforming algorithm, the target speech can be more accurately separated from the mixed speech.

[0124] Among them, it should be noted that in order to accurately separate the target speech from the mixed speech, the present application pre-constructs a multi-mode speech separation model, and does not limit the specific composition structure of the model. Next, this embodiment will introduce the construction process of the multi-mode speech separation model. An optional implementation method is that the construction process of the multi-mode alignment model can specifically include: first, the acoustic features of the sample speech and the intermediate features output by the cross-modal Transformer are input into the multi-mode VAD model to determine the speech existence probability of the sample speech using the above formula (1). Then, based on the speech existence probability of the sample speech, combined with the beamforming algorithm, the multi-mode beam features corresponding to the sample speech are calculated using the above formula (9). Next, the initial multimodal speech separation model is trained using the acoustic features of the sample speech, the intermediate features output by the cross-modal Transformer, the multimodal beam features corresponding to the sample speech, and the minimum mean squared error loss (MSE Loss) function until the preset conditions are met (the specific content is not limited and can be set according to actual conditions and experience, such as the value of the contrast loss function is very small and basically unchanged, or the preset maximum number of iterations (such as 100 times) is reached), then the model parameter update is stopped to obtain the trained multimodal speech separation model.

[0125] Among them, it should be noted that the initial multimodal speech separation model adopted in this application can be but not limited to a convolutional long short-term memory fully connected deep neural network (Convolutional, Long Short-Term Memory, Deep Neural Networks, referred to as CLDNN), specifically including but not limited to a 5-layer convolutional neural network (Convolutional Neural Network, CNN) and a 2-layer long short-term memory network (Long Short-Term Memory, LSTM), etc.

[0126] In summary, the present embodiment provides a method for speech separation, which first obtains a mixed speech and a target video to be separated; and extracts the acoustic features of the mixed speech; wherein the mixed speech contains a target speech emitted by a target user; the target video is a video containing the lip shape of the target user, and then the acoustic features of the mixed speech and the image of the lip shape of the target user contained in the target video are input into a pre-constructed multi-mode alignment model to obtain the lip shape features of the target user; then, the acoustic features of the mixed speech and the lip shape features of the target user are input into a pre-constructed multi-mode VAD model to obtain the separation probability of the mixed speech; then, based on the separation probability of the mixed speech and in combination with a beamforming algorithm, the multi-mode beam features corresponding to the mixed speech are calculated, and then the acoustic features of the mixed speech, the lip shape features of the target user, and the multi-mode beam features can be input into a pre-constructed multi-mode speech separation model, and the target speech can be separated from the mixed speech based on the masking signal output by the model.

[0127] It can be seen that since the present application utilizes a pre-built multi-mode alignment model, a multi-mode VAD model and a multi-mode speech separation model, it realizes an audio and video alignment based on the acoustic features of the mixed speech and the lip shape features of the target user (ie, the speaker) and a multi-mode separation method based on multi-mode directional features. This not only solves the problem in the prior art that the separation performance is reduced due to the unknown position and movement of the speaker, but also can more accurately separate the target speech from the mixed speech, thereby improving the separation effect of the target speech.

[0128] Second embodiment

[0129] This embodiment will introduce a speech separation device. For related content, please refer to the above method embodiment.

[0130] See also Figure 5 , is a schematic diagram of the composition of a speech separation device provided in this embodiment, the device 500 includes:

[0131] The first acquisition unit 501 is used to acquire a mixed speech to be separated and a target video; and extract the acoustic features of the mixed speech; the mixed speech includes the target speech emitted by the target user; and the target video is a video including the lip shape of the target user;

[0132] The first input unit 502 is used to input the acoustic features of the mixed speech and the image of the target user's lip shape contained in the target video into a pre-built multi-modal alignment model to obtain the lip shape features of the target user;

[0133] The second input unit 503 is used to input the acoustic features of the mixed speech and the lip shape features of the target user into a pre-built multi-modal voice activity detection VAD model to obtain the separation probability of the mixed speech;

[0134] A first calculation unit 504 is used to calculate the multi-mode beam characteristics corresponding to the mixed speech according to the separation probability of the mixed speech in combination with a beamforming algorithm;

[0135] The separation unit 505 is used to input the acoustic features of the mixed speech, the lip shape features of the target user, and the multi-mode beam features into a pre-constructed multi-mode speech separation model, and separate the target speech from the mixed speech according to the masking signal output by the model.

[0136] In an implementation of this embodiment, the multi-mode alignment model includes an audio branch, a video branch, and a cross-modal converter Transformer; the first input unit 502 includes:

[0137] A first input subunit, configured to input the acoustic features of the mixed speech into the audio branch of the multi-mode alignment model to obtain an audio coding feature vector;

[0138] A second input subunit is used to input the image of the target user's lip shape contained in the target video into the video branch of the multi-mode alignment model to obtain an image coding feature vector;

[0139] The third input subunit is used to input the audio coding feature vector and the image coding feature vector into the cross-modal Transformer of the multi-modal alignment model to obtain the lip shape features of the target user.

[0140] In an implementation of this embodiment, the device further includes:

[0141] The second acquisition unit is used to acquire a synchronized sample speech and sample video pair, and an unsynchronized sample speech and sample video pair; and extract the acoustic features of the sample speech; the target video is a video containing the sample user's lip shape;

[0142] A first training unit is used to train an initial multimodal alignment model using the acoustic features of the sample speech, the image of the sample user's lip shape contained in the sample video, and a contrast loss function to obtain the multimodal alignment model;

[0143] The initial multimodal alignment model includes an initial audio branch, an initial video branch, an initial cross-modal Transformer and a binary classification layer.

[0144] In an implementation of this embodiment, the first training unit includes:

[0145] A fourth input subunit, used for inputting the acoustic features of the sample speech into the initial audio branch of the initial multi-mode alignment model to obtain a sample audio coding feature vector;

[0146] A fifth input subunit, configured to input the image of the sample user's lip shape contained in the sample video into the initial video branch of the initial multi-modal alignment model to obtain a sample image encoding feature vector;

[0147] a sixth input subunit, configured to input the sample audio coding feature vector and the sample image coding feature vector into an initial cross-modal Transformer of the initial multimodal alignment model to obtain intermediate sample features output by the cross-modal Transformer;

[0148] The training subunit is used to input the intermediate sample features into the binary classification layer of the initial multimodal alignment model to obtain a synchronization score, and use the synchronization score and the contrast loss function to train the initial multimodal alignment model until a preset condition is met, then stop updating the model parameters to obtain a trained multimodal alignment model.

[0149] In an implementation of this embodiment, the device further includes:

[0150] The second training unit is used to train the initial multimodal VAD model using the acoustic features of the sample speech, the intermediate features output by the cross-modal Transformer and the cross-entropy loss function to obtain the multimodal VAD model; wherein the initial multimodal VAD model includes four convolutional layers and one fully connected layer.

[0151] In an implementation of this embodiment, the device further includes:

[0152] A third input unit is used to input the acoustic features of the sample speech and the intermediate features output by the cross-modal Transformer into a multimodal VAD model to obtain a speech existence probability of the sample speech;

[0153] A second calculation unit, configured to calculate a multimode beam feature corresponding to the sample speech according to the speech existence probability of the sample speech and in combination with a beamforming algorithm;

[0154] The third training unit is used to train the initial multimodal speech separation model by using the acoustic features of the sample speech, the intermediate features output by the cross-modal Transformer, the multimodal beam features corresponding to the sample speech, and the minimum mean square error loss function to obtain the multimodal speech separation model; wherein the initial multimodal speech separation model includes a five-layer convolutional neural network CNN and a two-layer long short-term memory network LSTM.

[0155] In an implementation of this embodiment, the acoustic feature of the mixed speech is a logarithmic power spectrum LPS.

[0156] Furthermore, the embodiment of the present application also provides a speech separation device, including: a processor, a memory, and a system bus;

[0157] The processor and the memory are connected via the system bus;

[0158] The memory is used to store one or more programs, and the one or more programs include instructions. When the instructions are executed by the processor, the processor executes any one of the implementation methods of the above-mentioned speech separation method.

[0159] Furthermore, an embodiment of the present application also provides a computer-readable storage medium, in which instructions are stored. When the instructions are executed on a terminal device, the terminal device executes any one of the implementation methods of the above-mentioned speech separation method.

[0160] Furthermore, an embodiment of the present application also provides a computer program product, which, when executed on a terminal device, enables the terminal device to execute any one of the implementation methods of the above-mentioned speech separation method.

[0161] It can be known from the description of the above implementation mode that those skilled in the art can clearly understand that all or part of the steps in the above-mentioned embodiment method can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product can be stored in a storage medium such as ROM / RAM, a disk, an optical disk, etc., including several instructions for enabling a computer device (which can be a personal computer, a server, or a network communication device such as a media gateway, etc.) to execute the methods described in the various embodiments of the present application or certain parts of the embodiments.

[0162] It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0163] It should also be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0164] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A speech separation method, characterized in that: include: Acquire a mixed speech to be separated and a target video; and extract acoustic features of the mixed speech; The mixed voice includes the target voice uttered by the target user; the target video is a video including the lip shape of the target user; Inputting the acoustic features of the mixed speech and the image of the target user's lip shape contained in the target video into a pre-built multi-modal alignment model to obtain the lip shape features of the target user; Inputting the acoustic features of the mixed speech and the lip shape features of the target user into a pre-built multi-modal voice activity detection VAD model to obtain a separation probability of the mixed speech; Calculate the multi-mode beam characteristics corresponding to the mixed speech according to the separation probability of the mixed speech and in combination with the beamforming algorithm; The acoustic features of the mixed speech, the lip shape features of the target user, and the multi-mode beam features are input into a pre-constructed multi-mode speech separation model, and the target speech is separated from the mixed speech according to the masking signal output by the model.

2. The method according to claim 1, characterized in that The multimodal alignment model includes an audio branch, a video branch, and a cross-modal transformer Transformer; the step of inputting the acoustic features of the mixed speech and the image of the lip shape of the target user included in the target video into the pre-built multimodal alignment model to obtain the lip shape features of the target user includes: Inputting the acoustic features of the mixed speech into the audio branch of the multi-mode alignment model to obtain an audio coding feature vector; Inputting the image of the target user's lip shape contained in the target video into the video branch of the multi-mode alignment model to obtain an image coding feature vector; The audio coding feature vector and the image coding feature vector are input into the cross-modal Transformer of the multi-modal alignment model to obtain the lip shape features of the target user.

3. The method according to claim 1, characterized in that The multi-mode alignment model is constructed as follows: Acquire a synchronized pair of sample speech and sample video, and acquire an unsynchronized pair of sample speech and sample video; and extract acoustic features of the sample speech; the target video is a video containing the sample user's lip shape; Using the acoustic features of the sample speech, the image of the sample user's lip shape contained in the sample video, and the contrast loss function, an initial multimodal alignment model is trained to obtain the multimodal alignment model; The initial multimodal alignment model includes an initial audio branch, an initial video branch, an initial cross-modal Transformer and a binary classification layer.

4. The method according to claim 3, characterized in that The method of training the initial multimodal alignment model by using the acoustic features of the sample speech, the image of the sample user's lip shape contained in the sample video, and the contrast loss function to obtain the multimodal alignment model includes: Inputting the acoustic features of the sample speech into the initial audio branch of the initial multi-mode alignment model to obtain a sample audio coding feature vector; Inputting the image of the sample user's lip shape contained in the sample video into the initial video branch of the initial multimodal alignment model to obtain a sample image encoding feature vector; Inputting the sample audio coding feature vector and the sample image coding feature vector into the initial cross-modal Transformer of the initial multimodal alignment model to obtain intermediate sample features output by the cross-modal Transformer; The intermediate sample features are input into the binary classification layer of the initial multimodal alignment model to obtain a synchronization score, and the initial multimodal alignment model is trained using the synchronization score and the contrast loss function until a preset condition is met, then the updating of the model parameters is stopped to obtain a trained multimodal alignment model.

5. The method according to claim 3, characterized in that: The multi-mode VAD model is constructed as follows: The initial multimodal VAD model is trained using the acoustic features of the sample speech, the intermediate features output by the cross-modal Transformer, and the cross-entropy loss function to obtain the multimodal VAD model; wherein the initial multimodal VAD model includes four convolutional layers and one fully connected layer.

6. The method according to claim 3, characterized in that The multi-mode speech separation model is constructed as follows: Inputting the acoustic features of the sample speech and the intermediate features output by the cross-modal Transformer into a multimodal VAD model to obtain a speech existence probability of the sample speech; Calculating the multi-mode beam characteristics corresponding to the sample speech according to the speech existence probability of the sample speech in combination with a beamforming algorithm; The initial multimodal speech separation model is trained using the acoustic features of the sample speech, the intermediate features output by the cross-modal Transformer, the multimodal beam features corresponding to the sample speech, and the minimum mean square error loss function to obtain the multimodal speech separation model; wherein the initial multimodal speech separation model includes a five-layer convolutional neural network CNN and a two-layer long short-term memory network LSTM.

7. The method according to any one of claims 1 to 6, characterized in that: The acoustic feature of the mixed speech is a logarithmic power spectrum LPS.

8. A speech separation device, characterized in that: include: A first acquisition unit is used to acquire a mixed speech to be separated and a target video; and extract acoustic features of the mixed speech; The mixed voice includes the target voice uttered by the target user; the target video is a video including the lip shape of the target user; A first input unit is used to input the acoustic features of the mixed speech and the image of the lip shape of the target user contained in the target video into a pre-built multi-modal alignment model to obtain the lip shape features of the target user; A second input unit is used to input the acoustic features of the mixed speech and the lip shape features of the target user into a pre-built multi-modal voice activity detection VAD model to obtain a separation probability of the mixed speech; A first calculation unit, configured to calculate a multimode beam feature corresponding to the mixed speech according to the separation probability of the mixed speech and in combination with a beamforming algorithm; The separation unit is used to input the acoustic features of the mixed speech, the lip shape features of the target user, and the multi-mode beam features into a pre-constructed multi-mode speech separation model, and separate the target speech from the mixed speech according to the masking signal output by the model.

9. A speech separation device, characterized in that: include: Processor, memory, system bus; The processor and the memory are connected via the system bus; The memory is used to store one or more programs, wherein the one or more programs include instructions, and when the instructions are executed by the processor, the processor executes the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, and when the instructions are executed on a terminal device, the terminal device executes the method according to any one of claims 1 to 7.

Images