Cross-modal loss based target audio separation method and system

Abstract

To provide a method and a system for separating target voice based on cross-modal loss.SOLUTION: A target voice separation method uses a pre-trained target voice separation model, acquires a separated target voice signal based on a lip sequence of a target speaker and a mixed audio signal, extracts visual embedding by introducing visual information of the target speaker to a model, adopts an audio embedding and self-extracting method instead of pre-registered voice, and avoids the problem of target confusion by introducing cross-modal loss and restricting extraction of speaker visual and auditory features. By creating two branch architectures, acquiring an interfering speaker voice signal to convert the same into auxiliary information, and optimizing voice extraction of first branch, use of auxiliary information which is difficult to acquire can be avoided and adaptability of target voice separation in a practical situation is improved.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to the technical field of speech processing, and in particular to a method and system for target speech separation based on cross-modal loss. [Background technology]

[0002] The description in this section merely provides background information related to the present invention and does not necessarily constitute prior art.

[0003] Target speech separation, which refers to extracting a specific target speaker's voice from mixed audio, is a branch of speech separation. In complex scenarios with multiple speakers, speech separation can effectively improve the performance of applications such as speech recognition and speaker identification, and is applicable to systems such as remote meeting recording and smart homes. Unlike multi-speaker separation, which extracts the voices of all speakers in a conversational scenario, target speech separation only has one output: the estimated voice of the target speaker. Target speech separation avoids the problem of multi-speaker separation, which is sequence ambiguity, and does not require prior knowledge of the number of speakers. However, a priori knowledge or auxiliary information must be provided to specify the target signal. Depending on the form of auxiliary information, target speech separation can be divided into methods such as visual-dependent extraction of the target speaker or extraction-assisted extraction based on enrolled speech fragments of the target speaker.

[0004] A target speech separation network typically consists of a main network and an auxiliary network, where the main network is used to train a target speaker mask and the auxiliary network is used to train a discriminative speaker embedding. In a target speech separation network, the output of the auxiliary network has important guiding meaning for the main network. If the speaker embedding generated by the auxiliary network cannot distinguish between the target speaker and the interfering speaker, a target confusion problem will occur, i.e., the interfering speaker will be incorrectly extracted, or the target speaker's speech quality will be poor. Furthermore, it may be difficult to obtain information such as speaker tags and pre-registered speaker fragments used in the auxiliary network, making it unsuitable for practical use. Therefore, by more appropriately utilizing auxiliary information to extract discriminative embeddings, the target speech separation performance can be further improved.

[0005] However, the inventors have recognized that the current target audio separation has the following problems.

[0006] (1) Target mixing is likely to occur. When the target speech and background noise or interference speech overlap in the spectrum, different sound sources in the mixed signal interfere with each other in the frequency domain, resulting in the extraction of the interference speech or the poor quality of the extracted target speech.

[0007] (2) It is difficult to obtain some auxiliary information. In the target speaker visual-dependent extraction assistance method, it is usually necessary to obtain speaker identity tags and calculate speaker classification loss during training to constrain speaker extraction. However, it is difficult to obtain identity tags in real life. In the target speaker enrollment speech fragment-dependent extraction assistance method, it is necessary to obtain the target speaker's pure speech in advance, which is not suitable for practical use. Summary of the Invention

[0008] To solve the above problems, the present invention provides a target speech separation method and system based on cross-modal loss, which extracts visual embeddings by introducing the target speaker's visual information into the model, uses audio embedding self-extraction instead of pre-registered speech, and replaces speaker cross-entropy loss with cross-modal loss, thereby avoiding the problem of target confusion and improving the suitability of target speech separation in practical situations.

[0009] According to some embodiments, the present invention adopts the following technical solutions.

[0010] obtaining a video of a target speaker and extracting a lip sequence of the target speaker; obtaining a mixed audio signal; obtaining a separated target speech signal based on the lip sequence of the target speaker and the mixed audio signal using a pre-trained target speech separation model, wherein the training process of the target speech separation model takes into account cross-modal loss; The target speech separation model has a two-branch architecture in the training process: The first branch is used to realize audiovisual multimodal target speech extraction, and obtains a separated target speech signal based on the target speaker's lip sequence and the mixed audio signal; The second branch is used to realize single-modal interference voice extraction. Based on the estimated target voice signal and the mixed audio signal of the first branch, the separated interference voice signal is obtained. The obtained interference voice signal is used as auxiliary information, and the cross-modal loss affects the extraction of the target voice of the first branch. Cross-modal loss is a target speech separation method based on cross-modal loss, which simultaneously affects the training process of two branches, performs metric learning between the visual features of the target speaker, the target speech features, and the interference speech features, shortens the distance between positive samples and increases the distance between negative samples, and constrains the extraction of visual and auditory features.

[0011] In an alternative embodiment, the first branch includes an audio encoder, a visual encoder, a speaker extractor, and an audio decoder; The audio encoder extracts audio embeddings from the mixed audio signal, and the visual encoder encodes the target speaker's lip sequence as a visual embedding; The speaker extractor receives the extracted audio embeddings and visual embeddings as inputs and includes multi-layer iterative extractor blocks, each of which includes a speaker encoder that stitches together the two modal embeddings in the time dimension to obtain a target speaker embedding that includes distinctive identity information, and a mask estimator that obtains a target speaker mask based on the visual embeddings, the target speaker embedding, and the output of a mask estimator from a previous layer; The audio decoder obtains a target speech signal based on a target speaker mask.

[0012] In an alternative embodiment, the second branch comprises an audio encoder, a speaker extractor, and an audio decoder connected in series; The second branch exists only in the training process.

[0013] In an alternative embodiment, the audio encoder of the second branch receives the difference between the mixed audio signal and the target speech signal estimated in the first branch, and the speaker encoder receives only the audio embedding and outputs the interference speaker embedding.

[0014] In an alternative embodiment, the audio encoder includes a one-dimensional convolution for converting a time-domain audio signal into a sequence of audio features.

[0015] In an alternative embodiment, the audio decoder performs an overlap-add operation to recreate the sequence of audio features as an audio signal.

[0016] In an optional embodiment, the visual encoder includes a sequentially connected 3D convolutional layer, a residual network layer, and a video-temporal convolutional layer, where the video-temporal convolutional layer includes a sequentially connected normalized linear unit, a batch normalization layer, and a depth-unit separable convolutional layer. The visual encoder receives the audio-synchronized cut target speaker's lip sequence, extracts visual embeddings through the network layers, and upsamples the output of the video-temporal convolutional layer to finally obtain the visual embeddings.

[0017] In an alternative embodiment, the speaker encoder includes three sequentially connected residual blocks and an adaptive average pooling layer, and the mask estimator includes a plurality of stacked temporal convolution blocks, each of which includes a plurality of sequentially connected one-dimensional convolution layers and a residual module.

[0018] As an optional embodiment, a cross-modal loss is introduced into the target speech separation algorithm, and metric learning is performed between the visual features of the target speaker, the target speech features, and the interfering speech features, constraining the extraction of visual and auditory features by audio-visual consistency, converting the interfering speaker information into auxiliary information, and optimizing the speech extraction of the first branch.

[0019] As an optional embodiment, in the training process of the target speech separation model, a multi-task learning framework including three objective functions is created, in which the scale-invariant signal-to-noise ratio loss in the first branch is used to judge the quality between the extracted target speech and the pure speech, the scale-invariant signal-to-noise ratio loss in the second branch is used to judge the quality between the extracted interference speech and the pure target speech, and the cross-modal loss is used to bring the target visual features and the estimated target speech signal audio features closer together and to move the target visual features and the estimated interference speech signal audio features further apart.

[0020] In an alternative embodiment, the training process of the target speech separation model uses the existing audio data and corresponding video data to perform optimization training using an Adam optimizer until the test requirements are met.

[0021] a data acquisition module configured to acquire a video of a target speaker, extract a lip sequence of the target speaker, and acquire a mixed audio signal; a target speech separation module configured to utilize a pre-trained target speech separation model and obtain a separated target speech signal based on a lip sequence of a target speaker and the mixed audio signal, wherein the training process of the target speech separation module takes into account cross-modal loss; The target speech separation model has a two-branch architecture in the training process: The first branch is used to realize audiovisual multimodal target speech extraction, and obtains a separated target speech signal based on the target speaker's lip sequence and the mixed audio signal; The second branch is used to realize single-modal interference voice extraction. Based on the estimated target voice signal and the mixed audio signal of the first branch, the separated interference voice signal is obtained. The obtained interference voice signal is used as auxiliary information, and the cross-modal loss affects the extraction of the target voice of the first branch. Cross-modal loss is a target speech separation system based on cross-modal loss, which simultaneously affects the training process of two branches, performs metric learning between the visual features of the target speaker, the target speech features, and the interference speech features, shortens the distance between positive samples and increases the distance between negative samples, and constrains the extraction of visual and auditory features.

[0022] An electronic device includes a memory, a processor, and computer commands stored in the memory and executed by the processor, the computer commands completing the steps of the above method when executed by the processor.

[0023] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0024] The present invention introduces the visual information of the target speaker into the model to extract the visual embedding, uses the method of self-extracting audio embedding instead of pre-registered speech, splices the self-extracted audio embedding and visual embedding and feeds it to the speaker encoder to obtain the speaker embedding as the attractor of the separation network, thereby avoiding the problem of target confusion.

[0025] This invention replaces the speaker cross-entropy loss with a cross-modal loss, and during the training process, shortens the distance between the audio embedding and the visual embedding of the target speaker, while increasing the distance between the audio embedding and the visual embedding of the interfering speaker, thereby encouraging the visual encoder to extract better visual embeddings and obtain more accurate speaker embeddings. Furthermore, by replacing the pre-registered voice with a self-extracted voice and repeating the speaker extractor network four times, a priori knowledge is obtained, solving the problem of the difficulty of obtaining auxiliary information.

[0026] In order to make the above objects, features and advantages of the present invention more comprehensible, preferred embodiments will be particularly described below with reference to the accompanying drawings.

[0027] The specification drawings that form a part of this invention are intended to provide a further understanding of the invention, and the illustrative embodiments of the invention and their descriptions are intended to interpret the invention and are not intended to unduly limit the invention. [Brief explanation of the drawings]

[0028] [Figure 1] FIG. 2 is a structural schematic diagram of the model of the present embodiment. [Figure 2] FIG. 2 is a structural schematic diagram of a mask estimator according to the present embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0029] In the following the invention will be further explained with reference to the figures and examples.

[0030] It should be noted that the following detailed description is for illustrative purposes only and is intended to further explain the present invention. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art.

[0031] It should be noted that the terminology used herein is for the purpose of describing specific embodiments only and is not intended to limit the exemplary embodiments according to the present invention. For example, unless the context clearly dictates otherwise, the singular forms used herein are intended to include the plural forms, and it should also be understood that the use of the terms "comprises" and / or "includes" herein indicates the presence of features, steps, operations, devices, assemblies, and / or combinations thereof.

[0032] The embodiments and features of the embodiments in the present application may be combined with each other without contradiction.

[0033] Example 1 obtaining a video of a target speaker and extracting a lip sequence of the target speaker; obtaining a mixed audio signal; obtaining a separated target speech signal based on the lip sequence of the target speaker and the mixed audio signal using a pre-trained target speech separation model, wherein the training process of the target speech separation model takes into account cross-modal loss; The target speech separation model has a two-branch architecture during the training process: The first branch is used to realize audiovisual multimodal target speech extraction, and obtains a separated target speech signal based on the target speaker's lip sequence and the mixed audio signal; The second branch is used to realize single-modal interference voice extraction. Based on the estimated target voice signal and the mixed audio signal of the first branch, the separated interference voice signal is obtained. The obtained interference voice signal is used as auxiliary information, and the cross-modal loss affects the extraction of the target voice of the first branch. Cross-modal loss is a target speech separation method based on cross-modal loss, which simultaneously affects the training process of two branches, performs metric learning between the visual features of the target speaker, the target speech features, and the interference speech features, shortens the distance between positive samples and increases the distance between negative samples, and constrains the extraction of visual and auditory features.

[0034] The pre-trained target speech separation model includes an audio encoder, a visual encoder, a speaker extractor, and an audio decoder; The audio encoder extracts audio embeddings from the mixed audio signal, and the visual encoder encodes the target speaker's lip sequence as a visual embedding; The speaker extractor receives the extracted audio embeddings and visual embeddings as inputs and includes multi-layer iterative extractor blocks, each of which includes a speaker encoder that stitches together the two modal embeddings in the time dimension to obtain a target speaker embedding that includes distinctive identity information, and a mask estimator that obtains a target speaker mask based on the visual embeddings, the target speaker embedding, and the output of a mask estimator from a previous layer; The audio decoder obtains a target speech signal based on a target speaker mask.

[0035] The target speech separation model of this embodiment is a two-branch speaker extraction network based on the time domain, and as shown in FIG. 1, it is mainly composed of four parts: an audio encoder, an audio decoder, a visual encoder, and a speaker extractor.

[0036] Here, both the first branch and the second branch include an audio encoder, an audio decoder, and a speaker extractor.

[0037] The audio encoder extracts audio embeddings from the input mixed audio signal, and the audio decoder can restore the audio embeddings to the audio signal. The visual encoder encodes the input target speaker's lip sequence as a visual embedding. The main task of the speaker extractor is to obtain a speaker mask. However, the structures of the two branches are slightly different, and the second branch only exists in the training phase and does not exist in the practical or testing phase.

[0038] Each part of the model is described in detail below.

[0039] For audio encoders, a one-dimensional convolution is used to generate a time-domain audio signal.

number

number

[0040] where N is the dimension of the audio embedding, T is the total number of audio samples, K = (2(TL)) / L + 1, and L is the size of the convolution kernel for one-dimensional convolution.

[0041] In the second branch, the input to the audio encoder is the difference between the mixed audio signal and the target audio signal estimated in the first branch.

[0042] For the audio decoder, we perform an overlap-add operation to decode the audio feature sequence

number

number

[0043] In the model, L is set to 40 and N is set to 256 in this example.

[0044] The visual encoder mainly consists of three parts: a 3D convolutional layer, a ResNet18 layer, and a video-temporal convolutional layer. The 3D convolutional layer and the ResNet18 layer are derived from a pre-trained lip-reading network, and their weights are constant during speaker extraction training. The video-temporal convolutional layer consists of a rectified linear unit (Relu), a batch normalization layer (BN), and a depth-separable convolutional layer (DS Conv1D). The visual encoder receives the audio-synchronized cut lip sequence of the target speaker and extracts the visual embedding through the network layers. To match the temporal resolution of the audio embedding, the output of the video-temporal convolutional layer must be further upsampled, ultimately obtaining a visual embedding V(t) of size 1024.

[0045] The speaker extractor is composed of four layers of repeated extractor blocks, each consisting of one speaker encoder and one mask estimator. The speaker encoder generates a speaker embedding A based on a dynamically estimated speech signal. r The mask estimator estimates a mask Mr(t) that passes only the target speech. The speaker encoder consists of three residual blocks and an adaptive average pooling layer. The two branches differ slightly: in the first branch, which includes visual information, the speaker encoder receives the visual embeddings output by the visual encoder and the dynamically estimated audio embeddings of the speech signal, stitches the two modal embeddings in the time dimension, and outputs a target speaker embedding containing distinctive identity information. The embedding size is 256. In the second branch, the speaker encoder receives only the audio embeddings as input and outputs an interfering speaker embedding.

[0046] As shown in Figure 2, the mask estimator is constructed based on a temporal convolutional network, stacking eight temporal convolutional blocks. Each temporal convolutional block consists of multiple one-dimensional convolutional layers and a residual module. The kernel size of the one-dimensional convolutional block is 3, and the expansion factor of the temporal convolutional block is 2. d The inputs to the mask estimator are the visual embedding V(t) and the target speaker embedding A r and the mask output from the extraction block of the previous layer, M r-1 (t), and the input to the mask estimator in the second branch is the same as in the first branch, except that it does not include the visual embedding.

[0047] In this embodiment, a multi-task learning framework is constructed using three objective functions: the scale-invariant signal-to-noise ratio (SISNR) loss in the first branch is used to judge the quality between the extracted target speech and the pure speech; the scale-invariant signal-to-noise ratio (SISNR) loss in the second branch is used to judge the quality between the extracted interference speech and the pure target speech; and the cross-modal loss is used to bring the target visual features closer to the estimated target speech signal audio features and move the target visual features away from the estimated interference speech signal audio features. The losses are defined as follows:

[0048]

number

[0049] where γ is a scale factor, which is set to 0.1 in this example.

[0050] Of course, in other embodiments, the scale factor of each loss can be set or adjusted depending on the requirements of the target audio separation, the application scenario, etc.

[0051] In Equation (6), dp represents the cosine distance between the visual feature and the estimated target speech signal audio feature, dn represents the cosine distance between the visual feature and the estimated interference speech signal audio feature, where dp, dn ∈ [0, 2], and margin is the margin between the two distances, which is set to 0.5.

[0052] In this example, the model is trained and evaluated using the VoxCeleb2 dataset. The VoxCeleb2 dataset contains over one million audio clips and corresponding videos extracted from YouTube. The training set contains 5,994 speakers, and the test set contains 118 speakers. Two audio clips are randomly selected from the raw training dataset each time and mixed directly to create a training set containing 20,000 mixed audio clips. Two audio clips are randomly selected from the raw test set each time and mixed directly to create a validation set containing 5,000 mixed audio clips. Similarly to the validation set, a test set containing 3,000 mixed audio clips is created. The dataset contains audio clips with a sampling frequency of 16 kHz and a duration of 4 seconds. The dataset contains audio clips with a video sampling rate of 25 FPS, and the video and audio clips are of equal and time-matched duration.

[0053] During model training, we use the Adam optimizer with an initial learning rate of 1e -4 The learning rate is set to , and if the loss on the validation set increases three consecutive epochs, the learning rate is reduced by 0.5. If the loss on the validation set increases six consecutive epochs, training is stopped. In the inference phase, only the speaker extraction process of the first branch is kept.

[0054] Of course, in the above process, the dataset can be trained by selecting other legitimate data from public sources, and the model optimizer, such as a momentum optimizer, are common choices that can be easily thought of by those skilled in the art, and detailed explanations will be omitted here.

[0055] In other embodiments, it is understood that any of the set values, taken values, etc. of the above parameters or thresholds can be adjusted or changed depending on the specific situation or circumstances, and these are all general choices that can be easily thought of by a person skilled in the art, and are included in the scope of protection of the present invention.

[0056] Example 2 a data acquisition module configured to acquire a video of a target speaker, extract a lip sequence of the target speaker, and acquire a mixed audio signal; a target speech separation module configured to utilize a pre-trained target speech separation model and obtain a separated target speech signal based on a lip sequence of a target speaker and the mixed audio signal, wherein the training process of the target speech separation module takes into account cross-modal loss; The target speech separation model has a two-branch architecture in the training process: The first branch is used to realize audiovisual multimodal target speech extraction, and obtains a separated target speech signal based on the target speaker's lip sequence and the mixed audio signal; The second branch is used to realize single-modal interference voice extraction. Based on the estimated target voice signal and the mixed audio signal of the first branch, the separated interference voice signal is obtained. The obtained interference voice signal is used as auxiliary information, and the cross-modal loss affects the extraction of the target voice of the first branch. Cross-modal loss is a target speech separation system based on cross-modal loss, which simultaneously affects the training process of two branches, performs metric learning between the visual features of the target speaker, the target speech features, and the interference speech features, shortens the distance between positive samples and increases the distance between negative samples, and constrains the extraction of visual and auditory features.

[0057] the trained target speech separation model includes an audio encoder, a visual encoder, a speaker extractor, and an audio decoder; The audio encoder extracts audio embeddings from the mixed audio signal, and the visual encoder encodes the target speaker's lip sequence as a visual embedding; The speaker extractor receives the extracted audio embeddings and visual embeddings as inputs and includes multi-layer iterative extractor blocks, each of which includes a speaker encoder that stitches together the two modal embeddings in the time dimension to obtain a target speaker embedding that includes distinctive identity information, and a mask estimator that obtains a target speaker mask based on the visual embeddings, the target speaker embedding, and the output of a mask estimator from a previous layer; The audio decoder obtains a target speech signal based on a target speaker mask.

[0058] In order to verify the effectiveness and performance of the technical solutions provided in the above examples, the following quality evaluations are carried out here.

[0059] In this example, the performance of the target speech separation algorithm was evaluated using the following four indices.

[0060] (1) Signal-to-Distortion Ratio Improvement (SDRi). SDRi is the difference between the signal-to-distortion ratios (SDR) of the estimated speech signal and the input mixed speech signal. It intuitively reflects the improvement in speech signal quality before and after separation by the algorithm. The greater the SDR improvement before and after separation by the algorithm, the larger the SDRi value, indicating better model performance. SDR represents the ratio of the target signal power to the power of other interference signals, reflecting the energy ratio between the target speech component and other interference speech components in the enhanced speech separated by the algorithm. The higher the ratio of the target effective signal, the higher the signal-to-distortion ratio, indicating that the target speech separated by the algorithm is closer to the real speech. The SDR calculation formula is as follows:

[0061]

number

[0062] (2) Scale Invariant Signal to Distortion Ratio improvement (SISDRi). SISDRi is the difference between the two scale invariant signal to distortion ratios (SISDR) of the estimated speech signal and the input mixed speech signal. SISDR is a more robust improvement evaluation index based on SDR, and aims to avoid discrepancies caused by scale mismatch. The higher the SISDR value, the closer the estimated speech separated by the separation algorithm is to the actual speech, and the higher the model performance. The formula for SISDR is as follows:

[0063]

number

[0064] (3) Perceptual Evaluation of Speech Quality (PESQ). PESQ is one of the most commonly used quality evaluation indices for speech enhancement, separation, and extraction. This index is based on the design of the subjective evaluation index MOS. MOS predicted values ​​are calculated using the noisy attenuated signal and the reference signal, and the score range is mapped to the MOS scale range. The PESQ value range is -0.5 to 4.5, with higher values ​​indicating better quality of the test speech.

[0065] (4) Short-Time Objective Intelligibility (STOI). STOI is an objective speech quality assessment index that is linearly correlated with speech signal intelligibility and is used to evaluate speech signal intelligibility. It is applied to situations such as speech recognition. STOI aims to evaluate the intelligibility of extracted speech by calculating the percentage of correctly readable words in the speech. Its range is 0 to 1, and the larger the STOI, the higher the intelligibility of the extracted speech.

[0066] The results of evaluating the model on the VoxCeleb2 dataset are shown in Table 1. After algorithm processing, the SISDR of the speech signal improved by 8.201 dB and the SDR improved by 8.777 dB. The model effectively extracts the target speech from the mixed speech signal without introducing complex target auxiliary information, avoiding the problem of target confusion and achieving good results in each evaluation index.

[0067] [Table 1]

[0068] It will be apparent to those skilled in the art that embodiments of the present invention may be provided as a method, a system, or a computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. The present invention may also take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, magnetic disk storage devices, CD-ROMs, optical storage devices, etc.) containing computer-usable program code.

[0069] The present invention has been described with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and combinations of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program commands. These computer program commands may be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, whereby the commands executed by the processor of the computer or other programmable data processing apparatus create means for implementing the functions specified in one or more flows of the flowcharts and / or one or more blocks of the block diagrams.

[0070] These computer program commands may be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to operate in a particular manner, whereby the commands stored in the computer readable memory create an article of manufacture that includes command means for implementing the functions specified in one or more flows of the flowcharts and / or one or more blocks of the block diagrams.

[0071] These computer program commands may be loaded into a computer or other programmable data processing apparatus to cause the computer or other programmable data processing apparatus to perform a series of operational steps to produce a computer-implemented process, whereby the commands executed on the computer or other programmable apparatus provide steps for implementing the functions specified in one or more flows of the flowcharts and / or one or more blocks of the block diagrams.

[0072] The above is only a preferred embodiment of the present invention, and is not intended to limit the present invention. Those skilled in the art can make various modifications and changes to the present invention. Any modifications, equivalent replacements, improvements, etc. made by those skilled in the art within the spirit and principle of the present invention without requiring creative effort shall all be included in the protection scope of the present invention.

Claims

1. obtaining a video of a target speaker and extracting a lip sequence of the target speaker; obtaining a mixed audio signal; inputting a lip sequence of a target speaker and a mixed audio signal into a pre-trained target speech separation model, obtaining a separated target speech signal as an output of the target speech separation model, and performing a training process of the target speech separation model, wherein the training process of the target speech separation model considers cross-modal loss; In the training process of the target speech separation model, the Adam optimizer is used to perform optimization training using the existing audio data and corresponding video data until the test requirements are met; The target speech separation model is a time-domain based bipartite speaker extraction network, The first branch is used to realize audiovisual multimodal target speech extraction, and obtains a separated target speech signal based on the target speaker's lip sequence and the mixed audio signal; The second branch is used to realize single-modal interference voice extraction, and obtains a separated interference voice signal based on the estimated target voice signal and the mixed audio signal of the first branch. The obtained interference voice signal is used as auxiliary information, and the cross-modal loss affects the extraction of the target voice of the first branch. The first branch and the second branch both comprise an audio encoder, an audio decoder, and a speaker extractor, and the first branch comprises a visual encoder, wherein: The audio encoder of the first branch extracts audio embeddings from the mixed audio signal, and the visual encoder extracts visual embeddings from the lip sequence of a target speaker; The speaker extractor receives the extracted audio embeddings and visual embeddings as inputs and includes multi-layer iterative extractor blocks, each of which includes a speaker encoder that stitches together the two modal embeddings in the time dimension to obtain a target speaker embedding that includes distinctive identity information, and a mask estimator that obtains a target speaker mask based on the visual embeddings, the target speaker embedding, and the output of a mask estimator from a previous layer; The audio decoder obtains a target speech signal based on a target speaker mask; an audio encoder of the second branch receives the difference between the mixed audio signal and the target speech signal estimated in the first branch, and extracts the audio embedding; The speaker encoder of the speaker extractor receives only the audio embeddings and outputs the interfering speaker embeddings, and the mask estimator in the speaker extractor obtains the interfering speaker mask based on the interfering speaker embeddings and the output of the mask estimator of the previous layer; The audio decoder obtains a separated interfering audio signal based on an interfering speaker mask; The audio encoders of the first and second branches generate time-domain audio signals by one-dimensional convolution. [Equation 8] to an audio feature sequence X(t), and the conversion process can be expressed as: [Equation 9] where N is the dimension of the audio embedding, T is the total number of audio samples, K=(2(T−L)) / L+1, and L is the size of the convolution kernel for one-dimensional convolution. The audio decoders of the first branch and the second branch perform an overlap-add operation to generate an audio feature sequence. [Equation 10] and the decoding process is as follows, [0011] In the training process of the target speech separation model, a multi-task learning framework is created, which includes three objective functions: The scale-invariant signal-to-noise ratio loss (SISNR) of the first branch is used to judge the signal quality between the extracted target speech and the pure target speech; The scale-invariant signal-to-noise ratio loss (SISNR) of the second branch is used to judge the signal quality between the extracted interference sound and the pure interference sound; The cross-modal loss is used to bring the target visual features closer to the estimated target speech signal audio features and to move the target visual features away from the estimated interfering speech signal audio features. The loss is defined as follows: [0012] Cross-modal loss simultaneously affects the training process of the two branches, performs metric learning between the visual features of the target speaker, the target speech features, and the interference speech features, and reduces the distance between positive samples and negative samples, thereby constraining the extraction of visual and auditory features. where γ is a scale factor, dp represents the cosine distance between the visual feature and the estimated target speech signal audio feature, dn represents the cosine distance between the visual feature and the estimated interference speech signal audio feature, dp, dn ∈ [0, 2], margin is the margin between the two distances, which is set to 0.5, A target speech separation method based on cross-modal loss, which introduces a cross-modal loss into the target speech separation algorithm, and the cross-modal loss simultaneously affects the training process of the two branches, performs metric learning between the visual features of the target speaker, the target speech features, and the interference speech features, shortens the distance between positive samples and increases the distance between negative samples, constrains the extraction of visual and auditory features through audio-visual consistency, converts the interference speaker information into auxiliary information, and optimizes the speech extraction of the first branch.

2. 2. The method for target speech separation based on cross-modal loss according to claim 1, wherein the visual encoder comprises a sequentially connected 3D convolutional layer, a residual network layer, and a video-temporal convolutional layer, wherein the video-temporal convolutional layer comprises a sequentially connected normalized linear unit, a batch normalization layer, and a depth-unit separable convolutional layer, wherein the visual encoder receives the audio-synchronized cut lip sequence of the target speaker, extracts visual embeddings through the network layers, and upsamples the output of the video-temporal convolutional layer to finally obtain the visual embeddings.

3. The target speech separation method based on cross-modal loss according to claim 1, characterized in that the speaker encoder includes three sequentially connected residual blocks and an adaptive average pooling layer, and the mask estimator includes a plurality of stacked temporal convolution blocks, each of which includes a plurality of sequentially connected one-dimensional convolution layers and a residual module.

4. a data acquisition module configured to acquire a video of a target speaker, extract a lip sequence of the target speaker, and acquire a mixed audio signal; a target speech separation module configured to input a lip sequence of a target speaker and a mixed audio signal into a pre-trained target speech separation model and obtain a separated target speech signal as an output of the target speech separation model, the target speech separation module performing a training process, the training process taking into account cross-modal loss; In the training process of the target speech separation model, the Adam optimizer is used to perform optimization training using the existing audio data and corresponding video data until the test requirements are met; The target speech separation model is a time-domain based bipartite speaker extraction network, where: The first branch is used to realize audiovisual multimodal target speech extraction, and obtains a separated target speech signal based on the target speaker's lip sequence and the mixed audio signal; The second branch is used to realize single-modal interference voice extraction, and obtains a separated interference voice signal based on the estimated target voice signal and the mixed audio signal of the first branch. The obtained interference voice signal is used as auxiliary information, and the cross-modal loss affects the extraction of the target voice of the first branch. The first branch and the second branch both comprise an audio encoder, an audio decoder, and a speaker extractor, and the first branch comprises a visual encoder, wherein: The audio encoder of the first branch extracts audio embeddings from the mixed audio signal, and the visual encoder extracts visual embeddings from the lip sequence of a target speaker; The speaker extractor receives the extracted audio embeddings and visual embeddings as inputs and includes multi-layer iterative extractor blocks, each of which includes a speaker encoder that stitches together the two modal embeddings in the time dimension to obtain a target speaker embedding that includes distinctive identity information, and a mask estimator that obtains a target speaker mask based on the visual embeddings, the target speaker embedding, and the output of a mask estimator from a previous layer; The audio decoder obtains a target speech signal based on a target speaker mask; an audio encoder of the second branch receives the difference between the mixed audio signal and the target speech signal estimated in the first branch, and extracts the audio embedding; The speaker encoder of the speaker extractor receives only the audio embeddings and outputs the interfering speaker embeddings, and the mask estimator in the speaker extractor obtains the interfering speaker mask based on the interfering speaker embeddings and the output of the mask estimator of the previous layer; The audio decoder obtains a separated interfering audio signal based on an interfering speaker mask; The audio encoders of the first and second branches generate time-domain audio signals by one-dimensional convolution. [0013] to an audio feature sequence X(t), and the conversion process can be expressed as: [0014] where N is the dimension of the audio embedding, T is the total number of audio samples, K=(2(T−L)) / L+1, and L is the size of the convolution kernel for one-dimensional convolution. The audio decoders of the first branch and the second branch perform an overlap-add operation to generate an audio feature sequence. [Equation 15] and the decoding process is as follows, [0016] In the training process of the target speech separation model, a multi-task learning framework is created, which includes three objective functions: The scale-invariant signal-to-noise ratio loss (SISNR) of the first branch is used to judge the signal quality between the extracted target speech and the pure target speech; The scale-invariant signal-to-noise ratio loss (SISNR) of the second branch is used to judge the signal quality between the extracted interference sound and the pure interference sound; The cross-modal loss is used to bring the target visual features and the estimated target speech signal audio features closer together and to move the target visual features and the estimated interfering speech signal audio features further apart. The loss is defined as: [Equation 17] Cross-modal loss simultaneously affects the training process of the two branches, performs metric learning between the visual features of the target speaker, the target speech features, and the interference speech features, and reduces the distance between positive samples and negative samples, thereby constraining the extraction of visual and auditory features. where γ is a scale factor, dp represents the cosine distance between the visual feature and the estimated target speech signal audio feature, dn represents the cosine distance between the visual feature and the estimated interference speech signal audio feature, dp, dn ∈ [0, 2], margin is the margin between the two distances, which is set to 0.5, A cross-modal loss-based target speech separation system is characterized in that it introduces a cross-modal loss into the target speech separation algorithm, which simultaneously affects the training process of the two branches, performs metric learning between the visual features of the target speaker, the target speech features, and the interference speech features, shortens the distance between positive samples and increases the distance between negative samples, constrains the extraction of visual and auditory features through audio-visual consistency, converts the interference speaker information into auxiliary information, and optimizes the speech extraction of the first branch.

5. 4. An electronic device comprising: a memory; a processor; and computer commands stored in the memory and executed by the processor, wherein when the computer commands are executed by the processor, the steps of the method of any one of claims 1 to 3 are completed.

Images