Audio separation method and device, electronic equipment and storage medium

Abstract

The invention relates to an audio separation method and device, electronic equipment and a storage medium. The audio separation method comprises the steps that audio and video data are acquired, target image features and mixed audio features of the audio and video data are determined, the audio and video data comprise human face pictures, the human face pictures comprise the target image features, and the mixed audio features comprise human voice audio features. Fusing the target image features and the mixed audio features to obtain fused features; and processing the fusion feature through a preset masking network to obtain a target audio feature, the preset masking network being used for obtaining the target audio feature from the fusion feature. And decoding the target audio feature to obtain a target audio, and outputting the target audio. According to the invention, voice separation is carried out according to the audio features and the video features in the audio and video data, and the accuracy of voice separation is ensured.

Description

Technical Field

[0001] This disclosure relates to the field of audio processing, and more particularly to audio separation methods, apparatus, electronic devices, and storage media. Background Technology

[0002] In real-world recordings of people speaking, the speaker's voice is often accompanied by acoustic interference. Acoustic interference refers to non-speech background noise and / or speech signals from non-speakers, which affects the viewer's ability to hear the speaker.

[0003] Among related technologies, joint audiovisual speech analysis is used to separate the voices of people in videos. Joint audiovisual speech analysis plays a crucial role in human speech communication systems, helping to focus attention on specific speakers and reduce the impact of acoustic background noise. Summary of the Invention

[0004] To overcome the problems existing in related technologies, this disclosure provides an audio separation method, apparatus, electronic device, and storage medium.

[0005] According to a first aspect of the present disclosure, an audio separation method is provided, comprising: acquiring audio and video data; determining target image features and mixed audio features of the audio and video data, wherein the audio and video data includes a person's facial image, the person's facial image includes target image features, and the mixed audio features include human voice audio features; fusing the target image features and the mixed audio features to obtain fused features; processing the fused features through a preset masking network to obtain target audio features, wherein the preset masking network is used to obtain target audio features from the fused features; decoding the target audio features to obtain target audio, and outputting the target audio.

[0006] In one embodiment, determining the target image features and mixed audio features of the audio and video data includes: converting the audio and video data according to a preset format to obtain converted audio and video data, and determining the frame rate of the audio and video data, wherein the format of the converted audio and video data conforms to the preset format; performing audio-video separation on the converted audio and video data to obtain the video data and audio data contained in the converted audio and video data, and determining the audio data contained in the converted audio and video data as the first audio data; determining the target image features in the video data corresponding to the facial image of the person according to the frame rate, and determining the mixed audio features of the first audio data according to the frame rate.

[0007] In one embodiment, determining the target image features corresponding to the facial image of the person in the video data based on the frame rate includes: acquiring the image of the person in each frame of the video data frame by frame according to the frame rate to obtain a target image, the target image including multiple frames arranged in sequence; acquiring the image features of the target image through a video encoder to obtain the target image features; determining the mixed audio features of the first audio data based on the frame rate includes: segmenting the first audio data according to the frame rate to obtain second audio data, the second audio data including multiple audio data segments of the first audio data arranged in sequence, each audio data segment of the multiple first audio data corresponding one-to-one with each frame of the multiple frames; acquiring the audio features of the second audio data through an audio encoder to obtain the mixed audio features.

[0008] In one embodiment, fusing the target image features and the mixed audio features to obtain fused features includes: fusing the target image features and the mixed audio features according to a multi-head attention mechanism to obtain fused features.

[0009] In one embodiment, processing the fused features to obtain target audio features includes: determining multiple mask matrices using a dual-path attention network and the fused features, wherein the multiple mask matrices correspond to facial images of people in the audio and video data; determining multiple audio features based on the correspondence between the multiple mask matrices and the fused features, and the multiple mask matrices, wherein the multiple audio features correspond to facial images of people in the audio and video data; and determining the target audio feature among the multiple audio features.

[0010] According to a second aspect of the present disclosure, an audio separation apparatus is provided, comprising: a determining unit, configured to acquire audio and video data, determine target image features and mixed audio features of the audio and video data, wherein the audio and video data includes a person's facial image, the person's facial image includes target image features, and the mixed audio features include human voice audio features; a fusing unit, configured to fuse the target image features and the mixed audio features to obtain fused features; a processing unit, configured to process the fused features through a preset masking network to obtain target audio features, wherein the preset masking network is used to obtain target audio features from the fused features; and an output unit, configured to decode the target audio features to obtain target audio and output the target audio.

[0011] In one embodiment, the determining unit determines the target image features and mixed audio features of the audio and video data in the following manner: The audio and video data are format-converted according to a preset format to obtain converted audio and video data, and the frame rate of the audio and video data is determined, wherein the format of the converted audio and video data conforms to the preset format; Audio and video separation is performed on the converted audio and video data to obtain the video data and audio data contained in the converted audio and video data, and the audio data contained in the converted audio and video data is determined as the first audio data; Based on the frame rate, the target image features corresponding to the facial image of the person in the video data are determined, and based on the frame rate, the mixed audio features of the first audio data are determined.

[0012] In one embodiment, the determining unit determines the target image features corresponding to the facial image of the person in the video data according to the frame rate in the following manner: Based on the frame rate, it acquires the image of the person in each frame of the video data frame by frame to obtain a target image, the target image comprising multiple frames arranged in sequence; it acquires the image features of the target image through a video encoder to obtain the target image features. The determining unit determines the mixed audio features of the first audio data according to the frame rate in the following manner: Based on the frame rate, it segments the first audio data to obtain second audio data, the second audio data comprising multiple audio data segments of the first audio data arranged in sequence, each audio data segment of the multiple first audio data corresponding one-to-one with each frame of the multiple frames; it acquires the audio features of the second audio data through an audio encoder to obtain the mixed audio features.

[0013] In one embodiment, the fusion unit fuses the target image features and the mixed audio features in the following manner to obtain fused features: according to the multi-head attention mechanism, the target image features and the mixed audio features are fused to obtain fused features.

[0014] In one embodiment, the processing unit processes the fused features in the following manner to obtain target audio features: using a dual-path attention network and the fused features, multiple mask matrices are determined, each mask matrix corresponding to a person's facial image in the audio / video data; based on the correspondence between the multiple mask matrices and the fused features, and the multiple mask matrices themselves, multiple audio features are determined, each audio feature corresponding to a person's facial image in the audio / video data; and a target audio feature is determined from among the multiple audio features.

[0015] According to a third aspect of the present disclosure, an electronic device is provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to: execute the audio separation method described in the first aspect or any embodiment of the first aspect.

[0016] According to a fourth aspect of the present disclosure, a storage medium is provided, the storage medium storing instructions that, when executed by a processor, enable the processor to perform the audio separation method described in the first aspect or any embodiment of the first aspect.

[0017] The technical solutions provided by the embodiments of this disclosure can include the following beneficial effects: After acquiring audio and video data, target image features corresponding to facial images are determined in the audio and video data, and audio features corresponding to human voices are determined. The target image features and mixed audio features are fused to obtain fused features. The fused features are processed through a preset masking network to obtain target audio features. The target audio features are decoded to obtain target audio, and the target audio is output. Through this disclosure, speech separation is performed based on audio and video features in the audio and video data, ensuring the accuracy of speech separation.

[0018] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0020] Figure 1 This is a flowchart illustrating an audio separation method according to an exemplary embodiment.

[0021] Figure 2 This is a block diagram illustrating an audio separation method according to an exemplary embodiment of the present disclosure.

[0022] Figure 3 This is a flowchart illustrating a method for determining target image features and mixed audio features of audio and video data according to an exemplary embodiment.

[0023] Figure 4 This is a flowchart illustrating a method for determining features of a target image according to an exemplary embodiment.

[0024] Figure 5 This is a flowchart illustrating a method for determining mixed audio features according to an exemplary embodiment.

[0025] Figure 6This is a flowchart illustrating a method for obtaining fusion features according to an exemplary embodiment.

[0026] Figure 7 This is a flowchart illustrating a method for acquiring target audio features according to an exemplary embodiment.

[0027] Figure 8 This is a flowchart illustrating an audio separation method according to an exemplary embodiment of the present disclosure.

[0028] Figure 9 This is a block diagram illustrating an audio separation device according to an exemplary embodiment.

[0029] Figure 10 This is a block diagram illustrating an audio separation apparatus according to an exemplary embodiment. Detailed Implementation

[0030] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure.

[0031] The video separation method provided in this disclosure is applied to scenarios where voice information of a target person in a video is obtained.

[0032] This disclosure primarily relates to the fields of speech separation, speaker separation, audio and video coding, and related artificial intelligence technologies. In particular, it relates to an audiovisual perception speaker locking and separation technology based on a time-domain transformer (Transformer). It can be extended to the fields of target speech detection, target speaker separation, and sound source separation.

[0033] In real-world environments, when a target person speaks, their speech is often accompanied by acoustic interference. This interference can be background noise (not spoken by the person) and / or speech signals from non-target speakers (such as those from competitors). Therefore, videos recorded in real-world scenarios will simultaneously contain the target person's speech and acoustic interference, making it difficult for other viewers to accurately hear the target person's voice. To improve the user's auditory experience, it is necessary to separate the audio signal in the video to obtain the target person's speech. Related technologies generally use sound separation methods based on deep-learning neural networks (DNNs) for this purpose.

[0034] In related technologies, audio separation techniques using deep-learning neural networks (DNNs) include methods for acquiring audio data from videos and performing sound separation (i.e., audio separation) based on this audio data, or even relying solely on the audio data contained within the audio and video data for audio separation. Audio separation methods relying solely on audio data use sound separation models built on deep neural networks to separate sound from mixed audio. These models include models using the time-domain-frequency domain (TF) signal of the audio signal and models using time-domain audio data. The former separates the speaker's speech by estimating the TF amplitude or amplitude-phase mask, while the latter performs sound separation (i.e., audio separation) based on time-domain audio data. However, audio separation methods relying solely on audio data can only achieve sound separation that meets user needs for sounds from different sound sources. In multi-person dialogues, separating the speech of different speakers based on audio data remains a challenge. When multiple speakers are present in mixed audio, audio separation techniques based on audio data cannot accurately capture the audio emitted by each speaker.

[0035] In related technologies, joint audiovisual speech analysis is used to separate the speech of a target person in video streams. Joint audiovisual speech analysis plays a crucial role in human speech communication systems, helping neural networks used for speech separation to focus on specific speakers and mitigate the impact of acoustic background noise. Furthermore, by integrating audio and visual information, it provides a more complete speech representation. For humans speaking in noisy environments, joint audiovisual speech analysis can improve the ability of neural network models to understand speech. However, video, as a multimodal data source, includes both audio and video modal data, requiring joint audiovisual speech analysis to consider both modalities simultaneously. Due to the complex relationship between audio and video modal data, effectively utilizing information from both modalities (i.e., audio and video modal data) remains a challenging task. It requires manual annotation to select target persons in the video stream and then correspondingly obtain their audio information; it cannot automatically obtain the audio information of the main speaker among multiple speakers. Furthermore, the relevant technologies have not been optimized for videos from different sources. Since the data formats of videos from different sources vary, audio separation processing for videos with different data formats requires audio separation models with a large amount of data, which leads to complex calculation processes. When the real-time requirements of the algorithm are high, more efficient algorithms and hardware support are needed.

[0036] In view of this, this disclosure proposes an audio separation method. After acquiring audio and video data, target image features corresponding to facial images of people are determined, and audio features corresponding to human voices are determined. The target image features and mixed audio features are fused to obtain fused features. The fused features are processed through a preset masking network to obtain target audio features. The target audio features are decoded to obtain and output the target audio. Through this disclosure, speech separation is performed based on audio and video features in the audio and video data, ensuring the accuracy of speech separation.

[0037] Figure 1 This is a flowchart illustrating an audio separation method according to an exemplary embodiment. Figure 1 As shown, the method includes steps S101 to S104.

[0038] In step S101, audio and video data are acquired, and the target image features and mixed audio features of the audio and video data are determined. The audio and video data includes a person's facial image, the person's facial image includes target image features, and the mixed audio features include human voice audio features.

[0039] In step S102, the target image features and the mixed audio features are fused to obtain the fused features.

[0040] In step S103, the fusion features are processed by a preset masking network to obtain the target audio features. The preset masking network is used to obtain the target audio features from the fusion features.

[0041] In step S104, the target audio features are decoded to obtain the target audio, and the target audio is output.

[0042] In this embodiment, the audio separation method for acquiring target audio is mainly used in scenarios such as video calls, video conferencing, and camera recording on electronic devices, and can also provide relevant functional interfaces for third-party apps. It primarily achieves voice separation of a target person based on audio and video data, and can send the separated audio data back to the original channel for subsequent downstream functions.

[0043] In this embodiment, the audio and video data for which audio separation is performed includes audio and video data from various sources, such as audio and video data recorded by the user or downloaded from various application software. It is understood that there is a correlation between the audio emitted by a person and the changes in their facial features (such as lip features, eye features, and overall facial features) when they emit the audio. Therefore, after acquiring the audio and video data, this disclosure uses a person recognition algorithm to identify the audio and video data frame by frame, determining the person's facial image in the audio and video data frame by frame, and extracting the person's facial image from the audio and video data frame by frame. By extracting features from each extracted frame of the person's facial image, the target image features corresponding to the person's facial image in the audio and video data are determined, and the audio features corresponding to the human voice data are also determined. The audio corresponding to the target person is obtained through an audio separation model that characterizes the correspondence between the audio emitted by a person and changes in the person's facial features.

[0044] In this embodiment, after acquiring target image features and mixed audio features, the two types of features are fused to obtain fused features. These fused features are then processed by a preset masking network to obtain target audio features. Finally, the target audio features are decoded to obtain the target audio, which is then output.

[0045] In this embodiment, considering that deep learning models based on the Transformer model can identify long-term correlations between audio and visual inputs, and that the parallel processing nature of the Transformer model makes it very suitable for audio-visual speech separation (AVSS, i.e., the audio separation scenario in this disclosure) applications requiring real-time processing, the Transformer model possesses powerful performance for sequential data processing tasks of speaker separation and audio-visual speech enhancement (AVSE). Therefore, the audio separation model used in this disclosure to perform the audio separation method is an audio separation model based on the Transformer model.

[0046] In an exemplary embodiment of this disclosure, audio data in audio and video data is defined as: Where S(t) represents the audio data and t represents the number of audio channels. This corresponds to audio from multiple speakers. n(t) represents background noise. The process of separating speaker speech and suppressing background noise by utilizing audio and combining it with video streams is represented as: S i = f(S,V), where V is the video stream, S is the audio stream, f represents a deep neural network parameterized by θ, whose goal is to remove background noise n(t) from the speech signal S(t), and θ represents all parameters in the deep neural network.

[0047] In an exemplary embodiment of this disclosure, a temporal audiovisual separation network based on a dual-path attention architecture is established, such as... Figure 2 The block diagram of the audio-video separation method shows that the separation of audio and video data is performed as follows: Audio and video data (such as user-recorded videos or audio-video data streams from other third-party applications) are acquired, along with the video stream (segmented facial images of the speaker) and audio stream (such as noisy mixed audio from multiple speakers). The audio stream is encoded to obtain audio features (corresponding to mixed audio features), and the video stream is encoded to obtain video features (corresponding to target video features). A fusion layer uses a cross-attention mechanism to combine the encoded audio and video features, and then fuses the video and audio features through an attention fusion layer to obtain the fused feature h. A masking network separates speaker features by estimating a separate mask for each speaker and applying it to the mixed feature matrix; that is, the masking network processes the fused features, and the target audio features are obtained based on the processed fused features and the original fused features. Finally, a decoding network composed of transposed convolutional layers processes the target audio features to obtain the temporal audio of the target speaker.

[0048] It is understood that the audio and video data in this disclosure includes audio and video data from various sources, and audio and video data from different sources have different formats. Therefore, it is necessary to unify audio and video data from different sources into a specific format. The following embodiments of this disclosure illustrate a method for determining target image features and mixed audio features of audio and video data.

[0049] Figure 3 This is a flowchart illustrating a method for determining target image features and mixed audio features of audio and video data according to an exemplary embodiment. Figure 3 As shown, the method includes steps S201 to S203.

[0050] In step S201, the audio and video data are converted according to a preset format to obtain the converted audio and video data, and the frame rate of the audio and video data is determined. The converted audio and video data format conforms to the preset format.

[0051] In step S202, the converted audio and video data is separated into audio and video data to obtain the video data and audio data contained in the converted audio and video data, and the audio data contained in the converted audio and video data is determined as the first audio data.

[0052] In step S203, the target image features corresponding to the facial images of the person in the video data are determined according to the frame rate, and the mixed audio features of the first audio data are determined according to the frame rate.

[0053] In this embodiment of the disclosure, audio and video data from multiple sources [such as audio and video data from third-party apps (e.g., social media videos, live streaming apps, etc.) and audio and video data from the mobile phone itself (remote audio and video calls, video conferencing)] are unified into the same data format (e.g., audio and video data are compressed to 16kHz, 8-bit data) to ensure data consistency before being fed into the audio separation model, thereby improving the versatility of this disclosure when used in various electronic devices.

[0054] It is understood that this disclosure converts the audio and video data into a specific data format before inputting it into the audio separation model. Correspondingly, the audio separation model used to perform audio separation is trained using audio and video data in that specific format. Given the use of audio and video data in a specific format as training data, the audio separation model in this disclosure has a smaller data volume and performs fewer computations during audio separation, thus ensuring that audio separation can be completed quickly and that the audio separation method in this disclosure performs audio separation in real time.

[0055] It is understood that a video consists of multiple consecutive image frames. Therefore, when performing feature extraction, it is necessary to determine the frame rate of the audio and video. Thus, this disclosure extracts mixed audio features and target image features frame by frame based on the frame rate. The following embodiments of this disclosure illustrate the method for determining target image features.

[0056] Figure 4 This is a flowchart illustrating a method for determining target image features according to an exemplary embodiment. Figure 4 As shown, the method includes steps S301 to S302.

[0057] In step S301, based on the frame rate, the image of the person in each frame of the video data is acquired frame by frame to obtain the target image, which includes multiple frames of images arranged in sequence.

[0058] In step S302, the image features of the target image are obtained by using a video encoder.

[0059] In this embodiment of the disclosure, the images of people in each frame of the video data are acquired sequentially, and multiple consecutive frames of images of people (target images) are input into the video encoder. The image features of the target images are obtained through the video encoder to obtain the target image features.

[0060] In an exemplary embodiment of this disclosure, for the video encoder, an audio encoder is configured using a model based on a specific architecture (such as the Lip2Wav architecture). The convolution kernel size in the video encoder is set to (1, 3). A single-pass Long Short-Term Memory (LSTM) is added after the second average pooling layer of the video encoder to achieve temporal dimension modeling of image features. By using residual features between modules to achieve cascading between modules of the video encoder, fast image feature extraction and streaming processing of the algorithm are achieved, ensuring the operating efficiency of the electronic device when performing audio separation on audio and video data. After passing through the video encoder, the speaker's face (i.e., the image of the person in each frame) presented in the mixed audio and video signal (i.e., audio and video data) is used as input V (T×3×200×200, i.e., time * number of image channels * image length * image width). A series of block sequences with 3D convolution, normalization, and rectified linear unit (ReLU) activation functions, as well as bidirectional LSTM, are used to further process the target image features. The encoded video matrices of all speakers are concatenated and represented as h′(Fv×Tv) in the feature dimension. The encoded video is upsampled in the time dimension to match the number of frames in the encoded audio. The final output is represented as hv(Fv×T′a) (i.e., target image features).

[0061] The following embodiments illustrate a method for determining mixed audio features.

[0062] Figure 5 This is a flowchart illustrating a method for determining mixed audio features according to an exemplary embodiment. Figure 5 As shown, the method includes steps S401 to S402.

[0063] In step S401, the first audio data is segmented according to the frame rate to obtain the second audio data. The second audio data includes multiple audio data segments of the first audio data arranged in sequence. Each audio data segment of the multiple first audio data corresponds one-to-one with each frame of the multiple frames.

[0064] In step S402, the audio features of the second audio data are obtained through the audio encoder to obtain the mixed audio features.

[0065] In this embodiment, audio signals and video frames are fed into separate encoders (audio encoder and video encoder) to transform the input data into a higher-dimensional feature space. The encoded video is upsampled to ensure frame count matching in the audio and visual features. The encoded audio and video are combined in an audiovisual fusion layer to produce fused features represented by the variable h. The fused features are then passed to a masking network to estimate the mask, and the estimated mask is multiplied by the fused feature matrix to separate the speaker's features.

[0066] Understandably, since the model output is an estimated waveform of a clean signal, this disclosure designs to train the model using speech level permutation invariant training (uPIT) and scale invariant signal-to-noise ratio (SI-SNR) loss functions.

[0067] In an exemplary embodiment of this disclosure, an audio encoder is constructed using one-dimensional convolution. The encoder uses convolution kernels of size k and strides of k / 2 to process mixed noise signals y in the time domain. The goal of constructing the audio encoder is to extract features from the noisy audio signal, denoted as hf(TxF).

[0068] In this embodiment of the disclosure, after obtaining the mixed audio features and the target image features, the mixed audio features and the target image features are fused to obtain fused features. The following embodiments of this disclosure illustrate the method for obtaining fused features.

[0069] Figure 6 This is a flowchart illustrating a method for obtaining fusion features according to an exemplary embodiment. Figure 6 As shown, the method includes steps S501 to S502.

[0070] In step S501, the target image features and mixed audio features of the audio and video data are determined.

[0071] In step S502, the target image features and mixed audio features are fused according to the multi-head attention mechanism to obtain fused features.

[0072] In this embodiment of the disclosure, after acquiring the target image features and mixed audio features of the audio and video data, the fusion of the target image features and mixed audio features is performed. An attention-based fusion network is used, and based on the attention mechanism, the target image features and mixed audio features are respectively input into separate multi-head attention blocks. The mixed audio features are designated as audio key vector Ka and audio digital vector Va, making the audio features the query vector Qa for the next multi-head attention MHA of the audio. Similarly, the target image features are designated as video key vector Kv and video digital vector Vv, making the target image features the query vector Qa for the next multi-head attention MHA of the video.

[0073] Right now: Among them, f a ' udio f is the result of processing mixed audio features. v ' ideo The result of processing the target image features.

[0074] In this embodiment of the disclosure, after obtaining the fusion features, a mask matrix corresponding to the target person is obtained. Based on the obtained mask matrix and the fusion features, the target audio features are obtained. The following embodiments of this disclosure illustrate the method for obtaining target audio features.

[0075] Figure 7 This is a flowchart illustrating a method for acquiring target audio features according to an exemplary embodiment. Figure 7 As shown, the method includes steps S601 to S603.

[0076] In step S601, multiple mask matrices are determined through a dual-path attention network and fusion features. These multiple mask matrices correspond to facial images of people in the audio and video data.

[0077] In step S602, multiple audio features are determined based on the correspondence between multiple mask matrices and fused features, and the multiple mask matrices themselves. These multiple audio features correspond to facial images of people in the audio and video data.

[0078] In step S603, the target audio feature is determined from among multiple audio features.

[0079] In this embodiment, a masking matrix is ​​generated for each speaker using a dual-path attention network. The fused audiovisual features are normalized and the result is passed to a linear layer. The output of the linear layer is segmented into temporal blocks, with overlapping regions between each pair of consecutive blocks (e.g., 50% overlap between each pair). The segmented temporal blocks are processed using a dual-path attention layer, which consists of two Transformer blocks with similar structures. The difference between the two Transformer blocks is that one block processes the time series, while the other processes the feature sequence. Each Transformer block consists of a layer normalization and a self-attention block, followed by another set of layer normalization and a feedforward neural network (FFN) layer. This disclosure optimizes training efficiency by introducing residual connections, allowing the gradients of the Transformer blocks to flow through each layer. The output of the feedforward neural network layer is fed into the overlapping summation block and the linear layer. The linear layer is combined with an activation function (such as ReLU) to estimate the mask matrix corresponding to each speaker. The masking network separates speaker features by estimating individual masks for each speaker and applying them to a hybrid feature matrix. Finally, the separated features (i.e., the target audio features) are transformed into the temporal domain in a decoder layer consisting of transposed convolutional layers.

[0080] In an exemplary embodiment of this disclosure, during model training, the influence of visual data on model performance is increased by dynamically scheduling data with different SNRs. The training process begins with a high level of background noise, which is gradually reduced over time. The information carried by the visual stream remains intact because it is unaffected by background noise or reverberation. The high noise level prevents audio features from dominating the model input, thus allowing the visual modality to play a more significant role in the early stages of training. As training progresses, the background noise energy level is gradually reduced, allowing audio features to regain dominance.

[0081] In an exemplary embodiment of this disclosure, the audio encoder is configured with 128 convolutional filters, a kernel size of 8, and a stride of 4. The video encoder has kernel size and stride values ​​of (1, 3) and (2, 2), respectively. A bidirectional LSTM is used in the last layer of the video encoder, with a hidden state size of 192. After concatenating the encoded features of the two speakers, a vector with a feature size of 384 is obtained. Partial model structure is shown below. Figure 2 As shown. During the training phase, the Adam optimizer was used with a learning rate of 0.0001. When the validation performance did not improve for five consecutive epochs, the learning rate was halved, and the model was trained using a halved batch size.

[0082] In an exemplary embodiment of this disclosure, such as Figure 8As shown in the flowchart of the audio separation method, this disclosure employs the following approach to separate audio and obtain the target audio: After video encoding, target image features are obtained; after audio encoding, mixed audio features are obtained. The target image features are upsampled, and the mixed audio features and target image features are processed based on Multi-Head Attention (MHA). Specifically, the encoded audio and audio features are input into separate MHA blocks. The audio features are then designated as audio key vector Ka and audio digit vector Va, serving as the query vector Qa for the next MHA of the audio. Similarly, the video features are designated as video key vector Kv and video digit vector Vv, serving as the query vector Qa for the next MHA of the video. The mixed audio features processed by MHA and the target image features are fused to obtain fused features. The fused features are processed by MHA to obtain multi-head attention-processed fused features. The multi-head attention-processed fused features are processed using a Transformer model to obtain intermediate features. The intermediate features are processed using a deep convolutional network with activation functions (DNN+ReLU) to obtain the target audio mask.

[0083] In this embodiment, after acquiring audio and video data, target image features corresponding to facial images and audio features corresponding to human voices are determined. The target image features and mixed audio features are fused to obtain fused features. The fused features are processed through a preset masking network to obtain target audio features. The target audio features are decoded to obtain target audio, which is then output. This disclosure achieves speech separation based on audio and video features in the audio and video data, ensuring accuracy. It also reduces the performance requirements of the audio separation method on the device, proposing a temporal audiovisual integration model for single-channel speaker separation, used to implement audio-video-based speaker locking and separation functionality in electronic devices.

[0084] Based on the same concept, this disclosure also provides an audio separation device 100.

[0085] It is understood that the audio separation device 100 provided in this disclosure includes hardware structures and / or software modules corresponding to each function in order to achieve the above-mentioned functions. In conjunction with the units and algorithm steps of the various examples disclosed in this disclosure, this disclosure can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed by hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the technical solutions of this disclosure.

[0086] Figure 9 This is a block diagram illustrating an audio separation device 100 according to an exemplary embodiment. (Refer to...) Figure 9 The device includes a determining unit 101, a fusion unit 102, a processing unit 103, and an output unit 104.

[0087] The determining unit 101 is used to acquire audio and video data, determine the target image features and mixed audio features of the audio and video data, the audio and video data includes a person's facial image, the person's facial image includes target image features, and the mixed audio features include human voice audio features.

[0088] The fusion unit 102 is used to fuse target image features and mixed audio features to obtain fused features.

[0089] The processing unit 103 is used to process the fusion features through a preset masking network to obtain target audio features. The preset masking network is used to obtain target audio features from the fusion features.

[0090] The output unit 104 is used to decode the target audio features, obtain the target audio, and output the target audio.

[0091] In one embodiment, the determining unit 101 determines the target image features and mixed audio features of the audio and video data in the following manner: The audio and video data are converted according to a preset format to obtain converted audio and video data, and the frame rate of the audio and video data is determined. The format of the converted audio and video data conforms to the preset format. Audio and video separation is performed on the converted audio and video data to obtain the video data and audio data contained in the converted audio and video data, and the audio data contained in the converted audio and video data is determined as the first audio data. Based on the frame rate, the target image features corresponding to the facial image of a person in the video data are determined, and based on the frame rate, the mixed audio features of the first audio data are determined.

[0092] In one embodiment, the determining unit 101 determines the target image features corresponding to a person's face in the video data according to the frame rate as follows: Based on the frame rate, the person's image in each frame of the video data is acquired frame by frame to obtain the target image, which includes multiple frames arranged in sequence. The image features of the target image are obtained through a video encoder to obtain the target image features. The determining unit 101 determines the mixed audio features of the first audio data according to the frame rate as follows: Based on the frame rate, the first audio data is segmented to obtain second audio data, which includes multiple audio data segments of the first audio data arranged in sequence, each audio data segment corresponding one-to-one with each frame in the multiple frames. The audio features of the second audio data are obtained through an audio encoder to obtain the mixed audio features.

[0093] In one embodiment, the fusion unit 102 fuses the target image features and the mixed audio features in the following manner to obtain fused features: according to the multi-head attention mechanism, the target image features and the mixed audio features are fused to obtain fused features.

[0094] In one embodiment, the processing unit 103 processes the fused features to obtain the target audio features as follows: Multiple mask matrices are determined using a dual-path attention network and the fused features, each mask matrix corresponding to a facial image in the audio / video data. Based on the correspondence between the multiple mask matrices and the fused features, and the multiple mask matrices themselves, multiple audio features are determined, each audio feature corresponding to a facial image in the audio / video data. The target audio feature is then determined from among the multiple audio features.

[0095] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0096] Figure 10 This is a block diagram illustrating an audio separation device 200 according to an exemplary embodiment. The device 200 can be provided as a terminal. For example, the device 200 can be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0097] Reference Figure 10 The device 200 may include one or more of the following components: processing component 202, memory 204, power component 206, multimedia component 208, audio component 210, input / output (I / O) interface 212, sensor component 214, and communication component 216.

[0098] Processing component 202 typically controls the overall operation of device 200, such as operations associated with display, telephone calls, data communication, camera operation, and recording. Processing component 202 may include one or more processors 220 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 202 may include one or more modules to facilitate interaction between processing component 202 and other components. For example, processing component 202 may include a multimedia module to facilitate interaction between multimedia component 208 and processing component 202.

[0099] Memory 204 is configured to store various types of data to support the operation of device 200. Examples of such data include instructions for any application or method operating on device 200, contact data, phonebook data, messages, pictures, videos, etc. Memory 204 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0100] The power supply component 206 provides power to the various components of the device 200. The power supply component 206 may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power to the device 200.

[0101] Multimedia component 208 includes a screen that provides an output interface between the device 200 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 208 includes a front-facing camera and / or a rear-facing camera. When the device 200 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0102] Audio component 210 is configured to output and / or input audio signals. For example, audio component 210 includes a microphone (MIC) configured to receive external audio signals when device 200 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 204 or transmitted via communication component 216. In some embodiments, audio component 210 also includes a speaker for outputting audio signals.

[0103] I / O interface 212 provides an interface between processing component 202 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0104] Sensor assembly 214 includes one or more sensors for providing status assessments of various aspects of device 200. For example, sensor assembly 214 may detect the on / off state of device 200, the relative positioning of components such as the display and keypad of device 200, changes in the position of device 200 or a component of device 200, the presence or absence of user contact with device 200, the orientation or acceleration / deceleration of device 200, and temperature changes of device 200. Sensor assembly 214 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 214 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 214 may also include an accelerometer, a gyroscope, a magnetometer, a pressure sensor, or a temperature sensor.

[0105] Communication component 216 is configured to facilitate wired or wireless communication between device 200 and other devices. Device 200 can access wireless networks based on communication standards, such as WiFi, 2G, or 3G, or combinations thereof. In one exemplary embodiment, communication component 216 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 216 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0106] In an exemplary embodiment, the apparatus 200 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0107] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 204 including instructions, which can be executed by a processor 220 of the device 200 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0108] It is understood that in this disclosure, "multiple" refers to two or more, and other quantifiers are similar. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. The singular forms "a," "the," and "the" are also intended to include the plural forms unless the context clearly indicates otherwise.

[0109] It is further understood that the terms "first," "second," etc., are used to describe various types of information, but this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another, and do not indicate a specific order or degree of importance. In fact, the expressions "first," "second," etc., are completely interchangeable. For example, without departing from the scope of this disclosure, first information can also be referred to as second information, and similarly, second information can also be referred to as first information.

[0110] It is further understood that the terms “center,” “longitudinal,” “lateral,” “front,” “rear,” “up,” “down,” “left,” “right,” “vertical,” “horizontal,” “top,” “bottom,” “inner,” and “outer,” etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this embodiment and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation.

[0111] It can be further understood that, unless otherwise specified, "connection" includes both direct connections where no other components exist between the two parties and indirect connections where other components exist between them.

[0112] It is further understood that although operations are described in a specific order in the accompanying drawings in the embodiments of this disclosure, this should not be construed as requiring these operations to be performed in the specific order or serial order shown, or requiring all of the shown operations to be performed to obtain the desired result. In certain environments, multitasking and parallel processing may be advantageous.

[0113] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein.

[0114] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. An audio separation method, characterized in that, include: Acquire audio and video data, determine the target image features and mixed audio features of the audio and video data, wherein the audio and video data includes a person's facial image, the person's facial image includes target image features, and the mixed audio features include human voice audio features; The target image features and the mixed audio features are fused to obtain the fused features; The fusion features are processed by a preset masking network to obtain target audio features, wherein the preset masking network is used to extract target audio features from the fusion features; The target audio features are decoded to obtain the target audio, and the target audio is output.

2. The method according to claim 1, characterized in that, Determining the target image features and mixed audio features of the audio and video data includes: The audio and video data are converted according to a preset format to obtain converted audio and video data, and the frame rate of the audio and video data is determined. The format of the converted audio and video data conforms to the preset format. The converted audio and video data is separated into audio and video data to obtain the video data and audio data contained in the converted audio and video data, and the audio data contained in the converted audio and video data is determined as the first audio data; Based on the frame rate, target image features corresponding to the facial image of the person in the video data are determined, and mixed audio features of the first audio data are determined based on the frame rate.

3. The method according to claim 2, characterized in that, The step of determining the target image features in the video data corresponding to the facial image of the person based on the frame rate includes: Based on the frame rate, the image of the person in each frame of the video data is obtained frame by frame to obtain the target image, which includes multiple frames of images arranged in sequence. The target image features are obtained by acquiring the image features of the target image through a video encoder; Determining the mixed audio features of the first audio data based on the frame rate includes: According to the frame rate, the first audio data is segmented to obtain second audio data. The second audio data includes multiple audio data segments of the first audio data arranged in sequence. Each audio data segment of the multiple first audio data corresponds one-to-one with each frame of the multiple frames of images. The audio features of the second audio data are obtained through an audio encoder to obtain the mixed audio features.

4. The method according to claim 1, characterized in that, The fusion of the target image features and the mixed audio features to obtain fused features includes: Based on the multi-head attention mechanism, the target image features and the mixed audio features are fused to obtain fused features.

5. The method according to claim 1, characterized in that, The process of processing the fused features to obtain target audio features includes: Multiple mask matrices are determined using a dual-path attention network and the fused features, and these multiple mask matrices correspond to facial images of people in the audio and video data; Based on the correspondence between the multiple mask matrices and the fused features, and the multiple mask matrices, multiple audio features are determined, and the multiple audio features correspond to the facial images of people in the audio and video data; Identify the target audio feature among the plurality of audio features.

6. An audio separation device, characterized in that, include: A determining unit is used to acquire audio and video data, determine the target image features and mixed audio features of the audio and video data, wherein the audio and video data includes a person's facial image, the person's facial image includes target image features, and the mixed audio features include human voice audio features; The fusion unit is used to fuse the target image features and the mixed audio features to obtain fused features; The processing unit is used to process the fusion features through a preset masking network to obtain target audio features, wherein the preset masking network is used to obtain target audio features from the fusion features; The output unit is used to decode the target audio features to obtain the target audio and output the target audio.

7. The apparatus according to claim 6, characterized in that, The determining unit determines the target image features and mixed audio features of the audio and video data in the following manner: The audio and video data are converted according to a preset format to obtain converted audio and video data, and the frame rate of the audio and video data is determined. The format of the converted audio and video data conforms to the preset format. The converted audio and video data is separated into audio and video data to obtain the video data and audio data contained in the converted audio and video data, and the audio data contained in the converted audio and video data is determined as the first audio data; Based on the frame rate, target image features corresponding to the facial image of the person in the video data are determined, and mixed audio features of the first audio data are determined based on the frame rate.

8. The apparatus according to claim 7, characterized in that, The determining unit determines the target image features corresponding to the person's facial image in the video data based on the frame rate in the following manner: Based on the frame rate, the image of the person in each frame of the video data is obtained frame by frame to obtain the target image, which includes multiple frames of images arranged in sequence. The target image features are obtained by acquiring the image features of the target image through a video encoder; The determining unit determines the mixed audio features of the first audio data based on the frame rate in the following manner: According to the frame rate, the first audio data is segmented to obtain second audio data. The second audio data includes multiple audio data segments of the first audio data arranged in sequence. Each audio data segment of the multiple first audio data corresponds one-to-one with each frame of the multiple frames of images. The audio features of the second audio data are obtained through an audio encoder to obtain the mixed audio features.

9. The apparatus according to claim 6, characterized in that, The fusion unit fuses the target image features and the mixed audio features in the following manner to obtain fused features: Based on the multi-head attention mechanism, the target image features and the mixed audio features are fused to obtain fused features.

10. The apparatus according to claim 6, characterized in that, The processing unit processes the fused features in the following manner to obtain the target audio features: Multiple mask matrices are determined using a dual-path attention network and the fused features, and these multiple mask matrices correspond to facial images of people in the audio and video data; Based on the correspondence between the multiple mask matrices and the fused features, and the multiple mask matrices, multiple audio features are determined, and the multiple audio features correspond to the facial images of people in the audio and video data; Identify the target audio feature among the plurality of audio features.

11. An electronic device, characterized in that, include: processor: Memory used to store processor-executable instructions; The processor is configured to execute the audio separation method according to any one of claims 1 to 5.

12. A storage medium, characterized in that, The storage medium stores instructions that, when executed by a processor, enable the processor to perform the audio separation method according to any one of claims 1 to 5.

Images