Audio extraction method and device, electronic equipment and storage medium

Abstract

Embodiments of the present disclosure provide an audio extraction method and device, electronic equipment and storage medium. The method comprises: obtaining a to-be-processed mixed audio, the to-be-processed mixed audio mixed with target audio corresponding to a target audio event; performing short-time Fourier transform processing on the to-be-processed mixed audio to obtain an audio amplitude spectrum feature of the to-be-processed mixed audio; obtaining a to-be-processed image corresponding to the target audio event; inputting the to-be-processed image into an image feature extraction network for processing to obtain an image feature of the to-be-processed image; inputting the audio amplitude spectrum feature of the to-be-processed mixed audio and the image feature of the to-be-processed image into an audio separation model to obtain the target audio. The method can accurately extract a predicted audio corresponding to a target object category from a mixed audio, improving the accuracy of audio extraction.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and more specifically, to an audio extraction method, an audio extraction device, an electronic device, and a computer-readable storage medium. Background Technology

[0002] With the rapid development of computer technology, audio, as one of the main carriers of information transmission, plays an increasingly important role in practical production and daily life. In real life, target audio signals are easily interfered with by environmental and background audio, necessitating the extraction of target audio from mixed audio.

[0003] The audio extraction process in related technologies mainly focuses on processing the voice of "humans", such as speech separation, speech enhancement and target speaker separation. This audio extraction process is less accurate when extracting audio from mixed audio containing multiple object categories, and it cannot extract the target audio corresponding to a certain object category. Summary of the Invention

[0004] This disclosure provides an audio extraction method, an audio extraction device, an electronic device, and a computer-readable storage medium. The method uses image features corresponding to a target audio event as a driving image for the predicted audio corresponding to the target audio event extracted from mixed audio, enabling the audio separation model to extract the predicted audio corresponding to the target audio event more accurately from mixed audio, thereby improving the accuracy of audio extraction.

[0005] This disclosure provides a training method for an audio extraction model. The method includes: acquiring a mixed audio to be processed, wherein the mixed audio contains a target audio corresponding to a target audio event; performing a short-time Fourier transform on the mixed audio to be processed to obtain the audio amplitude spectrum features of the mixed audio to be processed; acquiring an image to be processed corresponding to the target audio event; inputting the image to be processed into an image feature extraction network for processing to obtain image features of the image to be processed; and inputting the audio amplitude spectrum features of the mixed audio to be processed and the image features of the image to be processed into an audio separation model to obtain the target audio.

[0006] In some exemplary embodiments of this disclosure, the step of inputting the audio amplitude spectrum features of the mixed audio to be processed and the image features of the image to be processed into an audio separation model to obtain the target audio includes: inputting the audio amplitude spectrum features of the mixed audio to be processed and the image features of the image to be processed into an audio separation model to obtain the audio amplitude spectrum features of the target audio; and performing inverse short-time Fourier transform processing on the audio amplitude spectrum features of the target audio to obtain the target audio.

[0007] In some exemplary embodiments of this disclosure, the audio separation model includes multiple fusion modules, including a first fusion module, at least one intermediate fusion module, and a last fusion module. The step of inputting the audio amplitude spectrum features of the mixed audio to be processed and the image features of the image to be processed into the audio separation model to obtain the audio amplitude spectrum features of the target audio includes: inputting the audio amplitude spectrum features of the mixed audio to be processed and the image features of the image to be processed into the first fusion module for processing to obtain the output features of the first fusion module; inputting the image features of the image to be processed and the output features of the previous fusion module of each intermediate fusion module into the intermediate fusion module for processing to obtain the output features of the intermediate fusion module; and inputting the image features of the image to be processed and the output features of the previous fusion module of the last fusion module into the last fusion module for processing to obtain the audio amplitude spectrum features of the target audio.

[0008] In some exemplary embodiments of this disclosure, the first fusion module includes a first feature linear modulation layer, a multi-head self-attention layer, a convolutional layer, and a second feature linear modulation layer; wherein, the step of inputting the audio amplitude spectrum features of the mixed audio to be processed and the image features of the image to be processed into the first fusion module for processing to obtain the output features of the first fusion module includes: processing the audio amplitude spectrum features of the mixed audio to be processed and the image features of the image to be processed through the first feature linear modulation layer to obtain a first intermediate feature; processing the first intermediate feature through the multi-head self-attention layer and the convolutional layer to obtain a second intermediate feature; and processing the second intermediate feature and the image features of the image to be processed through the second feature linear modulation layer to obtain the output features of the first fusion module.

[0009] In some exemplary embodiments of this disclosure, the audio separation model is trained by: acquiring a plurality of first training audios corresponding to the target audio event, a plurality of second training audios corresponding to other audio events, and a plurality of training images corresponding to the target audio event; randomly determining a target first training audio from the plurality of first training audios, randomly determining at least one target second training audio from the plurality of second training audios, and randomly determining a target training image from the plurality of training images; mixing the target first training audio and the at least one target second training audio to obtain a training mixed audio; and performing a short-time Fourier transform on the training mixed audio. The training mixed audio is processed by a Fourier transform to obtain the audio amplitude spectrum features; the target training image is input into the image feature extraction network for processing to obtain the image features of the target training image; the audio amplitude spectrum features of the training mixed audio and the image features of the target training image are input into the audio separation network to be trained for processing to obtain the audio amplitude spectrum features of the predicted audio; the audio amplitude spectrum features of the predicted audio are processed by an inverse short-time Fourier transform to obtain the predicted audio corresponding to the target audio event; the audio separation model to be trained is trained based on the target first training audio and the predicted audio to obtain the audio separation model.

[0010] In some exemplary embodiments of this disclosure, training the audio separation model to be trained based on the target first training audio and the predicted audio to obtain the audio separation model includes: determining a first loss based on the target first training audio and the predicted audio; performing a short-time Fourier transform on the target first training audio to obtain the audio amplitude spectrum features of the target first training audio; determining a second loss based on the audio amplitude spectrum features of the target first training audio and the audio amplitude spectrum features of the predicted audio; and adjusting the model parameters of the audio separation model to be trained based on the first loss and the second loss to obtain the audio separation model.

[0011] In some exemplary embodiments of this disclosure, the step of mixing the target first training audio and the at least one target second training audio to obtain training mixed audio includes: mixing the target first training audio and the plurality of target second training audio based on a random signal-to-noise ratio to obtain the training mixed audio; wherein the plurality of target second training audio corresponds to at least two other audio events.

[0012] This disclosure provides an audio extraction device, comprising: an acquisition module configured to acquire a mixed audio to be processed, wherein the mixed audio contains a target audio corresponding to a target audio event; a processing module configured to perform a short-time Fourier transform on the mixed audio to be processed to obtain the audio amplitude spectrum features of the mixed audio to be processed; the acquisition module is further configured to acquire an image to be processed corresponding to the target audio event; the processing module is further configured to input the image to be processed into an image feature extraction network for processing to obtain image features of the image to be processed; and the acquisition module is configured to input the audio amplitude spectrum features of the mixed audio to be processed and the image features of the image to be processed into an audio separation model to obtain the target audio.

[0013] This disclosure provides an electronic device, including: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute the executable instructions to implement any of the audio extraction methods described above.

[0014] This disclosure provides a computer-readable storage medium that, when executed by a processor of an electronic device, enables the electronic device to perform any of the audio extraction methods described above.

[0015] This disclosure provides a computer program product, including a computer program that, when executed by a processor, implements any of the above-described audio extraction methods.

[0016] The audio extraction method provided in this disclosure involves acquiring mixed audio to be processed, which includes target audio corresponding to a target audio event; acquiring an image to be processed corresponding to the target audio event; performing a short-time Fourier transform on the mixed audio to obtain the audio amplitude spectrum features of the mixed audio; inputting the image to be processed into an image feature extraction network for processing to obtain the image features of the image to be processed; and inputting the audio amplitude spectrum features of the mixed audio and the image features of the image to be processed into an audio separation model to obtain the target audio. This method, in the audio extraction process, only requires inputting a target image corresponding to the target audio event to extract the target audio corresponding to the target audio event from the mixed audio, thus improving the accuracy, usability, simplicity, and efficiency of audio separation.

[0017] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0019] Figure 1 A schematic diagram of an exemplary system architecture to which the audio extraction method of the present disclosure can be applied is shown.

[0020] Figure 2 This is a flowchart illustrating an audio extraction method according to an exemplary embodiment.

[0021] Figure 3 This is a flowchart illustrating the training of an audio separation model according to an exemplary embodiment.

[0022] Figure 4 This is a schematic diagram illustrating an audio extraction and processing procedure based on an example.

[0023] Figure 5 This is a schematic diagram of the structure of an image feature extraction network as shown in an example.

[0024] Figure 6 This is a schematic diagram of the structure of each fusion module based on an example of an audio separation model.

[0025] Figure 7 This is a block diagram illustrating an audio extraction device according to an exemplary embodiment.

[0026] Figure 8 This is a schematic diagram illustrating the structure of an electronic device suitable for implementing exemplary embodiments of the present disclosure, according to an exemplary embodiment. Detailed Implementation

[0027] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, they are provided so that this disclosure will be thorough and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted.

[0028] The features, structures, or characteristics described in this disclosure can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced with one or more specific details omitted, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this disclosure.

[0029] The accompanying drawings are merely illustrative of this disclosure, and the same reference numerals in the drawings denote the same or similar parts, thus omitting repeated descriptions of them. Some block diagrams shown in the drawings do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in at least one hardware module or integrated circuit, or in different network and / or processor devices and / or microcontroller devices.

[0030] The flowchart shown in the accompanying drawings is merely illustrative and does not necessarily include all content and steps, nor does it require execution in the described order. For example, some steps may be broken down, while others may be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0031] In this specification, the terms “a,” “an,” “the,” “the,” and “at least one” are used to indicate the presence of at least one element / component / etc.; the terms “comprising,” “including,” and “having” are used to indicate an open-ended inclusion and to mean that there may be other elements / components / etc. in addition to the listed elements / components / etc.; the terms “first,” “second,” and “third,” etc., are used only as markings and are not a limitation on the number of objects.

[0032] Figure 1 A schematic diagram of an exemplary system architecture to which the audio extraction method of the present disclosure can be applied is shown.

[0033] like Figure 1 As shown, the system architecture may include server 101, network 102, terminal device 103, terminal device 104, and terminal device 105. Network 102 serves as the medium for providing a communication link between terminal device 103, terminal device 104, or terminal device 105 and server 101. Network 102 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0034] Server 101 can be a server that provides various services, such as a back-end management server that supports the devices operated by users using terminal devices 103, 104, or 105. The back-end management server can analyze and process received requests and other data, and feed back the processing results to terminal devices 103, 104, or 105.

[0035] Terminal devices 103, 104, and 105 can be smartphones, tablets, laptops, desktop computers, smart speakers, wearable smart devices, virtual reality devices, augmented reality devices, etc., but are not limited to these.

[0036] In this embodiment of the present disclosure, server 101 can obtain mixed audio to be processed from terminal device 103, wherein the mixed audio contains target audio corresponding to the target audio event; server 101 can obtain image to be processed corresponding to the target audio event from terminal device 103; server 101 can perform short-time Fourier transform processing on the mixed audio to be processed to obtain the audio amplitude spectrum features of the mixed audio to be processed; input the image to be processed into an image feature extraction network for processing to obtain image features of the image to be processed; input the audio amplitude spectrum features of the mixed audio to be processed and the image features of the image to be processed into an audio separation model to obtain the target audio; server 101 can return the obtained target audio to terminal device 103.

[0037] It should be understood that Figure 1 The number of terminal devices 103, 104, 105, network 102, and server 101 in the diagram is merely illustrative. Server 101 can be a single physical server, a server cluster consisting of multiple servers, or a cloud server. Depending on actual needs, it can have any number of terminal devices, networks, and servers.

[0038] The audio extraction method in the exemplary embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings and examples. The method provided in the embodiments of this disclosure can be executed by any electronic device, such as the one described above. Figure 1 The server and / or terminal equipment in the process, but this disclosure does not limit this.

[0039] Figure 2 This is a flowchart illustrating an audio extraction method according to an exemplary embodiment. Figure 2 As shown, the audio extraction method provided in this embodiment may include the following steps.

[0040] In step S210, the mixed audio to be processed is obtained, which contains the target audio corresponding to the target audio event.

[0041] In this embodiment of the disclosure, the mixed audio to be processed contains at least two audio components, one of which is a target audio corresponding to a target audio event. The target audio event can be any type of audio event, referring to different categories of sounds, such as singing, car sounds, animal sounds, airplane sounds, etc. In the following example, a dog barking is used as an example of the target audio event, but this disclosure is not limited to this.

[0042] In step S220, a short-time Fourier transform is performed on the mixed audio to be processed to obtain the audio amplitude spectrum characteristics of the mixed audio.

[0043] Figure 4 This is a schematic diagram illustrating an audio extraction and processing procedure, for reference only. Figure 4 The image feature extraction network 410 can be a pre-trained image model, such as a ResNet (residual network), EfficientNet (a convolutional neural network), or a Vit model or a Swin model based on Transformer. The image feature extraction network 410 is used to extract image features. The audio separation model 420 can be composed of multiple stacked FiLmConformer structures, where the FiLmConformer structure is a fusion structure composed of FiLm (Feature-wise Linearly modulated) layers and Conformer structures. The FiLm layers can achieve cross-modal feature fusion and use the image features (embedding) of the target image to guide audio separation, realizing the sound selector function. The Conformer structure is a "sandwich" structure, sandwiching the convolutional module and the self-attention module between two fully connected layers (FFN). The audio separation model 420 can also be other neural network models with audio separation functions such as Conv-TasNet, Soundfilter, ResUNet, etc., which are not limited in this disclosure.

[0044] refer to Figure 4 For example, a short-time Fourier transform (STFT) can be performed on the mixed audio 403 to obtain the audio amplitude spectrum characteristics Y of the mixed audio. mag-mix Its formal description is as follows:

[0045] Y mag-mix =STFT(MixSignal) (1)

[0046] Where MixSignal represents the mixed audio to be processed, and STFT represents the short-time Fourier transform.

[0047] In step S230, the image to be processed corresponding to the target audio event is obtained.

[0048] In this embodiment of the disclosure, the image to be processed corresponding to the target audio event refers to any image that includes the object corresponding to the target audio event; for example, if the target audio event is a dog barking, then any image including a dog can be obtained; as another example, if the target audio event is an airplane sound, then any image including an airplane can be obtained.

[0049] In step S240, the image to be processed is input into the image feature extraction network for processing to obtain the image features of the image to be processed.

[0050] Continue to refer to Figure 4 The image to be processed 401 is input into the image feature extraction network 410 for feature extraction, converting the image to be processed into image features (embedding) 402 containing image content. This image feature can be used as a sound selector to drive the audio separation model 420 to extract the target audio from the mixed audio to be processed; its formulaic description is as follows:

[0051] Im = VisualBranch(Image) (2)

[0052] Where Image is the input image, VisualBranch represents the image feature extraction network, and Im is the image feature output by the image feature extraction network, i.e., the embedding extracted by the image feature extraction network.

[0053] Figure 5 This is a schematic diagram of the structure of an image feature extraction network as shown in the example. (Refer to...) Figure 5 In this embodiment of the present disclosure, the image feature extraction network 410 may be, for example, a ResNet or EfficientNet model, which includes multiple CNN (Convolutional Neural Network) blocks; or, the image feature extraction network 410 may be, for example, a Transformer-based Vit model or a Transformer-based Swin model, which includes multiple Transformer blocks.

[0054] In this embodiment of the disclosure, a pre-trained image feature extraction network is used to extract image features as a sound selector. The dimension of the image features does not depend on the number of audio events input during training. Adding an audio event does not require modifying the model or retraining the model, which greatly improves the model's generalization ability, scalability, and versatility.

[0055] In step S250, the audio amplitude spectrum features of the mixed audio to be processed and the image features of the image to be processed are input into the audio separation model to obtain the target audio.

[0056] In this embodiment of the disclosure, the audio separation model can be a trained neural network model. After the audio separation model is trained, the image features of the image to be processed corresponding to the target audio event and the audio amplitude spectrum features of the mixed audio to be processed are input into the trained audio separation model, so that the target audio corresponding to the target audio event can be extracted from the mixed audio to be processed.

[0057] For example, by inputting the image features of the image to be processed, which includes a dog, and the audio amplitude spectrum features of the mixed audio to be processed, which includes the dog barking, into the audio separation model, the dog barking can be extracted; as another example, by inputting the image features of the image to be processed, which includes an airplane, and the audio amplitude spectrum features of the mixed audio to be processed, which includes the airplane sound, into the trained audio separation model, the airplane sound can be extracted.

[0058] The audio extraction method provided in this disclosure involves acquiring mixed audio to be processed, which includes target audio corresponding to a target audio event; acquiring an image to be processed corresponding to the target audio event; performing a short-time Fourier transform on the mixed audio to obtain the audio amplitude spectrum features of the mixed audio; inputting the image to be processed into an image feature extraction network for processing to obtain the image features of the image to be processed; and inputting the audio amplitude spectrum features of the mixed audio and the image features of the image to be processed into an audio separation model to obtain the target audio. This method, in the audio extraction process, only requires inputting a target image corresponding to the target audio event to extract the target audio corresponding to the target audio event from the mixed audio, thus improving the accuracy, usability, simplicity, and efficiency of audio separation.

[0059] In an exemplary embodiment, the process of inputting the audio amplitude spectrum features of the mixed audio to be processed and the image features of the image to be processed into an audio separation model to obtain the target audio includes: inputting the audio amplitude spectrum features of the mixed audio to be processed and the image features of the image to be processed into an audio separation model to obtain the audio amplitude spectrum features of the target audio; and performing inverse short-time Fourier transform processing on the audio amplitude spectrum features of the target audio to obtain the target audio.

[0060] refer to Figure 4 The audio amplitude spectrum characteristics Y of the mixed audio to be processed after STFT transformation are obtained. mag-mix Image feature 402 is used as input to audio separation model 420, which separates the audio amplitude spectrum features of the target audio related to the sound selector (image feature 402); its formulaic description is as follows:

[0061] X = Separate(Y) mag-mix ,Im) (3)

[0062] Where X represents the audio amplitude spectrum feature of the target audio extracted from the mixed audio to be processed, and Separate represents the audio separation model.

[0063] After obtaining the audio amplitude spectrum features of the target audio through the audio separation model 420, the audio amplitude spectrum features of the target audio can be processed by inverse Fourier transform (iSTFT) to obtain the target audio 404 corresponding to the target audio event.

[0064] In this embodiment, the mixed audio to be processed is subjected to Fourier transform to obtain the audio amplitude spectrum features of the mixed audio to be processed; the audio amplitude spectrum features of the mixed audio to be processed and the image features of the image to be processed are input into the audio separation model to obtain the audio amplitude spectrum features of the target audio; the audio amplitude spectrum features of the target audio are subjected to inverse short-time Fourier transform to obtain the target audio; that is, the mixed audio to be processed is converted into a feature vector that is easier for the audio separation model to process. After the audio separation model outputs the audio amplitude spectrum features of the target audio, the audio amplitude spectrum features of the target audio are then converted into the target audio, which improves the processing efficiency of the audio extraction model.

[0065] In an exemplary embodiment, the audio separation model includes multiple fusion modules, which include a first fusion module, at least one intermediate fusion module, and a last fusion module. The process of inputting the audio amplitude spectrum features of the mixed audio to be processed and the image features of the image to be processed into the audio separation model to obtain the audio amplitude spectrum features of the target audio includes: inputting the audio amplitude spectrum features of the mixed audio to be processed and the image features of the image to be processed into the first fusion module for processing to obtain the output features of the first fusion module; inputting the image features of the image to be processed and the output features of the previous fusion module of each intermediate fusion module into the intermediate fusion module for processing to obtain the output features of the intermediate fusion module; and inputting the image features of the image to be processed and the output features of the previous fusion module of the last fusion module into the last fusion module for processing to obtain the audio amplitude spectrum features of the target audio.

[0066] Specifically, the first fusion module in the audio separation model takes as input the audio amplitude spectrum features of the mixed audio to be processed and the image features of the image to be processed. The other fusion modules take as input the output features of the previous fusion module and the image features of the image to be processed. The output features of the last module are the audio amplitude spectrum features of the target audio.

[0067] In this embodiment, the audio separation model includes multiple fusion modules. The audio amplitude spectrum features of the mixed audio to be processed and the image features of the image to be processed are input into a first fusion module for processing to obtain the output features of the first fusion module. The image features of the image to be processed and the output features of the previous fusion module of each intermediate fusion module are input into an intermediate fusion module for processing to obtain the output features of the intermediate fusion module. The image features of the image to be processed and the output features of the previous fusion module of the last fusion module are input into the last fusion module for processing to obtain the audio amplitude spectrum features of the target audio. This audio separation model processes image features and the mixed audio to be processed through multiple fusion modules, improving the accuracy of audio extraction and thus improving the accuracy of model training.

[0068] Figure 6 This is a schematic diagram of the structure of each fusion module based on an example audio separation model, see reference. Figure 6 In this embodiment of the present disclosure, each fusion module may include a first feature linear modulation layer, a multi-head self-attention layer, a convolutional layer and a second feature linear modulation layer, and may also include other modules, which are not limited in this disclosure; the first fusion module is used as an example for the following description, and other fusion modules are similar, which will not be described in detail in this disclosure.

[0069] In an exemplary embodiment, the audio amplitude spectrum features of the mixed audio to be processed and the image features of the image to be processed are input into a first fusion module for processing to obtain the output features of the first fusion module. This may include: processing the audio amplitude spectrum features of the mixed audio to be processed and the image features of the image to be processed through a first feature linear modulation layer to obtain a first intermediate feature; processing the first intermediate feature through a multi-head self-attention layer and a convolutional layer to obtain a second intermediate feature; and processing the second intermediate feature and the image features of the image to be processed through a second feature linear modulation layer to obtain the output features of the first fusion module.

[0070] For example, refer to Figure 6Following the feature processing order, the first fusion module may include a first fully connected layer (FFN), a first feature linear modulation layer (FiLm), a multi-head self-attention layer (MSA with RE), a convolutional layer, a second fully connected layer (FFN), and a second feature linear modulation layer (FiLm). The first fully connected layer can be connected to a first residual connection and layer normalization (Add&Norm) module, with the output of the first residual connection and layer normalization module serving as the input of the first feature linear modulation layer. The multi-head self-attention layer can be connected to a second residual connection and layer normalization module, with the output of the second residual connection and layer normalization module serving as the input of the convolutional layer. The convolutional layer can be connected to a third residual connection and layer normalization module, with the output of the third residual connection and layer normalization module serving as the input of the second fully connected layer. The second fully connected layer can be connected to a fourth residual connection and layer normalization module, with the output of the fourth residual connection and layer normalization module serving as the input of the second feature linear modulation layer.

[0071] Specifically, the audio amplitude spectrum features of the mixed audio can be input into the first fully connected layer of the first fusion module for processing. The output features of the first fully connected layer and the image features of the target image can be input into the first feature linear modulation layer of the first fusion module for processing. The output features of the first feature linear modulation layer can be input into the multi-head self-attention layer of the first fusion module for processing. The output features of the multi-head self-attention layer can be input into the convolutional layer of the first fusion module for processing. The output features of the convolutional layer can be input into the second fully connected layer of the first fusion module for processing. The output features of the second fully connected layer and the image features of the target image can be input into the second feature linear modulation layer of the first fusion module for processing, thereby obtaining the output feature vector of the first fusion module.

[0072] In this embodiment of the present disclosure, the output feature vector of the first fusion module can be used as the input feature vector of the second fusion module, the output feature vector of the second fusion module can be used as the input feature vector of the third fusion module, and so on, to obtain the output feature vector of the last fusion module, which is then used as the audio amplitude spectrum feature of the predicted audio.

[0073] The formulaic description of each fusion module is as follows:

[0074]

[0075]

[0076] z″=Con(BN(z′+MSA(z′))) (6)

[0077] output=FiLm(LN(z″+0.5×FFN(z”)),Im) (7)

[0078] Wherein, FFN() represents a fully connected layer, MAS() represents a multi-head attention layer, FiLm() represents a feature linear modulation layer, LN() represents layer normalization, BN() represents batch normalization, Im represents image features, and z represents the output of the previous fusion module (if it is the first fusion module, z is the audio amplitude spectrum feature Y of the mixed audio to be processed). mag-mix ).

[0079] The feature linear modulation layer achieves cross-modal feature fusion, uses image features to guide audio separation, and realizes the sound selector function. The formulaic description of the feature linear modulation layer is as follows:

[0080] G = w_w × Im; B = w_b × Im (8)

[0081]

[0082] in, z is the output of the layer normalization in the fusion module (i.e., the output of the first residual connection and the layer normalization module), z′ is the output of the characteristic linear modulation layer, and w_w and w_b are the parameters learned by the network.

[0083] In this embodiment of the disclosure, each fusion module of the audio separation model includes a first feature linear modulation layer, a multi-head self-attention layer, a convolutional layer, and a second feature linear modulation layer; by combining the feature linear modulation layer with the self-attention layer and the convolutional layer, cross-modal feature fusion is achieved, thereby improving the accuracy of audio separation.

[0084] Figure 3 This is a flowchart illustrating the training of an audio separation model according to an exemplary embodiment. Figure 3 As shown, the process of training the audio separation model provided in this embodiment may include the following steps.

[0085] In step S310, multiple first training audios corresponding to the target audio event, multiple second training audios corresponding to other audio events, and multiple training images corresponding to the target audio event are acquired.

[0086] In this embodiment of the present disclosure, during the current round of training, a plurality of first training audios corresponding to the target audio event are acquired, and a plurality of second audios corresponding to other audio events are acquired, wherein the plurality of second audios may correspond to one or more other audio events; a plurality of images corresponding to the target audio event are acquired, wherein the plurality of images include at least one object corresponding to the target audio event.

[0087] For example, in the current training round, dog barks are used as the target audio event, and human voices, car sounds, and airplane sounds are used as other audio events; multiple dog bark audios (from different dogs) are acquired and used as [A1, A2, ..., A...]. 100 [O1, O2, ..., O2] indicates that multiple audio signals corresponding to other audio events (e.g., audio from people, cars, and airplanes) are retrieved using [O1, O2, ..., O2]. 1000 [D1, D2, ..., D5] represents the number of dogs to be captured (each image may include a different dog).

[0088] In this embodiment of the disclosure, the target audio event in each round of training can be re-determined from multiple audio events until all audio events have been trained.

[0089] In step S320, a target first training audio is randomly selected from a plurality of first training audios, at least one target second training audio is randomly selected from a plurality of second training audios, and a target training image is randomly selected from a plurality of training images.

[0090] In this embodiment of the disclosure, in each training iteration of each round of training, a target first training audio is determined from a plurality of first training audios, at least one target second training audio is determined from a plurality of second training audios, and a target training image is determined from a plurality of training images, forming a set of training data. The target first training audio and the target second training audio in this set of training data are used to generate training mixed audio, which is then input into the audio separation model. The target training image is used as a sound selector (i.e., the target image is used as guidance information) to drive the extraction of the predicted target first training audio from the training mixed audio. The target first training audio is also used as a training label to determine whether the audio predicted by the model is accurate.

[0091] Specifically, in the current training iteration of the current round of training, a first training audio is randomly selected from multiple first training audios as the target first training audio, multiple second training audios are randomly selected from multiple second training audios as the target second training audio, and an image is randomly selected from multiple training images as the target training image. These are used as a set of training data for training. In the next training iteration of the current round of training, a first training audio is randomly selected again from multiple first training audios as the target first training audio, multiple second training audios are randomly selected again as the target second training audio, and a training image is randomly selected again from multiple training images as the target training image. This process is repeated N times to complete the current round of training, where N is an integer.

[0092] For example, in the current training iteration of the current round of training, from the audio of the dog barking [A1, A2, ..., A...] 100 Randomly select one as the target and use it as the first training audio A. i Randomly select one image D from multiple dog images [D1, D2, ..., D5] as the target training image. j ; From multiple audio sources corresponding to other audio events (e.g., audio from people, cars, and airplanes) [O1, O2, ..., O 1000 Randomly select one or more as the target second training audio O k O k There can be one or more, and k can be the sum of multiple [k1,k2,k3,…,km].

[0093] In step S330, the target first training audio and at least one target second training audio are mixed to obtain training mixed audio.

[0094] In this embodiment of the present disclosure, in each training session of each round of training, the randomly selected target first training audio and the randomly selected target second training audio can be mixed to obtain mixed audio, which is then input into the audio separation model.

[0095] In this embodiment of the disclosure, multiple target training second audios randomly selected from multiple second training audios can correspond to at least two other audio events. For example, the target second training audios may include human voices, car sounds, and airplane sounds. This allows the resulting mixed audio to contain multiple other audio events, enabling the trained audio separation model to extract the target audio corresponding to the target audio event from the mixed audio containing multiple other audio events.

[0096] In an exemplary embodiment, mixing the target first training audio and at least one target second training audio to obtain training mixed audio includes: mixing the target first training audio and multiple target second training audio based on a random signal-to-noise ratio to obtain training mixed audio; wherein the multiple target second training audios correspond to at least two other audio events.

[0097] In this embodiment of the disclosure, when multiple target second training audios are randomly selected and these multiple target second training audios correspond to at least two other audio events, the selected target first training audios and the target second training audios corresponding to each other audio event can be mixed based on a random signal-to-noise ratio, so that the resulting mixed audio contains multiple other audio events.

[0098] For example, in the current training iteration of the current round of training, the randomly selected target first training audio A can be used.i And randomly selected target second training audio O k The training mixed audio is obtained by mixing (which can be understood as addition) with arbitrary signal-to-noise ratios; in the next training round, the target first training audio A can be randomly selected again. i And the second training audio O, which was randomly selected as the target. k The signal-to-noise ratio (SNR) can be mixed to obtain training mixed audio; the SNR is randomly determined in each training session.

[0099] In this embodiment of the disclosure, during each training process, a target first training audio is determined from multiple first training audios, a target second training audio is determined from multiple second training audios, and a target training image is determined from multiple training images based on random selection. The target first training audio and multiple target second training audios are mixed based on random signal-to-noise ratios, which can make the obtained training data more comprehensive and random, thereby improving the training effect of the audio extraction model. In addition, multiple target second training audios are determined from multiple second training audios, and the multiple target second training audios correspond to at least two other audio events. This can make the obtained training mixed audio contain a variety of other audio events, so that the trained audio separation model can extract the target audio corresponding to the target audio event from the mixed audio containing a variety of other audio events.

[0100] In step S340, the training mixed audio is subjected to short-time Fourier transform processing to obtain the audio amplitude spectrum features of the training mixed audio.

[0101] In step S350, the target training image is input into the image feature extraction network for processing to obtain the image features of the target training image.

[0102] In step S360, the audio amplitude spectrum features of the training mixed audio and the image features of the target training image are input into the audio separation network to be trained for processing to obtain the audio amplitude spectrum features of the predicted audio.

[0103] In step S370, the audio amplitude spectrum features of the predicted audio are subjected to inverse short-time Fourier transform processing to obtain the predicted audio corresponding to the target audio event.

[0104] In this embodiment of the disclosure, the steps described above in the model training process are similar to the corresponding steps in the model application process, and will not be repeated here.

[0105] In step S380, the audio separation model to be trained is trained based on the target first training audio and the predicted audio to obtain the audio separation model.

[0106] In this embodiment of the present disclosure, in the current round of model training, the target first training audio can be used as the training label. The loss value is calculated based on the target first training audio and the predicted audio extracted by the audio separation model. The model parameters of the audio separation model are adjusted according to the loss value until the loss value between the obtained predicted audio and the target first training audio is less than a preset value, and the current round of model training is completed. In the next round of model training, the target audio event is re-determined, and the audio separation model is trained again until all rounds of training are completed, and a trained audio separation model is obtained.

[0107] For example, in the current training round, dog barking is used as the target audio event, and human voices, car sounds, and airplane sounds are used as other audio events. The above steps are used to train the audio separation model so that it can extract dog barking from a mixed audio stream containing dog barking when an image of a dog is input. In the next training round, human voices are used as the target audio event, and dog barking, car sounds, and airplane sounds are used as other audio events. The above steps are used to train the audio separation model so that it can extract human voices from a mixed audio stream containing human voices when an image of a human is input. This process continues until all preset audio events have been trained. The resulting audio separation model can then extract the target audio corresponding to the object in the mixed audio stream based on the object included in the input image.

[0108] The audio extraction method provided in this disclosure determines a target first training audio as a training label from multiple first training audios corresponding to a target audio event; determines a target second training audio from multiple second training audios corresponding to other audio events, mixes the target second training audio with the target first training audio to obtain a training mixed audio, and uses it as input to an audio separation model; and determines a target training image from multiple training images corresponding to the target audio event as input to the audio separation model, thereby driving the audio separation model to extract the predicted audio corresponding to the target audio event from the training mixed audio. On the one hand, this method uses image features corresponding to the target audio event as the driving image for the predicted audio corresponding to the target audio event extracted from the training mixed audio, enabling the audio separation model to extract the predicted audio corresponding to the target audio event more accurately from the training mixed audio, thus improving the accuracy of the audio separation model training. On the other hand, the audio separation model obtained using this method can extract the target audio corresponding to the audio event to be extracted from the mixed audio by inputting a single target image corresponding to the audio event to be extracted in practical applications, improving the usability and simplicity of audio separation, thereby increasing audio separation efficiency.

[0109] In an exemplary embodiment, training an audio separation model to be trained based on a target first training audio and a predicted audio to obtain an audio separation model includes: determining a first loss based on the target first training audio and the predicted audio; performing a short-time Fourier transform on the target first training audio to obtain the audio amplitude spectrum features of the target first training audio; determining a second loss based on the audio amplitude spectrum features of the target first training audio and the audio amplitude spectrum features of the predicted audio; and adjusting the model parameters of the audio separation model to be trained based on the first loss and the second loss to obtain the audio separation model.

[0110] In this embodiment, a first loss can be determined based on the difference between the target first training audio and the predicted audio. For example, the SNR (signal-to-noise ratio) loss function can be used to calculate the first loss between the target first training audio and the predicted audio. The target first training audio can be subjected to Fourier transform processing to obtain the audio amplitude spectrum features of the target first training audio. A second loss can be determined based on the difference between the audio amplitude spectrum features of the target first training audio and the audio amplitude spectrum features of the predicted audio. For example, the L1 loss function (absolute value loss function) can be used to calculate the second loss between the audio amplitude spectrum features of the first training audio and the audio amplitude spectrum features of the predicted audio. The model parameters of the audio extraction model are adjusted based on the total loss of the first loss and the second loss. After each adjustment, the first loss and the second loss are recalculated until the training termination condition is met to obtain a trained audio extraction model. The training termination condition can be that the total loss of the first loss and the second loss is less than a preset value, or that the number of training iterations reaches a preset number.

[0111] Specifically, the loss function can be calculated by taking the mean absolute error (L1 loss) of the complex domain features and combining it with the signal-to-noise ratio (SNR) loss in the time domain. The L1 loss and SNR loss require that the energy and direction of the clean signal and the estimated signal from each source be essentially consistent; their formulas are as follows:

[0112]

[0113] Where α is a hyperparameter used to balance each loss term in the loss function, making these losses numerically uniform; S represents the audio amplitude spectrum feature of the first training audio. This represents the audio amplitude spectrum characteristics of the predicted audio estimated by the model. Let s represent the L1 loss and s represent the target first training audio. This represents the predicted audio value estimated by the model. SNR loss represents the difference between the target first training audio and the predicted audio.

[0114] In this embodiment of the disclosure, a first loss is determined based on the target first training audio and the predicted audio, and a second loss is determined based on the audio amplitude spectrum characteristics of the target first training audio and the predicted audio. The model parameters of the audio separation model are adjusted based on the first loss and the second loss, so that the energy and direction of the predicted audio extracted by the trained audio separation model are basically consistent with those of the target first training audio, thereby improving the accuracy of the audio separation model.

[0115] In this embodiment, using 20 common events from an audioSet (audio classification dataset), the image-driven general audio extraction method provided in this embodiment achieves a separation performance of over 10.1 dB under the conditions of using Efficient-b3 as the image feature extraction network, FiLmConformer structure as the audio separation network, and a parameter count of 43M. This demonstrates that the image-driven general audio extraction method provided in this embodiment is universal and effective. The specific event names, number of images, and audio segments are shown in Table 1.

[0116] Table 1

[0117]

[0118]

[0119] In the embodiments of this disclosure, the image feature extraction network in the audio extraction model can be ResNet, EfficientNet, Transformer-based Vit, Transformer-based Swin, CLIP-ResNet, or AudioCLIP-ResNet, and the audio separation network can be Conv-TasNet, Soundfilter, ResUNet, or FiLmConformer. Tables 2 and 3 show the separation performance of the audio extraction model trained using different image feature extraction networks and different audio separation networks, respectively.

[0120] Table 2

[0121]

[0122]

[0123] Table 3

[0124] Efficient-b3 Conv-TasNet 34 8.7 Efficient-b3 Soundfilter 30 8.7 Efficient-b3 ResUNet 76 9.4 Efficient-b3 FiLmConformer 43 10.1

[0125] As can be seen from Tables 2 and 3, the image-driven general audio separation method provided in this embodiment of the present disclosure, while keeping the structure of the audio separation network unchanged, achieves a separation performance of more than 8dB when using different image pre-trained models as image feature extraction models; and while keeping the image feature extraction model unchanged, achieves a separation performance of more than 8.7dB when using different audio separation networks.

[0126] It should also be understood that the above is only to help those skilled in the art better understand the embodiments of this disclosure, and is not intended to limit the scope of the embodiments of this disclosure. Those skilled in the art can obviously make various equivalent modifications or changes based on the examples given above. For example, some steps in the above methods may be unnecessary, or new steps may be added, etc. Alternatively, any combination of any two or more of the above embodiments may be used. Such modifications, changes, or combinations also fall within the scope of the embodiments of this disclosure.

[0127] It should also be understood that the above description of the embodiments of this disclosure focuses on highlighting the differences between the various embodiments. Similarities or differences not mentioned can be referred to each other, and for the sake of brevity, they will not be repeated here.

[0128] It should also be understood that the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this disclosure.

[0129] It should also be understood that, in the various embodiments of this disclosure, unless otherwise specified or in case of logical conflict, the terminology and / or descriptions between different embodiments are consistent and can be referenced by each other, and the technical features in different embodiments can be combined to form new embodiments according to their inherent logical relationships.

[0130] The foregoing section detailed examples of training methods for the audio extraction model provided in this disclosure. It is understood that, in order to implement the above functions, computer devices include corresponding hardware structures and / or software modules for performing each function. Those skilled in the art should readily recognize that, based on the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein, this disclosure can be implemented in hardware or a combination of hardware and computer software. Whether a function is implemented in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this disclosure.

[0131] The following are embodiments of the apparatus disclosed herein, which can be used to execute embodiments of the method disclosed herein. For details not disclosed in the apparatus embodiments of this disclosure, please refer to the embodiments of the method disclosed herein.

[0132] Figure 7 This is a block diagram illustrating an audio extraction apparatus according to an exemplary embodiment. (Refer to...) Figure 7 The device 700 may include an acquisition module 710, a processing module 720, and an acquisition module 730.

[0133] The acquisition module 710 is configured to acquire a mixed audio to be processed, wherein the mixed audio contains a target audio corresponding to a target audio event; the processing module 720 is configured to perform a short-time Fourier transform on the mixed audio to be processed to obtain the audio amplitude spectrum features of the mixed audio to be processed; the acquisition module 710 is further configured to acquire an image to be processed corresponding to the target audio event; the processing module 720 is further configured to input the image to be processed into an image feature extraction network for processing to obtain the image features of the image to be processed; and the acquisition module 730 is configured to input the audio amplitude spectrum features of the mixed audio to be processed and the image features of the image to be processed into an audio separation model to obtain the target audio.

[0134] In some exemplary embodiments of this disclosure, the processing module 720 is further configured to perform: inputting the audio amplitude spectrum features of the mixed audio to be processed and the image features of the image to be processed into an audio separation model to obtain the audio amplitude spectrum features of the target audio; and performing inverse short-time Fourier transform processing on the audio amplitude spectrum features of the target audio to obtain the target audio.

[0135] In some exemplary embodiments of this disclosure, the audio separation model includes multiple fusion modules, including a first fusion module, at least one intermediate fusion module, and a last fusion module; wherein, the processing module 720 is further configured to perform: inputting the audio amplitude spectrum features of the mixed audio to be processed and the image features of the image to be processed into the first fusion module for processing to obtain the output features of the first fusion module; inputting the image features of the image to be processed and the output features of the previous fusion module of each intermediate fusion module into the intermediate fusion module for processing to obtain the output features of the intermediate fusion module; and inputting the image features of the image to be processed and the output features of the previous fusion module of the last fusion module into the last fusion module for processing to obtain the audio amplitude spectrum features of the target audio.

[0136] In some exemplary embodiments of this disclosure, the first fusion module includes a first feature linear modulation layer, a multi-head self-attention layer, a convolutional layer, and a second feature linear modulation layer; wherein, the processing module 720 is further configured to perform: processing the audio amplitude spectrum features of the mixed audio to be processed and the image features of the image to be processed through the first feature linear modulation layer to obtain a first intermediate feature; processing the first intermediate feature through the multi-head self-attention layer and the convolutional layer to obtain a second intermediate feature; and processing the second intermediate feature and the image features of the image to be processed through the second feature linear modulation layer to obtain the output feature of the first fusion module.

[0137] In some exemplary embodiments of this disclosure, the audio separation model is trained by: acquiring a plurality of first training audios corresponding to the target audio event, a plurality of second training audios corresponding to other audio events, and a plurality of training images corresponding to the target audio event; randomly determining a target first training audio from the plurality of first training audios, randomly determining at least one target second training audio from the plurality of second training audios, and randomly determining a target training image from the plurality of training images; mixing the target first training audio and the at least one target second training audio to obtain a training mixed audio; and performing a short-time Fourier transform on the training mixed audio. The training mixed audio is processed by a Fourier transform to obtain the audio amplitude spectrum features; the target training image is input into the image feature extraction network for processing to obtain the image features of the target training image; the audio amplitude spectrum features of the training mixed audio and the image features of the target training image are input into the audio separation network to be trained for processing to obtain the audio amplitude spectrum features of the predicted audio; the audio amplitude spectrum features of the predicted audio are processed by an inverse short-time Fourier transform to obtain the predicted audio corresponding to the target audio event; the audio separation model to be trained is trained based on the target first training audio and the predicted audio to obtain the audio separation model.

[0138] In some exemplary embodiments of this disclosure, training the audio separation model to be trained based on the target first training audio and the predicted audio to obtain the audio separation model includes: determining a first loss based on the target first training audio and the predicted audio; performing a short-time Fourier transform on the target first training audio to obtain the audio amplitude spectrum features of the target first training audio; determining a second loss based on the audio amplitude spectrum features of the target first training audio and the audio amplitude spectrum features of the predicted audio; and adjusting the model parameters of the audio separation model to be trained based on the first loss and the second loss to obtain the audio separation model.

[0139] In some exemplary embodiments of this disclosure, the step of mixing the target first training audio and the at least one target second training audio to obtain training mixed audio includes: mixing the target first training audio and the plurality of target second training audio based on a random signal-to-noise ratio to obtain the training mixed audio; wherein the plurality of target second training audio corresponds to at least two other audio events.

[0140] It should be noted that the block diagrams shown in the above figures are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor terminal devices and / or microcontroller terminal devices.

[0141] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0142] The following reference Figure 8 To describe an electronic device 800 according to such an embodiment of the present disclosure. Figure 8 The electronic device 800 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.

[0143] like Figure 8 As shown, the electronic device 800 is presented in the form of a general-purpose computing device. The components of the electronic device 800 may include, but are not limited to: at least one processing unit 810, at least one storage unit 820, a bus 830 connecting different system components (including storage unit 820 and processing unit 810), and a display unit 840.

[0144] The storage unit stores program code, which can be executed by the processing unit 810 to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure. For example, the processing unit 810 can perform actions such as... Figure 2 The steps shown are as follows.

[0145] For example, electronic devices can achieve such Figure 2 The steps shown.

[0146] Storage unit 820 may include a readable medium in the form of a volatile storage unit, such as random access memory (RAM) 821 and / or cache memory 822, and may further include a read-only memory (ROM) 823.

[0147] The storage unit 820 may also include a program / utility 824 having a set (at least one) of program modules 825, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0148] Bus 830 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0149] Electronic device 800 can also communicate with one or more external devices 870 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 800, and / or with any device that enables electronic device 800 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 850. Furthermore, electronic device 800 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 860. As shown, network adapter 860 communicates with other modules of electronic device 800 via bus 830. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 800, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0150] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0151] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory including instructions that can be executed by a processor of the device to perform the described method. Optionally, the computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0152] In an exemplary embodiment, a computer program product is also provided, including a computer program / instructions, which, when executed by a processor, implement the training apparatus for the audio extraction model in the above embodiments.

[0153] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0154] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. An audio extraction method, characterized in that, include: Obtain the mixed audio to be processed, wherein the mixed audio to be processed contains target audio corresponding to the target audio event; The mixed audio to be processed is subjected to short-time Fourier transform to obtain the audio amplitude spectrum characteristics of the mixed audio to be processed; Obtain the image to be processed corresponding to the target audio event; The image to be processed is input into an image feature extraction network for processing to obtain the image features of the image to be processed; The audio amplitude spectrum features of the mixed audio to be processed and the image features of the image to be processed are input into the audio separation model. The image features and the mixed audio to be processed are processed by multiple fusion modules in the audio separation model to obtain the audio amplitude spectrum features of the target audio. The target audio is obtained by performing an inverse short-time Fourier transform on the audio amplitude spectrum features. In the audio separation model, the first fusion module takes the audio amplitude spectrum features of the mixed audio to be processed and the image features as inputs, the other fusion modules take the output features of the previous fusion module and the image features of the image to be processed as inputs, and the last fusion module takes the audio amplitude spectrum features of the target audio as output features. The first fusion module includes a first feature linear modulation layer, a multi-head self-attention layer, a convolutional layer, and a second feature linear modulation layer; wherein, obtaining the output features of the first fusion module includes: The first feature linear modulation layer processes the audio amplitude spectrum features of the mixed audio to be processed and the image features of the image to be processed to obtain a first intermediate feature; the first intermediate feature is processed by the multi-head self-attention layer and the convolutional layer to obtain a second intermediate feature; the second feature linear modulation layer processes the second intermediate feature and the image features of the image to be processed to obtain the output feature of the first fusion module.

2. The audio extraction method according to claim 1, characterized in that, The audio separation model includes multiple fusion modules, which include a first fusion module, at least one intermediate fusion module, and a last fusion module. The step of inputting the audio amplitude spectrum features of the mixed audio to be processed and the image features of the image to be processed into the audio separation model to obtain the audio amplitude spectrum features of the target audio includes: The audio amplitude spectrum features of the mixed audio to be processed and the image features of the image to be processed are input into the first fusion module for processing to obtain the output features of the first fusion module; The image features of the image to be processed and the output features of the previous fusion module of each intermediate fusion module are input into the intermediate fusion module for processing to obtain the output features of the intermediate fusion module; The image features of the image to be processed and the output features of the previous fusion module of the last fusion module are input into the last fusion module for processing to obtain the audio amplitude spectrum features of the target audio.

3. The audio extraction method according to claim 1, characterized in that, The audio separation model is trained in the following manner: Acquire multiple first training audios corresponding to the target audio event, multiple second training audios corresponding to other audio events, and multiple training images corresponding to the target audio event; Randomly determine a target first training audio from the plurality of first training audios, randomly determine at least one target second training audio from the plurality of second training audios, and randomly determine a target training image from the plurality of training images; The target first training audio and the at least one target second training audio are mixed to obtain training mixed audio; The training mixed audio is subjected to short-time Fourier transform processing to obtain the audio amplitude spectrum features of the training mixed audio; The target training image is input into the image feature extraction network for processing to obtain the image features of the target training image; The audio amplitude spectrum features of the training mixed audio and the image features of the target training image are input into the audio separation network to be trained for processing to obtain the audio amplitude spectrum features of the predicted audio. The inverse short-time Fourier transform is performed on the audio amplitude spectrum features of the predicted audio to obtain the predicted audio corresponding to the target audio event; The audio separation model to be trained is trained based on the target first training audio and the predicted audio to obtain the audio separation model.

4. The audio extraction method according to claim 3, characterized in that, The step of training the audio separation model to be trained based on the target first training audio and the predicted audio to obtain the audio separation model includes: A first loss is determined based on the target first training audio and the predicted audio; The target first training audio is subjected to short-time Fourier transform processing to obtain the audio amplitude spectrum features of the target first training audio; The second loss is determined based on the audio amplitude spectrum features of the first training audio and the audio amplitude spectrum features of the predicted audio. The model parameters of the audio separation model to be trained are adjusted based on the first loss and the second loss to obtain the audio separation model.

5. The audio extraction method according to claim 3, characterized in that, The step of mixing the target first training audio and the at least one target second training audio to obtain training mixed audio includes: The target first training audio and the plurality of target second training audios are mixed based on a random signal-to-noise ratio to obtain the training mixed audio; wherein the plurality of target second training audios correspond to at least two other audio events.

6. An audio extraction device, characterized in that, include: The acquisition module is configured to acquire mixed audio to be processed, wherein the mixed audio to be processed contains target audio corresponding to the target audio event; The processing module is configured to perform a short-time Fourier transform on the mixed audio to be processed to obtain the audio amplitude spectrum features of the mixed audio to be processed. The acquisition module is also configured to acquire the image to be processed corresponding to the target audio event; The processing module is further configured to perform processing on the image to be processed by an image feature extraction network to obtain the image features of the image to be processed; The acquisition module is configured to input the audio amplitude spectrum features of the mixed audio to be processed and the image features of the image to be processed into the audio separation model, and process the image features and the mixed audio to be processed through multiple fusion modules in the audio separation model to obtain the audio amplitude spectrum features of the target audio. The target audio is obtained by performing an inverse short-time Fourier transform on the audio amplitude spectrum features. In the audio separation model, the first fusion module takes the audio amplitude spectrum features of the mixed audio to be processed and the image features as inputs, the other fusion modules take the output features of the previous fusion module and the image features of the image to be processed as inputs, and the last fusion module takes the audio amplitude spectrum features of the target audio as output features. The first fusion module includes a first feature linear modulation layer, a multi-head self-attention layer, a convolutional layer, and a second feature linear modulation layer; wherein, the processing module is further configured to perform: The first feature linear modulation layer processes the audio amplitude spectrum features of the mixed audio to be processed and the image features of the image to be processed to obtain the first intermediate feature; The first intermediate feature is processed by the multi-head self-attention layer and the convolutional layer to obtain the second intermediate feature; The second feature linear modulation layer processes the second intermediate feature and the image features of the image to be processed to obtain the output features of the first fusion module.

7. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the executable instructions to implement the audio extraction method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, wherein instructions in the computer-readable storage medium, when executed by a processor of an electronic device, enable the electronic device to perform the audio extraction method as described in any one of claims 1 to 5.

Images