Unsupervised mixed audio cross-modal separation method, system and device and storage medium

Abstract

The invention discloses an unsupervised mixed audio cross-modal separation method, system and device, and a storage medium, and the method comprises the steps: detecting a sound source object in a video, and extracting a target object and visual features in the video through employing a target detection algorithm; preprocessing audio and video data; the cross-modal audio and visual semantic alignment learning comprises audio and video semantic consistency learning and audio and visual cross attention learning; performing unsupervised mixed audio preliminary separation to obtain a plurality of separated audio signals; performing independent adversarial learning, constructing a signal library, and setting a loss function to close the difference between the joint distribution of signals generated by the audio decoder and the joint distribution of samples in the random selection signal library so as to train the U-Net audio decoder to generate mutually independent separated audio signals; the invention relates to visually guided unsupervised audio separation. According to the method, on the basis of independent adversarial learning, independence of a single sound source generated after mixed audio separation and corresponding relevance between sound sources are considered, and unsupervised mixed audio separation is achieved.

Description

Technical Field

[0001] The present invention belongs to the technical field of audio processing, and in particular relates to an unsupervised mixed audio cross-modal separation method, system, device and storage medium. Background Art

[0002] Mixed audio separation technology is an advanced technology for processing and separating multiple sound sources. It aims to extract independent sound signals from mixed audio. This technology is widely used in speech recognition, music processing, audio editing, audio enhancement and other fields. However, mixed audio separation technology also faces many challenges. First of all, the audio signal is a complex time series signal. How to separate it without destroying the quality of the original audio is a difficult problem. Secondly, different sound source signals may have different spectral characteristics and time characteristics, which makes the separation process more complicated. In addition, there may be a variety of interference factors in actual applications, such as environmental noise, equipment quality, etc., which will affect the audio separation effect.

[0003] With the rapid development of artificial intelligence technology, audio separation technology has also made significant progress. Audio separation methods based on deep learning no longer rely on traditional signal processing technology and predefined statistical models. Instead, they collect a large amount of data to train the separation model and learn the inherent laws of audio signals, thereby achieving more accurate sound source separation. However, most existing separation models rely on a single real sound source as supervisory information, which is often scarce in practical applications. Summary of the invention

[0004] The main purpose of the present invention is to overcome the shortcomings and deficiencies of the prior art and to propose an unsupervised mixed audio cross-modal separation method, system, device and storage medium.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] An unsupervised mixed audio cross-modal separation method comprises the following steps:

[0007] Sound source object detection in the video: for videos containing mixed audio signals of multiple sound sources, target objects and visual features in the video are extracted using target detection algorithms;

[0008] Preprocessing of audio and video data, processing of audio mixed signals to obtain mixed audio features;

[0009] Cross-modal audio-visual semantic alignment learning, including audio-visual semantic consistency learning and audio-visual cross-attention learning; semantic alignment is performed through audio-video semantic consistency learning and the semantic expression ability of the two modalities of audio and video is enhanced; through audio-visual cross-attention learning, the characteristic semantics of the audio-visual modality is further refined;

[0010] Unsupervised mixed audio preliminary separation, using U-Net as an audio decoder to obtain several separated audio signals;

[0011] Independent adversarial learning, by randomly extracting signals from audio signals separated from different mixed audios to build a signal library, setting the loss function to bring the joint distribution of the signals generated by the audio decoder closer to the joint distribution of the samples in the randomly selected signal library, so as to train the U-Net audio decoder to generate independent separated audio signals;

[0012] Visually guided unsupervised audio separation uses visual target objects as virtual supervisory labels, sets the audio-visual semantic consistency discrimination loss function, and trains the U-Net audio decoder to generate audio signals corresponding to the target objects, thereby accurately separating the mixed audio.

[0013] The present invention also includes an unsupervised mixed audio cross-modal separation system, the system adopts the unsupervised mixed audio cross-modal separation method provided by the present invention, and the system includes sound source object detection, audio and video data preprocessing module, cross-modal audio-visual semantic alignment learning module, unsupervised mixed audio preliminary separation module, independent adversarial learning module and unsupervised audio separation module;

[0014] The sound source object detection module uses the target detection algorithm to extract the target object and visual features in the video;

[0015] The audio and video data preprocessing module processes the audio mixed signal to obtain the mixed audio features;

[0016] The cross-modal audio-visual semantic alignment learning module performs semantic alignment and enhances the semantic expression capabilities of the two modalities through audio-visual semantic consistency learning; and further refines the characteristic semantics of the audio-visual modality through audio-visual cross-attention learning;

[0017] Unsupervised mixed audio preliminary separation module, using U-Net as an audio decoder to obtain several separated audio signals;

[0018] An independent adversarial learning module is used to construct a signal library by randomly extracting signals from audio signals separated from different mixed audios, and to set a loss function to narrow the difference between the joint distribution of the signals generated by the audio decoder and the joint distribution of the samples in the randomly selected signal library, so as to train the U-Net audio decoder to generate independent separated audio signals;

[0019] The unsupervised audio separation module is used to use the visual target object as a virtual supervisory label, set the audio-visual semantic consistency discrimination loss function, train the U-Net audio decoder to generate the audio signal corresponding to the target object, and accurately separate the mixed audio.

[0020] The present invention also includes a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the unsupervised mixed audio cross-modal separation method provided by the present invention is implemented.

[0021] The present invention also includes a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the unsupervised mixed audio cross-modal separation method provided by the present invention is implemented.

[0022] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0023] 1. The present invention proposes an independent adversarial learning algorithm to balance the correlation and independence of the sound source after unsupervised separation, thereby improving the accuracy of unsupervised separation; the present invention adopts two processing methods to increase the independence of the separated audio signal; firstly, a mutual information loss function is set to minimize the mutual information of the separated audio to reduce the information overlap between the separated audio signals, thereby improving the independence of the separated audio; secondly, a reference library is formed by randomly selecting separated signals from different mixed audios; due to the random selection, the audio signals in the library are relatively independent; by projecting the feature distribution of the separated audio onto the feature distribution of the audio signal in the reference library, the mutual independence of the separated audio signals is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 is a flow chart of the method of the present invention;

[0025] Figure 2 Schematic diagram of cross-modal audio-visual semantic alignment learning in the present invention;

[0026] Figure 3 It is a schematic diagram of preliminary separation of unsupervised mixed audio in the present invention;

[0027] Figure 4 It is a schematic diagram of independent adversarial learning in the present invention. DETAILED DESCRIPTION

[0028] The present invention is further described in detail below in conjunction with embodiments and drawings, but the embodiments of the present invention are not limited thereto.

[0029] Example

[0030] like Figure 1 As shown, the present invention provides an unsupervised mixed audio cross-modal separation method, comprising the following steps:

[0031] S1, sound source object detection in the video, for the video containing audio mixed signals of multiple sound sources, the target object and visual features in the video are extracted using the target detection algorithm;

[0032] S2, audio and video data preprocessing, processing the audio mixed signal to obtain mixed audio features; taking the separation of mixed audio of two videos as an example, this embodiment is specifically as follows:

[0033] The audio signals in the two videos are set to A i1 ,A i2 , the audio mixed signal in the two videos is set to A mix =A i1 +A i2 ;

[0034] Obtaining the target object in the video and visual features After that, the mixed audio signal A is obtained by short-time Fourier transform (STFT) and Mel spectrum transform. mix Mel spectrum S mix , and S mix Input audio encoder to obtain mixed audio features

[0035] S3, cross-modal audio-visual semantic alignment learning, including audio-visual semantic consistency learning and audio-visual cross-attention learning;

[0036] Different from the traditional method that directly aligns audio and visual features by providing real semantic labels, the method of this embodiment uses an unsupervised learning method to perform modal semantic alignment and enhance the semantic expression capabilities of the two modalities of audio and visual through multimodal semantic consistency learning. Figure 2 As shown in Figure 2, audio and video semantic consistency learning is specifically as follows:

[0037] Fusion of target object visual features and audio mixing features Get fusion features

[0038] Contrastive learning is used to enhance the accuracy of visual features and to fuse features corresponding to the same sound source object. and the visual characteristics of the sound source object The InfoNCE is used as the contrastive learning loss function to optimize the visual feature expression. The contrastive learning loss function is:

[0039]

[0040] Where τ is the temperature hyperparameter, represents the negative sample space;

[0041] Audio-visual cross-attention learning is specifically:

[0042] like Figure 2 As shown, the enhanced fusion features and visual features It is used interactively as a query vector for interactive cross-modal attention learning. Through spatial attention and channel attention learning, the semantic consistency features of the audio-visual modality are further refined, the semantic accuracy of feature expression is improved, and the fusion of modal information is promoted. The fusion feature f with enhanced attention is output. c Among them, spatial attention aims to improve the feature expression of key spatial areas. During the calculation process, the original features are first pooled in the channel dimension, multi-channel features are spliced, and convolution and activation calculations are performed on the spliced ​​features. A weight mask is generated for each spatial position and weighted output, thereby enhancing the specific target area of ​​interest while weakening irrelevant background areas. When calculating channel attention, the spatial representation of the feature is compressed through average pooling, retaining the channel expression information, and the weight coefficient of each feature channel is automatically obtained through attention network learning to strengthen important channel features and suppress unimportant features.

[0043] S4, unsupervised mixed audio preliminary separation, using U-Net as an audio decoder to obtain several separated audio signals; Figure 3 As shown, specifically:

[0044] Use U-Net as the audio decoder to generate the target object with the sound source The corresponding separation mask By placing the mask Multiply it with the sound mixed spectrum to obtain the corresponding sound source target object Audio Mel Spectrum The calculation formula is:

[0045]

[0046] The spectrum Perform inverse transform from frequency domain to time domain to obtain time domain audio signal In order to avoid information loss during audio separation, the loss function L is set sep The signal-to-noise ratio (SNR) is set to evaluate the integrity and quality of the separated audio information generated by the U-Net audio decoder. The loss function L sep Specifically:

[0047]

[0048] Among them, A i is the real mixed audio data in the video, A i After unsupervised audio separation, several separated audio signals are obtained After integrating the separated audio signals, the mixed audio is

[0049] Mixed with real audio data A i The difference between them can reflect the functional performance of the separation model. Therefore, by maximizing the loss L sep Training the U-Net audio decoder ensures that the audio mixture A before and after separation i and The highest similarity is achieved, so that the loss of audio information during the separation process is zero or minimal.

[0050] S5, independent adversarial learning, by randomly extracting signals from audio signals separated from different mixed audios to build a signal library, setting the loss function to narrow the difference between the joint distribution of the signal generated by the audio decoder and the joint distribution of the samples in the randomly selected signal library, so as to train the U-Net audio decoder to generate independent separated audio signals;

[0051] The accuracy of the audio samples obtained through unsupervised preliminary separation has yet to be evaluated due to the lack of real audio data as a reference. Therefore, this embodiment also performs an independent adversarial learning algorithm to balance the correlation and independence of the sound sources after unsupervised separation, thereby improving the accuracy of unsupervised separation.

[0052] In this embodiment, if Figure 4 As shown in Figure 2, independent adversarial learning is specifically as follows:

[0053] For the separated audio signal use Represents two signals separated from a mixed audio. represents the joint probability distribution of two signals;

[0054] A signal library is constructed by randomly extracting signals from audio signals separated from different mixed audio. According to Bernoulli's law of large numbers, when the number of random selections is large enough, the distribution of signals in the signal library is approximately independent distribution; the signal library is defined as Randomly select two signals from the signal library, and Among them, m1,m2∈{1,…,M},m1≠m2, and the joint distribution of the two is defined as When the signal library samples are random enough, the two randomly selected signals are independent of each other and their joint distribution It is equivalent to the product of the marginal distributions of the two signals, expressed as follows:

[0055]

[0056] Among them, P m (·) indicates marginal distribution.

[0057] The purpose of independent adversarial learning is to guide the audio decoder to generate relatively independent audio signals and achieve mixed audio separation. The joint distribution of samples randomly selected from the signal library is approximately independent distribution, and this is used as the target distribution. The loss function L is set G To bring the joint distribution of the signal generated by the audio decoder closer to the joint distribution of samples in the randomly selected signal library, so as to train the U-Net audio decoder to generate independent separated audio signals Loss function L G It is expressed as:

[0058]

[0059] Among them, the function T can be Kullback-Leibler (KL) divergence, Jensen-Shannon (JS) divergence, Wasserstein distance and Pearson χ 2 Distance, etc.

[0060] In the process of audio separation, in order to prevent the loss of audio information, the consistency of the mixed audio is maintained before and after audio separation. Specifically, the separated audio signals are remixed and the distribution of the remixed audio is aligned with the distribution of the original mixed audio in the same video. The modified loss function L G It is expressed as:

[0061]

[0062] Independent adversarial learning methods focus on the independence of separated audio signals, which may lead to over-separation and may not cover scenes where the audio spectra partially overlap. Therefore, the audio separation operation needs to maintain a balance between the independence and integrity of the separated audio signals. In order to avoid over-separation, an independent decision loss function L is set. D Used to expand the distribution difference between the joint distribution of the signal generated by the audio decoder and the joint distribution of the samples in the randomly selected signal library, so as to maintain an appropriate balance between the independence and correlation of the separated audio signals;

[0063] Decision loss function L D Specifically:

[0064]

[0065] Where Φ represents the distribution of both the generated audio signal and the randomly selected signal Perform full convolution and activation operations; the decision loss emphasizes the distribution of the two , thus weakening the independence of the separated signals.

[0066] Loss function LG and L D Alternately train the U-Net audio decoder for adversarial learning. The loss L D Used to expand distribution The difference between the element pairs, and the loss L G By minimizing The probability distribution distance between them is used to train the cross-modal audio separation decoder to generate independent audio signals. Finally, adversarial training is used to achieve the best balance between independence and correlation in mixed audio separation.

[0067] S6, visually guided unsupervised audio separation, using the visual target object as a virtual supervisory label, setting the audio-visual semantic consistency discrimination loss function, training the U-Net audio decoder to generate an audio signal corresponding to the target object, and accurately separating the mixed audio; in this embodiment, specifically:

[0068] Taking the visual target object as the virtual supervision label, the audio-visual semantic consistency discrimination loss function L is set con , train the audio decoder to generate audio signals corresponding to the target object and accurately separate the mixed audio;

[0069] Audio-visual semantic consistency discrimination loss function L com Specifically:

[0070]

[0071] Among them, the function Represents fully connected convolution and category mapping, which is used to make category judgments on visual information and separated audio information;

[0072] The final complete loss function L used to train the U-Net audio decoder is defined as:

[0073] L=L av +λ1L sep +λ2(L G +L D )+λ3L con ;

[0074] Among them, λ1, λ2, λ3 represent the loss function parameters.

[0075] In order to better illustrate the technical effect of the method of the present invention, two large-scale audio and video datasets MUSIC and VGGSound are used, and four comparison methods are selected for comparison experiments. The source distortion ratio (SDR), source interference ratio (SIR) and source artifact ratio (SAR) are selected as evaluation criteria; the four comparison methods are:

[0076] Sound-of-Pixels algorithm, see the document "H Zhao, C Gan, A Rouditchenko, C Vondrick, JH. McDermott and A Torralba. The sound of pixels. In Proceedings of the European Conference on Computer Vision (ECCV), pages 587-604, 2018.".

[0077] Co-Separation algorithm, see the document "R Gao and K Grauman. Co-separating sounds of visual objects. In Proceedings of the IEEE / CVF International Conference on Computer Vision (ICCV), pages 3878-3887, 2019.".

[0078] MP-Net algorithm, see the literature "X Xu, B Dai and D Lin. Recursive visual soundseparation using minus-plus net. In Proceedings of the IEEE / CVF International Conference on Computer Vision 691 (ICCV), pages 882-891, 2019.".

[0079] For the FCSN algorithm, see the document "S Ma, Y ​​Ji, X Xu, and

[0080] The results are shown in Table 1, Table 2 and Table 3 below; wherein Table 1 is a comparison table of unsupervised audio separation results of the present invention and the comparative method on the mixed audio and video dataset VGGSound. Table 2 is a comparison table of unsupervised audio separation accuracy of the present invention and the comparative method on the mixed audio and video dataset MUSIC. Wherein, *algorithm+IAL represents the verification result of combining the independent adversarial learning (IAL) of the present invention as a plug-in with the existing algorithm in the unsupervised audio separation task. Table 3 is a comparison table of supervised audio separation accuracy of the present invention and the comparative method on the mixed audio and video dataset MUSIC.

[0081] Table 1 Comparison of unsupervised audio separation results of the present invention and the comparative method on the mixed audio and video dataset VGGSound

[0082]

[0083] Table 2 Comparison of the unsupervised audio separation accuracy of the present invention and the comparative method on the mixed audio and video dataset MUSIC

[0084]

[0085] Table 3 Comparison of supervised audio separation accuracy of the present invention and the comparative method on the mixed audio and video dataset MUSIC

[0086]

[0087] As shown in Table 1, Table 2 and Table 3, compared with these algorithms, the method of the present invention has obvious advantages in the sound source separation evaluation standards SDR, SIR and SAR. The experimental results of using the independent adversarial learning method as an algorithm plug-in show that independent adversarial learning has a significant improvement effect on audio separation. This proves the effectiveness of the audio separation algorithm based on independent adversarial learning proposed in the present invention.

[0088] In another embodiment, an unsupervised mixed audio cross-modal separation system is provided, the system adopts the unsupervised mixed audio cross-modal separation method of the above embodiment, and the system includes sound source object detection, audio and video data preprocessing module, cross-modal audio and video semantic alignment learning module, unsupervised mixed audio preliminary separation module, independent adversarial learning module and unsupervised audio separation module;

[0089] The sound source object detection module uses the target detection algorithm to extract the target object and visual features in the video;

[0090] The audio and video data preprocessing module processes the audio mixed signal to obtain the mixed audio features;

[0091] The cross-modal audio-visual semantic alignment learning module performs semantic alignment and enhances the semantic expression capabilities of the two modalities through audio-visual semantic consistency learning; and further refines the characteristic semantics of the audio-visual modality through audio-visual cross-attention learning;

[0092] Unsupervised mixed audio preliminary separation module, using U-Net as an audio decoder to obtain several separated audio signals;

[0093] An independent adversarial learning module is used to construct a signal library by randomly extracting signals from audio signals separated from different mixed audios, and to set a loss function to narrow the difference between the joint distribution of the signals generated by the audio decoder and the joint distribution of the samples in the randomly selected signal library, so as to train the U-Net audio decoder to generate independent separated audio signals;

[0094] The unsupervised audio separation module is used to use the visual target object as a virtual supervisory label, set the audio-visual semantic consistency discrimination loss function, train the U-Net audio decoder to generate the audio signal corresponding to the target object, and accurately separate the mixed audio.

[0095] In another embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the unsupervised mixed audio cross-modal separation method as described in the above embodiment is implemented.

[0096] The hardware entity of the computer device includes a processor, a memory and a communication interface; wherein the processor generally controls the overall operation of the computer device; the communication interface is used to enable the computer device to communicate with other terminals or servers through a network; the memory is configured to store instructions and applications executable by the processor, and can also cache data to be processed or processed by the processor and various modules in the computer device (including but not limited to image data, audio data, voice communication data and video communication data), which can be implemented through flash memory (FLASH) or random access memory (RAM).

[0097] Data can be transmitted between the processor, the communication interface and the memory via a bus, which may include any number of interconnected buses and bridges, and the bus connects various circuits of one or more processors and memories together.

[0098] In another embodiment, a computer-readable storage medium is provided, storing a computer program, and when the computer program is executed by a processor, the unsupervised mixed audio cross-modal separation method of the above embodiment is implemented.

[0099] The storage medium may be transient or non-transient. Exemplarily, the storage medium includes, but is not limited to, a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store computer program codes.

[0100] Exemplarily, the processor may be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP), or a field programmable gate array (FPGA).

[0101] It should also be noted that in this specification, terms such as "comprises", "includes" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of more restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0102] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An unsupervised mixed audio cross-modal separation method, characterized in that: The following steps are involved: Sound source object detection in the video: for videos containing mixed audio signals of multiple sound sources, target objects and visual features in the video are extracted using target detection algorithms; Preprocessing of audio and video data, processing of audio mixed signals to obtain mixed audio features; Cross-modal audio-visual semantic alignment learning, including audio-visual semantic consistency learning and audio-visual cross-attention learning; Through audio and video semantic consistency learning, semantic alignment is performed and the semantic expression ability of the two modalities is enhanced; through audio and visual cross-attention learning, the characteristic semantics of the audio and visual modalities are further refined; Unsupervised mixed audio preliminary separation, using U-Net as an audio decoder to obtain several separated audio signals; Independent adversarial learning, by randomly extracting signals from audio signals separated from different mixed audios to build a signal library, setting the loss function to bring the joint distribution of the signals generated by the audio decoder closer to the joint distribution of the samples in the randomly selected signal library, so as to train the U-Net audio decoder to generate independent separated audio signals; Visually guided unsupervised audio separation uses visual target objects as virtual supervisory labels, sets the audio-visual semantic consistency discrimination loss function, and trains the U-Net audio decoder to generate audio signals corresponding to the target objects, thereby accurately separating the mixed audio.

2. The unsupervised mixed audio cross-modal separation method according to claim 1, characterized in that: The specific preprocessing of audio and video data is as follows: The audio signals in multiple videos are set to A i1 ,A i2 ,…,A in , the audio mixed signal in multiple videos is set to A mix =A i1 +A i2 +…+A in ; Obtaining the target object in the video and visual features After that, the mixed audio signal A is obtained by short-time Fourier transform and Mel spectrum transform. mix Mel spectrum S mix , and S mix Input audio encoder to obtain mixed audio features 3. The unsupervised mixed audio cross-modal separation method according to claim 2, characterized in that: The specific steps of audio and video semantic consistency learning are: Fusion of target object visual features and audio mixing features Get fusion features Contrastive learning is used to enhance the accuracy of visual features and to fuse features corresponding to the same sound source object. and the visual characteristics of the sound source object The InfoNCE is used as the contrastive learning loss function to optimize the visual feature expression. The contrastive learning loss function is: Where τ is the temperature hyperparameter, represents the negative sample space; Audio-visual cross-attention learning is specifically: Enhanced fusion features and visual features It is used interactively as a query vector for interactive cross-modal attention learning. Through spatial attention and channel attention learning, the semantic consistency features of the audio-visual modality are further refined, the semantic accuracy of feature expression is improved, and the fusion of modal information is promoted. The fusion feature f with enhanced attention is output. c .

4. The unsupervised mixed audio cross-modal separation method according to claim 3, characterized in that: The initial separation of unsupervised mixed audio is as follows: Use U-Net as the audio decoder to generate the target object with the sound source The corresponding separation mask By placing the mask Multiply it with the sound mixed spectrum to obtain the corresponding sound source target object Audio Mel Spectrum It is expressed as: The spectrum Perform inverse transform from frequency domain to time domain to obtain time domain audio signal In order to avoid information loss during audio separation, the loss function L is set sep The integrity and quality of the separated audio information generated by the U-Net audio decoder can be evaluated by setting the signal-to-noise ratio, and the loss function L seo Specifically: Among them, A i is the real mixed audio data in the video, A i After unsupervised audio separation, several separated audio signals are obtained After integrating the separated audio signals, the mixed audio is 5. The unsupervised mixed audio cross-modal separation method according to claim 4, characterized in that: Independent adversarial learning is specifically as follows: For the separated audio signal use Represents two signals separated from a mixed audio. represents the joint probability distribution of two signals; A signal library is constructed by randomly extracting signals from audio signals separated from different mixed audio. According to Bernoulli's law of large numbers, when the number of random selections is large enough, the distribution of signals in the signal library is approximately independent distribution; the signal library is defined as Randomly select two signals from the signal library, and Among them, m1,m2∈{1,…,M},m1≠m2, and the joint distribution of the two is defined as When the signal library samples are random enough, the two randomly selected signals are independent of each other and their joint distribution It is equivalent to the product of the marginal distributions of the two signals, expressed as follows: Among them, P m (·) indicates marginal distribution; Set the loss function L G To bring the joint distribution of the signal generated by the audio decoder closer to the joint distribution of samples in the randomly selected signal library, so as to train the U-Net audio decoder to generate independent separated audio signals Loss function L G It is expressed as: Among them, function T uses Kullback-Leibler divergence, Jensen-Shannon divergence, Wasserstein distance and Pearsonχ 2 One of the distances; In the process of audio separation, in order to prevent the loss of audio information, the consistency of the mixed audio is maintained before and after audio separation. Specifically, the separated audio signals are remixed and the distribution of the remixed audio is aligned with the distribution of the original mixed audio in the same video. The modified loss function L G It is expressed as:

6. The unsupervised mixed audio cross-modal separation method according to claim 5, characterized in that: In order to avoid over-separation, an independent decision loss function L is set D Used to expand the distribution difference between the joint distribution of the signal generated by the audio decoder and the joint distribution of the samples in the randomly selected signal library, so as to maintain an appropriate balance between the independence and correlation of the separated audio signals; Decision loss function L D Specifically: Where Φ represents the distribution of both the generated audio signal and the randomly selected signal Perform full convolution and activation operations; the decision loss emphasizes the distribution of the two , thus weakening the independence of the separated signals.

7. The unsupervised mixed audio cross-modal separation method according to claim 6, characterized in that: Visually guided unsupervised audio separation is specifically: Taking the visual target object as the virtual supervision label, the audio-visual semantic consistency discrimination loss function L is set con , train the audio decoder to generate audio signals corresponding to the target object and accurately separate the mixed audio; Audio-visual semantic consistency discrimination loss function L con Specifically: Among them, the function Represents fully connected convolution and category mapping, which is used to make category judgments on visual information and separated audio information; The final complete loss function L used to train the U-Net audio decoder is defined as: L=L av +λ1L sep +λ2(L G +L D )+λ3L con ; Among them, λ1, λ2, λ3 represent the loss function parameters.

8. An unsupervised mixed audio cross-modal separation system, characterized in that: The system adopts the unsupervised mixed audio cross-modal separation method according to any one of claims 1 to 7, and the system includes sound source object detection, audio and video data preprocessing module, cross-modal audio and video semantic alignment learning module, unsupervised mixed audio preliminary separation module, independent adversarial learning module and unsupervised audio separation module; The sound source object detection module uses the target detection algorithm to extract the target object and visual features in the video; The audio and video data preprocessing module processes the audio mixed signal to obtain the mixed audio features; The cross-modal audio-visual semantic alignment learning module performs semantic alignment and enhances the semantic expression capabilities of the two modalities through audio-visual semantic consistency learning; and further refines the characteristic semantics of the audio-visual modality through audio-visual cross-attention learning; Unsupervised mixed audio preliminary separation module, using U-Net as an audio decoder to obtain several separated audio signals; An independent adversarial learning module is used to construct a signal library by randomly extracting signals from audio signals separated from different mixed audios, and to set a loss function to narrow the difference between the joint distribution of the signals generated by the audio decoder and the joint distribution of the samples in the randomly selected signal library, so as to train the U-Net audio decoder to generate independent separated audio signals; The unsupervised audio separation module is used to use the visual target object as a virtual supervisory label, set the audio-visual semantic consistency discrimination loss function, train the U-Net audio decoder to generate the audio signal corresponding to the target object, and accurately separate the mixed audio.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images