Music source extraction method, device and product based on reference audio and midi guidance

Abstract

The application provides a music source extraction method and device based on reference audio and MIDI guidance, and a product, the method comprising: processing the obtained mixed audio signal and reference audio signal through a preset music source extraction model to output a separated audio signal, the separated audio signal corresponding to the reference audio signal; the music source extraction model processing step comprising: splicing the MIDI features extracted from the mixed audio signal with the features extracted from the mixed audio signal and the reference audio signal to obtain multi-scale fusion features; performing multiple feature extractions on the multi-scale fusion features of different time resolutions to obtain cross-resolution representations; processing the cross-resolution representations through a multi-resolution mask decoder to obtain target masks of different time resolutions, and obtaining the required separated audio signal based on the target masks. The application effectively alleviates the ambiguity problem caused by spectral overlap by using MIDI features, and improves the fidelity of the separated audio signal by combining the reference audio signal as a timbre condition.

Description

Technical Field

[0001] This invention relates to the field of music source extraction technology, specifically to a method, apparatus, and product for music source extraction based on reference audio and MIDI guidance. Background Technology

[0002] The core objective of Music Source Extraction (MSE) technology is to accurately separate specific sound sources, such as vocals, drums, bass, and piano tracks, from a mono audio mix. Unlike traditional methods that focus on overall separation of multiple sound sources, MSE technology emphasizes targeted extraction of specific tracks, thus offering better adaptability in practical applications. For example, in karaoke and cover song production, it's often necessary to extract clean vocal tracks; while in beat detection or rhythm analysis scenarios, accurate separation of drum tracks is crucial. Compared to general sound source separation technologies, MSE technology offers significant advantages in computational efficiency and the purity of the output track.

[0003] However, due to the severe overlap in the spectrum of different sound sources and their complex temporal dynamic relationships, how to preserve the original timbre and details while achieving high-precision separation remains the main challenge facing this technology.

[0004] Current music source separation techniques mainly include two approaches: frequency domain-based and time domain-based. Frequency domain-based methods typically achieve separation using Short-Time Fourier Transform (STFT) and mask estimation, but source leakage occurs when the spectra of instruments and vocals overlap. Time domain-based methods circumvent the limitations of STFT by directly modeling the original waveform, but often lead to timbre confusion and noise residue. Furthermore, most existing methods presuppose a fixed number of sound sources and employ only a bottom-up hybrid model construction approach, lacking both external prior constraints and the ability to generalize across styles and timbres. Summary of the Invention

[0005] The primary objective of this invention is to provide a music source extraction method based on reference audio and MIDI guidance, thereby addressing the problems of sound source feature confusion and timbre mismatch in existing music source extraction methods.

[0006] A second objective of the present invention is to provide a computer device for implementing the above-described method for extracting music sources guided by reference audio and MIDI.

[0007] A third objective of this invention is to provide a computer-readable storage medium that implements the above-described method for extracting music sources guided by reference audio and MIDI.

[0008] To achieve the aforementioned first objective, this invention provides a music source extraction method based on reference audio and MIDI guidance, comprising the following steps: processing the obtained mixed audio signal and reference audio signal through a preset music source extraction model to output a separated audio signal, wherein the separated audio signal corresponds to the reference audio signal; the music source extraction model processing steps include: performing short-time Fourier transforms on the mixed audio signal and reference audio signal at different time resolutions, then encoding them separately through an audio encoder, concatenating the mixed features and reference features output by the audio encoder with the MIDI features extracted from the mixed audio signal to obtain multi-scale fusion features; averaging the multi-scale fusion features at different time resolutions to obtain initial features, performing multiple feature extractions on the initial features, averaging the results of the multiple feature extractions and concatenating them with the initial features to obtain a cross-resolution representation; processing the cross-resolution representation through a multi-resolution mask decoder to obtain target masks at different time resolutions, performing mask processing based on the target masks, and obtaining the separated audio signal through inverse short-time Fourier transform and averaging at different time resolutions.

[0009] As can be seen from the above scheme, the present invention is based on the dual constraints of MIDI features and reference audio signals. Through the structural constraints guided by MDID, it effectively alleviates the ambiguity problem caused by spectrum overlap. At the same time, by using reference audio as a timbre cues, it improves the fidelity of the separated audio signals and optimizes the timbre matching effect.

[0010] A further approach is to optimize the parameters of the music source extraction model through contrastive loss and scale-invariant signal-to-noise ratio loss in contrastive learning. The contrastive loss is used to optimize the acquisition of multi-scale fusion features and cross-resolution representations, while the scale-invariant signal-to-noise ratio loss is used to optimize the acquisition of separated audio signals.

[0011] Therefore, it can be seen that the present invention significantly improves the distinguishability between the target audio track and the interference audio track by jointly optimizing the music source extraction model by fusing contrast loss and scale-invariant signal-to-noise ratio loss.

[0012] A further approach is to express the total loss function of the contrast loss and the scale-invariant signal-to-noise ratio loss as follows: ,in, This represents the scale-invariant signal-to-noise ratio loss. Indicates comparative loss, This represents the weighting factor.

[0013] Therefore, by setting adjustable weight factors, it is possible to ensure that the music source model achieves a balance between waveform fidelity and feature discrimination, and to achieve robust music source extraction in complex mixed scenarios.

[0014] A further approach is that, during the training process of the music source extraction model, the input training data includes a mixed audio signal for training, a reference audio signal for training, a target audio signal for training, and a noise signal for training. Anchor samples are obtained based on the mixed audio signal for training and the reference audio signal for training, positive samples are obtained based on the target audio signal for training and the reference audio signal for training, and negative samples are obtained based on the noise signal for training and the reference audio signal for training.

[0015] Therefore,

[0016] A further approach is to use the Conformer extractor when performing multiple feature extractions on the initial features.

[0017] Therefore, it can be seen that by extracting through the Conformer extractor, the model can effectively preserve the rhythmic clues in MIDI and maintain its sensitivity to timbre changes in the mixed signal, thereby obtaining a more accurate voiceprint separation effect.

[0018] A further approach is to extract MIDI features from the mixed audio signals using a pre-defined multi-task, multi-track music transcription model.

[0019] A further approach involves performing short-time Fourier transforms on the mixed audio signal and the reference audio signal at different time resolutions, with window lengths of 23, 46, and 186 corresponding to the different time resolutions.

[0020] Therefore, by covering different temporal resolutions, the complementarity of features can be ensured. Using multi-resolution time-frequency transformations with different window lengths allows for the simultaneous capture of both short-term transient and long-term dependency features. Shorter windows better represent rapidly changing transient information such as drum beats and plosives; longer windows preserve long-term dependencies such as melody and harmony; and medium-sized windows serve a complementary role between the two. Through the fusion of multi-scale features, the model can focus on both local details and maintain global structure during separation, effectively mitigating spectral overlap and timbre confusion, and significantly improving the robustness and fidelity of separation.

[0021] A further approach is to concatenate the mixed features and reference features corresponding to the same time resolution output by the audio encoder with the time-aligned MIDI features extracted from the mixed audio signal to obtain multi-scale fusion features at the same time resolution.

[0022] Therefore, interpolation can be used to align the mixed features and the reference features, thereby better mitigating spectral overlap.

[0023] To achieve the second objective described above, the present invention provides a computer device comprising a processor and a memory, wherein: the memory stores a computer program, and when the computer program is executed by the processor, it implements the above-described method for extracting music sources based on reference audio and MIDI guidance.

[0024] To achieve the third objective mentioned above, the present invention provides a computer program product comprising computer instructions, wherein: when the computer instructions are executed by a processor, they implement the aforementioned method for extracting music sources based on reference audio and MIDI guidance. Attached Figure Description

[0025] Figure 1 This is a flowchart of an embodiment of the music source extraction method based on reference audio and MIDI guidance of the present invention.

[0026] Figure 2 This is an architecture diagram of the music source extraction model in an embodiment of the music source extraction method based on reference audio and MIDI guidance of the present invention.

[0027] The present invention will be further described below with reference to the accompanying drawings and embodiments. Detailed Implementation

[0028] The music source extraction method based on reference audio and MIDI guidance of the present invention achieves the separation of the desired target audio signal from the reference audio signal under the dual guidance of timbre cues provided by the reference audio signal and pitch and timing constraints provided by MIDI features.

[0029] MIDI (Musical Instrument Digital Interface) is an electronic communication protocol that establishes a unified technical standard for electronic musical instruments, computers, and other digital audio devices, enabling these devices to communicate with each other. MIDI does not transmit actual sound signals; instead, it transmits instruction data such as notes, pitch, and volume.

[0030] Example of a music source extraction method based on reference audio and MIDI guidance:

[0031] See Figure 1 This embodiment is implemented by executing a computer program, and specifically includes the following steps:

[0032] S1: Acquire the mixed audio signal.

[0033] The mixed audio signal is the signal that needs to be processed in this embodiment. This embodiment separates a specific sound source from the mixed audio signal.

[0034] For example, the mixed audio signal is a pop song, which includes audio signals corresponding to sound sources such as vocals, drums, bass, and piano.

[0035] S2: Obtain the reference audio signal.

[0036] The reference audio signal corresponds to the sound source that needs to be separated from the mixed audio signal.

[0037] For example, if it is necessary to separate the audio signal corresponding to the drum from the mixed audio signal, a section of the audio signal corresponding to the drum needs to be provided as a reference audio signal. The duration of the provided reference audio signal does not need to be the same as that of the mixed audio signal.

[0038] S3: Processes the mixed audio signal and reference audio signal through a preset music source extraction model, and outputs a separate audio signal.

[0039] The music source extraction model uses the reference audio signal as a timbre anchor and, guided by the MIDI features of the mixed audio, separates the desired audio signal from the mixed audio signal.

[0040] See Figure 2 The music source extraction model in this embodiment includes an audio encoding module, a MIDI extraction module, a feature extraction module, and a mask decoding module. The following section describes in detail the process of the music source extraction model processing mixed audio signals and reference audio signals, based on the functions of the audio encoding module, MIDI extraction module, feature extraction module, and mask decoding module.

[0041] The audio encoding module is used to encode the mixed audio signal and the reference audio signal into a time-frequency (TF) embedding, i.e., multi-scale fusion features, which preserves the time domain and timbre features.

[0042] The MIDI extraction module is used to extract MIDI features from the acquired mixed audio signal and align the extracted MIDI features according to the corresponding mixed features and reference feature time, providing melody and rhythm constraints as structured prior knowledge.

[0043] The feature extraction module is used to fuse reference audio signals, mixed audio signals, and MIDI features, and achieves cross-modal alignment by modeling long-range dependencies and local detail features.

[0044] The mask decoding module is used to estimate the target mask and generate a separate audio signal based on the target mask.

[0045] In the audio encoding module, the acquired mixed audio signal and reference audio signal are first subjected to short-time Fourier transforms at different time resolutions, and then encoded by the audio encoder 11. The mixed features and reference features output by the audio encoder 11 are concatenated with the MIDI features extracted from the mixed audio signal to obtain multi-scale fusion features.

[0046] In this embodiment, the preferred window lengths for the short-time Fourier transforms (SFTs) at different time resolutions are 23, 46, and 186. The audio encoder includes an amplitude extraction unit, a one-dimensional convolutional unit (Conv1D), a layer normalization unit (LN), and an activation unit (GELU). Thus, after the mixed audio signals undergo SFTs with window lengths of 23, 46, and 186 respectively, they are fed into the audio encoder to obtain the mixed features corresponding to different time resolutions. The reference audio signal was subjected to short-time Fourier transforms with window lengths of 23, 46, and 186, respectively, to obtain reference features corresponding to different time resolutions. ,in, This represents the window length for the short-time Fourier transform. It is relevant for mixed features at different time resolutions. and reference features MIDI features aligned with time are spliced ​​together to form multi-scale fusion features. .

[0047] In other embodiments, the window lengths corresponding to the short-time Fourier transforms at different time resolutions can be determined according to actual needs. At least two different window lengths need to be used to ensure different time resolutions and thus guarantee feature complementarity. Using three or more windows can further improve performance, but it will also increase computational overhead.

[0048] In the MIDI extraction module, a pre-defined multi-task, multi-track music transcription model is first used to transcribe the mixed audio signal into note events including pitch, start time, end time, and instrument information, which are then output as MIDI features. Next, to ensure time alignment, the MIDI features are interpolated to match their frame length with the mixed features and reference features at different time resolutions, as shown below: ,in, This represents the MIDI features output by the multi-task, multi-track music transcription model. This indicates time interpolation.

[0049] Therefore, the time-aligned MIDI features obtained by the MIDI extraction module can be concatenated with the corresponding temporal resolution blended features and reference features to obtain multi-scale fused features, which are represented as follows: By using the dual constraints of timbre anchors provided by the reference audio signal and pitch and rhythm anchors provided by MIDI features, spectral overlap and timbre confusion can be reduced.

[0050] The preferred multi-task, multi-track music transcription model in this embodiment is the MT3 model. The MT3 model, through joint training with multiple instruments and datasets, exhibits powerful transcription performance even with instruments facing limited resources. This makes it a reliable source of structural cues, providing accurate pitch prior knowledge for melodic instruments (such as vocals and bass) and setting rhythm prior knowledge for percussion instruments (such as drum kits). For different embodiments, other multi-task, multi-track music transcription models can be used according to different needs.

[0051] For the feature extraction module, the multi-scale fused features at different time resolutions are first averaged to obtain the initial features. Then, the initial features are extracted multiple times. The results of the multiple feature extractions are averaged and then concatenated with the initial features to obtain the cross-resolution representation.

[0052] In this embodiment, the multi-scale fusion features obtained at three time resolutions are averaged, that is, the three multi-scale fusion features corresponding to window lengths of 23, 46, and 186 are averaged to obtain the initial features. The initial features are processed by three Conformer extractors 21, or by one Conformer extractor three times, resulting in three extracted features. The average of the three extracted features is then compared with the initial features. The images are stitched together and then linearly projected to generate the final cross-resolution representation. .

[0053] It should be noted that, although Figure 2 The input to each Conformer extractor 21 is the same initial feature. However, since the parameters of each branch's Conformer extractor 21 are independent and undergo nonlinear transformation, the outputs will not be exactly the same. The number of Conformer extractors 21 can be flexibly adjusted as needed. The design of the multi-branch Conformer extractor aims to enhance feature representation capabilities, improve robustness, and fully utilize multi-scale information by fusing the outputs of different branches to improve the overall performance.

[0054] When performing multiple feature extractions on the initial features, this embodiment preferably uses the Conformer extractor. The Conformer extractor combines self-attention mechanisms with convolutional operations to achieve collaborative capture of global context and local spectral features. Using the Conformer extractor allows the music source extraction model to effectively preserve rhythmic cues in MIDI features while maintaining sensitivity to timbre changes in mixed audio signals, thus achieving more accurate voiceprint separation. The Conformer extractor can be implemented using the paper "Hybrid dual-path network: Singing voice separation in the waveform domain by combining conformer and transformer architecture". For other alternative feature extraction schemes, other existing feature extractors can also be used, such as feature extractors based on bidirectional long short-term memory networks (BLSTM).

[0055] For the mask decoding module, firstly, based on the cross-resolution representation obtained by the multi-resolution mask decoder 31, target masks with different time resolutions are obtained. Then, based on the target masks, mask processing is performed, and the separated audio signals are obtained by inverse short-time Fourier transform and averaging at different time resolutions.

[0056] In this embodiment, the multi-resolution mask decoder 31 processes and obtains the cross-resolution representation. This yields three target masks for different time resolutions (corresponding to window lengths of 23, 46, and 186). The target masks at different time resolutions are then applied element-wise to the corresponding complex spectrograms, as shown below: ,in, Indicates the target mask. This represents the spectral representation of mixed audio quotes at different STFT resolutions. This represents the element-wise product (Hadamard product). Choose 23, 46, or 186. Then, analyze the complex spectrum after masking. The waveform is reconstructed by performing an inverse short-time Fourier transform, and the waveforms at different time resolutions are averaged to obtain the final separated audio signal. Separate audio signals As a result of the music source extraction model, the target audio signal For objective audio signals Through continuous optimization of the loss function, the music source extraction model processes the separated audio signals. Approaching or even matching the target audio signal same.

[0057] Therefore, the music source extraction model in this embodiment can accurately separate specific sound sources from audio mixed signals.

[0058] The following details the setting of the loss function for the music source extraction model in this embodiment.

[0059] In this embodiment, the music source extraction model optimizes its parameters during training by using contrastive loss and scale-invariant signal-to-noise ratio (SI-SNR) loss. Contrast loss is used to optimize the acquisition of multi-scale fusion features and cross-resolution representations, while SI-SNR loss is used to optimize the acquisition of separated audio signals.

[0060] The input training data includes a mixed audio signal for training, a reference audio signal for training, a target audio signal for training, and a noise signal for training. Anchor samples are obtained based on the mixed audio signal for training and the reference audio signal for training. Positive samples are obtained based on the target audio signal for training and the reference audio signal for training. Negative samples are obtained based on the noise signal for training and the reference audio signal for training. Thus, the audio encoding module and the feature extraction module are optimized through contrastive learning.

[0061] See also Figure 2 Let x' be the mixed audio signal used for training, r' be the reference audio signal used for training, y' be the target audio signal used for training, and n' be the noise signal used for training.

[0062] The mixed audio signal x', the target audio signal y' used for training, and the training noise signal n' are input into the music source extraction model along with the reference audio signal r'. The resulting audio embedding vectors are obtained through the audio encoding module and the feature extraction module. The audio embedding vectors obtained based on the mixed audio signal x' and the training reference audio signal r' are used as anchor point samples, denoted as follows: Based on the target audio signal y' and the training reference audio signal r', the audio embedding vector is obtained as a positive sample, represented as follows: Negative samples are obtained based on the training noise signal n' and the reference audio signal r', denoted as... The similarity between the anchor sample and the positive sample, and the similarity between the anchor sample and each negative sample, are defined as follows: , ,in This represents the temperature hyperparameter. The loss function for contrastive learning is expressed as: ,in, This indicates the batch size. The contrastive learning loss function can effectively bring the predicted audio embedding vector closer to the target audio embedding vector, while simultaneously widening the gap between the predicted audio embedding vector and the non-target audio embedding vector, thereby enhancing the performance of the audio encoding module and the feature extraction module, and consequently improving the overall discriminative ability of the music source extraction model.

[0063] The total loss function of contrast loss and scale-invariant signal-to-noise ratio loss in this embodiment is expressed as follows: ,in, This represents the scale-invariant signal-to-noise ratio loss. Indicates comparative loss, This represents the weighting factor. The setting of the weighting factor allows the enhanced music source extraction model to achieve a balance between waveform fidelity and feature discrimination, enabling robust music source extraction in complex mixed scenarios.

[0064] The following details the data and parameter settings used in training the music source extraction model in this embodiment.

[0065] The music source extraction model in this embodiment is trained on the MUSDB18-HQ dataset. Each sample contains four independent voices: drums, bass, vocals, and other instruments. For each sample, the following data items are set: (1) Target voice: The real audio track of the target instrument (such as drums, bass, vocals) extracted from the mixed track, as positive supervision; (2) Reference audio: Audio clips randomly sampled from other songs of the same instrument category, providing cross-song timbre and style reference; (3) Non-target voice: The remaining voices in the same song other than the target voice, as negative samples to enhance discriminative power; (4) MIDI features: MIDI files automatically predicted from the mixed track by MT3, providing symbolic guidance for pitch and note hierarchy structure.

[0066] To increase data diversity and improve generalization ability, this embodiment also remixed the four tracks of 100 training songs, generating approximately 3,000 training samples. The same mixing strategy was also applied to the validation and test sets, generating 400 samples each, with a final total audio duration of approximately 500 hours.

[0067] During the training process based on the above training data, the Adam optimizer was used, and the initial learning rate was set to 10. -3 The ReduceLROnPlateau scheduler was used for tuning. The batch size was set to 8, and the maximum norm of gradient clipping was limited to 5.0. Short-term and long-term dependencies were captured by employing multi-scale short-time Fourier transforms with window lengths of 23, 46, and 186. The loss function combined contrastive loss and scale-invariant signal-to-noise ratio loss, with a weighting factor set to 0.1.

[0068] The trained music source extraction model is compared with baseline models implemented by three representative baseline methods: Open-Unmix (UMX), Chimera, and MRX. All three methods are based on the short-time Fourier transform features in the standard encoder-splitter-decoder framework, making them robust and comparable benchmark models.

[0069] Referring to Table 1, Table 1 shows the comparison results of the music source extraction model (MC-MRX) implemented in this embodiment with the models implemented by Open-Unmix (UMX), Chimera, and MRX in terms of the SI-SNRi index. As shown in Table 1, the model implemented by the method in this embodiment consistently outperforms all baseline models. The data in Table 1 show that the SI-SNRi of MC-MRX for bass, drums, other parts, and vocals are 11.45 dB, 10.21 dB, 10.12 dB, and 12.79 dB, respectively. Compared to UMX, these four indicators are improved by 0.30 dB, 0.16 dB, 0.75 dB, and 0.54 dB, respectively; compared to Chimera, the improvement is more significant, reaching 1.24 dB, 1.21 dB, 1.06 dB, and 1.86 dB, respectively; in MRX, this embodiment achieves the greatest advantage in all four evaluation indicators, obtaining gains of 2.51 dB, 2.24 dB, 3.51 dB, and 1.66 dB, respectively. These results fully demonstrate the effectiveness of methods such as MIDI structured priors and cross-resolution feature aggregation, while the contrastive learning mechanism further enhances the target sound source's ability to separate interference. It is worth noting that the MRX backbone network used in this invention has the inherent advantages of being lightweight and computationally efficient. By combining structured priors with discriminative learning, the lightweight MRX achieves performance comparable to classic frequency domain baseline models, highlighting the potential to integrate MIDI driving mechanisms into more advanced architectures.

[0070] Table 1. Evaluation results of SI-SNRi index

[0071]

[0072] The music source extraction model in this embodiment also underwent an ablation experiment, and the results are shown in Table 2. Regarding the SI-SNRi index, after removing the contrast loss, the bass and drum frequencies decreased to 10.48 (-0.97) dB and 9.16 (-1.05) dB respectively, indicating that this index plays an important role in constraining the embedding space and enhancing robustness. Further removal of the MIDI guide further reduced the bass and other frequencies to 9.35 (-1.13) dB and 7.71 (-2.33) dB respectively, confirming the importance of structured music a priori. When the reference audio signal was simultaneously removed, the bass frequency decreased to 8.94 (-0.41) dB, highlighting the effectiveness of the reference audio signal as a timbre anchor in reducing ambiguity.

[0073] Table 2. Results of ablation experiments on the SI-SNRi index

[0074]

[0075] A similar trend is evident in the SDRi metric, as shown in Table 3. Removing different modules consistently leads to a decrease in model performance. The most significant attenuation occurs when a reference audio signal is missing; for example, bass decreases from 7.66 dB to 5.58 dB, and vocals from 7.60 dB to 0.81 dB. This is primarily due to the SDRi metric being more sensitive to residual interference; therefore, timbre anchors are crucial for suppressing signal leakage and improving signal purity. Overall, although the SI-SNRi metric occasionally fluctuates, both metrics show a continuous downward trend with the removal of different modules. This not only highlights the complementary role of these three modules but also confirms the necessity of the music source extraction model design scheme of this invention.

[0076] Table 3. Results of ablation experiments on SDRi index

[0077]

[0078] Computer device embodiment:

[0079] The computer device in this embodiment includes a processor and a memory. The memory stores a computer program, and when the processor executes the computer program, it implements the various steps of the above-described embodiment of the music source extraction method based on reference audio and MIDI guidance.

[0080] A computer device may include, but is not limited to, a processor and memory. Those skilled in the art will understand that a computer device may include more or fewer components, or a combination of certain components, or different components; for example, a computer device may also include input / output devices, network access devices, buses, etc.

[0081] For example, a processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microcontroller or any conventional processor. The processor is the control center of a computer device, connecting all parts of the computer device through various interfaces and lines.

[0082] The memory can be used to store computer programs and / or modules. The controller implements various functions of the computer device by running or executing the computer programs and / or modules stored in the memory, and by accessing data stored in the memory. For example, the memory may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (e.g., sound receiving function, sound-to-text function, etc.), etc.; the data storage area may store data created based on the use of the mobile phone (e.g., audio data, text data, etc.). In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, RAM, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0083] If the modules integrated into the computer device in the above embodiments are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the process of the music source extraction method embodiment based on reference audio and MIDI guidance can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a controller, it can implement the steps of the music source extraction method embodiment based on reference audio and MIDI guidance. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The storage medium can include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content contained in computer-readable media may be appropriately added to or subtracted from the requirements of legislation and patent practice in a jurisdiction. For example, in some jurisdictions, computer-readable media may not include electrical carrier signals and telecommunication signals, in accordance with legislation and patent practice.

[0084] Example of a computer program product:

[0085] The computer program product of this embodiment includes computer instructions stored in a computer-readable storage medium. The processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the various steps of the above-described embodiment of the music source extraction method based on reference audio and MIDI guidance.

[0086] In summary, this invention addresses the problems of sound source feature confusion and timbre mismatch by integrating MIDI features as multimodal cues with the reference audio signal. It employs a combination of contrast loss and scale-invariant signal-to-noise ratio loss for optimization, thereby extracting the target audio track and enhancing its focus and fidelity.

[0087] Finally, it should be emphasized that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention can have various changes and modifications. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for extracting music sources based on reference audio and MIDI guidance, characterized in that, Includes the following steps: The mixed audio signal and reference audio signal obtained by processing the music source extraction model are processed to output a separated audio signal, which corresponds to the reference audio signal; The music source extraction model processing steps include: The mixed audio signal and the reference audio signal are subjected to short-time Fourier transforms at different time resolutions, and then encoded by audio encoders. The mixed features and reference features output by the audio encoders are concatenated with the MIDI features extracted from the mixed audio signal to obtain multi-scale fusion features. The multi-scale fusion features at different time resolutions are averaged to obtain initial features. The initial features are then subjected to multiple feature extractions. The results of the multiple feature extractions are averaged and then concatenated with the initial features to obtain a cross-resolution representation. The cross-resolution representation is processed by a multi-resolution mask decoder to obtain target masks with different time resolutions. Based on the target masks, mask processing is performed, and the separated audio signals are obtained by inverse short-time Fourier transform and averaging at different time resolutions.

2. The music source extraction method based on reference audio and MIDI guidance as described in claim 1, characterized in that: The music source extraction model optimizes its parameters using contrastive loss and scale-invariant signal-to-noise ratio loss from contrastive learning. The contrastive loss is used to optimize the acquisition of the multi-scale fusion features and the cross-resolution representation, while the scale-invariant signal-to-noise ratio loss is used to optimize the acquisition of the separated audio signals.

3. The music source extraction method based on reference audio and MIDI guidance as described in claim 2, wherein: The total loss function of the contrast loss and the scale-invariant signal-to-noise ratio loss is expressed as follows: ,in, This represents the scale-invariant signal-to-noise ratio loss. This represents the contrast loss. This represents the weighting factor.

4. The music source extraction method based on reference audio and MIDI guidance as described in claim 1, characterized in that: During the training process of the music source extraction model, the input training data includes a mixed audio signal for training, a reference audio signal for training, a target audio signal for training, and a noise signal for training. Anchor samples are obtained based on the mixed audio signal for training and the reference audio signal for training. Positive samples are obtained based on the target audio signal for training and the reference audio signal for training. Negative samples are obtained based on the noise signal for training and the reference audio signal for training.

5. The music source extraction method based on reference audio and MIDI guidance as described in claim 1, characterized in that: When the initial features are extracted multiple times, the Conformer extractor is used.

6. The music source extraction method based on reference audio and MIDI guidance as described in claim 1, characterized in that: The MIDI features extracted from the mixed audio signal are implemented through a preset multi-task multitrack music transcription model.

7. The music source extraction method based on reference audio and MIDI guidance as described in claim 1, characterized in that: During the process of performing short-time Fourier transforms on the mixed audio signal and the reference audio signal at different time resolutions, the window lengths corresponding to the short-time Fourier transforms at different time resolutions are 23, 46, and 186.

8. The music source extraction method based on reference audio and MIDI guidance as described in claim 1, characterized in that: The mixed features and reference features corresponding to the same time resolution output by the audio encoder are concatenated with the time-aligned MIDI features extracted from the mixed audio signal to obtain multi-scale fusion features at the same time resolution.

9. A computer device comprising a processor and a memory, characterized in that: The memory stores a computer program, which, when executed by the processor, implements the music source extraction method based on reference audio and MIDI guidance as described in any one of claims 1 to 8.

10. A computer program product comprising computer instructions, characterized in that: When the computer instructions are executed by the processor, they implement the music source extraction method based on reference audio and MIDI guidance as described in any one of claims 1 to 8.

Images