Music audio processing method and device, equipment and storage medium

Abstract

The invention discloses a music audio processing method and device, equipment and a storage medium. The music audio processing method comprises the following steps: acquiring a plurality of frames of target music signals in music audio; performing drum sound separation processing on each target music signal to obtain a drum sound track signal of each target music signal; performing drum sound detection processing on each drum sound track signal to obtain a drum sound detection result of each target music signal, the drum sound detection result of each target music signal including that the target music signal is in a drum sound state or the target music signal is in a non-drum state; and processing each target music signal based on the drum sound detection result of each target music signal and the drum sound track signal of each target music signal to obtain a music audio processing result. According to the scheme, the sound quality of the processing result of the music audio can be improved.

Description

Technical Field

[0001] The present application relates to the field of audio processing, and in particular to a music audio processing method, device, equipment and storage medium. Background Art

[0002] The processing of music audio is usually achieved by tuning the sound source through an equalizer (EQ) to enhance the expressiveness of the drum sound in the music audio. This method directly enhances all types of sound sources in the original music signal within a certain frequency range. All types of sound sources in the original music signal include human voices, drum sounds, bass sounds and other sound sources. However, this method may over-enhance certain frequencies of all types of sound sources, which may cause timbre distortion and affect the sound quality of the final playback effect of the music audio.

[0003] In view of the existing technical defects, how to provide a solution that can improve the sound quality of music audio is a technical problem that needs to be solved urgently by technical personnel in this field. Summary of the invention

[0004] The present application at least provides a music audio processing method, device, equipment and storage medium.

[0005] The present application provides a music audio processing method, comprising: acquiring several frames of target music signals in music audio; performing drum sound separation processing on each target music signal to obtain a drum sound track signal of each target music signal; performing drum sound detection processing on each drum sound track signal to obtain a drum sound detection result of each target music signal, wherein the drum sound detection result of each target music signal includes that the target music signal is in a state with drum sound or that the target music signal is in a state without drum sound; processing each target music signal based on the drum sound detection result of each target music signal and the drum sound track signal of each target music signal to obtain a music audio processing result.

[0006] In some embodiments, drum sound separation processing is performed on each target music signal to obtain a drum sound track signal of each target music signal, including: performing the following steps for each target music signal: performing Fourier transform on the target music signal to obtain a complex spectrum of the target music signal; processing the complex spectrum using a drum sound separation model to obtain a drum sound frequency domain signal output by the drum sound separation model; and performing inverse Fourier transform on the drum sound frequency domain signal to obtain a drum sound track signal of the target music signal.

[0007] In some embodiments, a complex spectrum is processed using a drum sound separation model to obtain a drum sound frequency domain signal output by the drum sound separation model, including: reconstructing sub-band features of the complex spectrum to obtain reconstructed sub-band features; performing feature modeling processing on the reconstructed sub-band features to obtain target modeling features; performing feature recombination processing on the target modeling features to obtain recombined features; and obtaining a drum sound frequency domain signal based on the complex spectrum and the recombined features.

[0008] In some embodiments, the step of performing feature modeling processing on the reconstructed sub-band features to obtain target modeling features is executed by a feature modeling module, the feature modeling module includes several target modules arranged in cascade, and the features output by the last target module are used as target modeling features. The feature modeling module performs feature modeling processing on the reconstructed sub-band features to obtain target modeling features, including: extracting the current target features as the input of the current target module to obtain extracted features, where the current target features are the reconstructed sub-band features or the features output by the previous target module; performing a first fusion processing on the current target features and the extracted features to obtain initial fusion features; performing a second fusion processing on the initial fusion features and the candidate fusion features to obtain advanced fusion features of the target music signal, where the candidate fusion features are advanced fusion features of the historical target music signal; performing a third fusion processing on the advanced fusion features and the initial fusion features of the target music signal to obtain target fusion features, and using the target fusion features as the features output by the current target module.

[0009] In some embodiments, feature recombination processing is performed on the target modeling features to obtain recombined features, including: feature splitting processing is performed on the target modeling features to obtain a number of modeling sub-band features; for each modeling sub-band feature, the modeling sub-band feature is subjected to preset feature transformation processing to obtain a sub-band mask feature of the modeling sub-band feature; the sub-band mask features are combined to obtain a full-band mask feature, and the full-band mask feature is used as the recombined feature.

[0010] In some embodiments, drum sound detection processing is performed on each drum track signal to obtain a drum sound detection result of each target music signal, including: performing the following steps for the drum track signal of each target music signal: calculating the signal energy of the drum track signal of the target music signal to obtain the initial signal energy of the target music signal; determining the target signal energy of the target music signal based on the difference between the initial signal energy and the candidate signal energy, the candidate signal energy being a preset signal energy or the initial energy signal of the target music signal of the previous frame; in response to the target signal energy of the target music signal being greater than an energy threshold, determining the drum sound detection result of the target music signal as the target music signal being in a state of having drum sound; or, in response to the target signal energy of the drum track signal being less than or equal to the energy threshold, determining the drum sound detection result of the target music signal as the target music signal being in a state of not having drum sound.

[0011] In some embodiments, several frames of target music signals are obtained by interval sampling of music audio including initial music signals, and each target music signal is processed based on the drum sound detection result of each target music signal and the drum sound track signal of each target music signal to obtain the processing result of the music audio, including: performing the following steps for the drum sound track signal of each target music signal: in response to the drum sound detection result of the target music signal being that the target music signal is in a drum sound state, superimposing the target music signal and the drum sound track signal of the target music signal to obtain a first processing result of the target music signal; and / or, in response to the drum sound detection result of the target music signal being different from the drum sound detection result of an adjacent target music signal, smoothing the signal to be processed based on the drum sound track signal of the target music signal to obtain a second processing result of the signal to be processed, the signal to be processed being an initial music signal between the target music signal and the adjacent target music signal.

[0012] The present application provides a music audio processing device, including: an acquisition module, a separation module, a detection module and a processing module; the acquisition module is used to acquire a plurality of frames of target music signals in the music audio; the separation module is used to perform drum sound separation processing on each target music signal to obtain a drum sound track signal of each target music signal; the detection module is used to perform drum sound detection processing on each drum sound track signal to obtain a drum sound detection result of each target music signal, and the drum sound detection result of each target music signal includes that the target music signal is in a state with drum sound or the target music signal is in a state without drum sound; the processing module is used to process each target music signal based on the drum sound detection result of each target music signal and the drum sound track signal of each target music signal to obtain a processing result of the music audio.

[0013] The present application provides an electronic device, including a memory and a processor, wherein the processor is used to execute program instructions stored in the memory to implement the above-mentioned music audio processing method.

[0014] The present application provides a computer-readable storage medium on which program instructions are stored. When the program instructions are executed by a processor, the above-mentioned music audio processing method is implemented.

[0015] In the above scheme, the drum sound track signal of each target music signal obtained by performing drum sound separation processing on each frame of the acquired target music signal is a purer signal related to the drum sound, and then drum sound detection processing is performed according to each drum sound track signal to obtain the drum sound detection result of each target music signal, so that the drum sound detection result of each target music signal obtained is more accurate, and then each target music signal is processed based on the drum sound detection result of each target music signal and the drum sound track signal of each target music signal, so as to obtain a processing result of music audio with higher sound quality.

[0016] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solution of the present application.

[0018] Figure 1 This is a flowchart of an embodiment of the music audio processing method of the present application. Figure 1 ;

[0019] Figure 2 This is a flowchart of an embodiment of the music audio processing method of the present application. Figure 2 ;

[0020] Figure 3 This is a flowchart of an embodiment of the music audio processing method of the present application. Figure 3 ;

[0021] Figure 4 This is a flowchart of an embodiment of the music audio processing method of the present application. Figure 4 ;

[0022] Figure 5 This is a flowchart of an embodiment of the music audio processing method of the present application. Figure 5 ;

[0023] Figure 6 This is a flowchart of an embodiment of the music audio processing method of the present application. Figure 6 ;

[0024] Figure 7a is a time domain waveform diagram of music audio in an embodiment of the music audio processing method of the present application;

[0025] Figure 7b is a time domain waveform diagram of a target music signal in an embodiment of the music audio processing method of the present application;

[0026] Figure 7c is a time domain waveform diagram of the processing result of the music audio in the first embodiment of the music audio processing method of the present application;

[0027] Figure 8 7 is a flowchart of an embodiment of a music audio processing method of the present application;

[0028] Figure 9a is a schematic diagram of a drum sound separation model in an embodiment of the music audio processing method of the present application;

[0029] Figure 9bis a schematic diagram of a complex spectrum in an embodiment of the music audio processing method of the present application;

[0030] Fig.9c is a schematic diagram of a drum track signal in an embodiment of the music audio processing method of the present application;

[0031] Fig.10 It is a structural schematic diagram of an embodiment of a music and audio processing device of the present application;

[0032] Fig.11 It is a structural schematic diagram of an embodiment of the electronic device of the present application;

[0033] Fig.12 It is a structural diagram of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0034] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.

[0035] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0036] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the objects associated before and after are in an "or" relationship. In addition, "many" in this article means two or more than two. In addition, the term "at least one" in this article means any combination of at least two of any one or more of a plurality of, for example, including at least one of A, B, and C, can mean including any one or more elements selected from the set consisting of A, B, and C.

[0037] The present application provides some music audio processing methods and music audio processing devices. The application scenarios of the music audio processing method include but are not limited to processing the music audio played in a vehicle. The executor of the music audio processing method may be a music audio processing device. For example, the music audio processing device may be arranged in a terminal device or a server or other processing device, wherein the terminal device may be a device for music audio processing, a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, etc. In some possible implementations, the music audio processing method may be implemented by a processor calling computer-readable instructions stored in a memory.

[0038] See also Figure 1 , Figure 1 This is a flowchart of an embodiment of the music audio processing method of the present application. Figure 1 Specifically, the music audio processing method may include the following steps:

[0039] Step S11: Acquire several frames of target music signals in the music audio.

[0040] The music audio may be a multi-track audio source related to music that needs to be played. For example, the music audio may be a song. The music audio includes vocal track audio sources, drum track audio sources, bass track audio sources, and other track audio sources. The music audio is a continuous audio signal. The music audio includes a continuous number of frames of initial music signals. The target music signals of several frames are at least part of the initial music signals of several frames.

[0041] Perform interval sampling on each initial music signal contained in the music audio to obtain at least one frame of initial music signal, and use the initial music signal obtained by the interval sampling as several frames of target music signal. The acquisition method in the above step S11 can be directly several frames of target music signal. The interval sampling method can be to divide the initial music signals contained in the music audio into frames according to the frame length and frame shift to obtain a series of frames, and determine each frame in the series of frames as each target music signal. The frame length can be obtained by determining the sampling points corresponding to each frame according to the sampling rate of the audio. Frame shift can refer to determining the overlap amount between adjacent frames. The frame length and frame shift are determined according to the sampling frequency in the music audio processing accuracy requirements.

[0042] Step S12: performing drum sound separation processing on each target music signal to obtain a drum sound track signal of each target music signal.

[0043] The drum track signal of the target music signal may be used to indicate a signal including only the drum track in the target music signal.

[0044] For each target music signal, the step of drum sound separation processing is performed on the target music signal to obtain the drum sound track signal of the target music signal. The drum sound separation processing method can be: separate the target music signal to obtain the drum sound track signal in the target music signal. Specifically, the drum sound separation processing method can be to first convert the target music signal to the frequency domain. Perform spectral analysis on the target music signal after frequency domain conversion to obtain spectral features. Then perform mask estimation on the spectral features to obtain mask features Mask. The mask feature Mask is used to separate the target signal (such as drum sound) and other signals from the target music signal. Use the mask feature Mask to separate the target music signal to obtain a spectral signal related to the drum sound. Then perform time domain conversion on the separated spectral signal related to the drum sound to obtain a drum sound track signal. Specifically, it is also possible to use a pre-trained deep learning model to perform drum sound separation on each target music signal to obtain the drum sound track signal of each target music signal.

[0045] It can be considered that at this time, the drum track signal of each target music signal is pure drum related information, avoiding interference from other information except the drum sound.

[0046] Step S13: performing drum sound detection processing on each drum sound track signal to obtain a drum sound detection result of each target music signal.

[0047] The drum sound detection result of each target music signal includes whether the target music signal is in a state with drum sound or in a state without drum sound.

[0048] The target music signal being in a drum sound state can be used to indicate that the energy or amplitude of the drum track signal of the target music signal reaches a threshold. The target music signal being in a no drum sound state can be used to indicate that the energy or amplitude of the drum track signal of the target music signal does not reach a threshold.

[0049] The drum sound detection processing method may be to determine whether there is a drum sound state according to the energy or amplitude of the drum sound track signal of each frame of the target music signal calculated. Specifically, the energy or amplitude of the drum sound track signal may be calculated by using short-time energy or amplitude envelope.

[0050] It can be considered that the drum sound detection result obtained by using a pure drum sound track signal to perform drum sound detection processing is more accurate.

[0051] Step S14: processing each target music signal based on the drum sound detection result of each target music signal and the drum sound track signal of each target music signal to obtain a processing result of the music audio.

[0052] The processing result of the music audio may represent the audio processing result obtained by superimposing at least part of the signals in the music audio. Specifically, the processing result of the music audio includes the processing results of each target music signal.

[0053] Among them, the way to obtain the processing result of the music audio may be to directly perform signal superposition processing on each target music signal to obtain the audio processing result, or to first perform signal upmix processing on each target music signal to obtain the upmix processing result of each target music signal, and then perform signal superposition processing on the upmix processing result of each target music signal to obtain the audio processing result. Specifically, the drum sound detection result of the target music signal may match at least one signal superposition processing method. In some application scenarios, when the drum sound detection result of the target music signal is that the target music signal is in a state without drum sound, the signal superposition processing method may be not to perform signal superposition processing on the target music signal, and only retain the target music signal. The processing result of the target music signal may only include the target music signal. In other application scenarios, when the drum sound detection result of the target music signal is that the target music signal is in a state without drum sound, the signal superposition processing method may be to superimpose the drum sound track signal of the target music signal and the target music signal. The processing result of the target music signal may include the superposition result of the target music signal and the drum sound track signal of the target music signal.

[0054] It can be considered that the sound quality of the music audio obtained by performing signal superposition processing on at least part of the target music signal in the music audio using the pure drum track signal is higher.

[0055] In the above scheme, the drum sound track signal of each target music signal obtained by performing drum sound separation processing on each frame of the acquired target music signal is a purer signal related to the drum sound, and then drum sound detection processing is performed according to each drum sound track signal to obtain the drum sound detection result of each target music signal, so that the drum sound detection result of each target music signal obtained is more accurate, and then each target music signal is processed based on the drum sound detection result of each target music signal and the drum sound track signal of each target music signal, so as to obtain a processing result of music audio with higher sound quality.

[0056] See also Figure 2 , Figure 2 This is a flowchart of an embodiment of the music audio processing method of the present application. Figure 2 .

[0057] In some embodiments, the above step S12 may include the following steps: for each target music signal, perform the following Figure 2The following steps are shown: Step S21: Perform Fourier transform on the target music signal to obtain the complex spectrum of the target music signal. Step S22: Use the drum sound separation model to process the complex spectrum to obtain the drum sound frequency domain signal output by the drum sound separation model. Step S23: Perform inverse Fourier transform on the drum sound frequency domain signal to obtain the drum sound track signal of the target music signal.

[0058] The Fourier transform may be a short-time Fourier transform (STFT), a fast Fourier transform, or a discrete Fourier transform. The complex spectrum of the target music signal may represent the frequency domain signal of the target music signal.

[0059] The drum sound separation model may be a pre-trained deep learning model for performing drum sound separation on the frequency domain signal of the target music signal. The drum sound separation model is used to separate the drum sound frequency domain signal from the frequency domain signal of the target music signal. The drum sound frequency domain signal is a frequency domain signal related only to the drum sound track in the frequency domain signal of the target music signal.

[0060] The inverse Fourier transform is the inverse process of the above-mentioned Fourier transform. According to different modes of Fourier transform, the inverse Fourier transform can be set accordingly. Exemplarily, the inverse Fourier transform can be set to an inverse short-time Fourier transform (ISTFT). The drum track signal of the target music signal can characterize the time domain signal related to the drum track in the target music signal. The drum frequency domain signal is converted into the time domain using the inverse Fourier transform to obtain the drum track signal of the target music signal.

[0061] For example, the target music signal can be represented as an input signal x, and the target music signal x is subjected to a short-time Fourier transform to obtain a complex spectrum X of the target music signal. The drum sound separation model can be represented as a Drum Separation model. The complex spectrum X of the target music signal is input into the drum sound separation model to obtain the separated drum sound frequency domain signal output by the drum sound separation model. Frequency domain signal of drum sound Perform inverse short-time Fourier transform to obtain the drum track signal

[0062] It can be considered that the drum track signal output by the drum sound separation model is a purer drum sound information in the target music signal.

[0063] See also Figure 3 , Figure 3 This is a flowchart of an embodiment of the music audio processing method of the present application. Figure 3 .

[0064] In some embodiments, the above step S22 may include the following steps: Step S31: reconstructing the sub-band features of the complex spectrum to obtain reconstructed sub-band features. Step S32: performing feature modeling processing on the reconstructed sub-band features to obtain target modeling features. Step S33: performing feature reorganization processing on the target modeling features to obtain reorganized features. Step S34: obtaining a drum sound frequency domain signal based on the complex spectrum and the reorganized features.

[0065] Step S31: reconstruct sub-band characteristics of the complex spectrum to obtain reconstructed sub-band characteristics.

[0066] Sub-band feature reconstruction may be to perform feature reconstruction processing on each sub-band feature corresponding to the complex spectrum. The above step S31 may be to perform feature reconstruction processing on each sub-band feature corresponding to the complex spectrum to obtain the reconstructed sub-band feature corresponding to the target music signal.

[0067] Specifically, the complex spectrum is subjected to sub-band splitting processing to obtain the sub-band features corresponding to the complex spectrum. The sub-band features corresponding to the complex spectrum are normalized to obtain the normalized sub-band features. The normalized sub-band features are subjected to feature transformation processing to obtain the reconstructed sub-band features.

[0068] Among them, each sub-band feature corresponding to the complex spectrum may be input into a normalization module for normalization processing to obtain each normalized sub-band feature output by the normalization module. The normalization module may be a module or network with a normalization function. Exemplarily, the normalization module may be a BN module. The BN module may be a (BN, Batch Normalization) module. The normalization module may also be an iLN module. The iLN module is an instant layer normalization (iLN, instant Layernormalization) module. Among them, each normalized sub-band feature may be input into a fully connected layer for feature transformation processing to obtain the features output by the fully connected layer, and the features output by the fully connected layer are used as the reconstructed sub-band features. The fully connected layer may refer to FC (FC, Full Connection).

[0069] Exemplarily, the above step S31 may be performed by a feature reconstruction module. The feature reconstruction module may be a Bandsplit Module. First, the complex spectrum X of the target music signal is input into the feature reconstruction, and the frequency band is split and the feature is transformed according to the predetermined sub-band bandwidth to obtain the reconstructed sub-band feature X. s ′ The specific steps are as follows:

[0070] The complex spectrum X is split into bands according to the predetermined sub-band bandwidth to obtain the sub-band feature X sAs shown in Table 1, the following table may be a table that takes a 44.1kHz music signal as an example and divides the frequency band into 22 sub-bands.

[0071] Table 1: Example of subband bandwidth settings

[0072] Band Width Subband Width Subband Num K ≥16kHz 6.05kHz 1 8kHz-16kHz 2kHz 4 4kHz-8kHz 1kHz 4 2kHz-4kHz 500Hz 4 1kHz-2kHz 250Hz 4 <1kHz 200Hz 5

[0073] Band Width, Subband Width, and Subband Num represent the music audio band, subband bandwidth, and number of subbands, respectively. K is used to represent the number of subbands. Subband feature X s After normalization and feature transformation through the normalization module and the fully connected layer, the reconstructed sub-band feature X is obtained. s ′ .

[0074] Step S32: performing feature modeling processing on the reconstructed sub-band features to obtain target modeling features.

[0075] The target modeling features are features obtained by performing single-frame feature modeling and / or inter-frame feature modeling on the reconstructed sub-band features.

[0076] Single-frame feature modeling and / or inter-frame feature modeling are performed on the reconstructed sub-band features to obtain target modeling features. In some application scenarios, single-frame feature modeling is performed on the reconstructed sub-band features to obtain target modeling features. Specifically, single-frame feature modeling may refer to directly extracting features from the reconstructed sub-band features to obtain extracted features corresponding to the reconstructed sub-band features. Among them, the extracted features corresponding to the reconstructed sub-band features are directly used as target modeling features. The method of extracting features from the reconstructed sub-band features may be to use a preset feature extraction network to extract harmonic information related to the frequency domain in the reconstructed sub-band features. The preset feature extraction network may be a network such as a convolutional neural network, a recurrent neural network, a long short-term memory network, a gated recurrent unit, a BiGRU neural network, etc. that can extract harmonic information related to the frequency domain. For example, the preset feature extraction network may include at least a BiGRU neural network. In other application scenarios, inter-frame feature modeling is performed on the reconstructed sub-band features to obtain target modeling features. Specifically, the method of performing inter-frame feature modeling on the reconstructed subband features can be: for each reconstructed subband feature of the target music signal, the reconstructed subband feature of the target music signal is input into the gated cycle unit for processing to obtain the features output by the gated cycle unit. Among them, the features output by the gated cycle unit are directly used as the target modeling features. In other application scenarios, single-frame feature modeling and inter-frame feature modeling are performed on the reconstructed subband features in turn to obtain the target modeling features. Specifically, feature extraction is performed on the reconstructed subband features to obtain the extracted features corresponding to the reconstructed subband features. The extracted features corresponding to the reconstructed subband features are input into the gated cycle unit for processing to obtain the features output by the gated cycle unit. And the features output by the gated cycle unit are used as the target modeling features. It can be understood that the hidden state in the gated cycle unit implicitly represents the features output by the gated cycle unit for the historical target music signal. The target music signal can be understood as the current target music signal. When the extracted features of the current target music signal are input into the gated loop unit, the extracted features of the current target music signal are fused with the features output by the gated loop unit for the historical target music signal, and the features output by the current target music signal are obtained after being processed by the gated loop unit. It can be understood that the present application takes the example of serially executing single-frame feature modeling and inter-frame feature modeling on the reconstructed sub-band features to obtain the target modeling features. In addition, the present application does not limit the order of executing the steps of executing single-frame feature modeling and executing the steps of executing inter-frame feature modeling.

[0077] See also Figure 4 , Figure 4 This is a flowchart of an embodiment of the music audio processing method of the present application. Figure 4 .

[0078] In some embodiments, the above step S32 is performed by a feature modeling module, which includes a plurality of target modules in a cascade arrangement, and the features output by the last target module are used as target modeling features. The way in which the feature modeling module performs the above step S32 may include the following: Figure 4 The following steps are shown: Step S41: The current target feature is extracted as the input of the current target module to obtain the extracted feature. The current target feature is the reconstructed subband feature or the feature output by the previous target module. Step S42: The current target feature and the extracted feature are subjected to a first fusion process to obtain an initial fusion feature. Step S43: The initial fusion feature and the candidate fusion feature are subjected to a second fusion process to obtain an advanced fusion feature of the target music signal. The candidate fusion feature is an advanced fusion feature of the historical target music signal. Step S44: The advanced fusion feature of the target music signal and the initial fusion feature are subjected to a third fusion process to obtain a target fusion feature. The target fusion feature is used as the feature output by the current target module.

[0079] The several target modules may be one target module or multiple target modules. The input of the feature modeling module is used as the input of the first target module. The features output by the last target module are used as the output of the feature modeling module. The network structures of the target modules are the same, but the network parameters may be different. The target module includes a first submodule and a second submodule. The first submodule is used to implement single-frame feature modeling. The second submodule is used to implement inter-frame feature modeling. Specifically, the first submodule includes a preset feature extraction network, a fully connected layer, a normalization module, a residual connection module, and a dimension permutation module in sequence. Specifically, the second submodule includes a gated recurrent unit, a fully connected layer, a normalization module, a residual connection module, and a dimension permutation module in sequence.

[0080] When the current target feature is a reconstructed subband feature, the current target module is the first target module. The reconstructed subband feature is processed by the first submodule and the second submodule in the first target module in turn to obtain the feature output by the first target module. When the current target feature is a feature output by the previous target module, the current target module is the first target module. The feature output by the previous target module is processed by the first submodule and the second submodule in the next target module in turn to obtain the feature output by the next target module.

[0081] Exemplarily, the current target module includes a current first submodule and a current second submodule. Specifically, the current first submodule includes a preset feature extraction network, a fully connected layer, a normalization module, a residual connection module, and a dimension permutation module. Specifically, the current second submodule includes a gated recurrent unit, a fully connected layer, a normalization module, a residual connection module, and a dimension permutation module.

[0082] In some application scenarios, the above step S41 may be a step in which the preset feature extraction network in the current first submodule directly extracts the current target feature as the input of the current target module to obtain the extracted feature. The above step S42 may be a step in which the residual connection module in the current first submodule performs a first fusion process on the current target feature and the extracted feature to obtain the initial fusion feature. The above step S43 may be a step in which the gated cycle unit in the current second submodule performs a second fusion process on the initial fusion feature and the candidate fusion feature to obtain the advanced fusion feature of the target music signal. The above step S44 may be a step in which the residual connection module in the current second submodule directly performs a third fusion process on the advanced fusion feature and the initial fusion feature of the target music signal to obtain the target fusion feature.

[0083] In some other application scenarios, the above step S41 may be to input the current target feature into the preset feature extraction network in the current first submodule to perform feature extraction processing on the current target feature to obtain the extracted feature. After the above step S41, the extracted feature is input into the fully connected layer in the current first submodule for feature transformation to obtain the feature output by the fully connected layer in the current first submodule. The feature output by the fully connected layer in the current first submodule is input into the normalization module in the current first submodule for normalization processing to obtain the feature output by the normalization module in the current first submodule. The above step S42 may include: using the current target feature and the feature output by the normalization module in the current first submodule as the input of the residual connection module in the current first submodule for the first fusion processing to obtain the initial fusion feature.

[0084] In some other application scenarios, after the above step S42, the features output by the residual connection module in the current first submodule, that is, the initial fusion features, are used as the input of the dimension permutation module in the current first submodule for dimension permutation processing to obtain the output of the dimension permutation module in the current first submodule. It can be considered that the feature dimension in the output of the dimension permutation module in the current first submodule satisfies the feature dimension requirement required for the input of the current second submodule. The above step S43 may also include: using the output of the dimension permutation module in the current first submodule as the input of the gated recurrent unit in the current second module, and performing a second fusion process with the candidate fusion features to obtain an advanced fusion feature corresponding to the target music signal.

[0085] In other application scenarios, after the above step S43, the advanced fusion features corresponding to the target music signal obtained in the above step S43 are input into the fully connected layer in the current second submodule for feature transformation to obtain the features output by the fully connected layer in the current second submodule. The features output by the fully connected layer in the current second submodule are input into the normalization module in the current second submodule for normalization processing to obtain the features output by the normalization module in the current second submodule. The above step S44 can be inputting the features output by the normalization module in the current second submodule and the initial fusion features into the residual connection module in the current second submodule for a third fusion processing to obtain the output of the residual connection module in the current second submodule, and using the features output by the residual connection module in the current second submodule as the target fusion features of the target music signal.

[0086] In other application scenarios, the features output by the residual connection module in the current second submodule are used as the input of the dimension permutation module in the current second submodule for dimension permutation processing to obtain the output of the dimension permutation module in the current second submodule, and the features output by the dimension permutation module in the previous second submodule are used as the target fusion features of the target music signal.

[0087] It can be understood that the first fusion process and the third fusion process are feature fusion processes related to residual connection. The second fusion process is feature fusion processing of the initial fusion features of the target music signal related to the gated recurrent unit and the advanced fusion features of the historical target music signal.

[0088] Exemplarily, the present application takes the number of target modules as 2 target modules as an example.

[0089] The above step S32 may be inputting the reconstructed sub-band feature into the feature modeling module, wherein the feature modeling module may be a Separation Module module. Specifically, the reconstructed sub-band feature X s ′ After the feature modeling module completes single-frame modeling and / or inter-frame modeling, the target modeling feature F is obtained. The steps are as follows: The target module can be represented as a DPRNN blocks module. The first submodule can be represented as an Intra RNN unit. The second submodule can be represented as an Inter RNN unit. Reconstruct the subband feature X s ′ After being processed by the first submodule in the first target module, the initial fusion feature is obtained Initial fusion features After the dimension permutation module in the first submodule performs dimension permutation, the second submodule in the first target module performs processing to obtain the target fusion feature output by the first target module. The target fusion feature output by the first target module After dimension permutation, the last target module is passed to obtain the target fusion feature output by the last module. The feature output by the last target module is used as the target modeling feature. The target modeling feature can be represented as F. For example, the Intra RNN unit uses a preset feature extraction network to model the spectral pattern of a single frame. The preset feature extraction network is set to BiGRU. The Inter RNN unit uses a gated recurrent unit GRU to model inter-frame features. It can be considered that both the preset feature extraction network and the gated recurrent unit are followed by a fully connected layer and a normalization module for feature transformation and normalization, and the external residual connection can alleviate the gradient disappearance.

[0090] Step S33: performing feature recombination processing on the target modeling features to obtain reorganized features.

[0091] The feature reorganization process may be to first perform sub-band splitting on the target modeling features, and then perform a preset transformation process on the target modeling features after the sub-band splitting process to obtain features as reorganized features. The preset transformation process may be to input the target modeling features after the sub-band splitting process into a preset fully connected module for feature transformation process. The preset fully connected module includes at least one fully connected layer.

[0092] In some embodiments, the above step S33 may include the following steps: first, perform feature splitting processing on the target modeling feature to obtain a plurality of modeling sub-band features. Then, for each modeling sub-band feature, perform a preset feature transformation processing on the modeling sub-band feature to obtain a sub-band mask feature of the modeling sub-band feature. Then, combine the sub-band mask features to obtain a full-band mask feature. And use the full-band mask feature as the recombined feature.

[0093] The number of modeling subband features may be the same as the number of subbands of the target music signal. Several may refer to one or more. Each modeling subband feature is sequentially input into two fully connected layers for feature transformation processing to obtain a subband mask feature of each modeling subband feature. The subband mask features of each modeling subband feature are combined in subband order to obtain a full-band mask feature. Among them, the full-band mask feature can be represented as a full-band Mask.

[0094] For example, the feature reorganization module can be represented as a Bandmerge Module module. After the modeling is completed, the target modeling feature F is input to estimate the Mask M of each subband s , reorganize to obtain the full-band mask feature Mask M F The specific steps are as follows: the target modeling feature F after modeling is split into the corresponding sub-band features after modeling. The sub-band features after modeling, that is, the sub-band features F sAfter being processed by the normalization module, the fully connected layer followed by the activation function Tanh, and the fully connected layer followed by the activation function GLU, the corresponding mask features of each sub-band Mask M are obtained. s Combine the mask features of each sub-band in order s Get the full-band mask feature Mask M F . And the full-band mask features are used as the reorganized features.

[0095] Step S34: Obtain the drum sound frequency domain signal based on the complex spectrum and the reorganized features.

[0096] In some application scenarios, the complex spectrum and the reorganized features are convolved to obtain a drum sound frequency domain signal. Specifically, a mask is generated through a deconvolution layer or a transposed convolution layer or the drum sound frequency domain signal is directly output. In other application scenarios, the complex spectrum and the reorganized features are directly multiplied to obtain a drum sound frequency domain signal. Exemplarily, the complex spectrum of the target music signal is multiplied point by point with the reorganized features to obtain a separated drum sound frequency domain signal.

[0097] In some embodiments, the music audio processing method may further include a training step for a drum sound separation model, the training step including: obtaining training data. Inputting the time-frequency converted training data into the drum sound separation model to obtain a sample drum sound frequency domain signal output by the drum sound separation model. Performing ISTFT conversion on the sample drum sound frequency domain signal to obtain a sample drum sound track signal. Determining a target loss based on the difference between the sample drum sound track signal and the sample label. Adjusting the parameters in the drum sound separation model based on the target loss to obtain a trained drum sound separation model. The above step S22 is performed by the trained drum sound separation model.

[0098] Among them, the time-frequency conversion can be the above-mentioned short-time Fourier transform. The ISTFT conversion can be the above-mentioned inverse short-time Fourier transform. The combined Loss band segmentation scheme may cause the initial phase inconsistency problem. In addition to the existing waveform reconstruction L1Loss loss, this application uses multiple STFT Loss losses with different analysis parameters to enable the model to better learn the time-frequency characteristics of the audio, avoid overfitting to a specific STFT representation, and thus improve performance. The loss function used by the final model is l.

[0099] Specifically, the process of determining the target loss can refer to formula (1), formula (2) and formula (3):

[0100] l=l MAE +l M_STFT Formula (1);

[0101]

[0102] Among them, lMAE It can represent the L1 Loss function of waveform reconstruction in the time domain. M_STFT It can represent the multi-resolution STFTLoss loss function. Where y represents the corresponding sample label in the training data as the pure drum reference signal, that is, the groundtruth Drums label. It can represent the output signal of the drum sound separation model, that is, the sample drum sound track signal obtained by the drum sound separation model Among them, the multi-resolution STFT Loss is obtained by summing the MAELoss of STFTs calculated with different FFT sizes, window sizes and step sizes (M represents the number of STFTs with different resolutions).

[0103] Split the stereo into left and right channels, and the superimposed signals of the left and right channels. When training the model, the extracted Drums sample signal is remixed into new training samples in proportion with randomly extracted Vocal / Bass / Other track segments of different songs in the same time period. Among them, the extracted Drums sample signal is used as the groundtruth Drums label in the training data, that is, the sample label represents the pure drum reference signal y. Remix it into new training data, and use the new training data as the input of the drum sound separation model. It can be considered that data enhancement of the training data improves the generalization of the drum sound separation model.

[0104] It can be considered that the drum sound separation model of the present application is a lightweight drum sound separation model. The drum sound separation model can be based on a frequency band splitting model and can separate the drum sound part of the music in real time and with fidelity.

[0105] See also Figure 5 , Figure 5 This is a flowchart of an embodiment of the music audio processing method of the present application. Figure 5 .

[0106] In some embodiments, the above step S13 may include the following steps: for each drum track signal of the target music signal, performing the following Figure 5The following steps are shown: Step S51: Calculate the signal energy of the drum track signal of the target music signal to obtain the initial signal energy of the target music signal. Step S52: Determine the target signal energy of the target music signal based on the difference between the initial signal energy and the candidate signal energy. The candidate signal energy is a preset signal energy or the initial energy signal of the target music signal of the previous frame. Step S53: In response to the target signal energy of the target music signal being greater than the energy threshold, determine the drum sound detection result of the target music signal as the target music signal being in a drum sound state. Step S54: In response to the target signal energy of the drum track signal being less than or equal to the energy threshold, determine the drum sound detection result of the target music signal as the target music signal being in a drum sound-free state.

[0107] The initial energy signal of the target music signal is used to represent the signal energy of the drum track signal of the target music signal. The signal energy of the drum track signal is obtained by using at least one method of calculating the energy or amplitude of the drum track signal, and the signal energy of the drum track signal is used as the initial energy signal of the target music signal.

[0108] The target signal energy of the target music signal is the absolute value of the difference between the initial signal energy and the candidate signal energy. When there is no historical target music signal before the target music signal, that is, when the target music signal is the first signal among several target music signals, the candidate signal energy is the preset signal energy. When the target music signal is not the first signal among several target music signals, the candidate signal energy is the initial signal energy of the target music signal of the previous frame. After step S52, it is determined whether the target signal energy of the target music signal is greater than the energy threshold. If the judgment result is yes, execute step S53. If the judgment result is no, execute step S54.

[0109] Exemplarily, when the drum sound detection result of the target music signal indicates that the target music signal is in a state of having drum sound, the drum sound detection result of the target music signal may be identified as 1. When the drum sound detection result of the target music signal indicates that the target music signal is in a state of not having drum sound, the drum sound detection result of the target music signal may be identified as 0.

[0110] Exemplarily, the drum track signal energy of the current frame is calculated. The process of determining the target signal energy of the target music signal can refer to formula (4) and formula (5):

[0111]

[0112] in, It can represent the initial signal energy of the target music signal. Let the drum track signal of the current frame be d i(n), n represents the number of sampling points, i represents the i-th frame signal (assuming the current frame is the i-th frame). i Used to indicate the calculation of the drum track signal energy of the current frame The energy of the drum track signal of the previous frame It can be understood that if the current frame i is the first frame, it is considered that the energy of the previous frame Each frame signal D i Together they form the drum track signal OF function.

[0113] D i Compared with the set threshold Th, if D i >Th, it is considered that the drum track signal d of the current frame i (n) If there is a drum sound, the drum sound detection result of the target music signal is determined as the target music signal being in a drum sound state, and the broadcast state is set to the drum sound state. Otherwise, it is considered that there is no drum sound, and the drum sound detection result of the target music signal is determined as the target music signal being in a no drum sound state, and the broadcast state is set to the no drum sound state. Store the drum sound track signal energy of the target music signal in the current frame Used to calculate the energy difference D of the next frame signal i+1 .

[0114] It can be considered that using a feature extraction method such as energy difference to perform activity detection on the separated pure drum sound can improve the accuracy of the drum sound detection result.

[0115] It can be considered that beat detection (Beats track) is used to implement the beat detection in the music information retrieval (MIR, Music Information Retrieval) task. The present application detects the drum rhythm in the music audio to obtain the predetermined rhythm of the music audio. Specifically, the rhythm detection can be achieved through the onset function (OF, Onset Function) and the tempo estimation (TE, Tempo Estimate). The onset function OF is used to extract signal features such as the energy envelope in the music. The tempo estimation TE is used to estimate the position where the beat appears. The drum track signal obtained by converting the drum frequency domain signal separated by the drum separation model of the present application into the time domain Therefore, the rhythm detection task can be simplified to drum activity detection, and the drum track signal is extracted by energy difference and other feature extraction methods. The OF is used to make threshold decisions, which can reduce the amount of calculation in the music audio processing process.

[0116] In some embodiments, a plurality of frames of target music signals are obtained by sampling music audio including each initial music signal at intervals, and the above step S14 may include the following steps: for each drum track signal of the target music signal, the following steps are performed: in response to the drum sound detection result of the target music signal being that the target music signal is in a drum sound state, the target music signal and the drum track signal of the target music signal are superimposed to obtain a first processing result of the target music signal. And / or, in response to the drum sound detection result of the target music signal being different from the drum sound detection result of the adjacent target music signal, the drum track signal of the target music signal is smoothed to obtain a second processing result of the signal to be processed. The signal to be processed is an initial music signal between the target music signal and the adjacent target music signal.

[0117] See also Figure 6 , Figure 6 This is a flowchart of an embodiment of the music audio processing method of the present application. Figure 6 .

[0118] For each drum track signal of the target music signal, the following is performed: Figure 6 The following steps are shown: Step S61: In response to the drum sound detection result of the target music signal being that the target music signal is in a drum sound state, the target music signal and the drum sound track signal of the target music signal are superimposed to obtain a first processing result of the target music signal. Step S62: In response to the drum sound detection result of the target music signal being different from the drum sound detection result of the adjacent target music signal, the signal to be processed is smoothed based on the drum sound track signal of the target music signal to obtain a second processing result of the signal to be processed. The signal to be processed is an initial music signal between the target music signal and the adjacent target music signal.

[0119] It is understandable that, in the case where the above step S14 includes step S61 and step S62, step S61 and step S62 may be executed serially or in parallel. In the case where step S61 and step S62 are executed in parallel, the order of step S61 and step S62 is not limited. In the case where the above step S14 includes step S61 or step S62, only step S61 or only step S62 may be executed. Here, the above step S14 including step S61 and step S62 is taken as an example. Please refer to the following content for the process of executing step S61 alone or executing step S62 alone, which will not be repeated later.

[0120] The processing result of the music audio includes the first processing result of each target music signal and / or the second processing result of each signal to be processed. The first processing result of the target music signal is used to characterize whether the drum track signal of the target music signal is used to perform superimposition processing on the target music signal. The second processing result of the signal to be processed is used to characterize whether the drum track signal of the target music signal is used to perform smoothing processing on the signal to be processed. The adjacent target music signal is used to represent the previous target music signal and / or the next target music signal adjacent to the current target music signal.

[0121] Before step S61, it is determined whether the drum sound detection result of the target music signal is the same as the drum sound detection result of the adjacent target music signal. The above step S61 may include the following steps: in response to the drum sound detection result of the target music signal being that the target music signal is in a drum sound state, the target music signal is superimposed with all contents in the drum sound track signal of the target music signal to obtain a first processing result of the target music signal.

[0122] In some application scenarios, before step S62, it is determined whether the drum sound detection result of the target music signal is the same as the drum sound detection result of the adjacent target music signal. The above step S62 can be to perform superposition processing on at least part of the target drum sound track signal and the signal to be processed according to the distance between the signal to be processed and the target music signal, so as to obtain a second processing result of the signal to be processed. Among them, the proportion of at least part of the target drum sound track signal in the target drum sound track signal is related to the distance between the signal to be processed and the target music signal. The target drum sound track signal is a music signal in which the drum sound detection result in the signal to be processed and the target music signal is in a state of having drum sound.

[0123] Exemplarily, when three consecutive frames of target music signals include a first target music signal, a second target music signal, and a third target music signal, the drum sound detection results of the three consecutive frames of target music signals can be identified as 1, 0, and 1, respectively. Specifically, the signal between the first target music signal and the second target music signal is a plurality of first signals to be processed. At least a portion of the first target drum track signal is superimposed with each first signal to be processed to obtain a second processing result of each first signal to be processed. The first target drum track signal is a drum track signal of the first target music signal. Among them, for each first signal to be processed, the closer the first signal to be processed is to the first target music signal, the greater the proportion of at least a portion of the first target drum track signal in the first target drum track signal. Specifically, the signal between the third target music signal and the second target music signal is a plurality of second signals to be processed. At least a portion of the third target drum track signal is superimposed with each second signal to be processed to obtain a second processing result of each second signal to be processed. The third target drum track signal is a drum track signal of the third target music signal. For each second signal to be processed, the closer the second signal to be processed is to the third target music signal, the greater the proportion of at least a portion of the third target drum track signal in the third target drum track signal.

[0124] In some other application scenarios, before step S62, it is determined whether the drum sound detection result of the target music signal is that the target music signal is in a drum sound state. In response to the drum sound detection result of the target music signal being that the target music signal is in a drum sound state, it is determined whether the drum sound detection result of the target music signal is the same as the drum sound detection result of an adjacent target music signal. The above step S62 may be to smooth at least part of the drum sound track signal of the target music signal on the signal to be processed to obtain a second processing result of the signal to be processed.

[0125] Exemplarily, when three consecutive frames of target music signals include a first target music signal, a second target music signal, and a third target music signal, and the drum sound detection results of the three consecutive frames of target music signals can be identified as 0, 1, and 0, respectively. At this time, the target music signal is the second target music signal. The target drum sound track signal is the drum sound track signal of the second target music signal. Specifically, the signal between the first target music signal and the second target music signal is a plurality of first signals to be processed. Directly superimpose at least a portion of the target drum sound track signal with each first signal to be processed to obtain a second processing result of each first signal to be processed. Among them, for each first signal to be processed, the closer the first signal to be processed is to the second target music signal, the greater the proportion of at least a portion of the target drum sound track signal in the first target drum sound track signal. Specifically, the signal between the third target music signal and the second target music signal is a plurality of second signals to be processed. Superimpose at least a portion of the target drum sound track signal with each second signal to be processed to obtain a second processing result of each second signal to be processed. For each second signal to be processed, the closer the second signal to be processed is to the second target music signal, the greater the proportion of at least a portion of the target drum track signal in the third target drum track signal.

[0126] It can be understood that, in the present application, when step S61 and step S62 are executed in parallel, the present application takes the example of first executing step S61 and then executing step S62.

[0127] In some other application scenarios, the drum sound field control module can be represented as a Drum Motion module. The drum sound field control module controls the drum track signal of each target music signal to realize the drum sound flowing in the sound field of the target vehicle at a predetermined rhythm. According to the drum sound detection appearance function OF, the drum sound detection result of each target music signal can be divided into the target music signal in a state without drum sound, the target music signal in a state with drum sound, the target music signal in the attack stage, and the target music signal in the release stage, wherein the drum sound detection result of each target music signal is one of them. Specifically, in the state without drum sound, all channels in the sound field play the same original music signal. In the state with drum sound, a specific channel in the sound field (called a drum sound channel) will superimpose the separated drum sound track signal on the basis of the target music signal. The transition state from the state without drum sound to the state with drum sound is called the attack stage. The attack stage can characterize the state change of the detection OF function from 0 to 1. A certain channel in the sound field is gradually transformed into a drum sound channel according to a pre-designed rhythm. The transition state from the drum sound state to the drum sound-free state is called the Release stage. The Release stage can represent the change of the detection OF function from 1 to 0, and the drum sound channel gradually changes to a normal channel.

[0128] It is understandable that when the drum sound detection result of the target music signal is that the target music signal is in the attack stage or the target music signal is in the release stage, the target music signal is used as the signal to be processed. The target music signal that is closest to the signal to be processed and is in the drum sound state is used as the reference music signal of the signal to be processed. The drum sound track signal of the reference music signal is obtained. At least part of the drum sound track signal of the reference music signal is processed by signal superposition with the signal to be processed or the signal to be processed after signal upmixing.

[0129] For example, let the target music signal be x(n), and the separated drum track signal be In the state without drum sound, all channels in the sound field play signal x(n). In the state with drum sound, the normal channels in the sound field play signal x(n), and the drum channel plays The target object in the target vehicle will perceive that the sound image of the drum sound in the sound field is concentrated on the drum sound channel. In the Attack stage, the signal played on the target channel is transitioned from x(n) to Specifically, the process of obtaining the second processing result of the signal to be processed in the Attack stage can refer to formula (6):

[0130]

[0131] Among them, x A (n) represents the signal broadcast by the target channel in the Attack phase. A Represents the smoothing coefficient, in the attack stage α A From 0 to 1. For example, α A The size of x(n) is related to the distance between the signal to be processed corresponding to x(n) and the reference music signal. The drum track signal may represent the reference music signal. x(n) may represent the signal to be processed or the signal to be processed after the signal upmixing process.

[0132] In the Release phase, the signal played by the target channel is changed from Transition to x(n). Specifically, the process of obtaining the second processing result of the signal to be processed in the Release phase can refer to formula (7):

[0133]

[0134] Among them, x R (n) represents the signal broadcast by the target channel in the Release phase, α R Represents the smoothing coefficient, in the Release stage αR Gradually changes from 1 to 0. For example, α R The size of x(n) is related to the distance between the signal to be processed corresponding to x(n) and the reference music signal. The drum track signal may represent the reference music signal. x(n) may represent the signal to be processed or the signal to be processed after the signal upmixing process.

[0135] The purpose of using Fade in or Fade out is to make the switching of drum sound states smoother and make the broadcast signal more natural. The implementation details in the Attack and Release stages include the application of Fade in and Fade out effects, and the updating of the drum sound channel according to the drum sound detection results and the predetermined rhythm, so that the playback effect in the target vehicle is the effect of the drum sound flowing through each channel.

[0136] In some embodiments, before the above step S14, the music audio processing method may further include the following steps: performing signal upmixing processing on each target music signal to obtain the upmixing processing result of each target music signal. The above step S14 may further include the following steps: processing the upmixing processing result of each target music signal based on the drum sound detection result of each target music signal and the drum sound track signal of each target music signal to obtain the processing result of the music audio.

[0137] Specifically, the above-mentioned step S61 may also include the following steps: in response to the drum sound detection result of the target music signal being that the target music signal is in a drum sound state, the upmix processing result of the target music signal and the drum sound track signal of the target music signal are superimposed to obtain a first processing result of the target music signal.

[0138] Specifically, the above step S62 may also include the following steps: in response to the drum sound detection result of the target music signal being different from the drum sound detection result of the adjacent target music signal, at least part of the target drum sound track signal and the advanced signal to be processed are superimposed and processed according to the distance of the signal to be processed from the target music signal to obtain a second processing result of the signal to be processed. The proportion of at least part of the target drum sound track signal in the target drum sound track signal is related to the distance of the signal to be processed from the target music signal. The target drum sound track signal is a music signal in which the drum sound detection result of the signal to be processed and the target music signal is in a state of having drum sound. The advanced signal to be processed is the upmix processing result of the initial music signal between the target music signal and the adjacent target music signal.

[0139] It can be understood that the music audio is a dual-channel sound source. The same music audio processing operation is performed on each channel sound source in the music audio to obtain the processing result of the music audio. This application takes the music audio processing operation of a channel sound source as an example, and several frames of target music signals in the music audio are signals of the same channel sound source. In the above step of performing signal upmixing processing on each target music signal to obtain the upmixing processing result of each target music signal, the upmixing processing result of each target music signal may be on at least one channel to be played. The above step S14 may also include the following steps: performing the following steps for each target music signal: in response to the drum sound detection result of the target music signal that the target music signal is in a drum sound state, selecting a channel to be played from several channels to be played as the target channel according to a predetermined rhythm. Based on the drum sound detection result of the target music signal and the drum sound track signal of the target music signal, the upmixing processing result of the target music signal on the target channel is processed to obtain the processing result of the music audio. The channel to be played may be an audio playback device such as a speaker or a loudspeaker. Exemplarily, the channel to be played may be a loudspeaker at different positions on the target vehicle.

[0140] See also Figure 7a , Figure 7b as well as Figure 7c 7a is a time domain waveform diagram of the music audio in the first embodiment of the music audio processing method of the present application. 7b is a time domain waveform diagram of the target music signal in the first embodiment of the music audio processing method of the present application. 7c is a time domain waveform diagram of the processing result of the music audio in the first embodiment of the music audio processing method of the present application.

[0141] Figure 7b as well as Figure 7c Figure 7a , Figure 7b as well as Figure 7c This is a time domain waveform comparison chart before and after processing. Specifically, the processing process is to play the drum sound in multiple channels according to the predetermined rhythm, so as to create a dynamic sound effect mode with a flowing spatial sound field. Figure 7a As shown in , the music audio is a two-channel stereo input of the original music. Figure 7b As shown, M1 and M2 are respectively the upmixing processing results of any two target music signals after the signal upmixing processing. Among them, M1 and M2 process two different target channels respectively. When the drum sound detection result of the target music signal M1 is that the target music signal is in a drum sound state, the upmixing processing result of the target music signal M1 is superimposed with the drum sound track signal of the target music signal M1 to obtain the first processing result of the target music signal M1. The first processing result of the target music signal M1 can be expressed as follows Figure 7cWhen the drum sound detection result of the target music signal M2 indicates that the target music signal is in a drum sound state, the upmix processing result of the target music signal M2 is superimposed with the drum sound track signal of the target music signal M2 to obtain a first processing result of the target music signal M2. The first processing result of the target music signal M2 can be expressed as follows Figure 7c m2 shown. It can be considered that the target channels of the target music signal M1 and the target music signal M2 are determined respectively according to the predetermined rhythm, so that the target music signal M1 and the target music signal M2 respectively realize the superposition of the target music signal itself or the upmixing processing result of the target music signal with the respective drum track signals on their respective target channels, and the separated drum track signals are convenient for superimposed playback on multiple channels, and the drum sound can be circulated according to the predetermined rhythm, so that the present application can present a dynamic sound effect mode of drum sound space circulation under the premise of ensuring the sound quality and with a small amount of calculation.

[0142] See also Figure 8 , Figure 8 This is flow chart seven of an embodiment of the music audio processing method of the present application.

[0143] The music audio is a dual-channel sound source. The same music audio processing operation is performed on each channel sound source in the music audio to obtain the music audio processing result. This application takes the music audio processing operation of any channel sound source as an example, and performs the following for the target music signal on each channel sound source. Figure 8 The following steps are shown:

[0144] Steps S81, S82, S83, S84 and S85 are performed in sequence. Step S81: Perform drum sound separation processing on the target music signal to obtain a drum sound track signal of the target music signal. Step S82: Perform drum sound detection processing on the drum sound track signal of the target music signal to obtain a drum sound detection result of the target music signal. Step S83: In response to the drum sound detection result of the target music signal being that the target music signal is in a drum sound state, a to-be-played channel is selected from a plurality of to-be-played channels according to a predetermined rhythm as a target channel. Step S84: In response to the drum sound detection result of the target music signal being that the target music signal is in a drum sound state, the upmix processing result of the target music signal on the target channel is superimposed with the target drum sound track signal to obtain a first processing result of the target music signal. Step S85: In response to the drum sound detection result of the target music signal being different from the drum sound detection result of the adjacent target music signal, smoothing the signal to be processed according to at least a portion of the target drum sound track signal to obtain a second processing result of the signal to be processed.

[0145] It can be understood that the processing result of the music audio includes the first processing result of each target music signal and the second processing result of each signal to be processed. Drum activity detection is performed. If drum sound is detected, it is input to the drum sound field control module, that is, the Drum motion module, and the drum track signal is The corresponding output channel of the target music signal x is superimposed on the target music signal x. The corresponding output channel of the target music signal x is the target channel selected according to the predetermined rhythm. On the basis of improving the sound quality of the processing result of the music audio, when playing the processing result of the music audio, the present application can realize that the drum sound will be played in multiple speakers in the car according to the preset rhythm, presenting a dynamic sound effect mode of the drum sound flowing in the space.

[0146] See also Figure 9a , Figure 9b as well as Fig.9c , Figure 9a is a schematic diagram of a drum sound separation model in an embodiment of the music audio processing method of the present application, Figure 9b is a schematic diagram of a complex spectrum in an embodiment of the music audio processing method of the present application, Fig.9c It is a schematic diagram of a drum track signal in an embodiment of the music audio processing method of the present application.

[0147] As shown in Figure 9, the drum sound separation model includes a feature reconstruction module, a feature modeling module and a feature recombination module. The functions performed by the feature reconstruction module, the feature modeling module and the feature recombination module are referred to above and will not be described here.

[0148] The complex spectrum is input into the drum sound separation model to obtain the drum sound frequency domain signal output by the drum sound separation model. Specifically, the complex spectrum is sequentially passed through the feature reconstruction module, the feature modeling module and the feature recombination module to obtain the full-band mask feature output by the feature recombination module. Then, the complex spectrum of the target music signal is multiplied point by point with the full-band mask feature to obtain the drum sound frequency domain signal output by the drum sound separation model.

[0149] In the above scheme, the drum sound track signal of each target music signal obtained by performing drum sound separation processing on each frame of the acquired target music signal is a purer signal related to the drum sound, and then drum sound detection processing is performed according to each drum sound track signal to obtain the drum sound detection result of each target music signal, so that the drum sound detection result of each target music signal obtained is more accurate, and then each target music signal is processed based on the drum sound detection result of each target music signal and the drum sound track signal of each target music signal, so as to obtain a processing result of music audio with higher sound quality.

[0150] See also Fig.10 , Fig.10It is a structural diagram of an embodiment of a music audio processing device of the present application. The music audio processing device 100 includes an acquisition module 101, a separation module 102, a detection module 103 and a processing module 104. The acquisition module 101 is used to acquire several frames of target music signals in the music audio; the separation module 102 is used to perform drum sound separation processing on each target music signal to obtain a drum sound track signal of each target music signal; the detection module 103 is used to perform drum sound detection processing on each drum sound track signal to obtain a drum sound detection result of each target music signal, and the drum sound detection result of each target music signal includes that the target music signal is in a state with drum sound or that the target music signal is in a state without drum sound; the processing module 104 is used to process each target music signal based on the drum sound detection result of each target music signal and the drum sound track signal of each target music signal to obtain a processing result of the music audio.

[0151] In the above scheme, the drum sound track signal of each target music signal obtained by performing drum sound separation processing on each frame of the acquired target music signal is a purer signal related to the drum sound, and then drum sound detection processing is performed according to each drum sound track signal to obtain the drum sound detection result of each target music signal, so that the drum sound detection result of each target music signal obtained is more accurate, and then each target music signal is processed based on the drum sound detection result of each target music signal and the drum sound track signal of each target music signal, so as to obtain a processing result of music audio with higher sound quality.

[0152] For the functions performed by each module, please refer to the music audio processing method, which will not be repeated here.

[0153] See also Fig.11 , Fig.11 1 is a schematic diagram of the structure of an embodiment of a vehicle of the present application. The electronic device 110 includes a memory 111 and a processor 112, and the processor 112 is used to execute program instructions stored in the memory 111 to implement the steps in the above-mentioned music audio processing method embodiment. In a specific implementation scenario, the electronic device 110 may include but is not limited to: a microcomputer, a server, and in addition, the electronic device 110 may also include a mobile device such as a laptop computer and a tablet computer, which is not limited here.

[0154] Specifically, the processor 112 is used to control itself and the memory 111 to implement the steps in the above-mentioned music audio processing method embodiment. The processor 112 can also be called a CPU (Central Processing Unit). The processor 112 may be an integrated circuit chip with signal processing capabilities. The processor 112 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 112 can be implemented by an integrated circuit chip.

[0155] In the above scheme, the drum sound track signal of each target music signal obtained by performing drum sound separation processing on each frame of the acquired target music signal is a purer signal related to the drum sound, and then drum sound detection processing is performed according to each drum sound track signal to obtain the drum sound detection result of each target music signal, so that the drum sound detection result of each target music signal obtained is more accurate, and then each target music signal is processed based on the drum sound detection result of each target music signal and the drum sound track signal of each target music signal, so as to obtain a processing result of music audio with higher sound quality.

[0156] See also Fig.12 , Fig.12 The computer-readable storage medium 120 stores program instructions 1201, which, when executed by a processor, implement the steps in any of the above-mentioned music and audio processing method embodiments.

[0157] In the above scheme, the drum sound track signal of each target music signal obtained by performing drum sound separation processing on each frame of the acquired target music signal is a purer signal related to the drum sound, and then drum sound detection processing is performed according to each drum sound track signal to obtain the drum sound detection result of each target music signal, so that the drum sound detection result of each target music signal obtained is more accurate, and then each target music signal is processed based on the drum sound detection result of each target music signal and the drum sound track signal of each target music signal, so as to obtain a processing result of music audio with higher sound quality.

[0158] In some embodiments, the functions or modules included in the system provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0159] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0160] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0161] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0162] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

Claims

1. A music audio processing method, characterized in that: The method comprises: Acquire several frames of target music signals in the music audio; Performing drum sound separation processing on each of the target music signals to obtain a drum sound track signal of each of the target music signals; Performing drum sound detection processing on each of the drum sound track signals to obtain drum sound detection results of each of the target music signals, wherein the drum sound detection results of each of the target music signals include the target music signal being in a state of having drum sound or the target music signal being in a state of not having drum sound; Each of the target music signals is processed based on the drum sound detection result of each of the target music signals and the drum sound track signal of each of the target music signals to obtain the processing result of the music audio.

2. The method according to claim 1, characterized in that The step of performing drum sound separation processing on each of the target music signals to obtain a drum sound track signal of each of the target music signals comprises: For each target music signal, the following steps are performed: Performing Fourier transform on the target music signal to obtain a complex spectrum of the target music signal; Processing the complex spectrum using a drum sound separation model to obtain a drum sound frequency domain signal output by the drum sound separation model; Perform inverse Fourier transform on the drum sound frequency domain signal to obtain a drum sound track signal of the target music signal.

3. The method according to claim 2, characterized in that The method of processing the complex spectrum by using the drum sound separation model to obtain a drum sound frequency domain signal output by the drum sound separation model includes: Reconstructing sub-band features of the complex spectrum to obtain reconstructed sub-band features; Performing feature modeling processing on the reconstructed sub-band features to obtain target modeling features; Performing feature recombination processing on the target modeling features to obtain reorganized features; The drum sound frequency domain signal is obtained based on the complex spectrum and the reorganized features.

4. The method according to claim 3, characterized in that The step of performing feature modeling processing on the reconstructed sub-band features to obtain target modeling features is performed by a feature modeling module, wherein the feature modeling module includes a plurality of target modules arranged in cascade, and the features output by the last target module are used as the target modeling features. The feature modeling module performs the feature modeling process on the reconstructed sub-band features to obtain the target modeling features, including: The current target feature is used as the input of the current target module for extraction processing to obtain the extracted feature, wherein the current target feature is the reconstructed sub-band feature or the feature output by the previous target module; Performing a first fusion process on the current target feature and the extracted feature to obtain an initial fusion feature; Performing a second fusion process on the initial fusion feature and the candidate fusion feature to obtain an advanced fusion feature of the target music signal, wherein the candidate fusion feature is an advanced fusion feature of the historical target music signal; The advanced fusion feature of the target music signal and the initial fusion feature are subjected to a third fusion process to obtain a target fusion feature, and the target fusion feature is used as the feature output by the current target module.

5. The method according to claim 3, characterized in that: The step of performing feature recombination processing on the target modeling features to obtain reorganized features includes: Performing feature splitting processing on the target modeling feature to obtain a plurality of modeling sub-band features; For each of the modeling sub-band features, performing a preset feature transformation process on the modeling sub-band feature to obtain a sub-band mask feature of the modeling sub-band feature; The sub-band mask features are combined to obtain a full-band mask feature, and the full-band mask feature is used as the recombined feature.

6. The method according to any one of claims 1 to 5, characterized in that The performing drum sound detection processing on each of the drum sound track signals to obtain the drum sound detection result of each of the target music signals includes: For each drum track signal of the target music signal, the following steps are performed: Calculating the signal energy of the drum track signal of the target music signal to obtain the initial signal energy of the target music signal; Determine the target signal energy of the target music signal based on the difference between the initial signal energy and the candidate signal energy, wherein the candidate signal energy is a preset signal energy or an initial energy signal of the target music signal of the previous frame; In response to the target signal energy of the target music signal being greater than an energy threshold, determining the drum sound detection result of the target music signal as the target music signal being in a drum sound state; or, In response to the target signal energy of the drum sound track signal being less than or equal to an energy threshold, the drum sound detection result of the target music signal is determined as the target music signal being in a no-drum sound state.

7. The method according to any one of claims 1 to 5, characterized in that The target music signals of the plurality of frames are obtained by sampling the music audio including the initial music signals at intervals, and the target music signals are processed based on the drum sound detection results of the target music signals and the drum sound track signals of the target music signals to obtain the processing results of the music audio, including: For each drum track signal of the target music signal, the following steps are performed: In response to the drum sound detection result of the target music signal being that the target music signal is in a drum sound state, superimposing the target music signal and the drum sound track signal of the target music signal to obtain a first processing result of the target music signal; and / or, In response to the drum sound detection result of the target music signal being different from the drum sound detection result of an adjacent target music signal, a signal to be processed is smoothed based on the drum sound track signal of the target music signal to obtain a second processing result of the signal to be processed, wherein the signal to be processed is an initial music signal between the target music signal and the adjacent target music signal.

8. A music audio processing device, characterized in that: include: An acquisition module, used to acquire a plurality of frames of target music signals in the music audio; A separation module, used for performing drum sound separation processing on each of the target music signals to obtain a drum sound track signal of each of the target music signals; A detection module, configured to perform drum sound detection processing on each of the drum sound track signals to obtain a drum sound detection result of each of the target music signals, wherein the drum sound detection result of each of the target music signals includes whether the target music signal is in a state of having drum sound or the target music signal is in a state of not having drum sound; The processing module is used to process each of the target music signals based on the drum sound detection results of each of the target music signals and the drum sound track signals of each of the target music signals to obtain the processing results of the music audio.

9. An electronic device, characterized in that: include: A memory and a processor, wherein the memory stores program instructions, and the processor retrieves the program instructions from the memory to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having program instructions stored thereon, characterized in that: When the program instructions are executed by a processor, they are used to implement the method according to any one of claims 1 to 7.

Images