Audio separation method and device, computer device and storage medium

Abstract

The invention relates to an audio separation method and device, a computer device and a storage medium. Relates to the field of deep learning, and solves the problem of difficulty in processing complex and rapidly changing music signals. The method comprises the following steps: extracting time domain features and frequency domain features of original audio data; according to the time domain features and the frequency domain features, high-dimensional feature vectors adapted to an application scene are generated, and the high-dimensional feature vectors indicate the contribution degree of each feature to audio separation; and performing sound source splitting on the high-dimensional feature vector to obtain human voice audio data and accompaniment audio data. The technical scheme provided by the invention is suitable for multi-sound-source splitting in audio, and a lightweight, accurate and rapid in-vehicle complex music signal splitting mechanism is realized.

Description

Technical Field

[0001] This disclosure relates to the field of deep learning, and more particularly to an audio separation method, apparatus, computer device, and storage medium. Background Technology

[0002] As users' demands for in-car entertainment functions continue to increase, in-car microphone-free karaoke is gradually gaining popularity as an emerging entertainment method. Traditional in-car entertainment systems can only play music and cannot provide karaoke functionality. Microphone-free karaoke uses audio processing algorithms to separate the accompaniment and vocals in a song, allowing users to sing directly using the car's audio system without holding a microphone, thus providing a completely new entertainment experience.

[0003] Commonly used accompaniment / vocals separation algorithms mainly include time-domain algorithms and frequency-domain algorithms. Time-domain algorithms focus on the temporal characteristics of the signal, processing the audio signal directly on the time axis. However, they struggle to effectively separate accompaniment and vocals when dealing with complex music signals. Frequency-domain algorithms, on the other hand, process the audio signal by converting it to the frequency domain, allowing for more precise analysis of the frequency components. However, they also struggle to accurately separate accompaniment and vocals when processing rapidly changing music signals. Therefore, both time-domain and frequency-domain algorithms have difficulty handling complex and rapidly changing music signals, thus affecting the effectiveness of accompaniment separation. Summary of the Invention

[0004] To overcome the problems existing in related technologies, this disclosure provides an audio separation method, apparatus, computer device, and storage medium. By combining the temporal and frequency domain characteristics of audio data, a high-dimensional feature vector adapted to the application scenario is generated, and sound source segmentation is performed accordingly. This achieves a lightweight, accurate, and fast mechanism for segmenting complex in-vehicle music signals, solving the problem of processing complex and rapidly changing music signals.

[0005] According to a first aspect of the present disclosure, an audio separation method is provided, comprising: Extract the time-domain and frequency-domain features of the raw audio data; Based on the time-domain features and the frequency-domain features, a high-dimensional feature vector adapted to the application scenario is generated. The high-dimensional feature vector indicates the contribution of each feature to audio separation. The high-dimensional feature vector is split into sound sources to obtain human voice audio data and accompaniment audio data.

[0006] Furthermore, the frequency domain features are extracted in the following manner: The original audio data is divided into multiple short time frames, with overlapping windows between adjacent short time frames; Perform a Fourier transform on each of the short-time frames according to the following expression: , in, This represents the k-th frequency component in the frequency domain. This represents the nth short-time frame signal in the time domain. The window function is represented by N, which represents the frame length of the short-time frame. The amplitude and / or phase information of the frequency domain signal of each of the short time frames is extracted frame by frame; The amplitude information and / or phase information of each of the short-time frames are spliced ​​together in chronological order to obtain the frequency domain features.

[0007] Furthermore, the step of generating a high-dimensional feature vector adapted to the application scenario based on the time-domain features and the frequency-domain features includes: Obtain at least one temporal vector of the temporal feature; Obtain at least one frequency domain vector of the frequency domain feature; The time-domain vector and the frequency-domain vector are concatenated to obtain a high-dimensional feature matrix; The weights of the high-dimensional feature matrix are obtained through an attention network, and the weights reflect the contribution of each feature in the high-dimensional feature matrix to audio separation. The weights are applied to the high-dimensional feature matrix, and the weighted high-dimensional feature matrix is ​​fused according to the following expression to obtain the high-dimensional feature vector: , in, It is the high-dimensional feature vector obtained after fusion. It is the weighted high-dimensional feature matrix, where [i, :] represents... The i-th row, m represents the total number of short time frames.

[0008] Furthermore, the weights are attention weights, and the step of generating the weights of the high-dimensional feature matrix through the attention network includes: Obtain the weight matrix; The attention weights are calculated according to the following expression: Attention(F) = softmax(WF + b) Where Attention(F) represents the attention weight, W represents the weight matrix, F represents the high-dimensional feature matrix, b represents the bias vector, and softmax represents the normalization function.

[0009] Furthermore, the step of performing sound source segmentation on the high-dimensional feature vector to obtain human voice audio data and accompaniment audio data includes: Deep feature encoding is performed on the high-dimensional feature vector to obtain a deep feature vector; The deep feature vectors are split into sound source features to obtain the human voice feature matrix and / or accompaniment feature matrix; The human voice feature matrix is ​​decoded to obtain human voice data, and the accompaniment feature matrix is ​​decoded to obtain accompaniment data.

[0010] Furthermore, the step of performing sound source feature decomposition on the deep feature vector to obtain the human voice feature matrix and / or accompaniment feature matrix includes: The deep feature vector containing deep features of mixed sound sources is split using a semantic segmentation model to obtain a human voice feature matrix containing human voice information and an accompaniment feature matrix containing accompaniment information.

[0011] Furthermore, the method also includes: Update parameter configuration, wherein the parameter configuration includes at least one or more of the following parameters: The parameters of the attention network that provide weights when generating the high-dimensional feature vector. The parameters of the encoder that performs deep feature encoding during sound source separation. The parameters of a semantic segmentation model that performs sound source feature decomposition during sound source separation. The parameters of the decoder used in the sound source separation process.

[0012] According to a second aspect of the embodiments of this disclosure, an audio separation apparatus is provided, comprising: The multi-domain feature extraction module is used to extract the time-domain and frequency-domain features of the original audio data; The feature fusion module is used to generate a high-dimensional feature vector adapted to the application scenario based on the time-domain features and the frequency-domain features. The high-dimensional feature vector indicates the contribution of each feature to audio separation. The audio separation module is used to split the high-dimensional feature vector into sound sources to obtain human voice audio data and accompaniment audio data.

[0013] According to a third aspect of the embodiments of this disclosure, a computer apparatus is provided, comprising: processor; Memory used to store processor-executable instructions; The processor is configured to execute the audio separation method described above.

[0014] According to a fourth aspect of the embodiments of the present disclosure, a non-transitory computer-readable storage medium is provided, which, when instructions in the storage medium are executed by a computer's processor, enables the computer to perform the above-described audio separation method.

[0015] The technical solutions provided by the embodiments of this disclosure can include the following beneficial effects: extracting the temporal and frequency domain features of the original audio data; then generating a high-dimensional feature vector adapted to the application scenario based on the temporal and frequency domain features, wherein the high-dimensional feature vector indicates the contribution of each feature to audio separation; and then performing sound source segmentation on the high-dimensional feature vector to obtain human voice audio data and accompaniment audio data. By combining temporal and frequency domain features for sound source segmentation, a lightweight, accurate, and fast mechanism for segmenting complex in-vehicle music signals is achieved, solving the problem of difficulty in processing complex and rapidly changing music signals.

[0016] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0018] Figure 1 This is a flowchart illustrating an audio separation method according to an exemplary embodiment.

[0019] Figure 2 This is a flowchart illustrating an audio separation method according to an exemplary embodiment.

[0020] Figure 3 This is a flowchart illustrating an audio separation method according to an exemplary embodiment.

[0021] Figure 4 This is a flowchart illustrating an audio separation method according to an exemplary embodiment.

[0022] Figure 5 This is a flowchart illustrating an audio separation method according to an exemplary embodiment.

[0023] Figure 6 This is a block diagram illustrating an audio separation system according to an exemplary embodiment.

[0024] Figure 7 This is a block diagram illustrating an audio separation device according to an exemplary embodiment. Detailed Implementation

[0025] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0026] Commonly used accompaniment / vocals separation algorithms mainly include time-domain algorithms and frequency-domain algorithms. Time-domain algorithms focus on the temporal characteristics of the signal, processing the audio signal directly on the time axis. However, they struggle to effectively separate accompaniment and vocals when dealing with complex music signals. Frequency-domain algorithms, on the other hand, process the audio signal by converting it to the frequency domain, allowing for more precise analysis of the frequency components. However, they also struggle to accurately separate accompaniment and vocals when processing rapidly changing music signals. Therefore, both time-domain and frequency-domain algorithms have difficulty handling complex and rapidly changing music signals, thus affecting the effectiveness of accompaniment separation.

[0027] To address the aforementioned problems, embodiments of this disclosure provide an audio separation method, apparatus, computer device, and storage medium. Sound source segmentation is performed based on time-domain and frequency-domain characteristics. The processing system employs a lightweight design suitable for vehicle processing capabilities, achieving efficient and accurate audio signal separation while fully considering the characteristics of the audio signal.

[0028] An exemplary embodiment of this disclosure provides an audio separation method, the process of separating vocals and accompaniment from an audio file using this method is as follows: Figure 1 As shown, it includes: Step 101: Extract the time-domain and frequency-domain features of the original audio data.

[0029] In this step, the raw audio data is input into the system. The raw audio data includes vocals and background accompaniment.

[0030] The raw audio data is in time-domain signal format. The sampling rate of the time-domain signal can be determined according to the standard configuration of the car audio system.

[0031] According to one exemplary implementation, signal amplitude normalization processing can be performed on the raw audio data. For example, the maximum amplitude value of the time-domain sampling points of the raw audio data is calculated, and the time-domain sampling points are divided by the maximum value to normalize the time-domain signal.

[0032] According to an exemplary embodiment, the original audio data is divided into multiple short-time frames, each of which can be used as a temporal feature vector. The framing parameters for dividing the short-time frames are the same as those used when extracting frequency domain features.

[0033] The frequency domain features are extracted in the following manner, with the specific process as follows: Figure 2 As shown, it includes: Step 201: Divide the original audio data into multiple short frames.

[0034] In this step, the original audio data is divided into multiple short-time frames, with overlapping windows between adjacent short-time frames. According to one exemplary implementation, a Hamming window overlap window is used, multiplying the short-time frames by a Hamming window to suppress spectral leakage.

[0035] Step 202: Convert the short time frame into a frequency domain signal.

[0036] In this step, a Fourier transform is performed on each of the short-time frames according to the following expression: , in, This represents the k-th frequency component in the frequency domain. This represents the nth short-time frame signal in the time domain. Let N represent the window function, and let k represent the frame length of the short-time frame. k is the frequency point index, ranging from 0 ≤ k ≤ N-1 (where N is the frame length of the short-time frame).

[0037] According to one exemplary implementation, This represents the complex spectrum value corresponding to the k-th frequency point in the frequency domain. It is the output of the Fourier transform and is used for subsequent calculations of amplitude and phase information. Let n represent the nth short-time frame signal in the time domain, where n is the index of the time-domain sampling point, and 0 ≤ n ≤ N-1. This represents a window function, such as the Hamming window function, which smooths the edges of short-time frames. N represents the frame length of the short-time frame, which can be the number of temporal sampling points within a single short-time frame (e.g., a value of 512).

[0038] Fourier transform converts the windowed short-frame time-domain signal into a frequency-domain signal, providing a basis for subsequent extraction of amplitude and phase information.

[0039] Step 203: Extract the amplitude information and / or phase information of the frequency domain signal of each short time frame frame by frame.

[0040] In this step, the frequency domain signal is filtered for spectral information to obtain amplitude information (such as amplitude spectrum) and phase information (such as phase spectrum).

[0041] Step 204: Concatenate the amplitude information and / or phase information of each short-time frame in chronological order to obtain the frequency domain features.

[0042] In this step, the amplitude and phase information of all short frames are spliced ​​together in chronological order to form frequency domain features.

[0043] According to one exemplary implementation, the frequency domain feature is a frequency domain feature matrix. The frequency domain feature matrix is ​​obtained by concatenating amplitude information, and the phase information is stored as a phase matrix.

[0044] Step 102: Generate a high-dimensional feature vector adapted to the application scenario based on the time-domain features and the frequency-domain features.

[0045] The high-level feature vector indicates the contribution of each feature to audio separation.

[0046] In this step, the importance of time-domain features and frequency-domain features is dynamically adjusted, for example, by assigning weights to time-domain features and frequency-domain features, and then fusing the time-domain features and frequency-domain features to adapt to the current application scenario (such as fast-paced songs, songs with complex arrangements, etc.).

[0047] According to an exemplary implementation, in this step, the weight allocation of time-domain features and frequency-domain features can be achieved through an attention network.

[0048] Step 103: Perform sound source segmentation on the high-dimensional feature vector to obtain human voice audio data and accompaniment audio data.

[0049] In this step, deep feature encoding is performed on the high-dimensional feature vector to obtain deep feature information, which is used to filter noise and aggregate key features. Then, the deep feature information output by deep feature encoding is used for sound source separation, and then decoded to recover playable human voice audio data and accompaniment audio data.

[0050] An exemplary embodiment of this disclosure provides an audio separation method, the process of multi-domain feature fusion using this method is as follows: Figure 3 As shown, it includes: Step 301: Obtain at least one time-series vector of the time-domain feature.

[0051] According to one exemplary implementation, the temporal features are one or more one-dimensional temporal vectors whose dimension depends on the sampling rate of the original audio data.

[0052] Step 302: Obtain at least one frequency domain vector of the frequency domain feature.

[0053] According to one exemplary implementation, the frequency domain feature includes one or more frequency domain vectors.

[0054] Frequency domain features can be represented as a carrier frequency feature matrix, with dimensions equal to the number of short-time frames. The frequency dimension is a frequency domain vector containing the same number of data as the number of short-time frames.

[0055] According to one exemplary implementation, the frequency dimension is consistent with the dimension of the time sequence vector.

[0056] Step 303: Concatenate the time-domain vector and the frequency-domain vector to obtain a high-dimensional feature matrix.

[0057] In this step, the time-domain vectors and frequency-domain vectors are concatenated to generate a high-dimensional feature matrix. For example, concatenating n time-domain vectors and frequency-domain vectors by their column dimensions, where the time dimension of each vector is d, results in a high-dimensional feature matrix with dimension d. .

[0058] The first row of the high-dimensional feature matrix obtained after concatenation corresponds to the temporal and frequency domain features of the same short frame, thus determining the alignment of the temporal and frequency domain features in the time dimension.

[0059] Step 304: Generate the weights of the high-dimensional feature matrix through an attention network, and use the weights to reflect the contribution of each feature in the high-dimensional feature matrix to audio separation.

[0060] This step is detailed as follows: Figure 4 As shown, it includes: Step 401: Obtain the weight matrix.

[0061] In this step, the weight matrix of the attention network is obtained, matching the dimension of the high-dimensional feature matrix. Each element in the weight matrix defines the weight of the corresponding element in the high-dimensional feature matrix. In this step, the bias vector can also be obtained.

[0062] The weight matrix can be obtained through training.

[0063] Step 402: Calculate attention weights.

[0064] In this step, the attention weights are calculated according to the following expression: Attention(F) = softmax(WF + b) Where Attention(F) represents the attention weights, W represents the weight matrix, F represents the high-dimensional feature matrix, b represents the bias vector, and softmax represents the normalization function. The normalization function ensures that the sum of the weights of each feature is 1.

[0065] Therefore, the attention weight vector α = Attention(F) can be obtained.

[0066] Step 305: Apply the weights to the high-dimensional feature matrix and fuse the weighted high-dimensional feature matrix to obtain the high-dimensional feature vector.

[0067] In this step, the attention weight vector α is applied to the high-dimensional feature matrix through element-wise multiplication to obtain the weighted high-dimensional feature matrix.

[0068] For example, the calculated weights are applied to the high-dimensional feature matrix using the following expression: F′=α⊙F, Where F′ represents the weighted high-dimensional feature matrix, and ⊙ represents element-wise multiplication.

[0069] Then, the weighted high-dimensional feature matrix is ​​fused according to the following expression to obtain the high-dimensional feature vector: , in, It is the high-dimensional feature vector obtained after fusion. It is the weighted high-dimensional feature matrix, where [i, :] represents... The i-th row, m represents the total number of short time frames.

[0070] An exemplary embodiment of this disclosure also provides an audio separation method, which, based on the combination of multi-domain feature fusion, splits the original audio data to obtain high-quality accompaniment audio data. The specific process is as follows: Figure 5 As shown, it includes: Step 501: Perform deep feature encoding on the high-dimensional feature vector to obtain a deep feature vector.

[0071] In this step, deep feature encoding is performed on the high-dimensional feature vector to extract and encode the deep feature information of the high-dimensional feature vector, filter noise, and perform deep abstraction of the features to reduce the redundancy of the high-dimensional feature vector while improving the discriminative power of the features.

[0072] According to one exemplary implementation, a deep feature vector is obtained through a deep feature encoder. The deep feature encoder can be implemented using a multi-layer convolutional neural network. The dimension of the deep feature vector is adapted to the input dimension of the sound source feature decomposition process.

[0073] Step 502: Perform sound source feature decomposition on the deep feature vector to obtain the human voice feature matrix and / or accompaniment feature matrix.

[0074] In this step, data is split using a semantic segmentation model. According to one exemplary implementation, the deep feature vector containing deep features of mixed sound sources is split using the U-Net model to obtain a voice feature matrix containing human voice information and an accompaniment feature matrix containing accompaniment information.

[0075] A separation network is constructed based on a semantic segmentation model. It takes deep feature vectors as input and outputs two separation feature matrices: a human voice feature matrix and an accompaniment feature matrix.

[0076] Step 503: Decode the human voice feature matrix to obtain human voice data, and decode the accompaniment feature matrix to obtain accompaniment data.

[0077] In this step, the U-Net vocal feature matrix and / or accompaniment feature matrix are restored to frequency domain signals, and then decoded by inverse Fourier transform to convert them into time domain vocal and accompaniment data to ensure that they can be played by car audio systems.

[0078] For accompaniment data, it can be converted to an audio format supported by the car audio system and transmitted to the accompaniment channel of the car karaoke system for playback output. For vocal data, users can choose not to output it or output it through the mixing channel, superimposing the user's real-time singing voice.

[0079] An exemplary embodiment of this disclosure also provides an audio separation method, the process of training the entire audio separation system and updating its parameter configuration is as follows: The audio separation method provided in this disclosure employs a system architecture of multi-domain feature fusion and semantic model separation to separate human voices from accompaniment in audio data, and is applied to in-vehicle karaoke systems. Based on the audio separation results, feedback can be generated to optimize the parameters of the audio separation system, better adapting to the separation requirements of in-vehicle scenarios.

[0080] According to one exemplary implementation, end-to-end joint training is performed on the entire audio separation system, and the models involved are simultaneously optimized.

[0081] According to one exemplary implementation, in the initial stage of the system, sample audio data (including vocal and accompaniment data) can be used as training data. The audio separation system forward propagates to generate separation results. The separation results are compared with the vocal and accompaniment data separated from the sample audio data to quantitatively evaluate the separation effect and calculate the loss. Based on the loss, the parameters of the entire audio separation system are optimized and updated through backpropagation.

[0082] According to one exemplary implementation, for an audio separation system, parameter configuration is updated, the parameter configuration including at least one or more of the following parameters: The parameters of the attention network that provide weights when generating the high-dimensional feature vector. The parameters of the encoder that performs deep feature encoding during sound source separation. The parameters of a semantic segmentation model that performs sound source feature decomposition during sound source separation. The parameters of the decoder used in the sound source separation process.

[0083] Specifically, the semantic segmentation model can be the U-Net model.

[0084] The parameters of the attention network, encoder, U-Net model, and decoder are trained synchronously to avoid poor module adaptability caused by independent training. The training data includes in-vehicle noise and karaoke audio without a microphone to ensure that the model is adapted to real-world scenarios.

[0085] The training process employs lightweight configurations, such as low-complexity loss functions and the Adam optimizer, to adapt to the training resources of in-vehicle terminals.

[0086] An exemplary embodiment of this disclosure also provides an audio separation system, the system framework of which is as follows: Figure 6 As shown, it includes: The frequency domain feature extraction module is used to acquire the raw audio data of the input audio, vocals and accompaniment mix. Through frequency domain feature extraction technology, the time domain signal is converted into a frequency domain signal so as to better analyze and process the frequency components in the audio signal.

[0087] Specifically, the conversion from time-domain signals to frequency-domain signals can be achieved using Short-Time Fourier Transform (STFT), including: 1. Framing: Divide the audio signal into multiple short frames using Hamming windows with overlapping windows.

[0088] 2. Fourier Transform: Perform a Fourier transform on each frame to obtain the spectrum.

[0089] , in, This represents the k-th frequency component in the frequency domain. This represents the nth short-time frame signal in the time domain. The window function is represented by N, which represents the frame length of the short-time frame.

[0090] The frequency components of short time frames are combined to form a spectrum.

[0091] 3. Amplitude and Phase: Extract amplitude and phase information of the frequency domain signal.

[0092] Multi-domain feature fusion module: The extracted frequency domain features and the time domain features of the original audio data are input into the multi-domain feature fusion module. Through the fusion of multi-domain features, a high-dimensional feature vector is formed. The specific implementation of multi-domain feature fusion is as follows: 1. Feature concatenation: Concatenate multiple extracted feature vectors (including time-domain and frequency-domain feature vectors) into a high-dimensional feature vector F. For example, if there are n feature vectors, each with dimension d, then the shape of F is n×d.

[0093] 2. Attention Mechanism Calculation: An attention network is used to calculate the weights of each time-domain and frequency-domain feature, reflecting the importance of the corresponding time-domain or frequency-domain feature. For example, if the parameters of the attention network are W and b, the calculation process is as follows: Attention(F) = softmax(WF + b) Where W is the weight matrix, b is the bias vector, and the softmax function is used to normalize the output to a probability distribution, ensuring that the weights sum to 1.

[0094] 3. Weights: Apply the calculated weights α to the high-dimensional feature matrix F to obtain the weighted high-dimensional feature matrix F′: F′=α⊙F, Here, ⊙ represents element-wise multiplication.

[0095] 4. Feature Fusion: The weighted feature matrices F′ are fused according to the following expression: , in, It is the high-dimensional feature vector obtained after fusion. It is the weighted high-dimensional feature matrix, where [i, :] represents... The i-th row, m represents the total number of short time frames.

[0096] Deep feature encoder: Receives high-dimensional feature vectors, extracts and encodes deep feature information, implements this through a multi-layer convolutional neural network, and inputs the results into a separate core network.

[0097] The core separation network is the central processing unit of the entire algorithm, responsible for separating the audio signal. It is implemented using the U-Net model, which decomposes the mixed audio signal into different sound sources.

[0098] The vocal and accompaniment decoders, by receiving the processing results from the separation core network, decode and recover the vocal and accompaniment parts using an inverse network transform implemented with a feature encoder. Finally, the separated accompaniment and vocals are output as audio signals. By employing a multi-domain feature fusion technique with an attention mechanism, the importance of different features is dynamically adjusted, enabling the model to adaptively select the most effective features in different scenarios. This better captures multifaceted information from the audio signal, improving the accuracy and quality of accompaniment separation.

[0099] Table 1 shows a comparison of the separation performance of pure temporal and multi-domain feature fusion methods under the same architecture. It can be seen that multi-domain feature fusion improves upon the signal-to-distortion ratio (SDR), scale-invariant signal-to-distortion ratio (SI_SDR), audio loss (e.g., Aura_mrstft), and fullness by more than 1 dB.

[0100] Table 1. Comparison of separation effects between pure temporal domain and multi-domain feature fusion methods

[0101] An exemplary embodiment of this disclosure also provides an audio separation device, the structure of which is as follows: Figure 7 As shown, it includes: The multi-domain feature extraction module 701 is used to extract the time-domain and frequency-domain features of the original audio data; The feature fusion module 702 is used to generate a high-dimensional feature vector adapted to the application scenario based on the time-domain features and the frequency-domain features. The high-dimensional feature vector indicates the contribution of each feature to audio separation. The audio separation module 703 is used to perform sound source splitting on the high-dimensional feature vector to obtain human voice audio data and accompaniment audio data.

[0102] The aforementioned device can be integrated into a vehicle control system, which then performs the corresponding functions. The specific methods by which each module performs its operations have been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0103] An exemplary embodiment of this disclosure also provides a computer apparatus, including: processor; Memory used to store processor-executable instructions; The processor is configured to execute the audio separation method provided in the embodiments of this disclosure.

[0104] An exemplary embodiment of this disclosure also provides a non-transitory computer-readable storage medium, wherein instructions in the storage medium, when executed by a computer's processor, enable the computer to perform the audio separation method provided by embodiments of this disclosure.

[0105] Embodiments of this disclosure provide an audio separation method, apparatus, computer device, and storage medium. The method extracts temporal and frequency domain features from raw audio data. Then, based on the temporal and frequency domain features, a high-dimensional feature vector adapted to the application scenario is generated. This high-dimensional feature vector indicates the contribution of each feature to audio separation. The high-dimensional feature vector is then used for sound source segmentation to obtain human voice audio data and accompaniment audio data. By combining temporal and frequency domain features for sound source segmentation, a lightweight, accurate, and fast mechanism for segmenting complex in-vehicle music signals is achieved, solving the problem of processing complex and rapidly changing music signals.

[0106] The algorithm for separating accompaniment in karaoke without microphone is based on multi-domain feature fusion, which significantly improves the effect of accompaniment separation and optimizes the overall effect of karaoke without microphone.

[0107] By simultaneously considering time and frequency domain features, employing multi-domain feature fusion technology, and using an attention mechanism to dynamically adjust the importance of different features, the model can adaptively select the most effective features in different scenarios. This allows for better capture of multifaceted information from the audio signal, improving the accuracy and quality of accompaniment separation. The attention mechanism adaptively adjusts the importance of multi-domain features, resulting in better model performance under different inputs and scenarios.

[0108] Those skilled in the art will also understand that the various illustrative logical blocks and steps listed in the embodiments of this disclosure can be implemented by electronic hardware, computer software, or a combination of both. Whether such functionality is implemented in hardware or software depends on the specific application and the overall system design requirements. Those skilled in the art can implement the described functionality using various methods for each specific application, but such implementation should not be construed as exceeding the scope of protection of the embodiments of this disclosure.

[0109] Furthermore, the term “exemplary” is used herein to mean serving as an example, instance, or illustration. Any aspect or design described herein as “exemplary” is not necessarily to be construed as advantageous compared to other aspects or designs. Rather, the use of the term “exemplary” is intended to present the concept in a concrete manner. As used herein, the term “or” is intended to mean an inclusive “or” rather than an exclusive “or.” That is, unless otherwise specified or clear from the context, “X applies A or B” is intended to mean any of the natural inclusive arrangements. That is, “X applies A or B” satisfies any of the foregoing instances if X applies A; X applies B; or both X applies A and B. Additionally, unless otherwise specified or clear from the context to refer to the singular form, the articles “a” and “an” as used in this application and the appended claims are generally understood to mean “one or more.”

[0110] Similarly, although this disclosure has been shown and described with respect to one or more implementations, equivalent variations and modifications will occur to those skilled in the art upon reading and understanding this specification and the accompanying drawings. This disclosure includes all such modifications and variations and is limited only by the scope of the claims. In particular, with respect to the various functions performed by the components described above (e.g., elements, resources, etc.), unless otherwise indicated, the terminology used to describe such components is intended to correspond to any component (functionally equivalent) that performs the specific function of the described component, even if structurally not equivalent to the disclosed structure. Furthermore, although specific features of this disclosure may have been disclosed with respect to only one of several implementations, such features may be combined with one or more other features of other implementations, as may be desired and advantageous to any given or particular application. Moreover, with regard to the terms “comprising,” “owning,” “having,” “having,” or variations thereof as used in the detailed description or claims, such terms are intended to be inclusive in a manner similar to the term “including.”

[0111] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0112] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. An audio separation method, characterized in that, include: Extract the time-domain and frequency-domain features of the raw audio data; Based on the time-domain features and the frequency-domain features, a high-dimensional feature vector adapted to the application scenario is generated. The high-dimensional feature vector indicates the contribution of each feature to audio separation. The high-dimensional feature vector is split into sound sources to obtain human voice audio data and accompaniment audio data.

2. The audio separation method according to claim 1, characterized in that, The frequency domain features are extracted using the following method: The original audio data is divided into multiple short time frames, with overlapping windows between adjacent short time frames; Based on the following expression, perform a Fourier transform on each of the short-time frames to obtain the frequency domain signal of each short-time frame: , in, This represents the k-th frequency component in the frequency domain. This represents the nth short-time frame signal in the time domain. The window function is represented by N, which represents the frame length of the short-time frame. The amplitude and / or phase information of the frequency domain signal of each of the short time frames is extracted frame by frame; The amplitude information and / or phase information of each of the short-time frames are spliced ​​together in chronological order to obtain the frequency domain features.

3. The audio separation method according to claim 1, characterized in that, The step of generating a high-dimensional feature vector adapted to the application scenario based on the time-domain features and the frequency-domain features includes: Obtain at least one temporal vector of the temporal feature; Obtain at least one frequency domain vector of the frequency domain feature; The time-domain vector and the frequency-domain vector are concatenated to obtain a high-dimensional feature matrix; The weights of the high-dimensional feature matrix are generated by an attention network, and the weights reflect the contribution of each feature in the high-dimensional feature matrix to audio separation. The weights are applied to the high-dimensional feature matrix, and the weighted high-dimensional feature matrix is ​​fused according to the following expression to obtain the high-dimensional feature vector: ， in, It is the high-dimensional feature vector obtained after fusion. It is the weighted high-dimensional feature matrix, where [i, :] represents... The i-th row, m represents the total number of short time frames.

4. The audio separation method according to claim 3, characterized in that, The weights are attention weights, and the step of generating the weights of the high-dimensional feature matrix through the attention network includes: Obtain the weight matrix; The attention weights are calculated according to the following expression: Attention(F) = softmax(WF + b) Where Attention(F) represents the attention weight, W represents the weight matrix, F represents the high-dimensional feature matrix, b represents the bias vector, and softmax represents the normalization function.

5. The audio separation method according to claim 1, characterized in that, The step of performing sound source segmentation on the high-dimensional feature vector to obtain human voice audio data and accompaniment audio data includes: Deep feature encoding is performed on the high-dimensional feature vector to obtain a deep feature vector; The deep feature vectors are split into sound source features to obtain the human voice feature matrix and / or accompaniment feature matrix; The human voice feature matrix is ​​decoded to obtain human voice data, and the accompaniment feature matrix is ​​decoded to obtain accompaniment data.

6. The audio separation method according to claim 5, characterized in that, The step of performing sound source feature decomposition on the deep feature vector to obtain the human voice feature matrix and / or accompaniment feature matrix includes: The deep feature vector containing deep features of mixed sound sources is split using a semantic segmentation model to obtain a human voice feature matrix containing human voice information and an accompaniment feature matrix containing accompaniment information.

7. The audio separation method according to claim 1, characterized in that, The method further includes: Update parameter configuration, wherein the parameter configuration includes at least one or more of the following parameters: The parameters of the attention network that provide weights when generating the high-dimensional feature vector. The parameters of the encoder that performs deep feature encoding during sound source separation. The parameters of a semantic segmentation model that performs sound source feature decomposition during sound source separation. The parameters of the decoder used in the sound source separation process.

8. An audio separation device, characterized in that, include: The multi-domain feature extraction module is used to extract the time-domain and frequency-domain features of the original audio data; The feature fusion module is used to generate a high-dimensional feature vector adapted to the application scenario based on the time-domain features and the frequency-domain features. The high-dimensional feature vector indicates the contribution of each feature to audio separation. The audio separation module is used to split the high-dimensional feature vector into sound sources to obtain human voice audio data and accompaniment audio data.

9. A computer device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to perform the audio separation method as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium, characterized in that, When the instructions in the storage medium are executed by the processor of a computer, the computer is able to perform the audio separation method as described in any one of claims 1 to 7.

Images