An audio analysis method, apparatus, device and storage medium

Abstract

The application relates to the technical field of audio processing, in particular to an audio analysis method and device, equipment and a storage medium. The method comprises the following steps: obtaining target audio to be analyzed; obtaining a complex spectrum of the target audio; inputting the complex spectrum into an encoder in a separation model to obtain first features; inputting the first features into a separator of the separation model, performing at least one time-frequency domain modeling on the first features by using the separator to obtain second features; wherein performing one time-frequency domain modeling comprises the following steps: performing frequency domain modeling on the first intermediate features to obtain second intermediate features; weighting the first intermediate features and the second intermediate features to obtain third intermediate features; performing time domain modeling on the third intermediate features to obtain fourth intermediate features; weighting the fourth intermediate features and the third intermediate features to obtain fifth intermediate features; and inputting the second features into a decoder of the separation model to obtain at least two audio tracks.

Description

Technical Field

[0001] This application relates to the field of audio processing technology, and in particular to an audio analysis method, apparatus, device, and computer-readable storage medium. Background Technology

[0002] Audio analysis refers to analyzing different sound sources in mixed audio and extracting the corresponding audio tracks. Traditional separation algorithms struggle to accurately separate pure sound sources when faced with complex in-vehicle environments such as engine noise, wind noise interference, and multi-instrument stereo mixing, often resulting in sound source residue or distortion. Summary of the Invention

[0003] One objective of this application is to provide an audio analysis method to solve the problem of poor sound source separation accuracy in the prior art; another objective is to provide an audio analysis device; a third objective is to provide an audio analysis equipment; a fourth objective is to provide a computer-readable storage medium; and a fifth objective is to provide a computer program.

[0004] To achieve the above objectives, the technical solution adopted in this application is as follows: This application provides an audio analysis method, which includes: acquiring a target audio to be analyzed; acquiring a complex spectrum of the target audio; inputting the complex spectrum into an encoder in a separation model to obtain a first feature; inputting the first feature into a separator of the separation model, and using the separator to perform time-frequency domain modeling on the first feature at least once to obtain a second feature; wherein performing time-frequency domain modeling once includes: performing frequency domain modeling on the first intermediate feature to obtain a second intermediate feature; weighting the first intermediate feature and the second intermediate feature to obtain a third intermediate feature; performing time-domain modeling on the third intermediate feature to obtain a fourth intermediate feature, and weighting the fourth intermediate feature and the third intermediate feature to obtain a fifth intermediate feature; the fifth intermediate feature is used as the first intermediate feature in the next time-frequency domain modeling; wherein, in the first time-frequency domain modeling, the first intermediate feature is the first feature; in the last time time-frequency domain modeling, the fifth intermediate feature obtained is the second feature; and inputting the second feature into a decoder of the separation model to obtain at least two audio tracks.

[0005] This application utilizes at least one separation module to perform modeling on the target audio from both the frequency domain and the time domain, which can fully extract the feature information of the audio from both dimensions, thereby capturing the time-frequency structure of the sound source more comprehensively and improving the accuracy and robustness of speech separation.

[0006] In some embodiments, the separation module includes a first separation module and a second separation module, both of which are time-domain and frequency-domain dual-path separation modules; to obtain a second intermediate feature by performing frequency-domain modeling on the first intermediate feature, the method includes: using the first frequency-domain modeling module of the first separation module and the second frequency-domain modeling module of the second separation module to sequentially perform frequency-domain modeling on the first intermediate feature to obtain the second intermediate feature; to obtain a fourth intermediate feature by performing time-domain modeling on the third intermediate feature, the method includes: using the first time-domain modeling module of the first separation module and the second time-domain modeling module of the second separation module to sequentially perform time-domain modeling on the third intermediate feature to obtain the fourth intermediate feature.

[0007] This application employs a separator composed of at least one stacked separation module, which can simultaneously and fully exploit the temporal dependencies of audio signals in the time domain and the spectral distribution characteristics in the frequency domain, thereby enhancing the model's ability to model and represent complex time-frequency information.

[0008] In some embodiments, segmentation processing is performed on the target mixed audio to obtain a plurality of audio segments arranged in chronological order; wherein the target audio is one of the plurality of audio segments; the target audio includes at least one audio frame.

[0009] This application divides long-term audio into short-term units suitable for model processing, reducing the computational pressure and memory usage of the model in processing long-sequence signals. At the same time, it uses one of multiple audio segments as the target audio and at least one audio frame to constitute the target audio, which can realize refined and framed feature extraction and analysis of audio signals, improve the model's ability to capture local time-frequency features, and thus improve the accuracy and processing efficiency of subsequent audio separation.

[0010] In some embodiments, inputting the complex spectrum into the encoder in the separation model to obtain a first feature includes: using a sparse banding strategy to divide the complex spectrum into multiple first sub-bands along the frequency dimension; downsampling each first sub-band by a corresponding multiple to obtain a first sub-feature corresponding to each first sub-band; and concatenating multiple first sub-features to obtain a first feature.

[0011] This application significantly reduces feature dimensions and model computation through a sparse banding strategy, thereby improving overall processing efficiency. At the same time, banding processing can better adapt to the signal distribution characteristics of different frequency regions, reduce unnecessary redundant information transmission and computation, avoid excessive compression of high-frequency information, and improve the model running speed while effectively ensuring the integrity and reconstruction accuracy of audio time-frequency features, thus enhancing the robustness of sound source separation.

[0012] In some embodiments, the decoder includes an audio track generation module; inputting the second feature into the decoder of the separation model to obtain at least two audio tracks includes: inputting the second feature into the decoder of the separation model, processing the second feature using the audio track generation module to obtain at least two audio features; and performing frequency domain to time domain conversion processing on the at least two audio features to obtain at least two audio tracks.

[0013] In some embodiments, before mapping the second feature to at least two audio tracks using the audio track generation module, the method further includes: employing a sparse banding strategy to divide the second feature according to the frequency range to obtain multiple second sub-bands; performing upsampling processing on each second sub-band by a corresponding multiple to obtain a second sub-feature corresponding to each second sub-band; concatenating multiple second sub-features to obtain a sixth intermediate feature; and processing the second feature using the audio track generation module to obtain at least two audio features, including: processing the sixth intermediate feature using the audio track generation module to obtain at least two audio features.

[0014] This application inputs the second feature into the decoder of the separation model and processes the second feature using the audio track generation module, which can output at least two audio features with discriminative power, thus achieving preliminary decoupling of signals from different sound sources. Then, by performing frequency-domain to time-domain transformation on each audio feature, at least two independent audio tracks can be directly reconstructed. While completing efficient frequency-domain modeling and separation, it achieves a complete mapping from frequency-domain features to time-domain waveforms, ensuring the clarity and integrity of the audio track reconstruction, simplifying the signal reconstruction process, and improving the practicality of the audio separation results.

[0015] This application significantly reduces feature dimension and model computation through a sparse banding strategy, thereby improving overall processing efficiency.

[0016] In some embodiments, before inputting the complex spectrum into the encoder of the separation model, the method further includes: training the separation model based on a loss function; wherein the loss function is obtained by weighting at least two of a frequency domain complex spectrum loss function, an SDR normalized loss function, and a time domain compression loss function.

[0017] This application incorporates a composite loss function that integrates time-frequency domain information, takes into account both amplitude and phase characteristics, and introduces an adaptive weighting mechanism to more comprehensively measure the difference between the estimated signal and the target signal, thereby improving model performance.

[0018] This application provides an audio analysis apparatus, comprising: a first acquisition unit for acquiring target audio to be analyzed; a second acquisition unit for acquiring the complex spectrum of the target audio; a first determination unit for inputting the complex spectrum into an encoder in a separation model to obtain a first feature; a second determination unit for inputting the first feature into a separator of the separation model, and performing time-frequency domain modeling on the first feature at least once using the separator to obtain a second feature; wherein performing time-frequency domain modeling once includes: performing frequency domain modeling on the first intermediate feature to obtain a second intermediate feature; weighting the first intermediate feature and the second intermediate feature to obtain a third intermediate feature; performing time-domain modeling on the third intermediate feature to obtain a fourth intermediate feature; and weighting the fourth intermediate feature and the third intermediate feature to obtain a fifth intermediate feature; the fifth intermediate feature is used as the first intermediate feature in the next time-frequency domain modeling; wherein, in the first time-frequency domain modeling, the first intermediate feature is the first feature; and in the last time time-frequency domain modeling, the fifth intermediate feature obtained is the second feature; and a third determination unit for inputting the second feature into a decoder of the separation model to obtain at least two audio tracks.

[0019] This application provides an audio analysis device, including one or more processors and a memory; the memory is used to store one or more programs, which, when executed by one or more processors, cause the emulation device to implement the method described above.

[0020] This application provides a computer-readable storage medium storing a computer program thereon, which, when executed by a computer's processor, causes the computer to perform any of the methods described above. Attached Figure Description

[0021] Figure 1 This is one of the flowcharts illustrating an audio analysis method provided in an embodiment of this application; Figure 2 This is one of the schematic diagrams of a separate module provided in an embodiment of this application; Figure 3 This is a second schematic diagram of a separate module provided in an embodiment of this application; Figure 4 This is one of the flowcharts illustrating an audio analysis method provided in an embodiment of this application; Figure 5 This is a second schematic flowchart of an audio analysis method provided in an embodiment of this application; Figure 6 This is the third flowchart illustrating an audio analysis method provided in this application embodiment; Figure 7 This is a schematic diagram of the structure of an audio analysis device provided in an embodiment of this application; Figure 8This is a schematic diagram of the structure of an audio analysis device provided in an embodiment of this application. Detailed Implementation

[0022] The embodiments of this application will be described below with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be understood that the preferred embodiments are only for illustrating this application and are not intended to limit the scope of protection of this application.

[0023] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this application. Therefore, the drawings only show the components related to this application and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0024] Audio analysis refers to analyzing different sound sources in mixed audio and extracting the corresponding audio tracks. Traditional separation algorithms, such as non-negative matrix factorization (NMF) or independent component analysis (ICA), struggle to accurately separate pure sound sources when faced with complex in-vehicle environments such as engine noise, wind noise interference, and multi-instrument stereo mixing, often resulting in sound source residue or distortion.

[0025] To this end, embodiments of this application provide an audio analysis method, apparatus, device, and computer-readable storage medium that analyzes audio using a separation model to obtain audio tracks from different sound sources. The separation model includes an encoder, a separator, and a decoder. The separator includes a time-domain modeling module for extracting time-domain features and a frequency-domain modeling module for extracting frequency-domain features. This allows for the simultaneous and comprehensive extraction of audio features, achieving complementary modeling, effectively improving the separation accuracy and demixing capability of different sound sources in mixed audio, and reducing distortion and artifacts in the separated audio tracks.

[0026] The following description, in conjunction with the accompanying drawings and embodiments, introduces an audio analysis method, apparatus, device, and storage medium provided in this application. Figure 1 This is a flowchart illustrating an audio analysis method provided in an embodiment of this application, as shown below. Figure 1 As shown, the method includes: S101, Obtain the target audio to be analyzed.

[0027] In some embodiments, the target audio is a mixture of music that the user is currently playing or will be playing. For example, after the user clicks play through the vehicle's infotainment system interface, the audio data is decoded by the player and preprocessed through standardization, noise reduction, and other steps to obtain the mixed audio to be analyzed. The mixed audio can be understood as audio obtained by mixing multiple sound sources. These sound sources include, but are not limited to, vocals, drums, piano, guitar, and bass.

[0028] In other embodiments, the target audio is one of multiple audio segments obtained after segmenting the mixed audio. For example, the mixed audio is segmented according to a preset duration to obtain multiple audio segments, and the target audio is one of these segments. The preset duration can be, for example, 1.28 seconds, 2.56 seconds, or 5.12 seconds. This is done to reduce complexity and improve processing efficiency.

[0029] In some embodiments, the target audio includes one audio frame, wherein an audio frame refers to a short-time discrete signal unit obtained by segmenting the mixed audio into frames according to a preset frame length, and each audio frame corresponds to a time-domain signal of fixed duration. In other embodiments, the target audio includes multiple audio frames arranged in chronological order. These multiple audio frames are arranged sequentially in chronological order to collectively constitute a temporal signal sequence of the target audio. That is, the target audio includes at least one audio frame.

[0030] S102, Obtain the complex spectrum of the target audio.

[0031] In some embodiments, a short-time fourier transform (STFT) is performed on the target audio to obtain its complex spectrum. The complex spectrum characterizes the signal distribution of the target audio in a two-dimensional space of time and frequency. For example, one dimension represents the time axis, reflecting the change of audio over time; the other dimension represents the frequency axis, reflecting the distribution of different frequency components. Each moment and each frequency point corresponds to a complex number, which contains both amplitude and phase information to characterize the energy magnitude and phase information of the frequency component at that moment. For example, the complex spectrum of the target audio at the time-frequency point (t, f) can be represented as: X = a + bj, where a is the real part and b is the imaginary part. The amplitude and phase of the frequency component can be calculated from the real and imaginary parts. The amplitude characterizes the energy magnitude of the frequency component, reflecting the intensity of the sound; the phase characterizes the temporal position and waveform alignment of the frequency component, reflecting the temporal structure of the sound. In other words, the complex spectrum consists of the real and imaginary parts at different time points and frequencies. For example, the complex spectrum can be a complex spectrum matrix. In a complex spectrum matrix, rows represent different time points, columns represent different frequency points, and each element represents the complex spectrum at the corresponding time and frequency point. In some embodiments, the complex spectrum can also be referred to as a time-frequency graph.

[0032] As mentioned earlier, the target audio includes at least one audio frame, and the complex spectrum of the target audio includes multiple complex spectra that correspond one-to-one with the at least one audio frame. In other words, the complex spectrum of the target audio is composed of the complex spectra of at least one audio frame.

[0033] S103, input the complex spectrum into the encoder in the separation model to obtain the first feature.

[0034] In this embodiment, both the first feature and the second feature are complex spectrum features obtained by feature processing the complex spectrum of the target audio. The complex spectrum is a time-frequency representation that simultaneously characterizes the distribution information of the audio signal in both the time and frequency dimensions; the resulting complex spectrum features possess corresponding time-frequency distribution characteristics. The intermediate features mentioned below are also complex spectrum features obtained by feature processing the complex spectrum.

[0035] In some embodiments, the encoder includes a downsampling module for processing the complex spectrum of the target audio to obtain a first feature.

[0036] In some embodiments, the downsampling module may include a convolutional layer for performing convolutional downsampling on the complex spectrum to obtain a first feature. Optionally, the downsampling module may also include a normalization layer and an activation function. The normalization layer normalizes the features output by the convolutional layer. The activation function introduces a nonlinear transformation to enhance the expressive power of the features. The activation function may be a gated linear unit (GLU).

[0037] In some embodiments, the encoder includes at least one downsampling module connected in series.

[0038] In some embodiments, the complex spectrum is input into the encoder in the separation model to obtain the first feature. For example, it can be: using a sparse banding strategy and a complex spectrum to obtain multiple first sub-bands, and performing convolutional downsampling processing on each first sub-band according to different downsampling factors to obtain the first feature.

[0039] For example, electronic devices employ a sparse banding strategy to divide the complex spectrum, resulting in multiple first sub-bands. For instance, along the frequency dimension, the electronic device divides the complex spectrum into low-frequency, mid-frequency, and high-frequency complex spectra. That is, the low-frequency, mid-frequency, and high-frequency complex spectra are three first sub-bands obtained by dividing the entire frequency band complex spectrum. For example, the complex spectrum with frequencies in the 0Hz-250Hz range is divided into the low-frequency complex spectrum, the complex spectrum with frequencies in the 250Hz-4kHz range is divided into the mid-frequency complex spectrum, and the complex spectrum with frequencies in the 4kHz-20kHz range is divided into the high-frequency complex spectrum. For example, the complex spectrum corresponding to low-frequency sound sources such as bass, kick drum, and cello is the low-frequency complex spectrum, whose frequency range typically covers 0Hz to 250Hz. The low-frequency complex spectrum concentrates the fundamental frequency and main harmonic energy of these instruments. The complex spectrum corresponding to mid-frequency sound sources such as vocals, guitars, and pianos is the mid-frequency complex spectrum, which typically covers a frequency range of 250Hz to 4kHz. The mid-frequency complex spectrum concentrates the fundamental frequency and main harmonic energy of these instruments. The complex spectrum corresponding to high-frequency sound sources such as cymbals is the high-frequency complex spectrum, which typically covers a frequency range of 4kHz to 20kHz. The high-frequency complex spectrum concentrates the fundamental frequency and main harmonic energy of these instruments. Furthermore, electronic devices can also divide the complex spectrum into five first sub-bands along the frequency dimension, such as the low-frequency complex spectrum, mid-low-frequency complex spectrum, mid-frequency complex spectrum, mid-high-frequency complex spectrum, and high-frequency complex spectrum.

[0040] For each of the multiple first sub-bands, the electronic device performs convolutional downsampling processing on each first sub-band using a downsampling factor corresponding to that sub-band. The downsampling factor is related to the frequency range of the first sub-band. For example, the higher the frequency range, the larger the corresponding downsampling factor. For instance, the downsampling factors corresponding to the low-frequency complex spectrum, mid-frequency complex spectrum, and high-frequency complex spectrum are 1x, 4x, and 16x, respectively. For example, the electronic device uses at least one downsampling module to perform at least one convolutional downsampling process on each first sub-band according to the downsampling factor corresponding to that first sub-band, obtaining the first sub-feature corresponding to each first sub-band. For example, the encoder includes N downsampling modules connected in series, each downsampling module including a convolutional layer. For instance, if downsampling module 1 processes the low-frequency complex spectrum, mid-frequency complex spectrum, and high-frequency complex spectrum using downsampling factors of 1x, 4x, and 16x respectively, it obtains low-frequency complex spectrum feature 1, mid-frequency complex spectrum feature 1, and high-frequency complex spectrum feature 1. Next, downsampling module 2 processes low-frequency complex spectrum feature 1, mid-frequency complex spectrum feature 1, and high-frequency complex spectrum feature 1 with downsampling factors of 1x, 4x, and 16x respectively, to obtain low-frequency complex spectrum feature 2, mid-frequency complex spectrum feature 2, and high-frequency complex spectrum feature 2. This process continues, and the final low-frequency complex spectrum feature N, mid-frequency complex spectrum feature N, and high-frequency complex spectrum feature N obtained after processing by the Nth downsampling module are the first sub-features corresponding to each first sub-band. Through continuous convolutional downsampling, the encoder module reduces the frequency dimension and computational complexity of the model while extracting high-dimensional features suitable for the separation task.

[0041] It should be understood that convolutional downsampling processing includes using convolutional layers for convolution and downsampling, and may also include using normalization layers and activation functions for normalization and nonlinear transformation.

[0042] The electronic device concatenates multiple first sub-features to obtain the first feature after encoder processing. For example, the electronic device merges the first sub-features corresponding to the low-frequency complex spectrum, the mid-frequency complex spectrum, and the high-frequency complex spectrum to obtain the first feature. This merging can be achieved through concatenation, addition, weighted fusion, attention fusion, etc.

[0043] For the first feature output by the encoder, the electronic device can process the first feature based on the separator in the separation model to obtain the second feature. For example, the electronic device can execute S104.

[0044] S104, input the first feature into the separator of the separation model, and use the separator to perform time-frequency domain modeling on the first feature at least once to obtain the second feature.

[0045] In this embodiment, a first feature is input into a separator, and the separator performs at least one time-frequency domain modeling on the first feature to obtain a second feature. The time-frequency domain modeling includes: performing frequency domain modeling on the first intermediate feature to obtain a second intermediate feature; weighting the first and second intermediate features to obtain a third intermediate feature; performing time domain modeling on the third intermediate feature to obtain a fourth intermediate feature; and weighting the fourth and third intermediate features to obtain a fifth intermediate feature. The fifth intermediate feature is used as the first intermediate feature in the next time-frequency domain modeling. Specifically, in the first time-frequency domain modeling, the first intermediate feature is the first feature; and in the last time-frequency domain modeling, the fifth intermediate feature obtained is the second feature.

[0046] In some embodiments, the separator includes at least one separation module, each separation module being used to perform one time-frequency domain modeling. For example... Figure 2 As shown, a separate module includes a time-domain modeling module, a frequency-domain modeling module, and a weighting module.

[0047] The frequency domain modeling module is used to perform frequency domain modeling on the input first intermediate feature to obtain the second intermediate feature. The input to the first separation module is the first feature. That is, the first intermediate feature input to the first separation module is the first feature. The first intermediate feature of each separation module other than the first separation module is the output of its preceding or next-upper-level separation module.

[0048] The weighting module is used to weight the first and second intermediate features to obtain the third intermediate feature.

[0049] The temporal modeling module is used to perform temporal modeling on the third intermediate feature to obtain the fourth intermediate feature.

[0050] The weighting module also weights the fourth and third intermediate features to obtain the fifth intermediate feature. This fifth intermediate feature is then used as input to the frequency domain modeling module in the next or next-level separation module. In other words, the feature obtained by the previous separation module, such as the fifth intermediate feature, is the input to the next separation module. The output of the last separation module is the second feature. That is, the fifth intermediate feature output by the last separation module is the final second feature.

[0051] For example, the electronic device inputs a first intermediate feature into the frequency domain modeling module of the separation module to obtain a second intermediate feature. Then, the electronic device uses a weighting module to weight the first and second intermediate features to obtain a third intermediate feature. Next, the electronic device inputs the third intermediate feature into the time domain modeling module to obtain a fourth intermediate feature. Finally, the electronic device uses a weighting module to weight the third and fourth intermediate features to obtain a fifth intermediate feature. This fifth intermediate feature is used as input to the next separation module or as the second feature.

[0052] Frequency domain modeling models the distribution characteristics of intermediate features along the frequency dimension, uncovering the harmonic structure, sub-band correlations, and energy distribution patterns between different frequency components to distinguish the spectral characteristics of different sound sources such as vocals and accompaniment. Temporal domain modeling models the distribution characteristics of intermediate features along the time dimension, capturing the dynamic correlations, transient changes, and inter-frame dependencies of audio over time to characterize the evolution of sound over time. The processes of frequency domain modeling and temporal domain modeling can be found in existing techniques and will not be elaborated upon here.

[0053] In some embodiments, the frequency domain modeling module is obtained by concatenating the first frequency domain modeling module of the first separation module and the second frequency domain modeling module of the second separation module, and the time domain modeling module is obtained by concatenating the first time domain modeling module of the first separation module and the second time domain modeling module of the second separation module. Both the first and second separation modules are dual-path separation modules in both the time and frequency domains. For example, the first separation module is a gated attention unit (GAU), and the second separation module is a bidirectional long short-term memory network (BLSTM). Both GAU and BLSTM contain frequency domain paths and time domain paths, i.e., a dual-path structure. Figure 3 As shown, the frequency domain modeling module in the separation module includes a first frequency domain modeling module and a second frequency domain modeling module connected in series, and the time domain modeling module in the separation module includes a first time domain modeling module and a second time domain modeling module connected in series. The first frequency domain modeling module can be, for example, the frequency domain path of a GAU, and the second frequency domain modeling module can be, for example, the frequency domain path of a BLSTM. Similarly, the first time domain modeling module can be, for example, the time domain path of a GAU, and the second time domain modeling module can be, for example, the time domain path of a BLSTM. In other words, the separator is obtained by stacking at least one separation module. A separation module includes a GAU, a BLSTM, and a weighting module, where both the GAU and BLSTM are dual-path components in both the time and frequency domains.

[0054] In this embodiment, the second intermediate feature is obtained by frequency domain modeling of the first intermediate feature. Specifically, this can be achieved by sequentially performing frequency domain modeling on the obtained first intermediate feature using the first frequency domain modeling module of the first separation module and the second frequency domain modeling module of the second separation module. The fourth intermediate feature is obtained by time domain modeling of the third intermediate feature. Specifically, this can be achieved by sequentially performing time domain modeling on the obtained third intermediate feature using the first time domain modeling module of the first separation module and the second time domain modeling module of the second separation module.

[0055] For example, the electronic device sequentially inputs the first intermediate feature into the frequency domain path of the GAU and the frequency domain path of the BLSTM in the separation module to obtain the second intermediate feature. Specifically, the frequency domain path of the GAU in the separation module performs frequency domain modeling on the first intermediate feature to obtain a frequency domain feature. Then, the frequency domain path of the BLSTM in the separation module performs frequency domain modeling on this frequency domain feature again to obtain the second intermediate feature. Next, the electronic device uses a weighting module to weight the second intermediate feature and the first intermediate feature to obtain the third intermediate feature. The electronic device simultaneously inputs the third intermediate feature into the time domain path of the GAU and the time domain path of the BLSTM in the separation module to obtain the fourth intermediate feature. Specifically, the time domain path of the GAU in the separation module performs time domain modeling on the third intermediate feature to obtain a time domain feature. Then, the time domain path of the BLSTM in the separation module performs time domain modeling on this time domain feature again to obtain the fourth intermediate feature. The electronic device can also use a weighting module to weight the fourth intermediate feature and the third intermediate feature to obtain a fifth intermediate feature. The fifth intermediate feature serves as the input feature or the second feature for the next separation module.

[0056] In this embodiment of the application, weighting may be, for example, adding the elements of a feature matrix one by one.

[0057] S105, input the second feature into the decoder of the separation model to obtain at least two audio tracks.

[0058] In some embodiments, the decoder includes an audio track generation module for mapping the second feature to at least two audio tracks. The audio track generation module may, for example, be a two-dimensional convolutional layer. The two-dimensional convolutional layer is used to extract features from the second feature, capturing the time-frequency structure of different sound sources such as human voices and drum sounds, and generating multi-channel audio features through multiple convolutional kernels.

[0059] For example, a two-dimensional convolutional layer includes at least two convolutional kernels to map the second feature to at least two audio features corresponding to at least two audio tracks. For example, the two-dimensional convolutional layer includes convolutional kernel A, convolutional kernel B, and convolutional kernel C. Convolutional kernel A generates audio features corresponding to a human voice, which are used to extract the human voice from the target audio. Convolutional kernel B generates audio features corresponding to a drum sound, which are used to extract the drum sound from the target audio. Convolutional kernel C generates audio features corresponding to a piano sound, which are used to extract the piano sound from the target audio. Convolutional kernels A, B, and C are different. Next, the electronic device performs frequency-domain to time-domain conversion on the at least two audio features to obtain at least two audio tracks. The frequency-domain to time-domain conversion can be, for example, the inverse short-time fourier transform (ISFTF).

[0060] In some embodiments, the electronic device can determine the tag corresponding to each audio track based on the channel order of at least two audio features output by the decoder. The tag indicates the type of the audio track, or the type of sound source for that track. For example, different convolution kernels correspond to different channels, and different channels correspond to different tags. Tags may include, but are not limited to, vocals, lead instrument, bass, drum kit, backing vocals, etc.

[0061] In some embodiments, before mapping the second feature to at least two audio tracks using the track generation module, at least one convolutional upsampling process can be performed on the second feature. In this embodiment, the decoder includes an upsampling module for processing the second feature. In some embodiments, the upsampling module includes a transposed convolutional layer. The transposed convolutional layer is used to perform convolutional upsampling on the input feature. Optionally, the upsampling module may also include a normalization layer and an activation function. The activation function may be, for example, GLU. In some embodiments, the decoder includes multiple upsampling modules connected in series.

[0062] For example, at least one upsampling module is used to perform convolutional upsampling processing on the second feature to obtain the sixth intermediate feature, and then the audio track generation module is used to map the sixth intermediate feature into at least two audio tracks.

[0063] For example, in order to reduce the amount of data processing and improve the processing effect, a sparse banding strategy can be used to divide the second feature into multiple second sub-bands. For each second sub-band, convolutional upsampling processing is performed according to different upsampling factors to obtain the sixth intermediate feature. Then, the audio track generation module is used to map the sixth intermediate feature into at least two audio tracks.

[0064] For example, an electronic device may divide the second feature into three sub-bands based on the frequency dimension, such as high-frequency feature, mid-frequency feature, and low-frequency feature. Alternatively, an electronic device may divide the second feature into five sub-bands based on the frequency dimension, such as high-frequency feature, mid-high-frequency feature, mid-frequency feature, mid-low-frequency feature, and low-frequency feature.

[0065] For each of the multiple second sub-bands, the electronic device performs convolutional upsampling processing on each second sub-band using an upsampling factor corresponding to that sub-band. The upsampling factor is related to the frequency range of the second sub-band. For example, the higher the frequency range, the larger the corresponding upsampling factor. For instance, the upsampling factors corresponding to low-frequency features, mid-frequency features, and high-frequency features are 1x, 4x, and 16x, respectively. For example, the electronic device uses at least one upsampling module to perform at least one convolutional upsampling process on each second sub-band according to the upsampling factor corresponding to that sub-band, obtaining the second sub-feature corresponding to each second sub-band. For example, the decoder includes N upsampling modules connected in series, each upsampling module including a transposed convolutional layer. For example, upsampling module 1 processes low-frequency features, mid-frequency features, and high-frequency features using upsampling factors of 1x, 4x, and 16x, respectively, to obtain low-frequency feature 1, mid-frequency feature 1, and high-frequency feature 1. Next, upsampling module 2 processes low-frequency feature 1, mid-frequency feature 1, and high-frequency feature 1 with upsampling factors of 1x, 4x, and 16x respectively, obtaining low-frequency feature 2, mid-frequency feature 2, and high-frequency feature 2. This process continues, and the low-frequency feature N, mid-frequency feature N, and high-frequency feature N obtained after processing by the Nth upsampling module are the second sub-features corresponding to each second sub-band. The decoder module, through continuous convolutional upsampling, reduces the frequency dimension and computational complexity of the model while extracting high-dimensional features suitable for the separation task. It should be understood that convolutional upsampling includes using convolutional layers for convolution and upsampling, and may also include using normalization layers and activation functions for normalization and nonlinear transformations.

[0066] As can be seen, the audio analysis method provided in this application uses at least one separation module to perform modeling on the target audio from both the frequency domain and the time domain. This allows for the full extraction of audio feature information from both dimensions, thereby capturing the time-frequency structure of the sound source more comprehensively and improving the accuracy and robustness of speech separation.

[0067] In some embodiments, the separation model needs to be trained before inputting the complex spectrum of the target audio into the encoder of the separation model; that is, the separation model needs to be trained before using it to analyze the target audio. The method for training the separation model is described below with reference to specific embodiments.

[0068] In some embodiments, a separation model is trained using training samples, which include mixed audio and real audio tracks extracted from the mixed audio. For example, the mixed audio is input into the separation model to obtain the predicted audio tracks output by the separation model. Based on the predicted audio tracks, the real audio tracks, and a loss function, the separation model is trained. In some embodiments, this application uses an open-source music track separation training set, which contains over 150 full-length songs of different genres (approximately 10 hours in duration), along with separately recorded vocals, piano, drums, bass, etc., for each song. To increase the variety of music sources to be separated, 150 mixed tracks and subjectively selected open-source music source separation tools are added as supplementary training sets. In some embodiments, this application employs data augmentation, specifically shuffling the track composition of all songs into bass, drums, vocals, and other audio track libraries. Each training session randomly selects one audio track from four libraries and then applies random volume gain (e.g., using gain multiples from 0.25 to 1.25) to each track. The signal in the time domain is randomly masked (e.g., randomly selecting a signal from 0.5 to 2 seconds and setting it to zero). Finally, the enhanced signals of the four tracks are added together to obtain the final mixed signal, which is then fed into the separation model to train the separation model to separate the four audio tracks.

[0069] In some embodiments, the loss function is obtained by weighting at least two of the frequency domain complex spectrum loss function, the SDR normalized loss function, and the time domain compression loss function.

[0070] For example, the loss function is obtained by weighting the frequency domain complex spectral loss function and the SDR normalized loss function.

[0071] For another example, the loss function is obtained by weighting the frequency domain complex spectrum loss function and the time domain compression loss function.

[0072] For another example, the loss function is obtained by weighting the frequency domain complex spectrum loss function and the time domain compression loss function.

[0073] For example, the loss function is obtained by weighting the frequency domain complex spectrum loss function, the signal-to-distortion ratio (SDR) normalized loss function, and the time domain compression loss function. Alternatively, a composite loss function can be used that integrates time and frequency domain information, considers both amplitude and phase characteristics, and introduces an adaptive weighting mechanism to more comprehensively measure the difference between the estimated signal and the target signal, thereby improving model performance. The loss function is achieved through the weighted fusion of three sub-losses: the frequency domain complex spectrum loss... SDR normalized loss and temporal compression loss Specifically, the loss function can be expressed by formula (1).

[0074] , formula (1).

[0075] Where loss is the total loss; It is the frequency domain complex spectrum loss; It is the SDR normalized loss; It is a time-domain compression loss; , , These are weighting coefficients, which are preset values. In some embodiments, , , w_3 adjusts the training of multiple models under different loss functions based on the actual test set performance, and compares the signal-to-noise ratio, sound quality, etc. of the signals output by the multiple models to determine the final weights, which are used to balance the contributions of different sub-losses. For example, , , The values ​​are 1.0, 0.2, and 1.5, respectively.

[0076] In some embodiments, To constrain the consistency between the predicted complex spectrum and the true complex spectrum in the time-frequency domain, the error loss is typically constructed for the real and imaginary parts. For example, it can be calculated as follows: .

[0077] First, calculate the complex spectrum and amplitude spectrum, such as by using formula (2) and formula (3).

[0078] , formula (2).

[0079] in, It is a true complex spectrum. The predicted complex spectrum is obtained based on the separation model. It is a predicted audio track obtained based on a separation model; It's a real audio track. T( This is a short-time Fourier transform (STFT), outputting a complex spectrum. .

[0080] , , formula (3).

[0081] in, It is the true amplitude spectrum; The predicted amplitude spectrum is obtained based on the separation model. Re(C) and Im(C) are the real and imaginary parts of the complex spectrum, respectively. It is a numerically stable small constant, such as .

[0082] Next, spectral compression, including polar coordinate normalization and compression, is performed. For example... , , , .in, It is the spectral compression coefficient, such as .

[0083] Finally, the formula (4) is used to calculate. .

[0084] , formula (4).

[0085] in, It is the amplitude / phase loss weight. ; This is the global mean function.

[0086] In some embodiments, This is used to reflect the degree of distortion between the predicted audio track and the actual audio track. For example, it can be calculated using formula (5). .

[0087] , formula (5).

[0088] In some embodiments, This is used to constrain the consistency of the waveform of the predicted audio track from the separated output with that of the real audio track. For example, it can be calculated in the following way. .

[0089] First, time-domain amplitude compression is performed using formulas (6) and (7).

[0090] , formula (6) , formula (7) Finally, the formula (8) is used to calculate. .

[0091] , formula (8).

[0092] The following section describes an audio analysis method provided by an embodiment of this application, using specific application scenarios as examples.

[0093] This application relates to the fields of in-vehicle audio signal processing and artificial intelligence technology, specifically a stereo sound source analysis module that integrates neural network models and is applied to in-vehicle scenarios. Its core functionality is to achieve integrated processing of music source separation and music genre classification.

[0094] Existing sound source analysis solutions in in-vehicle audio systems mainly revolve around basic noise reduction, simple separation, and style recognition, primarily serving to improve the in-vehicle listening experience. However, they are generally limited in functionality and accuracy, suffering from the following main shortcomings: Insufficient separation accuracy: Traditional separation algorithms struggle to accurately separate pure sound sources in complex in-vehicle environments such as engine noise, wind noise interference, and multi-instrument stereo mixing, easily resulting in sound source residue or distortion. Disjointed classification and separation: Sound source separation and music classification are treated as independent modules, failing to effectively utilize the features of the separated pure sound sources. This leads to classification results being affected by mixed audio, resulting in low accuracy in style and emotion recognition. Poor scene adaptability: Traditional models have weak generalization capabilities and cannot adapt to the stereo analysis needs of different music styles such as rock, classical, and pop, or different in-vehicle acoustic environments such as reverberation or echo.

[0095] To address the aforementioned issues, this application proposes an integrated sound source analysis module based on a large model. By leveraging the strong feature extraction and generalization capabilities of the large model, and through the concatenation, separation, and classification processes, the accuracy and efficiency of stereo analysis in in-vehicle scenarios are improved.

[0096] This application aims to overcome the shortcomings of existing technologies and provide an intelligent, adaptive, integrated sound source analysis solution based on a large model. Its core objective is not only to achieve audio track separation and genre identification, but also to improve the accuracy and generalization ability of the stereo analysis module by strengthening the interaction between the separation and classification modules. Ultimately, the separated clean audio tracks and style tags are fed to subsequent modules for sound field modeling to output high-quality panoramic audio with immersiveness, clear positioning, and content adaptability.

[0097] This application proposes a stereo resolution method, such as... Figure 4 As shown, the method includes: S401, acquire the mixed audio and perform preprocessing.

[0098] The system uses an audio input module and a preprocessing module to receive the raw stereo signal and perform preprocessing such as standardization and noise reduction. This mixed audio can be, for example, the target audio mentioned earlier.

[0099] S402 uses a separation model to obtain multiple audio tracks and the tag for each track.

[0100] The tags are used to identify the type of the corresponding audio track. Audio tracks include, but are not limited to, vocals, lead instruments, bass, drums, backing vocals, etc. Each audio track has its own type tag.

[0101] S403 uses multiple audio tracks, each track's tag, and a classification model to determine the genre of mixed audio.

[0102] Genres include, but are not limited to, blues, classical, country, disco, hip hop, jazz, metal, pop, reggae, and rock.

[0103] S404 utilizes a mixing model for sound field modeling and panoramic sound signal synthesis.

[0104] In this embodiment, the mixing model receives clean audio tracks and tags separated by the separation model, and performs sound field modeling and panoramic sound signal synthesis based on genre characteristics and the features of each audio track. For details, please refer to existing technologies. Figure 5 This is a schematic diagram of a separation model processing provided in an embodiment of this application, such as... Figure 5 As shown, the separation model includes an encoder, a separator, and a decoder. The encoder is responsible for extracting the feature information of the original audio, the separator is used for time-frequency domain modeling of the signal, and the decoder reconstructs the modeled features into a time-domain signal, i.e., the audio bitstream of the sound source object track. Figure 5 The encoder shown includes three downsampling modules connected in series, and the decoder includes three upsampling modules connected in series. Figure 5 This explanation uses a separator that includes a separation module as an example.

[0105] S1, perform a fast Fourier transform on the target audio to obtain the complex spectrum of the target audio.

[0106] The target audio is a time-domain mixed signal, such as a signal of multiple audio tracks mixed and superimposed.

[0107] S2, the encoder performs convolutional downsampling on the complex numbers of the target audio to obtain the first feature.

[0108] The target audio is a temporal mixed signal. After short-time Fourier transform, the complex spectrum of the mixed signal is obtained. The complex spectrum of the mixed signal is fed into the encoder. The encoder module consists of three cascaded downsampling modules stacked together. Each upsampling module includes a convolutional layer, a normalization layer, and an activation function. For example, S2 may include S21-S23. Through continuous convolutional downsampling, the encoder reduces the frequency dimension and computational complexity of the model on the one hand, and extracts high-dimensional features suitable for the separation task on the other.

[0109] S21 employs a sparse banding strategy to decompose the complex spectrum of the mixed signal into three sub-bands.

[0110] For example, a sparse banding strategy is used to decompose the complex spectrum of the mixed signal into three sub-bands, such as the low-frequency complex spectrum, the mid-frequency complex spectrum, and the high-frequency complex spectrum.

[0111] S22 uses three downsampling modules to perform three convolutional downsampling operations on the three sub-bands based on different multiples to obtain the first sub-feature corresponding to each sub-band.

[0112] S23, merge each first sub-feature to obtain the first feature.

[0113] Optionally, after merging each first sub-feature, a two-dimensional convolutional layer can be used to process the merged features to obtain the first feature.

[0114] S3, the first feature is processed by a separator to obtain the second feature.

[0115] The separator is composed of stacked separation modules. The separation modules mainly perform time-frequency domain modeling on the high-dimensional features extracted by the encoder, including GAU and BLSTM, and each GAU and BLSTM contains a frequency domain path and a time domain path (dual-path structure).

[0116] Dual-path GAU and BLSTM achieve differentiated attention to the importance of musical features by dynamically assigning weights to different time and frequency positions. For example, GAU and BLSTM automatically assign higher weights to spectrally rich segments (such as the chorus) or key frequency ranges to enhance the priority of these regions in the separation process, thereby improving the effectiveness of music source separation. Simultaneously, the multi-head attention mechanism in GAU captures multi-dimensional dependencies and importance in the input sequence, further enhancing the model's ability to represent and process complex musical data.

[0117] In some embodiments, dual-path GAU and BLSTM achieve differentiated attention to the importance of musical features by assigning dynamic weights to different time and frequency locations. For example, the model automatically assigns higher weights to spectrally rich segments (such as the chorus) or key frequency ranges to enhance the priority of these regions during the separation process, thereby improving the effectiveness of music source separation. For instance, each sound source has its own spectral distribution characteristics; for example, bass is mainly concentrated in the mid-low frequencies, vocals are mainly concentrated in the low frequencies, drums are mainly concentrated across the entire frequency band and change rapidly, and other instruments such as guitars and pianos also have their own characteristics. The model autonomously learns these differences in spectral distribution and assigns different weights and attention levels in the frequency domain to different audio tracks, achieving better separation results.

[0118] Simultaneously, a multi-head attention mechanism is employed to capture multi-dimensional dependencies and importance within the input sequence. This mechanism extracts local and global information from the audio signal through convolutional or fully connected layers, calculates the correlation between different frequency points and time frames, and constructs a correlation matrix. Larger matrix values ​​indicate higher correlation, resulting in higher weights during model training, thus enhancing the model's temporal and frequency domain modeling capabilities and audio separation performance. The multi-head attention mechanism can be viewed as a combination of multiple parallel attention units, learning the correlation matrix from multiple dimensions and then concatenating and fusing them to obtain the final weight distribution. This allows the model to focus on key frequency points and key frames from multiple perspectives, further enhancing feature focusing capabilities and separation accuracy. This further improves the model's representation and processing capabilities for complex music data. Further details can be found in related technologies and will not be elaborated upon here.

[0119] For example, S3 may include S31-S33.

[0120] S31, the frequency domain modeling module of GAU and the frequency domain modeling module of BLSTM in the separation module are used to perform frequency domain modeling on the first feature to obtain the second intermediate feature.

[0121] S32, using the weighting module to weight the first feature and the second intermediate feature, to obtain the third intermediate feature.

[0122] S33, the third intermediate feature is modeled in the time domain using the GAU time domain modeling module and the BLSTM time domain modeling module in the separation module to obtain the fourth intermediate feature.

[0123] S34, using the weighting module to weight the third and fourth intermediate features, the second feature is obtained.

[0124] S4 uses a decoder to perform convolutional upsampling on the second feature to obtain multiple audio features.

[0125] The decoder can be viewed as the inverse process of the encoder. It restores the high-dimensional features modeled by the separation modules to the complex spectrum form of the separated signal through convolutional upsampling, and then obtains the time-domain sampled signals of each instrument track through inverse short-time Fourier transform (ISTFT). The decoder is composed of multiple stacked upsampling modules.

[0126] For example, S4 can include S41-S44. The encoder reduces the frequency dimension and computational complexity of the model through continuous convolutional downsampling, while extracting high-dimensional features suitable for the separation task.

[0127] S41 employs a sparse banding strategy to decompose the second feature into three sub-bands.

[0128] For example, a sparse banding strategy is adopted to decompose the second feature into three sub-bands, such as low-frequency feature, mid-frequency feature and high-frequency feature.

[0129] S42 employs three upsampling modules to perform three convolutional upsampling operations on the three sub-bands based on different multiples, thereby obtaining the second sub-feature corresponding to each sub-band.

[0130] S43, merge each second sub-feature to obtain the sixth intermediate feature.

[0131] S44 uses the audio track generation module to process the sixth intermediate feature and obtain multiple audio features.

[0132] S5 performs a fast inverse Fourier transform on multiple audio features to obtain multiple audio tracks.

[0133] In some embodiments, the classification model comprises stacked two-dimensional convolutional layers, max-pooling layers, fully connected layers, dropout layers, and activation functions. The convolutional layers are used for feature extraction, the max-pooling layers for downsampling, the fully connected layers for capturing global information, the dropout layers to prevent overfitting, and the final output layer uses an activation function for classification. Figure 6 As shown, multiple audio tracks and their respective tags and classification models are used to determine the genre of the mixed audio, specifically including S601-S603. S601 performs a fast Fourier transform on the mixed audio and the multiple audio tracks included in the mixed audio to obtain the complex spectrum of the mixed audio and the multiple audio tracks.

[0134] S602, extracting Mel spectrum based on complex spectrum.

[0135] S603, input the Mel spectrum into the classification model to obtain the genre label.

[0136] The classification model is based on the pitch sequence, rhythmic features, harmonic progressions, and spectral energy distribution of music. It uses a classification neural network to identify music genres (such as classical, rock, jazz, and electronic) and infer their emotional characteristics (such as exciting, soothing, and melancholic).

[0137] The classification model uses Mel spectra as input. Different music genres have some unique differences in Mel spectra. This model cleverly transforms the music genre classification problem into an image classification task.

[0138] In some embodiments, a classification model is trained using an open-source dataset. The training configuration for the music genre classification model is as follows: the dataset includes 10 different genres (blues, classical, country, disco, hip hop, jazz, metal, pop, reggae, and rock), with 100 songs in each genre (30 seconds per sample). In some embodiments, the loss function of the classification model is the classification cross-entropy function.

[0139] Compared with existing technologies, the significant advantages of this application are: high separation accuracy, such as the strong feature extraction capability based on a large model, combined with in-vehicle scene optimization, to achieve accurate sound source separation in complex environments; improved classification accuracy, through an integrated separation, fusion, and classification process, utilizing pure sound source features to improve classification reliability; strong adaptability to in-vehicle scenes, with optimized models for in-vehicle noise and reverberation environments, supporting customized sound source splitting and in-vehicle adapted label output, adapting to different vehicle models and usage scenarios; and good interface compatibility, with standardized output format and interface design, which can seamlessly connect to existing in-vehicle audio rendering systems, facilitating engineering implementation.

[0140] The method embodiments of this application have been described above with reference to the accompanying drawings and examples. The device embodiments of this application will be described below with reference to the accompanying drawings and examples.

[0141] Figure 7 This is a schematic diagram of the structure of an audio analysis device provided in an embodiment of this application, as shown below. Figure 7 As shown, the audio analysis device 700 includes: a first acquisition unit 710, a second acquisition unit 720, a first determination unit 730, a second determination unit 740, and a third determination unit 750.

[0142] The first acquisition unit 710 is used to acquire the target audio to be analyzed.

[0143] The second acquisition unit 720 is used to acquire the complex spectrum of the target audio.

[0144] The first determining unit 730 is used to input the complex spectrum into the encoder in the separation model to obtain the first feature. The second determining unit 740 is used to input the first feature into the separator of the separation model, and use the separator to perform time-frequency domain modeling on the first feature at least once to obtain the second feature; wherein performing time-frequency domain modeling once includes: performing frequency domain modeling on the first intermediate feature to obtain the second intermediate feature; weighting the first intermediate feature and the second intermediate feature to obtain the third intermediate feature; performing time domain modeling on the third intermediate feature to obtain the fourth intermediate feature; and weighting the fourth intermediate feature and the third intermediate feature to obtain the fifth intermediate feature; the fifth intermediate feature is used as the first intermediate feature in the next time-frequency domain modeling; wherein, in the first time time-frequency domain modeling is performed, the first intermediate feature is the first feature; and in the last time time-frequency domain modeling is performed, the fifth intermediate feature is the second feature.

[0145] The third determining unit 750 is used to input the second feature into the decoder of the separation model to obtain at least two audio tracks.

[0146] In some embodiments, the second determining unit 740 is further configured to use the first frequency domain modeling module of the first separation module and the second frequency domain modeling module of the second separation module to sequentially perform frequency domain modeling on the first intermediate feature to obtain the second intermediate feature; and use the first time domain modeling module of the first separation module and the second time domain modeling module of the second separation module to sequentially perform time domain modeling on the third intermediate feature to obtain the fourth intermediate feature.

[0147] In some embodiments, the first acquisition unit 710 is further configured to perform segmentation processing on the target mixed audio to obtain a plurality of audio segments arranged in chronological order; wherein the target audio is one of the plurality of audio segments; the target audio includes at least one audio frame.

[0148] In some embodiments, the first determining unit 730 is further configured to adopt a sparse banding strategy to divide the complex spectrum into multiple first sub-bands along the frequency dimension; to downsample each first sub-band by a corresponding multiple to obtain a first sub-feature corresponding to each first sub-band; and to concatenate multiple first sub-features to obtain a first feature.

[0149] In some embodiments, the third determining unit 750 is further configured to input the second feature into the decoder of the separation model, process the second feature using the audio track generation module to obtain at least two audio features, and perform frequency domain to time domain conversion processing on the at least two audio features to obtain at least two audio tracks.

[0150] In some embodiments, the third determining unit 750 is further configured to adopt a sparse banding strategy to divide the second feature according to the frequency range to obtain multiple second sub-bands; to perform upsampling processing on each second sub-band by a corresponding multiple to obtain a second sub-feature corresponding to each second sub-band; to concatenate multiple second sub-features to obtain a sixth intermediate feature; and to process the sixth intermediate feature using the audio track generation module to obtain at least two audio features.

[0151] In some embodiments, the audio analysis apparatus further includes a training unit for training a separation model based on a loss function; wherein the loss function is obtained by weighting at least two of a frequency domain complex spectrum loss function, an SDR normalized loss function, and a time domain compression loss function.

[0152] Figure 8 This is a schematic diagram of the structure of an audio analysis device provided in an embodiment of this application, as shown below. Figure 8 As shown, the audio analysis device 800 includes one or more processors 810 and one or more memories 820.

[0153] The processor 810 can support the audio analysis device 800 in implementing the methods described in the preceding method embodiments.

[0154] The memory 820 stores a program that can be executed by the processor 810, causing the processor 810 to perform the methods described in the preceding method embodiments. The memory 820 can be independent of the processor 810 or integrated into the processor 810.

[0155] Optionally, the audio analysis device 800 may also include a transceiver 830. The processor 810 can communicate with other devices or chips via the transceiver 830. For example, the processor 810 can send and receive data with other devices or chips via the transceiver 830.

[0156] This application provides a computer storage medium that stores one or more programs, which can be executed by one or more processors to implement the steps of the methods described in any of the above embodiments.

[0157] It should be noted that the descriptions of the storage medium and device embodiments above are similar to the descriptions of the method embodiments above, and have similar beneficial effects. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0158] The aforementioned processor can be at least one of the following: application-specific integrated circuit (ASIC), digital signal processor (DSP), digital signal processing device (DSPD), programmable logic device (PLD), field-programmable gate array (FPGA), central processing unit (CPU), control module, microcontroller module, and microprocessor. It is understood that other electronic devices can also implement the functions of the aforementioned processor, and this application does not specifically limit the specific implementation.

[0159] The aforementioned computer storage media / memory can be read-only memory, programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), ferromagnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD ROM), etc.

[0160] This application provides a computer program including computer-readable code. When the computer-readable code runs in an electronic device, the processor in the electronic device executes some or all of the steps in the above-described method.

[0161] This application provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program. When the computer program is read and executed by a computer, it implements some or all of the steps in the above-described method. This computer program product can be implemented specifically through hardware, software, or a combination thereof. In some embodiments, the computer program product is specifically embodied as a computer storage medium; in other embodiments, the computer program product is specifically embodied as a software product, such as a software development kit (SDK), etc.

[0162] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above steps / processes do not imply a sequential order of execution; the execution order of each step / process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above embodiments of this application are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0163] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0164] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0165] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0166] In addition, each functional unit in the various embodiments of this application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0167] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, read-only memory (ROM), magnetic disks, or optical disks.

[0168] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence or the part that contributes to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an in-vehicle terminal (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, magnetic disks, or optical disks.

[0169] The above are merely embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

[0170] The above embodiments are merely preferred embodiments provided to fully illustrate this application, and the scope of protection of this application is not limited thereto. Equivalent substitutions or modifications made by those skilled in the art based on this application are all within the scope of protection of this application.

Claims

1. An audio analysis method, characterized in that, The method includes: Obtain the target audio to be analyzed; Obtain the complex spectrum of the target audio; The complex spectrum is input into the encoder in the separation model to obtain the first feature; The first feature is input into the separator of the separation model, and the first feature is modeled in the time-frequency domain at least once using the separator to obtain the second feature; The process of performing time-frequency domain modeling includes: performing frequency domain modeling on a first intermediate feature to obtain a second intermediate feature; weighting the first intermediate feature and the second intermediate feature to obtain a third intermediate feature; performing time domain modeling on the third intermediate feature to obtain a fourth intermediate feature; and weighting the fourth intermediate feature and the third intermediate feature to obtain a fifth intermediate feature; the fifth intermediate feature is used as the first intermediate feature in the next time the time-frequency domain modeling is performed. Specifically, during the first execution of the time-frequency domain modeling, the first intermediate feature is the first feature; and during the last execution of the time-frequency domain modeling, the fifth intermediate feature obtained is the second feature. The second feature is input into the decoder of the separation model to obtain at least two audio tracks.

2. The method according to claim 1, characterized in that, The separator includes a first separation module and a second separation module, both of which are time-domain and frequency-domain dual-path separation modules; The step of obtaining the second intermediate feature by performing frequency domain modeling on the first intermediate feature includes: Using the first frequency domain modeling module of the first separation module and the second frequency domain modeling module of the second separation module, the first intermediate feature is sequentially modeled in the frequency domain to obtain the second intermediate feature; The process of obtaining the fourth intermediate feature by performing time-domain modeling on the third intermediate feature includes: The fourth intermediate feature is obtained by sequentially performing time-domain modeling on the third intermediate feature row using the first time-domain modeling module of the first separation module and the second time-domain modeling module of the second separation module.

3. The method according to claim 1, characterized in that, The method further includes: The target mixed audio is segmented to obtain multiple audio segments arranged in chronological order; wherein the target audio is one of the multiple audio segments; the target audio includes at least one audio frame.

4. The method according to any one of claims 1-3, characterized in that, The step of inputting the complex spectrum into the encoder of the separation model to obtain the first feature includes: A sparse banding strategy is adopted to divide the complex spectrum into multiple first sub-bands along the frequency dimension; For each of the first sub-bands, downsample by a corresponding multiple to obtain the first sub-feature corresponding to each of the first sub-bands; By concatenating multiple first sub-features, the first feature is obtained.

5. The method according to any one of claims 1-3, characterized in that, The decoder includes an audio track generation module; The second feature is input into the decoder of the separation model to obtain at least two audio tracks, including: The second feature is input into the decoder of the separation model, and the second feature is processed by the audio track generation module to obtain at least two audio features. Perform frequency-domain to time-domain conversion on the at least two audio features to obtain at least two audio tracks.

6. The method according to claim 5, characterized in that, Before using the track generation module to map the second feature to at least two tracks, the method further includes: By employing a sparse banding strategy, the second feature is divided according to frequency range to obtain multiple second sub-bands; Each second sub-band is upsampled by a corresponding multiple to obtain the second sub-feature corresponding to each second sub-band; By concatenating multiple second sub-features, a sixth intermediate feature is obtained; The second feature is processed using the audio track generation module to obtain at least two audio features, including: The sixth intermediate feature is processed using the audio track generation module to obtain at least two audio features.

7. The method according to any one of claims 1-3, characterized in that, Before inputting the complex spectrum into the encoder in the separation model, the method further includes: The separation model is trained based on a loss function; wherein the loss function is obtained by weighting at least two of the following: a frequency domain complex spectrum loss function, an SDR normalized loss function, and a time domain compression loss function.

8. An apparatus for analyzing audio, characterized in that, The device includes: The first acquisition unit is used to acquire the target audio to be analyzed; The second acquisition unit is used to acquire the complex spectrum of the target audio. The first determining unit is used to input the complex spectrum into the encoder in the separation model to obtain the first feature; The second determining unit is configured to input the first feature into the separator of the separation model, and use the separator to perform at least one time-frequency domain modeling on the first feature to obtain a second feature; wherein performing one time-frequency domain modeling includes: performing frequency domain modeling on the first intermediate feature to obtain a second intermediate feature; weighting the first intermediate feature and the second intermediate feature to obtain a third intermediate feature; performing time domain modeling on the third intermediate feature to obtain a fourth intermediate feature; and weighting the fourth intermediate feature and the third intermediate feature to obtain a fifth intermediate feature; the fifth intermediate feature is used as the first intermediate feature in the next time the time-frequency domain modeling is performed. Specifically, during the first execution of the time-frequency domain modeling, the first intermediate feature is the first feature; and during the last execution of the time-frequency domain modeling, the fifth intermediate feature obtained is the second feature. The third determining unit is used to input the second feature into the decoder of the separation model to obtain at least two audio tracks.

9. An audio analysis device, characterized in that, It includes one or more processors and a memory; the memory is used to store one or more programs, which, when executed by the one or more processors, cause the device to perform the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by the computer's processor, causes the computer to perform the method according to any one of claims 1 to 7.

Images