Audio processing method and device, electronic equipment, storage medium and program product

Abstract

The invention discloses an audio processing method and device, electronic equipment, a storage medium and a program product, and relates to the technical field of media data processing, and the method comprises the steps: obtaining a first audio, and extracting a first amplitude spectrum of the first audio; performing sub-band division on the first amplitude spectrum based on the sampling frequency of the first audio to obtain a first sub-band; extracting a first spectrum feature corresponding to the first sub-band, and splicing all the first spectrum features to obtain a sub-band feature of the first audio; modeling the sub-band features of the first audio by using a first dual-path modeling module to obtain a first modeling result, the first dual-path modeling module being configured to have a first sub-band modeling module and a first cross-sub-band modeling module; and performing phase processing based on the first modeling result to obtain a target audio corresponding to the first audio. The method can improve the effect of the obtained target audio.

Description

Technical Field

[0001] This application relates to the field of media data processing technology, specifically to audio processing methods, apparatus, electronic devices, storage media, and program products. Background Technology

[0002] In some audio processing tasks, such as music synthesis, speech synthesis, sound effects generation, and speech enhancement, phase generation algorithms are generally used to convert the amplitude spectrum into a time-domain waveform. For example, ... Figure 1 As shown, in the audio processing task, the time-domain waveform is obtained by estimating the amplitude spectrum of the original audio and synthesizing the phase spectrum by combining the phase generation algorithm.

[0003] However, although through Figure 1 The method shown can effectively synthesize the corresponding phase spectrum from the amplitude spectrum, but the quality of the final audio is not good. Summary of the Invention

[0004] In view of this, this application provides an audio processing method, apparatus, electronic device, storage medium, and program product to solve the problem of poor audio processing effect.

[0005] In a first aspect, this application provides an audio processing method, including: Acquire the first audio signal and extract the first amplitude spectrum of the first audio signal; The first sub-band is obtained by dividing the first amplitude spectrum into sub-bands based on the sampling frequency of the first audio. Extract the first spectral features corresponding to the first sub-band, and concatenate all the first spectral features to obtain the sub-band features of the first audio. The first dual-path modeling module is used to model the sub-band features of the first audio to obtain a first modeling result. The first dual-path modeling module is configured to have a first sub-band modeling module and a first cross-sub-band modeling module. The first sub-band modeling module is configured to model the sub-band features of the first audio along the time dimension and input the modeling result to the first cross-sub-band modeling module. The first cross-sub-band modeling module is configured to model the obtained modeling result along the sub-band dimension. Phase processing is performed based on the first modeling result to obtain the target audio corresponding to the first audio.

[0006] Secondly, this application provides an audio processing apparatus, comprising: The first audio acquisition module is used to acquire the first audio and extract the first amplitude spectrum of the first audio. The first sub-band division module is used to divide the first amplitude spectrum into sub-bands based on the sampling frequency of the first audio, so as to obtain the first sub-band; The first feature extraction module is used to extract the first spectral features corresponding to the first sub-band and to concatenate all the first spectral features to obtain the sub-band features of the first audio. The first modeling module is used to model the sub-band features of the first audio using the first dual-path modeling module to obtain a first modeling result. The first dual-path modeling module is configured to have a first sub-band modeling module and a first cross-sub-band modeling module. The first sub-band modeling module is configured to model the sub-band features of the first audio along the time dimension and input the modeling result to the first cross-sub-band modeling module. The first cross-sub-band modeling module is configured to model the obtained modeling result along the sub-band dimension. The first phase processing module is used to perform phase processing based on the first modeling result to obtain the target audio corresponding to the first audio.

[0007] Thirdly, this application provides an electronic device, including: a memory and a processor, which are communicatively connected to each other. The memory stores computer instructions, and the processor executes the computer instructions to perform the audio processing method of the first aspect or any corresponding embodiment described above.

[0008] Fourthly, this application provides a computer-readable storage medium storing computer instructions for causing a computer to perform the audio processing method of the first aspect or any corresponding embodiment described above.

[0009] Fifthly, this application provides a computer program product, including computer instructions for causing a computer to execute the audio processing method described in the first aspect or any corresponding embodiment thereof.

[0010] The audio processing method provided in this application involves acquiring a first audio signal and extracting its first amplitude spectrum; dividing the first amplitude spectrum into sub-bands based on the sampling frequency of the first audio signal to obtain a first sub-band; extracting the first spectral features corresponding to the first sub-band and concatenating all the first spectral features to obtain the sub-band features of the first audio signal; modeling the sub-band features of the first audio signal using a first dual-path modeling module to obtain a first modeling result. The first dual-path modeling module is configured to have a first sub-band modeling module and a first cross-sub-band modeling module. The first sub-band modeling module is configured to model the sub-band features of the first audio signal along the time dimension and input the modeling result to the first cross-sub-band modeling module. The first cross-sub-band modeling module is configured to model the obtained modeling result along the sub-band dimension; and performing phase processing based on the first modeling result to obtain the target audio signal corresponding to the first audio signal. In this method, since the importance of information varies in different frequency domain intervals, dividing the first amplitude spectrum into sub-bands allows for spectral feature extraction based on frequency domain intervals, preserving important information in the first audio signal and providing a foundation for subsequent phase processing, etc. Furthermore, by setting up the first dual-path modeling module, we can focus on modeling using independent sub-band features on the one hand, and modeling through information interaction between sub-bands on the other hand, which improves the accuracy of sub-band feature modeling. On this basis, phase processing can improve the effect of the obtained target audio. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the specific embodiments or related technologies of this application, the drawings used in the description of the specific embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0012] Figure 1 This is a schematic diagram of audio processing in related technologies; Figure 2 This is a schematic diagram illustrating an application scenario according to an embodiment of this application; Figure 3 This is a schematic flowchart of a first audio processing method according to an embodiment of this application; Figure 4 This is a schematic diagram of a second type of audio processing method according to an embodiment of this application; Figure 5 This is a schematic diagram of a frequency band decomposition module according to an embodiment of this application; Figure 6 This is a schematic diagram of a dual-path modeling module according to an embodiment of this application; Figure 7This is a schematic diagram of a sub-band modeling module according to an embodiment of this application; Figure 8 This is a schematic diagram of a cross-subband modeling module according to an embodiment of this application; Figure 9 This is a schematic diagram of a phase estimation module according to an embodiment of this application; Figure 10 This is a training schematic diagram of the second dual-path modeling module according to an embodiment of this application; Figure 11 This is a schematic diagram of phase convolution according to an embodiment of this application; Figure 12 This is a schematic diagram of audio processing according to an embodiment of this application; Figure 13 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0013] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0014] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0015] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0016] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0017] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0018] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0019] In related technologies, deep learning and other techniques are used to process audio tasks, such as deep learning-based phase generation algorithms. These algorithms can effectively convert amplitude spectrum to waveform; however, they are only applicable to a single type of audio processing task, and their generalization ability across different audio scenarios needs improvement.

[0020] Furthermore, deep learning-based phase generation algorithms only support audio input with a single sampling rate. When it is necessary to adapt to scenarios with different sampling rates, the network models involved often need to be restructured and retrained.

[0021] Based on this, embodiments of this application provide an audio processing method, which involves acquiring a first audio signal and extracting its first amplitude spectrum; dividing the first amplitude spectrum into sub-bands based on the sampling frequency of the first audio signal to obtain a first sub-band; extracting the first spectral features corresponding to the first sub-band and concatenating all the first spectral features to obtain the sub-band features of the first audio signal; modeling the sub-band features of the first audio signal using a first dual-path modeling module to obtain a first modeling result, wherein the first dual-path modeling module is configured to have a first sub-band modeling module and a first cross-sub-band modeling module, wherein the first sub-band modeling module is configured to model the sub-band features of the first audio signal along the time dimension and input the modeling result to the first cross-sub-band modeling module, wherein the first cross-sub-band modeling module is configured to model the obtained modeling result along the sub-band dimension; and performing phase processing based on the first modeling result to obtain the target audio signal corresponding to the first audio signal.

[0022] This method addresses the varying importance of information across different frequency ranges by dividing the first amplitude spectrum into sub-bands. This allows for spectral feature extraction based on frequency ranges, preserving crucial information from the first audio and providing a foundation for subsequent phase processing. Furthermore, the first dual-path modeling module enables both focused modeling using independent sub-band features and modeling through inter-sub-band information interaction, improving the accuracy of sub-band feature modeling. Phase processing based on this foundation enhances the quality of the resulting target audio.

[0023] As one optional application scenario of this disclosure embodiment, such as Figure 2As shown, application 201 is installed in terminal device 210, and user 230 can interact with application 201 through terminal device 210 and / or access device of terminal device 210.

[0024] For example, application 201 can be any application that provides audio processing tasks. For instance, application 201 could be a music generation application. Figure 1 In the application scenario shown, if application 201 is active, the terminal device 210 can display the interface 202 of application 201. The interface 202 may include various pages that application 201 can provide, such as interactive pages, settings pages, query pages, etc.

[0025] In some embodiments, terminal device 210 is communicatively connected to server 220 to provide services to application 201. Terminal device 210 may be a mobile terminal, fixed terminal, or portable terminal, etc., including but not limited to mobile phones, desktop computers, laptop computers, multimedia tablets, e-book devices, gaming devices, or any combination thereof, including accessories and peripherals of these devices or any combination thereof. In some embodiments, terminal device 210 may also support any type of interface, and server 220 may be various types of computing systems or servers capable of providing computing power, including but not limited to mainframes, edge computing nodes, computing devices in cloud environments, etc.

[0026] It should be noted that, Figure 2 This is merely an example of an application scenario and does not limit the scope of protection of this disclosure.

[0027] The embodiments of this disclosure will now be described with reference to the accompanying drawings. It should be understood that the pages shown in the drawings are merely examples, and various page designs are possible in practice. The various graphic elements on the page may have different arrangements and different visual representations, one or more elements may be omitted or replaced, and one or more other elements may also be present; no limitations are imposed on the embodiments described in this disclosure. Furthermore, the embodiments described below primarily pertain to the server-side.

[0028] For ease of description below, the terms used in the embodiments of this application are explained as follows: Frequency domain characteristics: The characteristics of a time-domain waveform converted into a frequency-domain waveform through Fourier transform, such as amplitude spectrum and phase spectrum.

[0029] Amplitude spectrum: describes the energy intensity of different frequency components, determining the "timbre" and "content" of speech, such as the harmonic distribution of vowels.

[0030] Phase spectrum: describes the initial phase of different frequency components, determines the "shape details" of the waveform, and affects the "naturalness" and "coherence" of speech.

[0031] The relationship between amplitude, phase, and waveform: The amplitude spectrum determines "likeness," the phase generation algorithm determines "smoothness," and the time-domain waveform is the final result. The amplitude spectrum and phase spectrum together determine the time-domain waveform, and the phase generation algorithm is the key to connecting the frequency domain features and the time-domain waveform, determining whether the waveform is natural. These three form a complete link of "feature generation → phase estimation → waveform reconstruction."

[0032] According to an embodiment of this application, an audio processing method embodiment is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0033] This embodiment provides an audio processing method that can be used in the aforementioned server-side application. Figure 3 This is a flowchart of an audio processing method according to an embodiment of this application, such as... Figure 3 As shown, the process includes the following steps: Step S301: Obtain the first audio and extract the first amplitude spectrum of the first audio.

[0034] The first audio file can be a standalone audio file or audio from a video, etc. The type of the first audio file can be speech, music, ambient sound effects, etc. There are no restrictions on the source and type of the first audio file; you can set it according to your actual needs.

[0035] In this embodiment, there are no restrictions on the sampling rate of the first audio, which can be 8 kHz, 16 kHz, 24 kHz, 32 kHz, and 48 kHz, etc.

[0036] The first audio signal is in the time domain, while the amplitude spectrum is a characteristic of the frequency domain. Therefore, it is necessary to convert the first audio signal from the time domain to the frequency domain. This conversion can be achieved through Fourier transform or short-time Fourier transform, etc.

[0037] For the time-domain signal of the first audio, a short-time Fourier transform is performed, using a fixed-time window length, frame shift, and a number of Fourier transform points equal to a preset multiple of the window length, to transform the time-domain signal into the frequency domain and obtain its frequency domain characteristics. ,in, , These represent the magnitude in the frequency and time dimensions, respectively.

[0038] For the first audio file at different sampling rates, since the window length and frame shift duration remain constant, and the ratio of Fourier points to window length points is fixed, the time and frequency resolution are identical across different sampling rates. Specifically, the time resolution is determined by the fixed window length / frame shift duration and is independent of the sampling rate; the frequency resolution is determined by the fixed frequency interval (Δf), and the sampling rate is canceled out in the Δf calculation formula, so it is also independent of the sampling rate. Therefore, audio files at different sampling rates will have identical time and frequency resolution.

[0039] For example, extracting the first amplitude spectrum from the first audio signal can involve digitization, preprocessing (DC removal, framing, windowing), Fourier transform, amplitude (magnitude) calculation, and spectral processing. Specifically, the first audio signal is digitized into a digital signal. Since directly performing a Fourier transform on the digital signal can lead to errors due to signal non-stationarity or phase issues, preprocessing of the digital signal is necessary, including but not limited to DC removal, framing, and windowing. The digital signal may contain zero-frequency (DC) components (e.g., a non-zero average signal value), which can interfere with low-frequency spectral analysis. Therefore, DC removal can be performed by subtracting the average signal value. Since the audio signal is time-varying and has a long time span, it needs to be segmented into short frames for subsequent processing. Furthermore, to avoid losing frame boundary information, a certain degree of overlap between frames must be ensured during framing. Further, directly truncating the signal into frames can cause spectral leakage; windowing can mitigate this by smoothing the start / end of frames. Hanning windows, Hamming windows, and rectangular windows can be used for windowing.

[0040] Perform a Fourier transform on the windowed signal of each frame to obtain a complex array. Then, calculate the magnitude of each element in the complex array to obtain the amplitude of the corresponding frequency. Optionally, the amplitude value obtained by the Fourier transform is related to the signal energy and the number of Fourier points. Therefore, the first amplitude spectrum of the first audio frequency can be obtained by normalization.

[0041] Step S302: Divide the first amplitude spectrum into sub-bands based on the sampling frequency of the first audio signal to obtain the first sub-band.

[0042] Subbanding the first amplitude spectrum involves dividing the continuous frequency axis into several frequency intervals, or subbands, according to specific rules. These specific rules include the number of subbands and the subbanding rules themselves. The number of subbands can be set according to requirements; more subbands result in higher frequency resolution, but also greater computational complexity. The subbanding rules characterize the distribution of subband bandwidth, and can be uniform or non-uniform, etc.

[0043] Since the frequency range of the amplitude spectrum is from 0 to the Nyquist frequency (i.e., half the sampling frequency), the frequency range of the first amplitude spectrum can be obtained using the sampling frequency of the first audio signal. Then, by combining this with specific rules, the first amplitude spectrum can be subdivided into k first sub-bands. There are multiple first sub-bands obtained; that is, they are obtained by dividing the first amplitude spectrum according to the division rules. In other words, each first sub-band represents an amplitude spectrum within a specific frequency range.

[0044] Step S303: Extract the first spectral features corresponding to the first sub-band, and concatenate all the first spectral features to obtain the sub-band features of the first audio.

[0045] For each first sub-band, spectral features are extracted to obtain a first spectral feature corresponding to each first sub-band. Then, all the first spectral features are concatenated according to the order of the first sub-bands to obtain the sub-band features of the first audio. For example, the sub-band feature Z of the first audio can be represented as: For example, in the formula, k represents the number of the first sub-bands, C represents the size of the channel dimension, and T represents the size of the time dimension.

[0046] Specifically, based on the sub-band division results, the amplitude spectrum within each sub-band is extracted; each sub-band is processed individually to calculate its unique features, obtaining the first spectral feature; the first spectral features of all sub-bands are concatenated into a single overall feature vector, which serves as the input for subsequent tasks. The first spectral feature includes, but is not limited to, energy and power-related features (e.g., sub-band energy, sub-band power, and sub-band energy ratio), frequency component characteristics (e.g., sub-band center frequency, sub-band peak frequency, and sub-band bandwidth), and spectral shape features (e.g., sub-band spectral slope, sub-band spectral entropy, and sub-band spectral flatness), etc. In practical applications, the choice of which feature(s) to use as the first spectral feature is determined based on actual needs, and no restrictions are imposed here.

[0047] Step S304: The first dual-path modeling module is used to model the sub-band features of the first audio to obtain the first modeling result.

[0048] The first dual-path modeling module is configured to have a first sub-band modeling module and a first cross-sub-band modeling module, and the output of the first sub-band modeling module is used as the input of the first cross-sub-band modeling module.

[0049] In this step, the number of first dual-path modeling modules used can be one or more, and there is no specific limit to their exact number. If multiple first dual-path modeling modules are used, they are stacked, meaning the output of one first dual-path modeling module is connected to the input of the next. This connection can be direct or processed through a transpose module before connection.

[0050] The first dual-path modeling module has a first sub-band modeling module and a first cross-sub-band modeling module. The output of the first sub-band modeling module is used as the input of the first cross-sub-band modeling module. Specifically, the first sub-band modeling module obtains the corresponding modeling result by modeling independent sub-band features; the first cross-sub-band modeling module performs inter-sub-band modeling based on the output (i.e., the modeling result) of the first sub-band modeling module.

[0051] In some alternative implementations, the first sub-band modeling module is used to model the sub-band features of the first audio from a temporal dimension. Exemplarily, one-dimensional convolution, activation functions, and normalization layers are used to model the sub-band features of the first audio along the temporal dimension. For a feature input of k×C×T, the first sub-band modeling module processes all first sub-bands in parallel, with no information exchange between sub-bands.

[0052] The first sub-band modeling module is used to model the obtained modeling results from the sub-band dimension. For example, for The input features are first transformed into their shape using transpose. Then, a bidirectional long short-term memory recurrent neural network (BiLSTM) is used to model the features along the subband dimension. For features with different sampling rates, the traversal length of the BiLSTM varies due to the different number of subbands k. The first subband modeling module, for... The feature input is T, which is the batch dimension, and k is the time recursion term in LSTM, that is, recursively along the sub-bands, where there is information interaction between the sub-bands.

[0053] The sub-band features of the first audio are processed by at least one first dual-path modeling module to obtain the first modeling result. As described above, the first dual-path modeling modules are stacked. For different first audios, the module output level can be selected, that is, the output of the first dual-path modeling module at a specified level can be set as the first modeling result. There are no restrictions on this; it can be set according to actual needs.

[0054] Step S305: Perform phase processing based on the first modeling result to obtain the target audio corresponding to the first audio.

[0055] The first modeling result represents the result after processing by the phase processing algorithm. Further phase processing, including but not limited to phase estimation and phase reconstruction, is then performed on this basis to obtain the target audio corresponding to the first audio. The specific implementation of phase processing is not limited here; it can be set according to actual needs.

[0056] The audio processing method provided in this embodiment addresses the varying importance of information across different frequency ranges. By dividing the first amplitude spectrum into sub-bands, spectral features can be extracted based on these frequency ranges, preserving crucial information in the first audio and providing a foundation for subsequent phase processing. Furthermore, the first dual-path modeling module allows for both focused modeling using independent sub-band features and modeling through information interaction between sub-bands, improving the accuracy of sub-band feature modeling. Based on this, phase processing enhances the quality of the obtained target audio.

[0057] This embodiment provides an audio processing method that can be used in the aforementioned server-side application. Figure 4 This is a flowchart of an audio processing method according to an embodiment of this application, such as... Figure 4 As shown, the process includes the following steps: Step S401: Obtain the first audio signal and extract its first amplitude spectrum. See details... Figure 3 Step S301 of the illustrated embodiment will not be described again here.

[0058] Step S402: Divide the first amplitude spectrum into sub-bands based on the sampling frequency of the first audio signal to obtain the first sub-band.

[0059] Specifically, step S402 includes: Step S4021: Determine the upper limit of the spectrum bandwidth based on the sampling rate of the first audio.

[0060] The sampling rate of the first audio file can be obtained from the audio metadata, or it can be obtained by parsing the header of the first audio file using an audio processing library to extract the sampling rate parameter, and so on. Of course, other methods can also be used, and no restrictions are placed on them here.

[0061] As described above, the upper limit of the spectral bandwidth is related to the audio sampling rate; that is, the upper limit of the spectral bandwidth is half of the audio sampling rate. For example, if the sampling rate of the first audio is 8kHz, the corresponding upper limit of the spectral bandwidth is 4kHz, meaning that the spectral bandwidth range corresponding to the first audio is 0~4kHz.

[0062] Step S4022: Based on the upper limit of the spectrum bandwidth and the preset sub-band division method, determine the first sub-band division method corresponding to the first audio.

[0063] The first sub-band division method includes the number of first sub-bands and the frequency range corresponding to the first sub-band.

[0064] A preset subband partitioning method can be used to characterize the rules for subband partitioning. For example, according to a partitioning form from low frequency to high frequency and from fine to coarse, it gives the subband bandwidth from low frequency to high frequency, as well as the number of subbands corresponding to each subband bandwidth, and correspondingly, characterizes the effective bandwidth range corresponding to each subband bandwidth. For example, Table 1 shows the preset subband partitioning methods: Table 1 Preset Subband Division Methods

[0065] As described in Table 1, a subband bandwidth of 100Hz corresponds to 11 subbands, with an effective bandwidth range of 0~1.1KHz; a subband bandwidth of 250Hz corresponds to 20 subbands, with an effective bandwidth range of 1.1~6.1KHz.

[0066] Therefore, the upper limit of the spectrum bandwidth can be used to determine the spectrum bandwidth range. By matching the spectrum bandwidth range with the preset sub-band division method, the first sub-band division method corresponding to the first audio can be determined. For example, if the sampling rate of the first audio is 8kHz, and the spectrum bandwidth range corresponding to the first audio is 0~4kHz, then there are 11 sub-bands with a sub-band bandwidth of 100Hz and 12 sub-bands with a sub-band bandwidth of 250Hz. Therefore, there are a total of 11+12 first sub-bands.

[0067] Step S4023: Divide the first amplitude spectrum into sub-bands based on the first sub-band division method to obtain the first sub-band.

[0068] After determining the subband width and the corresponding number of subbands, the first amplitude spectrum is divided into subbands to obtain the corresponding number of first subbands.

[0069] Step S403: Extract the first spectral features corresponding to the first sub-band, and concatenate all the first spectral features to obtain the sub-band features of the first audio.

[0070] After obtaining the corresponding number of subbands, the first spectral features can be extracted using the coding units that correspond one-to-one with each subband. Referring to Table 1, there are 55 subbands within the effective bandwidth, therefore 55 coding units can be set, meaning there is a one-to-one correspondence between subbands and coding units.

[0071] like Figure 5 As shown, the extraction of spectral features is achieved through a frequency band decomposition module, which contains coding units that correspond one-to-one with sub-bands. Amplitude spectrum. Accordingly, there are k sub-bands, and the amplitude spectrum of each first sub-band can be expressed as: , , ,...as well as The number of coding units required for spectral feature extraction is determined by the number of the first sub-bands. Correspondingly, the first spectral feature for each sub-band is obtained, which can be expressed as: , , ,...as well as The first spectral features corresponding to k first sub-bands are concatenated to obtain the sub-band features of the first audio. .

[0072] Step S404: The first dual-path modeling module is used to model the sub-band features of the first audio to obtain the first modeling result.

[0073] The first dual-path modeling module is configured to have a first sub-band modeling module and a first cross-sub-band modeling module, and the output of the first sub-band modeling module is used as the input of the first cross-sub-band modeling module.

[0074] It should be noted that the model structure of the first dual-path sub-band modeling module here is the same as that of the second dual-path sub-band modeling module described below; the difference lies in the model parameters. For ease of description, both will be collectively referred to as dual-path sub-band modeling modules. Accordingly, the first sub-band modeling module and the second sub-band modeling module will be collectively referred to as sub-band modeling modules, and the first cross-sub-band modeling module and the second cross-sub-band modeling module will be collectively referred to as cross-sub-band modeling modules.

[0075] For example, such as Figure 6 As shown, the dual-path modeling module includes a sub-band modeling module, a transpose module, a cross-sub-band modeling module, and a transpose module. Specifically, for the sub-band modeling module, the given size is... The input features, after processing by the sub-band modeling module and transformation by transposition, become... The features, after being processed and transposed by the cross-subband modeling module, yield the following: The modeling results.

[0076] like Figure 7 As shown, the subband modeling module includes L stacked settings of depthwise one-dimensional convolution (DConvId), activation function (GELU), separable one-dimensional convolution (PConvId), activation function (GELU), global response normalization (GRN), and separable one-dimensional convolution (PConvId).

[0077] like Figure 8 As shown, the cross-subband modeling module includes a bidirectional long short-term memory recurrent neural network (BiLSTM), a fully connected layer (FC), and a normalized layer (LN).

[0078] In some alternative implementations, multiple first dual-path modeling modules are configured as a first modeling module group, and the multiple first modeling module groups are stacked together, and the multiple first dual-path modeling modules within the first modeling module group are stacked together.

[0079] When there are multiple first dual-path modeling modules, these modules can be grouped to obtain multiple first modeling module groups. As described above, the multiple first dual-path modeling modules are stacked, and grouping does not affect the model structure. Therefore, the resulting multiple first modeling module groups are also stacked, and the multiple first dual-path modeling modules within each first modeling module group are also stacked.

[0080] To facilitate differentiation, the first dual-path modeling modules can be numbered according to their connection order, i.e., the nth first dual-path modeling module. If there are N first dual-path modeling modules in total, divided into R groups, then each group contains N / R first dual-path modeling modules. Similarly, to distinguish different groups of first modeling modules, they can also be numbered according to their connection order, i.e., the mth group of first modeling modules. The larger the value of m, the deeper the model hierarchy of the first modeling module group.

[0081] For example, step S404 above includes: Step S4041: Determine the target sequence number identifier of the first modeling module group used for modeling result output.

[0082] When performing phase processing using the first modeling module group, the target sequence identifier for the first modeling module group used for modeling result output can be selected as needed. For example, if there are 4 groups, the selectable sequence identifiers are {1, 2, 3, 4}. That is, given a first audio value, the user can select the sequence identifier from {1, 2, 3, 4} to determine the target sequence identifier.

[0083] Step S4042: Input the sub-band features of the first audio into the first dual-path modeling module to obtain the output of the first modeling module group corresponding to the target sequence number identifier, and obtain the first modeling result.

[0084] The sub-band features of the first audio are input into the first dual-path modeling module. The module processes the data and obtains the output from the first modeling module group corresponding to the target sequence number, which is then used as the first modeling result.

[0085] For example, if there are four groups of first modeling modules, and the output of the third group is selected as the first modeling result, then during the processing of the first modeling modules, only the first modeling modules of the first to third groups can work, while the first modeling module of the fourth group does not participate in this processing.

[0086] Step S405: Perform phase processing based on the first modeling result to obtain the target audio corresponding to the first audio.

[0087] Phase processing of the first modeling result can be implemented based on the phase estimation module. The processing method of the phase estimation module is similar to that of the frequency band decomposition module described above. That is, the phase estimation module and the frequency band decomposition module adopt the same sub-band division mechanism and restore the sub-band features to the phase spectrum of the corresponding sub-band, and finally splice them together to obtain the complete phase spectrum.

[0088] Specifically, step S405 includes: Step S4051: Divide the first modeling result into subbands based on the subband division method of the first subband to obtain the second subband.

[0089] The first modeling result is a holistic feature representation. This first result needs to be further subdivided according to the sub-banding method used in the first sub-banding to obtain the second sub-band. The number of second sub-bands is the same as the number of first sub-bands. The specific sub-banding method is described above in the section on first sub-banding and will not be repeated here.

[0090] Step S4052: Perform feature recovery on the second sub-band to obtain the first phase spectrum of the second sub-band.

[0091] For each second sub-band, decoding is performed using the corresponding decoding unit to recover features and obtain the first phase spectrum of the second sub-band. For example, ... Figure 9 As shown, there are k second sub-bands. The k decoding units in the phase estimation module are used to decode each sub-band to obtain the first phase spectrum of each sub-band. For example, the input to the decoding unit for the first second sub-band is... The output is The second sub-band, the input of the corresponding decoding unit is The output is ; and so on, to obtain the first phase spectrum corresponding to each second sub-band.

[0092] Step S4053: The first phase spectra of all the second sub-bands are spliced ​​together to obtain the second phase spectrum corresponding to the first audio.

[0093] Since the first phase spectrum in step S4052 corresponds to each of the second sub-bands, the first phase spectra of all the second sub-bands are spliced ​​together to obtain the second phase spectrum corresponding to the first audio. For example, as shown... Figure 9 As shown, the second phase spectrum corresponding to the first audio frequency is .

[0094] Step S4054: Phase reconstruction is performed based on the second phase spectrum to obtain the target audio.

[0095] Reconstructing a time-domain waveform from the phase spectrum requires combining the first amplitude spectrum with an inverse Fourier transform. Specifically, by acquiring the first amplitude spectrum and the second phase spectrum, the two are recombined into a complex-form spectrum. Then, an inverse Fourier transform is performed on the complex-form spectrum to obtain the reconstructed discrete-time signal. Fitting this signal yields the continuous-time signal, i.e., the target audio.

[0096] Of course, other methods can also be used to achieve phase reconstruction. No restrictions are placed on these methods here. The specific method can be set according to the actual needs.

[0097] The audio processing method provided in this embodiment can adaptively divide subbands for first audio at different sampling rates, thereby satisfying the processing needs of audio with multiple sampling rates. By determining the target sequence identifier of the first modeling module group, the output of the first modeling module group corresponding to the target sequence identifier is used as the first modeling result, and the model order is selected as needed. Based on the subband division and the recovery of the phase spectrum of the corresponding subband from the subband features, the complete second phase spectrum is finally spliced ​​to ensure the transmission of important information.

[0098] The first dual-path modeling module in this embodiment is obtained by training the second dual-path modeling module with multiple types and resolutions of second audio. That is, the audio sample set includes second audio of multiple audio types and multiple sampling rates; during training, the second audio can be randomly selected for model training. The first dual-path modeling module trained in this way can handle audio tasks with multiple sampling rates and multiple audio types. Specifically, the training process includes: Step a1: Obtain the second audio from the audio sample set and extract the second amplitude spectrum of the second audio. The audio sample set contains second audio of multiple audio types and second audio of multiple sampling rates.

[0099] Step a2: Divide the second amplitude spectrum into sub-bands based on the sampling frequency of the second audio to obtain the third sub-band.

[0100] Step a3: Extract the second spectral features corresponding to the third sub-band, and concatenate all the second spectral features to obtain the sub-band features of the second audio.

[0101] Step a4: The sub-band features of the second audio are modeled using the second dual-path modeling module to obtain the second modeling result. The second dual-path modeling module has the same structure as the first dual-path modeling module.

[0102] Step a5: Perform phase processing based on the second modeling result to obtain the predicted audio corresponding to the second audio.

[0103] Step a6: Calculate the loss based on the predicted audio and the target audio of the second audio, and update the parameters of the second dual-path modeling module based on the loss calculation results to obtain the first dual-path modeling module.

[0104] It should be noted that the parameter update of the second dual-path modeling module described in this embodiment refers to updating the parameters of all second dual-path modeling modules used for audio processing tasks, not just one. For example, if the model has N stacked second dual-path modeling modules, then the parameters of all N second dual-path modeling modules are updated during the parameter update.

[0105] Steps a1 to a5 above are similar to the reasoning process described above and will not be repeated here. The training process mainly involves loss calculation, and the parameters of the second dual-path modeling module are updated based on the loss calculation results to obtain the first dual-path modeling module. The iteration stopping condition for parameter updates can be that the number of iterations reaches the maximum number of iterations, the loss calculation result is less than the loss threshold, etc., and can be set according to actual needs.

[0106] In this embodiment, loss is calculated by comparing the predicted audio with the target audio of the second audio, and the parameters of the second dual-path modeling module are updated accordingly. The loss calculation can be based on the difference between the predicted audio and the target audio. Alternatively, other losses can be combined for calculation, such as phase estimation loss, phase reconstruction loss, etc.

[0107] During training, since the importance of information varies across different frequency ranges, the second amplitude spectrum is sub-banded. This allows for spectral feature extraction based on frequency ranges, preserving crucial information in the second audio and providing a foundation for subsequent phase processing. Furthermore, the second dual-path modeling module allows for both focused modeling of independent sub-band features and modeling through inter-sub-band information interaction, improving the accuracy of sub-band feature modeling. Updating module parameters based on this improves the reliability of the resulting first dual-path modeling module. Ultimately, this enables high-quality phase spectrum generation adapted to multiple sampling rates through a single training iteration.

[0108] In some optional implementations, multiple second dual-path modeling modules are configured as a second modeling module group, which has the same structure as the first modeling module group. The structure of the first modeling module group is described above and will not be repeated here. Based on this, step a4 above includes: Step a41: Determine the sampling probability of the second modeling module group based on the number of the second modeling module group and the serial number identifier of the second modeling module group.

[0109] Step a42: Determine the second modeling module group for the modeling result output based on the sampling probability.

[0110] Step a43: Obtain the output of the second modeling module group used for modeling result output, and obtain the second modeling result.

[0111] During the training of the second dual-path modeling module, the group of second modeling modules used to output the modeling results in the current iteration can be determined based on sampling probabilities. Specifically, the sampling probability of each second modeling module group is determined according to the number of second modeling module groups and their sequence numbers.

[0112] For example, the sampling probability can be represented by one of the following three sampling probability density functions: Equation (1) Equation (2) Equation (3) In the formula, These represent three different sampling probability density functions, r represents the serial number of the second modeling module group, and R represents the number of the second modeling module groups.

[0113] For example, such as Figure 10 As shown, there are N dual-path modeling modules, which are divided into R second-level modeling module groups, i.e., group 1 to group R. Therefore, each second-level modeling module group contains N / R dual-path modeling modules. During training, the modeling results from group 1 to group R are selected and fed to the phase estimation module through selective output. The phase estimation module uses the results of the selective output to perform phase estimation to obtain the estimated phase, and then performs phase reconstruction on the estimated phase to obtain the phase reconstruction result. Finally, the phase reconstruction result and a single amplitude spectrum are processed by inverse short-time fourier transform (iSTFT) to output the target audio.

[0114] The sampling probability of the second modeling module group is determined based on the number of the second modeling module group and the serial number of the second modeling module group. That is, during the training process, dynamic training is achieved through sampling probability, which improves the training effect of the model.

[0115] In some optional implementations, the loss calculation result includes two parts of loss: phase reconstruction loss and adversarial generation loss. The fusion result of these two parts of loss is used as the final loss calculation result. Specifically, step a6 above includes: Step a61: Obtain the second modeling result and perform phase processing to obtain the predicted phase.

[0116] Step a62: Calculate the phase reconstruction loss based on the predicted phase and the target phase corresponding to the second audio, and obtain the first loss.

[0117] Step a63: Calculate the adversarial generation loss based on the predicted audio and the target audio to obtain the second loss.

[0118] Step a64: Based on the fusion result of the first loss and the second loss, the loss calculation result is obtained.

[0119] Phase processing is performed based on the second modeling result to obtain the predicted phase. The first loss is obtained by calculating the phase reconstruction loss between the predicted phase and the target phase. The second loss is obtained by processing the adversarial generation loss using a multi-period discriminator. The first and second losses are then fused using a loss weighting method to obtain the final loss calculation result. Of course, other methods can also be used to fuse the two losses, and no specific limitations are imposed on them.

[0120] Loss calculations are performed from the perspectives of phase reconstruction and adversarial generation, and the fusion result of the two losses is used as the final loss calculation result. Loss calculations are performed from multiple perspectives to ensure the processing reliability of the first dual-path modeling module obtained after training.

[0121] In some alternative implementations, step a62 above includes: Step a621: Perform convolution operations on the predicted phase and the target phase based on the preset convolution kernel to obtain the first differential phase and the second differential phase.

[0122] Step a622: Perform unwinding processing based on the first differential phase and the second differential phase to obtain the first processing result.

[0123] Step a623: The first processing result is weighted and calculated based on the second amplitude spectrum to obtain the first loss.

[0124] In this embodiment, an amplitude-weighted omnidirectional phase loss function is used to calculate the first loss. For example, as shown... Figure 11 As shown, by designing a set of fixed convolution kernels This allows for the rapid calculation of adjacent phase differences and transient phases in various directions. These are the time and frequency indices t and f, respectively. Subsequently, the loss is calculated using an unwinding function and by introducing amplitude weighting. The loss function is defined as follows: Equation (4) Equation (5) in, and These represent the target phase and the predicted phase, respectively. and Represent the target phase and prediction phase with the convolution kernel, respectively. The difference phases after the convolution operation, i.e., the first difference phase and the second difference phase. "*" represents a two-dimensional convolution operation, unwinding the phases. Defined as In the formula, round(.) represents the rounding operation. This is the first processing result. This is the first loss.

[0125] By performing convolution operations on the predicted phase and the target phase using preset convolution kernels, the adjacent phase differences and transient phases in each direction can be quickly calculated. Based on this, combined with amplitude weighting, the network can focus on the phase recovery of the higher energy part.

[0126] In some alternative implementations, step a63 above includes: Step a631: Obtain the period value of the multi-period discriminator corresponding to the sampling rate of the second audio.

[0127] Step a632: Perform the first adversarial loss for the discriminator part based on the period value of the multi-period discriminator, the predicted audio, and the target audio.

[0128] Step a633: Perform a second adversarial loss for the generator part based on the period values ​​of the multi-period discriminator and the predicted audio.

[0129] Step a634: Calculate the feature matching loss based on the period value of the multi-period discriminator, the predicted audio, and the target audio.

[0130] Step a635: Based on the fusion result of the first adversarial loss, the second adversarial loss, and the feature matching loss, the second loss is obtained.

[0131] In this embodiment, data at different sampling rates uses different groups of multi-period discriminators, and the period values ​​of each group of discriminators are configured differently. For example, each group of discriminators contains three discriminators with different scales but identical structures. For instance, the period values ​​at sampling rates of 8 kHz, 16 kHz, 24 kHz, 32 kHz, and 48 kHz are set to {1, 2, 3}, {2, 3, 4}, {3, 4, 6}, {4, 6, 8}, and {6, 8, 12}, respectively.

[0132] Based on this, the first adversarial loss of the discriminator section is calculated using the following formula: Equation (6) In the formula, The first adversarial loss is given by H, where H is the number of discriminators. , These represent the target audio and the predicted audio, respectively. , They represent , The discriminant output after passing through the sub-discriminator labeled h.

[0133] The second adversarial loss for the generator part is calculated using the following formula: Equation (7) In the formula, This represents the second wave of losses.

[0134] In addition, feature matching loss is incorporated and calculated using the following formula: Equation (8) In the formula, Let E represent the output feature of the h-th sub-discriminator at the e-th layer, where E is the layer number of each sub-discriminator.

[0135] The first adversarial loss in the discriminator part is used to train the discriminator to distinguish between real and generated audio. The second adversarial loss in the generator part is used to train the generator to generate audio with quality close to that of real audio. The feature matching loss is used to stabilize training and improve the generation quality. By fusing these three factors as the loss of the adversarial generation part, the reliability of the obtained first dual-path modeling module can be further guaranteed.

[0136] The first dual-path modeling module trained in the above manner can achieve high-quality phase spectrum generation for audio with multiple sampling rates through a single training iteration. Furthermore, the trained first dual-path modeling module can support multiple levels of complexity or audio quality selection. The multiple levels are achieved by selecting the sequence number of the first modeling module used to output the modeling results; different sequence numbers correspond to different model levels.

[0137] As a specific application embodiment of this application, in the audio processing task of speech enhancement, the user, through, as follows: Figure 2 The interface 202 shown provides an interaction with a first audio file that needs speech enhancement. After acquiring the first audio file, the terminal device 210 sends it to the server 220 for processing. The server 220 is equipped with the audio processing method described in this embodiment. After processing the first audio file, the server obtains the speech-enhanced target audio file and displays it on the interface 202. Furthermore, the user can trigger the playback of the target audio file through interaction with the interface 202.

[0138] This embodiment also provides an audio processing apparatus for implementing the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0139] This embodiment provides an audio processing device, such as... Figure 12 As shown, it includes: The first audio acquisition module 1201 is used to acquire the first audio and extract the first amplitude spectrum of the first audio.

[0140] The first sub-band division module 1202 is used to divide the first amplitude spectrum into sub-bands based on the sampling frequency of the first audio signal to obtain the first sub-band.

[0141] The first feature extraction module 1203 is used to extract the first spectral features corresponding to the first sub-band and splice all the first spectral features to obtain the sub-band features of the first audio.

[0142] The first modeling module 1204 is used to model the sub-band features of the first audio using the first dual-path modeling module to obtain the first modeling result. The first dual-path modeling module is configured to have a first sub-band modeling module and a first cross-sub-band modeling module. The output of the first sub-band modeling module is used as the input of the first cross-sub-band modeling module.

[0143] The first phase processing module 1205 is used to perform phase processing based on the first modeling result to obtain the target audio corresponding to the first audio.

[0144] In some alternative implementations, a plurality of first dual-path modeling modules are configured as a first modeling module group, the plurality of first modeling module groups are stacked among each other, and the plurality of first dual-path modeling modules within a first modeling module group are stacked among each other.

[0145] In some alternative implementations, the first sub-band modeling module is configured to model the sub-band features of the first audio along the time dimension and input the modeling results to the first cross-sub-band modeling module, which is configured to model the obtained modeling results along the sub-band dimension.

[0146] In some alternative implementations, the first modeling module 1204 includes: The identifier determination unit is used to determine the target sequence identifier of the first modeling module group used for modeling result output.

[0147] The output acquisition unit is used to input the sub-band features of the first audio into the first dual-path modeling module, obtain the output of the first modeling module group corresponding to the target sequence number identifier, and obtain the first modeling result.

[0148] In some optional implementations, the first sub-band partitioning module 1202 includes: The upper limit determination unit is used to determine the upper limit of the spectral bandwidth based on the sampling rate of the first audio.

[0149] The first sub-band division method determination unit is used to determine the first sub-band division method corresponding to the first audio based on the upper limit of the spectrum bandwidth and the preset sub-band division method. The first sub-band division method includes the number of first sub-bands and the frequency range corresponding to the first sub-band.

[0150] The first sub-band division unit is used to divide the first amplitude spectrum into sub-bands based on the first sub-band division method to obtain the first sub-band.

[0151] In some alternative implementations, the first phase processing module 1205 includes: The second sub-band partitioning unit is used to partition the first modeling result into sub-bands based on the sub-band partitioning method of the first sub-band, so as to obtain the second sub-band.

[0152] The feature recovery unit is used to perform feature recovery on the second sub-band to obtain the first phase spectrum of the second sub-band.

[0153] The splicing unit is used to splice the first phase spectrum of all the second sub-bands to obtain the second phase spectrum corresponding to the first audio.

[0154] The phase reconstruction unit is used to perform phase reconstruction based on the second phase spectrum to obtain the target audio.

[0155] In some alternative implementations, it also includes: The second audio acquisition module is used to acquire the second audio and extract the second amplitude spectrum of the second audio.

[0156] The third sub-band division module is used to divide the second amplitude spectrum into sub-bands based on the sampling frequency of the second audio, so as to obtain the third sub-band.

[0157] The second feature extraction module is used to extract the second spectral features corresponding to the third sub-band, and to concatenate all the second spectral features to obtain the sub-band features of the second audio.

[0158] The second modeling module is used to model the sub-band features of the second audio using the second dual-path modeling module to obtain the second modeling result. The second dual-path modeling module has the same structure as the first dual-path modeling module.

[0159] The second phase processing module is used to perform phase processing based on the second modeling result to obtain the predicted audio corresponding to the second audio.

[0160] The parameter update module is used to calculate the loss based on the predicted audio and the target audio of the second audio, and update the parameters of the second dual-path modeling module based on the loss calculation results to obtain the first dual-path modeling module.

[0161] In some optional implementations, a plurality of the second dual-path modeling modules are configured as a second modeling module group, the second modeling module comprising: The sampling probability determination unit is used to determine the sampling probability of the second modeling module group based on the number of the second modeling module group and the serial number identifier of the second modeling module group.

[0162] The second modeling module group determination unit is used to determine the second modeling module group for modeling result output based on the sampling probability.

[0163] The second modeling result acquisition unit is used to acquire the output of the second modeling module group used for modeling result output, and obtain the second modeling result.

[0164] In some optional implementations, the parameter update module includes: The predictive phase processing unit is used to obtain the second modeling result, perform phase processing, and obtain the predicted phase.

[0165] The first loss unit is used to calculate the phase reconstruction loss based on the predicted phase and the target phase corresponding to the second audio, and obtain the first loss.

[0166] The second loss unit is used to calculate the adversarial generation loss based on the predicted audio and the target audio, thus obtaining the second loss.

[0167] The loss fusion unit is used to obtain the loss calculation result based on the fusion result of the first loss and the second loss.

[0168] In some alternative implementations, the first loss unit includes: The convolutional subunit is used to perform convolution operations on the predicted phase and the target phase based on a preset convolutional kernel to obtain the first differential phase and the second differential phase.

[0169] The processing subunit is used to perform unwinding processing based on the first differential phase and the second differential phase to obtain the first processing result.

[0170] The first loss subunit is used to perform weighted calculations on the first processing result based on the second amplitude spectrum to obtain the first loss.

[0171] In some alternative implementations, the second loss unit includes: The period value acquisition subunit is used to acquire the period value of the multi-period discriminator corresponding to the sampling rate of the second audio.

[0172] The first adversarial loss subunit is used to perform the first adversarial loss of the discriminator part based on the period value of the multi-period discriminator, the predicted audio, and the target audio.

[0173] The second adversarial loss subunit is used to perform a second adversarial loss on the generator part based on the period value of the multi-period discriminator and the predicted audio.

[0174] The feature matching loss subunit is used to calculate the feature matching loss based on the period value of the multi-period discriminator, the predicted audio, and the target audio.

[0175] The feature fusion subunit is used to obtain the second loss based on the fusion result of the first adversarial loss, the second adversarial loss, and the feature matching loss.

[0176] The audio processing apparatus provided in this disclosure can execute the audio processing method provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects for executing the method. Further functional descriptions of the various modules and units described above are the same as in the corresponding embodiments described above, and will not be repeated here.

[0177] Figure 13 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure.

[0178] The following is a detailed reference. Figure 13The diagram illustrates a structural schematic suitable for implementing an electronic device according to embodiments of the present disclosure. The electronic device may include a processor (e.g., a central processing unit, graphics processor, etc.) 1301, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 1302 or a program loaded from memory 1308 into random access memory (RAM) 1303. The RAM 1303 also stores various programs and data required for the operation of the electronic device. The processor 1301, ROM 1302, and RAM 1303 are interconnected via a bus 1304. An input / output (I / O) interface 1305 is also connected to the bus 1304.

[0179] Typically, the following devices can be connected to I / O interface 1305: input devices 1306 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 1307 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; memory devices 1308 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1309. Communication device 1309 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 13 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown, and more or fewer devices may be implemented or have instead.

[0180] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 1309, or installed from memory 1308, or installed from ROM 1302. When the computer program is executed by processor 1301, it performs the functions defined in the audio processing method of embodiments of this disclosure.

[0181] Figure 13 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0182] This application also provides a computer-readable storage medium. The methods described in this application can be implemented in hardware or firmware, or implemented as recordable on a storage medium, or implemented as computer code downloaded via a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and subsequently stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code. When the software or computer code is accessed and executed by the computer, processor, or hardware, the audio processing method shown in the above embodiments is implemented.

[0183] A portion of this application can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to this application through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions, or the computer compiling the instructions and then executing the corresponding compiled program, or the computer reading and executing the instructions, or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.

[0184] Although embodiments of this application have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of this application, and all such modifications and variations fall within the scope defined by the appended claims.

Claims

1. An audio processing method, characterized by, The method comprises: obtaining a first audio and extracting a first amplitude spectrum of the first audio; performing sub-band division on the first amplitude spectrum based on a sampling frequency of the first audio to obtain first sub-bands; extracting first spectral features corresponding to the first sub-bands and splicing all the first spectral features to obtain sub-band features of the first audio; modeling the sub-band features of the first audio by using a first dual-path modeling module to obtain a first modeling result, the first dual-path modeling module being configured to have a first sub-band modeling module and a first cross-sub-band modeling module, an output of the first sub-band modeling module being used as an input of the first cross-sub-band modeling module; performing phase processing based on the first modeling result to obtain a target audio corresponding to the first audio.

2. The method of claim 1, wherein, A plurality of first dual-path modeling modules are configured as a first modeling module group, and the plurality of first modeling module groups are arranged in a stacked manner, and the plurality of first dual-path modeling modules in the first modeling module group are arranged in a stacked manner.

3. The method of claim 2, wherein, The first sub-band modeling module is configured to model the sub-band features of the first audio along a time dimension and input a modeling result to the first cross-sub-band modeling module, and the first cross-sub-band modeling module is configured to model the obtained modeling result along a sub-band dimension.

4. The method of claim 2, wherein, The first dual-path modeling module is configured to model the sub-band features of the first audio to obtain a first modeling result, comprising: determining a target serial number identifier of a first modeling module group for modeling result output; inputting the sub-band features of the first audio into the first dual-path modeling module to obtain an output of a first modeling module group corresponding to the target serial number identifier to obtain the first modeling result.

5. The method of claim 1, wherein, The first dual-path modeling module is configured to model the sub-band features of the first audio to obtain a first modeling result, comprising: determining a target serial number identifier of a first modeling module group for modeling result output; inputting the sub-band features of the first audio into the first dual-path modeling module to obtain an output of a first modeling module group corresponding to the target serial number identifier to obtain the first modeling result. The first dual-path modeling module is configured to model the sub-band features of the first audio to obtain a first modeling result, comprising:

6. The method of claim 1, wherein, determining a target serial number identifier of a first modeling module group for modeling result output; inputting the sub-band features of the first audio into the first dual-path modeling module to obtain an output of a first modeling module group corresponding to the target serial number identifier to obtain the first modeling result. The method further comprises: obtaining a second audio in an audio sample set and extracting a second amplitude spectrum of the second audio, the audio sample set having second audios of multiple audio types and second audios of multiple sampling rates; performing sub-band division on the second amplitude spectrum based on a sampling frequency of the second audio to obtain third sub-bands; 7. The method of claim 1, wherein, ​ ​ ​ extract a second spectral feature corresponding to the third subband, and splice all the second spectral features to obtain a subband feature of the second audio; model the subband feature of the second audio by using a second dual-path modeling module to obtain a second modeling result, the second dual-path modeling module having the same structure as the first dual-path modeling module; perform phase processing based on the second modeling result to obtain a predicted audio corresponding to the second audio; perform loss calculation based on the predicted audio and a target audio of the second audio, and update parameters of the second dual-path modeling module based on a loss calculation result to obtain the first dual-path modeling module.

8. The method of claim 7, wherein, The plurality of second dual-path modeling modules are configured as a second modeling module group, and the modeling of the subband feature of the second audio by using the second dual-path modeling module to obtain the second modeling result includes: determining a sampling probability of the second modeling module group based on the number of the second modeling module group and a serial number identifier of the second modeling module group; determining a second modeling module group for modeling result output based on the sampling probability; obtaining an output of the second modeling module group for modeling result output to obtain the second modeling result.

9. The method of claim 7, wherein, The loss calculation based on the predicted audio and the target audio of the second audio includes: obtaining a predicted phase by performing phase processing on the second modeling result; performing phase reconstruction loss calculation based on the predicted phase and a target phase corresponding to the second audio to obtain a first loss; performing calculation of a generative adversarial loss based on the predicted audio and the target audio to obtain a second loss; obtaining the loss calculation result based on a fusion result of the first loss and the second loss.

10. The method of claim 9, wherein, The phase reconstruction loss calculation based on the predicted phase and the target phase corresponding to the second audio to obtain the first loss includes: performing convolution operations on the predicted phase and the target phase based on a preset convolution kernel to obtain a first differential phase and a second differential phase; performing unwrapping processing based on the first differential phase and the second differential phase to obtain a first processing result; performing weighted calculation on the first processing result based on the second amplitude spectrum to obtain the first loss.

11. The method of claim 9, wherein, The calculation of the generative adversarial loss based on the predicted audio and the target audio to obtain the second loss includes: obtaining a period value of a multi-period discriminator corresponding to a sampling rate of the second audio; performing a first adversarial loss of a discriminator part based on the period value of the multi-period discriminator, the predicted audio, and the target audio; performing a second adversarial loss of a generator part based on the period value of the multi-period discriminator and the predicted audio; calculating a feature matching loss based on the period value of the multi-period discriminator, the predicted audio, and the target audio; obtaining the second loss based on a fusion result of the first adversarial loss, the second adversarial loss, and the feature matching loss.

12. An audio processing apparatus, characterized by comprising: The method includes: a first audio acquisition module configured to acquire a first audio and extract a first amplitude spectrum of the first audio; The first subband division module is configured to divide the first amplitude spectrum into subbands based on a sampling frequency of the first audio to obtain first subbands. The first feature extraction module is configured to extract first spectral features corresponding to the first subbands and splice all the first spectral features to obtain subband features of the first audio. The first modeling module is configured to model the subband features of the first audio by using a first dual-path modeling module to obtain a first modeling result, the first dual-path modeling module being configured to have a first subband modeling module and a first cross-subband modeling module, an output of the first subband modeling module being used as an input of the first cross-subband modeling module. The first phase processing module is configured to perform phase processing based on the first modeling result to obtain a target audio corresponding to the first audio.

13. An electronic device, comprising: The memory and the processor are communicatively connected, and the memory stores computer instructions. The processor executes the computer instructions to perform the audio processing method in any one of claims 1 to 11. The computer readable storage medium stores computer instructions for causing a computer to perform the audio processing method in any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions for causing a computer to perform the audio processing method in any one of claims 1 to 11.

15. A computer program product, characterised in that, ​

Images