Audio signal processing method, electronic device, and storage medium

Abstract

The present disclosure relates to the fields of large model technology and audio processing, and provides an audio signal processing method, an electronic device, and a storage medium. The method comprises: acquiring a raw audio signal; performing domain conversion processing on the raw audio signal to obtain a conversion result, the conversion result being used for representing a time domain signal and a frequency domain signal in a plurality of representation forms obtained from the conversion of the raw audio signal; and performing audio encoding processing on the conversion result to obtain an encoding result, the encoding result being used for restoring the raw audio signal in a preset application scenario. The present disclosure solves the technical problems in the related art of a low encoding utilization rate and poor encoding quality when performing quantization and encoding processing on an audio signal.

Description

Audio signal processing method, electronic device and storage medium Technical Field

[0001] The present disclosure relates to large-scale model technology and the field of audio processing, and more specifically, to an audio signal processing method, an electronic device, and a storage medium. Background Art

[0002] As one of the most commonly used data modalities, audio is widely present in daily life and plays an indispensable role. With the rise of large language models (LLM), the field of artificial intelligence has set off a wave of multimodal unified modeling. As an important data modality, audio has important research significance. In the related art, a variety of different audio quantization coding models have been developed to realize the quantization coding processing of audio signals, that is, to convert continuous audio signals into discrete digital signals, thereby reducing the amount of data required for storage or transmission while maintaining the quality of the original audio signal as much as possible. However, the audio quantization coding model in the related art mainly uses the time domain structure of the audio signal for encoding, which leads to low coding utilization and poor quantization coding quality.

[0003] To address the above-mentioned problems, no effective solutions have been proposed so far.

[0004] Summary of the Invention

[0005] The embodiments of the present disclosure provide an audio signal processing method, an electronic device, and a storage medium to at least solve the technical problems of low coding utilization and poor coding quality when performing quantization coding processing on audio signals in related technologies.

[0006] According to one aspect of an embodiment of the present disclosure, there is provided an audio signal processing method, comprising: obtaining an original audio signal; performing domain conversion processing on the original audio signal to obtain a conversion result, wherein the conversion result is used to represent time domain signals and frequency domain signals in multiple representation forms obtained by converting the original audio signal; performing audio encoding processing on the conversion result to obtain an encoding result, wherein the encoding result is used to restore the original audio signal in a preset application scenario.

[0007] According to another aspect of the embodiments of the present disclosure, an audio signal processing method is also provided, including: obtaining an encoding result, wherein the encoding result is used to restore the original audio signal under a preset application scenario, the encoding result is obtained by performing audio encoding processing on the conversion result, the conversion result is obtained by performing domain conversion processing on the original audio signal, and the conversion result is used to represent time domain signals and frequency domain signals in various representation forms obtained by converting the original audio signal; performing speech reconstruction processing on the encoding result to obtain a reconstructed audio signal.

[0008] According to another aspect of the embodiments of the present disclosure, an audio signal processing method is also provided, including: obtaining an original audio signal; performing domain conversion processing on the original audio signal using a target audio signal processing model to obtain a conversion result, performing audio encoding processing on the conversion result to obtain an encoding result, and performing speech reconstruction processing on the encoding result to obtain a reconstructed audio signal, wherein the conversion result is used to represent time domain signals and frequency domain signals in multiple representation forms obtained by converting the original audio signal, and the encoding result is used to restore the original audio signal under a preset application scenario; and playing the reconstructed audio signal.

[0009] According to another aspect of the embodiments of the present disclosure, an electronic device is provided, including: a memory storing an executable program; and a processor for running the program, wherein when the program is run, the audio signal processing method of any one of the embodiments of the present disclosure is executed.

[0010] According to another aspect of the embodiments of the present disclosure, a computer-readable storage medium is further provided, which includes a stored executable program, wherein when the executable program is running, the device where the computer-readable storage medium is located is controlled to execute any one of the audio signal processing methods in the embodiments of the present disclosure.

[0011] In the embodiment of the present disclosure, by obtaining the original audio signal and performing domain conversion processing on the original audio signal to obtain a conversion result, and finally performing audio encoding processing on the conversion result to obtain an encoding result, the purpose of reliably restoring the original audio signal in a preset application scenario is achieved, thereby achieving the technical effect of improving the encoding utilization rate and encoding quality of the audio signal, and further solving the technical problems of low encoding utilization rate and poor encoding quality when performing quantization encoding processing on audio signals in related technologies.

[0012] It is easy to note that the above general description and the following detailed description are only for the purpose of exemplifying and explaining the present disclosure, and do not constitute a limitation of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The drawings described herein are used to provide a further understanding of the present disclosure and constitute a part of the present disclosure. The exemplary embodiments of the present disclosure and their descriptions are used to explain the present disclosure and do not constitute an improper limitation of the present disclosure. In the drawings:

[0014] FIG1 is a schematic diagram of an application scenario of an audio signal processing method according to Embodiment 1 of the present disclosure;

[0015] FIG2 is a flowchart of an audio signal processing method according to Embodiment 1 of the present disclosure;

[0016] FIG3 is a schematic diagram of an audio signal processing method according to Embodiment 1 of the present disclosure;

[0017] FIG4 is a flowchart of an audio signal processing method according to Embodiment 2 of the present disclosure;

[0018] FIG5 is a flowchart of an audio signal processing method according to Embodiment 3 of the present disclosure;

[0019] FIG6 is a result block diagram of an audio signal processing apparatus according to Embodiment 1 of the present disclosure;

[0020] FIG7 is a result block diagram of an audio signal processing apparatus according to Embodiment 2 of the present disclosure;

[0021] FIG8 is a result block diagram of an audio signal processing apparatus according to Embodiment 3 of the present disclosure;

[0022] FIG9 is a structural block diagram of a computer terminal according to Embodiment 5 of the present disclosure. DETAILED DESCRIPTION

[0023] In order to enable those skilled in the art to better understand the solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the embodiments described are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present disclosure.

[0024] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0025] The technical solution provided by the present disclosure is mainly implemented using large-scale model technology. The large model here refers to a deep learning model with large-scale model parameters, which can usually contain hundreds of millions, tens of billions, hundreds of billions, trillions, or even more than ten trillion model parameters. The large model can also be called a cornerstone model / foundation model (Foundation Model). The large model is pre-trained by large-scale unlabeled corpus to produce a pre-trained model with more than 100 million parameters. This model can adapt to a wide range of downstream tasks and has good generalization capabilities, such as large-scale language models (LLMs) and multi-modal pre-training models.

[0026] It should be noted that when the large model is actually applied, the pre-trained model can be fine-tuned through a small number of samples, so that the large model can be applied to different tasks. For example, the large model can be widely used in natural language processing (NLP), computer vision, speech processing and other fields. Specifically, it can be applied to computer vision tasks such as visual question answering (VQA), image description (IC), image generation, etc. It can also be widely used in natural language processing tasks such as text-based sentiment classification, text summary generation, and machine translation. Therefore, the main application scenarios of the large model include but are not limited to digital assistants, intelligent robots, search, online education, office software, e-commerce, intelligent design, etc. In the embodiment of the present disclosure, data processing through an audio signal processing model in voice communication and voice storage scenarios is used as an example for explanation.

[0027] First, some nouns or terms that appear in the description of the embodiments of the present disclosure are subject to the following explanations:

[0028] Audio Codec: The process of converting a continuous audio signal sequence into a quantized code sequence.

[0029] Quantization code, token: a bounded non-negative integer representing the subscript of the quantized audio representation in the dictionary.

[0030] Token rate (TKR): The number of quantized codes required to quantize and encode an audio signal of 1 second in length at a preset audio sampling rate.

[0031] Bitrate: The number of bits required to quantize and encode a 1-second audio signal at a preset audio sampling rate.

[0032] The following four audio quantization coding models have been developed in the related art to implement quantization coding processing for audio signals. Common open source audio quantization coding models include encoders (Encodec), audio streams (SoundStream), and common open source audio quantization coding projects include audio decoders (AudioDec) and audio codecs (Descript-audio-codec). However, the audio quantization coding models in the related art generally directly model the original audio time domain signal, lack effective utilization of the frequency domain structure of the audio signal, and do not utilize the semantic information contained in the speech signal, but only encode it from an acoustic perspective. The coding dictionary in the related art adopts a random initialization method. In the subsequent update process, even if some codewords are rarely activated, they will not be reassigned, resulting in low codebook utilization. In addition, the training process of AudioDec is relatively cumbersome and is specifically divided into two stages. The first stage trains the autoencoder, and the second stage uses the first stage as the initialization and adds a vector quantizer for discrete encoding, and then further trains the encoder and decoder. At the same bit rate, Encodec, SoundStream, and Descript-audio-codec use too fine a time granularity, resulting in insufficient coding utilization and poor encoded speech quality.

[0033] In summary, the related art has the technical problems of low coding utilization and poor coding quality when performing quantization coding processing on audio signals. Currently, no effective solution has been proposed to the above problems.

[0034] Example 1

[0035] According to an embodiment of the present disclosure, a method for processing an audio signal is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0036] Taking into account the huge amount of model parameters of the large model and the limited computing resources of the mobile terminal, the above-mentioned audio signal processing method provided in the embodiment of the present disclosure can be applied to the application scenario shown in Figure 1, but is not limited thereto. In the application scenario shown in Figure 1, the large model is deployed in the server 10, and the server 10 can be connected to one or more client devices 20 through a local area network connection, a wide area network connection, an Internet connection, or other types of data networks. The client devices 20 here can include but are not limited to: smart phones, tablet computers, laptops, PDAs, personal computers, smart home devices, car-mounted devices, etc. The client device 20 can interact with the user through a graphical user interface to implement the call to the large model, thereby implementing the method provided in the embodiment of the present disclosure.

[0037] In an embodiment of the present disclosure, a system composed of a client device and a server can perform the following steps: the client device obtains the original audio signal and sends it to the server, the server performs domain conversion processing on the original audio signal to obtain a conversion result, wherein the conversion result is used to represent the time domain signal and frequency domain signal in various representation forms obtained by converting the original audio signal, and further, the server performs audio encoding processing on the conversion result to obtain an encoding result, wherein the encoding result is used to restore the original audio signal in a preset application scenario. It should be noted that if the operating resources of the client device can meet the deployment and operation conditions of the large model, the embodiment of the present disclosure can be carried out in the client device.

[0038] In the above operating environment, the present disclosure provides an audio signal processing method as shown in Figure 2. Figure 2 is a flow chart of the audio signal processing method according to embodiment 1 of the present disclosure. As shown in Figure 2, the method may include the following steps:

[0039] Step S21, obtaining an original audio signal;

[0040] Step S22, performing domain conversion processing on the original audio signal to obtain a conversion result, wherein the conversion result is used to represent a time domain signal and a frequency domain signal in a plurality of representation forms obtained by converting the original audio signal;

[0041] Step S23 , performing audio encoding processing on the conversion result to obtain an encoding result, wherein the encoding result is used to restore the original audio signal in a preset application scenario.

[0042] The aforementioned raw audio signals include speech signals and other audio signals. Specifically, speech signals are signals generated by the human voice and contain semantic and acoustic information. Semantic information refers to the meaning, implications, and information conveyed through language, including vocabulary, grammar, semantics, logical relationships, and so on. Semantic information can be understood through the structure and context of language, thereby conveying the specific meaning and information the speaker intends to convey. Acoustic information refers to the physical characteristics and waveform of sound, including frequency, amplitude, duration, pitch, volume, and speaking rate. Acoustic information is understood by analyzing and processing the physical characteristics of sound and can convey the sound quality and characteristics of the sound. Other audio signals refer to all audio signals other than speech signals, including but not limited to music and acoustic events such as knocking sounds, applause, and the sound of a door opening. These audio signals typically lack clear linguistic features and only contain acoustic information. They can be used for tasks such as audio processing, audio analysis, and audio enhancement.

[0043] There are many ways to obtain the above-mentioned original audio signal. For example, use a professional recording device or a built-in microphone of a device such as a smartphone to record, place the microphone near the sound source to be recorded, and then start recording using recording software, so that the original audio signal of the sound source can be directly captured. For another example, connect the output of an audio device (such as a musical instrument, computer output, etc.) to the audio input interface of a computer, and then use audio recording software to record. This method can capture the original audio signal output by the device. For another example, directly use audio editing software to open a preset audio file and export the preset audio file as an original audio signal.

[0044] Performing domain conversion on the original audio signal can transform it into the desired representation. The conversion results include, but are not limited to, the time domain, short-time frequency domain, amplitude-phase spectrum, and amplitude-angle spectrum corresponding to the original audio data. Specifically, the time domain refers to the temporal variations of the audio signal, which can be represented by a waveform. The time domain can represent the amplitude values ​​of the original audio signal at different time points. The short-time frequency domain divides the original audio signal into several time windows and performs frequency domain analysis within each window to obtain the spectral information of the original audio signal within that window. This representation can reveal the energy distribution of the original audio signal at different frequencies. A common method for frequency domain analysis is to use fast Fourier transform (FFT). The amplitude-phase spectrum represents the spectral information of the original audio signal in the form of amplitude and phase. The amplitude spectrum represents the energy level of the original audio signal at different frequencies, and the phase spectrum represents the phase information of the original audio signal at different frequencies. The amplitude-angle spectrum represents the spectral information of the original audio signal in the form of amplitude and angle. The amplitude spectrum represents the energy level of the original audio signal at different frequencies, and the angle spectrum represents the phase information of the original audio signal at different frequencies. Compared with the amplitude-phase spectrum, the amplitude-angle spectrum pays more attention to the representation of phase information.

[0045] Exemplary methods for implementing domain conversion processing include but are not limited to Fourier transform, fast Fourier transform, discrete cosine transform, etc. These domain conversion methods can convert the original audio signal from the time domain to the frequency domain to obtain the spectrum information of the original audio signal, or convert the original audio signal from the frequency domain to the time domain to obtain the time domain waveform. In the frequency domain, the audio signal can be subjected to spectrum analysis, filtering, noise reduction, compression, and other processing, while in the time domain, the audio signal can be subjected to time domain filtering, time domain modification, and other processing. Domain conversion processing has important applications in audio signal processing, and can extract the characteristics of the original audio signal, improve the sound quality, and realize the compression of the original audio signal and other functions.

[0046] Furthermore, the conversion result is subjected to audio encoding to obtain an encoded result, which is used to restore the original audio signal in a preset application scenario. Specifically, audio quantization encoding is the process of converting a continuous audio signal into a discrete digital signal. During the quantization encoding process, the amplitude of the original audio signal can be discretized and mapped into a series of discrete numerical values ​​to represent the amplitude of the original audio signal.

[0047] The encoding result consists of multiple sets of discrete codes of equal length, representing the audio signal. These codes are typically represented in binary form and stored or transmitted to the receiver for decoding to restore the original audio signal. Audio encoding can minimize the amount of data required for storage or transmission while preserving the quality of the original audio signal.

[0048] The above-mentioned preset application scenarios include but are not limited to voice communication, voice storage, language large models and other scenarios. Specifically, in the voice communication scenario, by performing audio coding processing on the original recorded voice signal, the voice signal can be compressed and encoded into a coding result and then transmitted. The receiving end can restore the coding result to the original recorded voice signal, thereby effectively reducing the bandwidth required for transmission and improving transmission efficiency. In the voice storage scenario, the recorded original voice signal, such as the customer service communication record, is audio coded and stored quantized, and the coding result is restored to the original voice signal when retrieval and playback are required. In the language large model scenario, the voice signal is audio coded to obtain discrete semantic tokens, which can be modeled uniformly with the text, thereby adding voice understanding and generation capabilities to the large model.

[0049] Based on the above steps S21 to S23, by obtaining the original audio signal and performing domain conversion processing on the original audio signal to obtain a conversion result, and finally performing audio encoding processing on the conversion result to obtain an encoding result, the purpose of reliably restoring the original audio signal in a preset application scenario is achieved, thereby achieving the technical effect of improving the encoding utilization rate and encoding quality of the audio signal, and further solving the technical problems of low encoding utilization rate and poor encoding quality when performing quantization encoding processing on audio signals in related technologies.

[0050] The above-mentioned audio signal processing method provided by the embodiments of the present disclosure can be applied to, but is not limited to, application scenarios involving voice communication, voice storage, and large voice models in the fields of e-commerce services, educational services, legal services, medical services, conference services, social network services, financial product services, logistics services, and navigation services, and the embodiments of the present disclosure are not limited thereto.

[0051] The audio signal processing method in the embodiment of the present disclosure is further introduced below.

[0052] In an optional embodiment, the audio signal processing method is applied to an audio signal coding model, and the audio signal coding model includes: a domain converter. In step S22, domain conversion processing is performed on the original audio signal, and the conversion result obtained includes:

[0053] The domain converter is used to perform multiple dimensional domain conversion processing on the original audio signal to obtain a conversion result, wherein the conversion result includes some or all of the following conversion results:

[0054] A first conversion result obtained by performing a short-time Fourier transform on the original audio signal;

[0055] a second conversion result obtained by performing amplitude-phase conversion on the first conversion result;

[0056] A third conversion result is obtained by performing amplitude-angle conversion on the first conversion result.

[0057] The domain converter in the above audio coding model can be used as a component of the LLM to implement multi-dimensional domain conversion processing on the original audio signal to obtain a conversion result.

[0058] Specifically, the original audio signal is subjected to a short-time Fourier transform to obtain a first conversion result. The short-time Fourier transform (STFT) is a method for performing time-frequency analysis on a signal. It can decompose the original audio signal x into a series of short-time segments whose spectra vary with time, that is, obtain the short-time frequency domain X. X = STFT(x)

[0059] Specifically, the original audio signal is divided into multiple overlapping time windows. The signal within each time window is called a frame, and the frame length is typically fixed, such as 10 milliseconds. Each frame is then windowed, using commonly used window functions such as Hamming windows and rectangular windows. Each frame is then Fourier transformed to convert the time-domain signal into the frequency domain, resulting in a spectral representation of each frame, known as a spectrogram. The result of this initial transformation is a spectrogram for each frame, which can be used to analyze the signal's energy distribution or frequency profile at different frequencies.

[0060] Performing amplitude-phase transformation on the first conversion result X can obtain the amplitude-phase spectrum, that is, the second conversion result X is obtained. mag,pha :

[0061] Among them, X i ,X r represent the real and imaginary parts of the complex spectrum, respectively.

[0062] Performing amplitude-angle transformation on the first conversion result X can obtain the amplitude-angle spectrum, that is, the third conversion result X is obtained. mag,ang : X mag,ang =log(|X|),angle(X i ,X r )

[0063] Based on the above optional embodiment, a domain converter is used to perform multi-dimensional domain conversion processing on the original audio signal to obtain a conversion result, thereby quickly extracting the spectral characteristics of the original audio signal and further improving the processing efficiency of the original audio signal.

[0064] In an optional embodiment, the audio signal coding model further includes: an encoder, which performs audio coding processing on the conversion result in step S23 to obtain the coding result, including: using the encoder to perform audio coding processing on the conversion result to obtain the coding result, wherein the coding result is used to determine the hidden layer representation obtained by encoding the time domain signal and the frequency domain signal in multiple representation forms.

[0065] The encoder in the above audio coding model can be used as a component of the LLM to implement audio coding processing on the conversion result to obtain a coding result.

[0066] In an optional embodiment, the audio signal coding model further includes: a quantizer, and the audio signal processing method in the embodiment of the present disclosure further includes: using the quantizer to perform layer-by-layer quantization processing on the coding result to obtain a quantization result, wherein the quantization result is used to represent the discrete code corresponding to the original audio signal.

[0067] The quantizer in the above audio coding model can be used as a component of the LLM to implement layer-by-layer quantization processing of the coding result to obtain a quantized result.

[0068] Specifically, the conversion results are subjected to audio encoding, resulting in a relatively compact latent representation encoded from various representations of time and frequency domain signals, suitable for audio processing, transmission, or storage. The compact latent representation is then quantized layer by layer to obtain multiple sets of discrete codes of equal length, representing the original audio signal.

[0069] Based on the above optional embodiment, an encoder is used to perform audio encoding processing on the conversion result to obtain an encoding result, and then the encoding result is quantized layer by layer to obtain a quantization result. In this way, the original audio signal can be converted into a more compact and efficient representation, thereby reducing the amount of data and improving the transmission and storage efficiency of the audio signal.

[0070] In an optional embodiment, in step S23, performing audio coding processing on the conversion result to obtain the coding result includes: performing multi-layer downsampling convolution coding processing on frequency domain signals in multiple representation forms in the conversion result to obtain the coding result.

[0071] Specifically, the audio encoding process for audio data in time domain form can refer to audio encoding methods in related technologies and will not be described in detail in the present embodiment. For frequency domain signals represented in the form of short-time frequency domain, amplitude-phase spectrum, amplitude-angle spectrum, etc., multi-layer downsampling convolutional coding can be performed to obtain the encoding result. The specific encoder structure is shown in Table 1.

[0072] Table 1 Encoder structure

[0073] In the encoder structure shown in Table 1, the network layer names (Layer Name) include: Domain Transformation module (Domain Trans), pre-two-dimensional convolution (PreConv2D), encoder network block (EncBlock), two-dimensional convolution module 1 (Conv2D_1), two-dimensional convolution module 2 (Conv2D_2), down-sampled two-dimensional convolution (Conv2D_ds), dimension conversion module (Reshape), long short-term network layer (Long Short-Term Memory, LSTM), linear output layer (OutLinear). Table 1 also includes the input (Inputs), output (Outputs), network layer parameters (Kernel), and network layer stride (Stride) of the encoder structure. T in Table 1 is used to represent the sequence length after the domain transformation module, and C b represents the number of channels of the convolutional module in the b-th encoder network block, T b represents the sequence length of the output of the convolutional module in the b-th encoder network block, F b represents the frequency dimension size of the convolution module in the b-th encoder network block, S b T represents the stride size of the convolution module in the b-th encoder network block. B represents the sequence length after B encoder network blocks, and D represents the vector dimension of the linear output layer output.

[0074] By stacking multiple layers of downsampling convolution Conv2D_ds in a large model, the temporal resolution is halved, and the quantizer resolution can be doubled at the same bit rate, thereby achieving better encoding quality and increasing the encoding rate.

[0075] In an optional embodiment, in step S232, the encoding result is quantized layer by layer, and the quantized result obtained includes:

[0076] Step S2321, determining an initial codebook to be used according to a preset data selection method;

[0077] Step S2322: in response to the activation count of some codewords in the initial codebook in the unit cluster data being lower than a preset threshold, the some codewords are updated to obtain a target codebook;

[0078] Step S2323 : performing layer-by-layer residual quantization processing on the encoding result based on the target codebook to obtain a quantization result.

[0079] Specifically, the preset data selection method can be a K-means clustering method, which uses the K-means clustering method to cluster the first unit cluster data (mini-batch) in the codebook, and uses the resulting cluster center as the initial codebook to be used. The codebook is subsequently updated using an exponential moving average (EMA) method. If the number of activations of a codeword in a mini-batch is lower than a preset threshold, it is reassigned to obtain the target codebook, that is, the quantization codebook. Using the K-means clustering results instead of random numbers to initialize the codebook of the multi-layer quantizer and reassigning codewords with lower activation levels during subsequent updates can effectively improve codebook utilization.

[0080] Furthermore, the residual vector quantization (RVQ) is used to perform layer-by-layer residual quantization on the coding result based on the target codebook to obtain the quantization result Q n .

[0081] RVQ quantizes the data into discrete codewords by splitting the input data into multiple subvectors and matching each subvector with a vector in a quantization codebook. RVQ differs from traditional vector quantization algorithms in that it uses a residual vector to represent the difference between the input data and the quantization codebook, effectively reducing compression error and improving the compression ratio.

[0082] Based on the above optional embodiment, the initial codebook to be used is determined according to a preset data selection method, and then in response to the activation number of some codewords in the initial codebook in the unit cluster data being lower than a preset threshold, the some codewords are updated to obtain a target codebook. Finally, the encoding result is subjected to layer-by-layer residual quantization processing based on the target codebook to obtain a quantization result. This can further improve the codebook utilization rate, and by reducing the time resolution, the resolution of the quantizer is correspondingly increased, thereby obtaining a higher encoding utilization rate.

[0083] In an optional embodiment, the audio signal processing method in the embodiment of the present disclosure further includes: in response to the original audio signal containing semantic information, extracting semantic information from the original audio signal by performing semantic extraction processing on the encoding result.

[0084] Specifically, when the original audio signal is a speech signal containing semantic information, the encoding result is processed into semantic tokens to extract semantic information from the speech signal. This can reduce the content that needs to be modeled by the encoder, and only needs to focus on other acoustic information in the speech signal besides semantic information, such as timbre, pitch, etc., thereby improving the encoding efficiency through semantic decoupling.

[0085] In an optional embodiment, in step S2323, performing layer-by-layer residual quantization processing on the encoding result based on the target codebook to obtain the quantization result includes: performing layer-by-layer residual quantization processing on the encoding result and semantic information based on the target codebook and a preset semantic augmentation method to obtain the quantization result.

[0086] Specifically, in the embodiment of the present disclosure, for the semantic information of the speech signal, the target codebook and the following three preset semantic augmentation methods can be used to perform layer-by-layer residual quantization processing on the encoding result and the semantic information to obtain the quantization result: cat (V a ,V s )=Concat(RVQ(V a ),V s ) f add (V a ,V s )=RVQ(V a )+V s f res (V a ,V s )=RVQ(V a -V s )+V s

[0087] Among them, V a Represents the output of the encoder, V s Represents semantic information, including but not limited to semantically quantized tokens such as frame-aligned monophone categories.

[0088] Based on the above optional embodiment, based on the target codebook and the preset semantic augmentation method, the encoding result and the semantic information are subjected to layer-by-layer residual quantization processing to obtain a quantization result, which can simultaneously utilize the semantic information and acoustic information contained in the speech signal to further improve the quantization processing effect.

[0089] In an optional embodiment, the audio signal processing method in the embodiment of the present disclosure further includes:

[0090] Step S24, performing audio decoding processing on the quantization result to obtain a decoding result, wherein the audio decoding structure corresponding to the decoding result and the audio encoding structure corresponding to the encoding result are mirror images of each other;

[0091] Step S25: performing domain inversion processing on the decoding result to obtain a reconstructed audio signal.

[0092] Specifically, the audio decoding structure and the audio encoding structure are mirror images of each other. Multiple groups of discrete codes in the quantization results are subjected to audio decoding processing, thereby obtaining audio signals in different representations. Furthermore, the audio signals in different representations are subjected to domain inversion processing, thereby restoring them to reconstructed audio signals that can be perceived by the human ear.

[0093] Based on the above optional embodiment, by performing audio decoding processing on the quantization result to obtain a decoding result, and then performing domain inversion processing on the decoding result to obtain a reconstructed audio signal, the original audio signal can be reliably restored in a preset application scenario.

[0094] Optionally, a graphical user interface is provided by the terminal device, and the content displayed by the graphical user interface at least partially includes an e-commerce service query scenario. The audio signal processing method in the embodiment of the present disclosure further includes:

[0095] Step S31, responding to a control operation performed on a voice recording control, recording an original voice message of an e-commerce service query;

[0096] Step S32, in response to the control operation performed on the query control, performing domain conversion processing on the e-commerce service query original voice to obtain an e-commerce service query conversion result, and performing audio encoding processing on the e-commerce service query conversion result to obtain an e-commerce service query encoding result;

[0097] Step S33: receiving e-commerce service query description information from the cloud service end, wherein the e-commerce service query description information is determined based on the e-commerce service query reconstructed speech, and the e-commerce service query reconstructed speech is obtained by performing domain inversion processing on the e-commerce service query encoding result;

[0098] Step S34: display the e-commerce service query description information in the graphical user interface.

[0099] The aforementioned voice input control allows users to interact with their computers through voice input. It can be used in voice recognition applications, intelligent assistants, and voice control systems to provide a more natural and convenient input method, enabling users to interact with their computers without a keyboard or mouse. Control operations performed on the aforementioned voice input control include, but are not limited to, single-clicking, long-pressing, and other operations to input raw voice for e-commerce service queries.

[0100] The above-mentioned query control can be a graphical button, check box, text box, drop-down menu, etc., or a non-graphic scroll bar, progress bar, etc. The user can interact with the query control to trigger domain conversion processing of the original voice of the e-commerce service query to obtain the e-commerce service query conversion result, and audio encoding processing of the e-commerce service query conversion result to obtain the e-commerce service query encoding result.

[0101] The e-commerce service query description information may be product introduction information, which can be determined based on the e-commerce service query reconstructed voice. The e-commerce service query reconstructed voice is obtained by performing domain inversion processing on the e-commerce service query encoding result.

[0102] Based on the above optional embodiments, by responding to the control operation performed on the voice input control, the original voice of the e-commerce service query is recorded, and then responding to the control operation performed on the query control, the domain conversion processing is performed on the original voice of the e-commerce service query to obtain the e-commerce service query conversion result, and the audio encoding processing is performed on the e-commerce service query conversion result to obtain the e-commerce service query encoding result, and then the e-commerce service query description information is received from the cloud service end, and the e-commerce service query description information is displayed in the graphical user interface, which can improve the user's query experience and convenience.

[0103] FIG3 is a schematic diagram of an audio signal processing method according to embodiment 1 of the present disclosure. As shown in FIG3 , after obtaining the original audio signal, the domain conversion module is used to perform domain conversion processing on the original audio signal to obtain time domain signals and frequency domain signals in various representation forms, and then these signals are input into the encoder for audio encoding processing to obtain an encoding result. If the original audio signal contains semantic information, the semantic information is extracted from the original audio signal by performing semantic extraction processing on the encoding result, and then the encoding result and the semantic information are subjected to layer-by-layer residual quantization processing based on the target codebook and the preset semantic augmentation method to obtain a quantization result. Finally, the quantization result is subjected to audio decoding processing to obtain a decoding result, wherein the audio decoding structure corresponding to the decoding result and the audio encoding structure corresponding to the encoding result are mirror images of each other, and the decoding result is subjected to domain inversion processing to obtain a reconstructed audio signal.

[0104] The domain conversion module and domain inversion module in the disclosed embodiments transform audio signals into more structurally distinct representations, such as the short-time frequency domain, amplitude-phase spectrum, and amplitude-angle spectrum. Furthermore, the use of a two-dimensional convolution-based encoder for audio encoding more effectively utilizes the time-frequency structure of the audio signal. By transforming the audio signal's representation and stacking more downsampling convolution modules, the temporal resolution is reduced, correspondingly increasing the quantizer's resolution and achieving higher coding efficiency.

[0105] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0106] It should be noted that for the aforementioned method embodiments, for simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present disclosure is not limited by the order of the actions described, because according to the present disclosure, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present disclosure.

[0107] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present disclosure.

[0108] Example 2

[0109] FIG4 is a flow chart of an audio signal processing method according to Embodiment 2 of the present disclosure. As shown in FIG4 , the method may include the following steps:

[0110] Step S41: Obtaining an encoding result, wherein the encoding result is used to restore the original audio signal in a preset application scenario, the encoding result is obtained by performing audio encoding processing on the conversion result, and the conversion result is obtained by performing domain conversion processing on the original audio signal, and the conversion result is used to represent a time domain signal and a frequency domain signal in multiple representation forms obtained by converting the original audio signal;

[0111] Step S42: Perform speech reconstruction processing on the encoding result to obtain a reconstructed audio signal.

[0112] Based on the above steps S41 to S42, by obtaining the encoding result and then performing speech reconstruction processing on the encoding result, a reconstructed audio signal is obtained, thereby achieving the purpose of reliably restoring the original audio signal in a preset application scenario, thereby achieving the technical effect of improving the encoding utilization rate and encoding quality of the audio signal, and thus solving the technical problems of low encoding utilization rate and poor encoding quality when performing quantization encoding processing on audio signals in related technologies.

[0113] The above-mentioned audio signal processing method provided by the embodiments of the present disclosure can be applied to, but is not limited to, application scenarios involving voice communication, voice storage, and large voice models in the fields of e-commerce services, educational services, legal services, medical services, conference services, social network services, financial product services, logistics services, and navigation services, and the embodiments of the present disclosure are not limited thereto.

[0114] The audio signal processing method in the embodiment of the present disclosure is further introduced below.

[0115] In an optional embodiment, the audio signal processing method is applied to an audio signal decoding model, the audio signal decoding model including: a decoder and a domain inverter. In step S42, speech reconstruction processing is performed on the encoding result to obtain a reconstructed audio signal including:

[0116] Step S421: Using a decoder to perform audio decoding processing on the encoding result to obtain a decoding result, wherein the audio decoding structure corresponding to the decoding result and the audio encoding structure corresponding to the encoding result are mirror images of each other;

[0117] Step S422: A domain invertor is used to perform domain inversion processing on the decoding result to obtain a reconstructed audio signal.

[0118] Based on the above optional embodiment, by performing audio decoding processing on the encoding result to obtain a decoding result, and then performing domain inversion processing on the decoding result to obtain a reconstructed audio signal, the original audio signal can be reliably restored in a preset application scenario.

[0119] For the parts not described in detail in the above embodiments of the present disclosure, please refer to the relevant description of Embodiment 1 and will not be repeated here.

[0120] Example 3

[0121] FIG5 is a flow chart of an audio signal processing method according to Embodiment 3 of the present disclosure. As shown in FIG5 , the method may include the following steps:

[0122] Step S51, obtaining an original audio signal;

[0123] Step S52: performing domain conversion processing on the original audio signal using the target audio signal processing model to obtain a conversion result, performing audio encoding processing on the conversion result to obtain an encoding result, and performing speech reconstruction processing on the encoding result to obtain a reconstructed audio signal, wherein the conversion result is used to represent time domain signals and frequency domain signals in multiple representation forms obtained by converting the original audio signal, and the encoding result is used to restore the original audio signal in a preset application scenario;

[0124] Step S53: play the reconstructed audio signal.

[0125] Based on the above steps S51 to S53, by obtaining the original audio signal, and then using the target audio signal processing model to perform domain conversion processing on the original audio signal to obtain a conversion result, the conversion result is subjected to audio encoding processing to obtain an encoding result, and the encoding result is subjected to speech reconstruction processing to obtain a reconstructed audio signal, and finally the reconstructed audio signal is played, thereby achieving the technical effect of improving the encoding utilization rate and encoding quality of the audio signal, and thus solving the technical problems of low encoding utilization rate and poor encoding quality when performing quantization encoding processing on the audio signal in the related art.

[0126] The above-mentioned audio signal processing method provided by the embodiments of the present disclosure can be applied to, but is not limited to, application scenarios involving voice communication, voice storage, and large voice models in the fields of e-commerce services, educational services, legal services, medical services, conference services, social network services, financial product services, logistics services, and navigation services, and the embodiments of the present disclosure are not limited thereto.

[0127] The audio signal processing method in the embodiment of the present disclosure is further introduced below.

[0128] In an optional embodiment, the audio signal processing method in the embodiment of the present disclosure further includes:

[0129] Step S521, performing audio reconstruction processing on the original training signal using the initial audio signal processing model to obtain a reconstructed training signal;

[0130] Step S522: determining a target loss based on the original training signal and the reconstructed training signal, wherein the target loss includes at least one of the following: a distance loss between the original training signal and the reconstructed training signal, a frequency domain reconstruction loss, and a discriminator adversarial loss;

[0131] Step S523 : updating the model parameters of the initial audio signal processing model by using the target loss to generate a target audio signal processing model.

[0132] Specifically, the target loss includes at least one of the following: the distance loss L between the original training signal and the reconstructed training signal t, frequency domain reconstruction loss L f , discriminator adversarial loss L adv / L feat , the quantization loss L of the similarity between the constrained quantization result and the corresponding encoding result cm Specifically, the calculation process of the target loss L is as follows: L = λ t L t +λ f L f +λ adv L adv +λ feat L feat +λ cm L cm

[0133] Wherein, λ is the loss weight. For the frequency domain reconstruction loss, the embodiment of the present disclosure adds the reconstruction loss of the energy spectrum domain S, thereby increasing the attention of the target audio signal processing model to the mid-high frequency signals and improving the overall audio quality. The frequency domain reconstruction loss L f Obtained through the following calculation process:

[0134] Among them, S i and M i Represents logarithmic compression power and Mel spectrum.

[0135] Discriminator adversarial loss L adv / L feat Obtained through the following calculation process:

[0136] Among them, D k,t represents the output of discriminator k at time step t.

[0137] The quantization loss L that constrains the similarity between the quantization result and the corresponding encoding result cm It can be obtained through the following calculation process:

[0138] Where V represents the input of RVQ.

[0139] Based on the above optional embodiments, by adding multiple losses and optimizing the multi-layer quantizer, the training process can be effectively streamlined and the training efficiency can be further improved.

[0140] For the parts not described in detail in the above embodiments of the present disclosure, please refer to the relevant description of Embodiment 1 and will not be repeated here.

[0141] Example 4

[0142] According to an embodiment of the present disclosure, an audio signal processing device for implementing the above-mentioned audio signal processing method is further provided. FIG6 is a block diagram of an audio signal processing device according to Embodiment 1 of the present disclosure. As shown in FIG6 , the device includes:

[0143] An acquisition module 601 is configured to acquire an original audio signal;

[0144] The conversion module 602 is configured to perform domain conversion processing on the original audio signal to obtain a conversion result, wherein the conversion result is used to represent a time domain signal and a frequency domain signal in multiple representation forms obtained by converting the original audio signal;

[0145] The encoding module 603 is configured to perform audio encoding processing on the conversion result to obtain an encoding result, wherein the encoding result is used to restore the original audio signal in a preset application scenario.

[0146] Optionally, the conversion module 602 is also configured to: use a domain converter to perform multiple dimensional domain conversion processing on the original audio signal to obtain a conversion result, wherein the conversion result includes some or all of the following conversion results: a first conversion result obtained by performing a short-time Fourier transform on the original audio signal; a second conversion result obtained by performing an amplitude-phase transform on the first conversion result; and a third conversion result obtained by performing an amplitude-angle transform on the first conversion result.

[0147] Optionally, the encoding module 603 is further configured to: use an encoder to perform audio encoding processing on the conversion result to obtain an encoding result, wherein the encoding result is used to determine a hidden layer representation obtained by encoding time domain signals and frequency domain signals in multiple representation forms.

[0148] Optionally, the audio signal processing apparatus further includes a quantization module 604 configured to use a quantizer to perform layer-by-layer quantization processing on the encoding result to obtain a quantization result, wherein the quantization result is used to represent a discrete code corresponding to the original audio signal.

[0149] Optionally, the encoding module 603 is further configured to perform multi-layer downsampling convolution encoding processing on the frequency domain signals in multiple representation forms in the conversion result to obtain an encoding result.

[0150] Optionally, the quantization module 604 is further configured to: determine an initial codebook to be used according to a preset data selection method; in response to the activation number of some codewords in the initial codebook in the unit cluster data being lower than a preset threshold, update some codewords to obtain a target codebook; and perform layer-by-layer residual quantization processing on the encoding result based on the target codebook to obtain a quantization result.

[0151] Optionally, the audio signal processing apparatus further includes: a processing module 605 configured to extract semantic information from the original audio signal by performing semantic extraction processing on the encoding result in response to the original audio signal containing semantic information.

[0152] Optionally, the encoding module 603 is further configured to: perform layer-by-layer residual quantization processing on the encoding result and the semantic information based on the target codebook and a preset semantic augmentation method to obtain a quantization result.

[0153] Optionally, the processing module 605 is further configured to: perform audio decoding processing on the quantization result to obtain a decoding result, wherein the audio decoding structure corresponding to the decoding result and the audio encoding structure corresponding to the encoding result are mirror images of each other; and perform domain inversion processing on the decoding result to obtain a reconstructed audio signal.

[0154] Optionally, the audio signal processing device also includes: an input module 605, which is configured to respond to the control operation performed on the voice input control and input the original voice of the e-commerce service query; the processing module 604 is also configured to respond to the control operation performed on the query control, perform domain conversion processing on the original voice of the e-commerce service query to obtain an e-commerce service query conversion result, and perform audio encoding processing on the e-commerce service query conversion result to obtain an e-commerce service query encoding result; the audio signal processing device also includes: a receiving module 606, which is configured to receive e-commerce service query description information from the cloud service end, wherein the e-commerce service query description information is determined based on the e-commerce service query reconstructed voice, and the e-commerce service query reconstructed voice is obtained by performing domain inversion processing on the e-commerce service query encoding result; the display module 607 is configured to display the e-commerce service query description information in the graphical user interface.

[0155] In the embodiment of the present disclosure, by obtaining the original audio signal and performing domain conversion processing on the original audio signal to obtain a conversion result, and finally performing audio encoding processing on the conversion result to obtain an encoding result, the purpose of reliably restoring the original audio signal in a preset application scenario is achieved, thereby achieving the technical effect of improving the encoding utilization rate and encoding quality of the audio signal, and further solving the technical problems of low encoding utilization rate and poor encoding quality when performing quantization encoding processing on audio signals in related technologies.

[0156] It should be noted that the acquisition module 601, conversion module 602, and encoding module 603 correspond to steps S21 to S23 in Example 1. The three modules and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in Example 2. It should be noted that the modules or units can be hardware components or software components stored in a memory and processed by one or more processors. The modules can also be part of the device and can be run in the computer terminal provided in Example 1.

[0157] It should be noted that the preferred implementation scheme involved in the above embodiments of the present disclosure is the same as the solution provided in Example 1, as well as the application scenario and implementation process, but is not limited to the solution provided in Example 1.

[0158] FIG7 is a result block diagram of an audio signal processing device according to Embodiment 2 of the present disclosure. As shown in FIG7 , the device includes:

[0159] an acquisition module 701 configured to acquire an encoding result, wherein the encoding result is used to restore the original audio signal in a preset application scenario, the encoding result being obtained by performing audio encoding processing on the conversion result, the conversion result being obtained by performing domain conversion processing on the original audio signal, and the conversion result being used to represent a time domain signal and a frequency domain signal in multiple representation forms obtained by converting the original audio signal;

[0160] The reconstruction module 702 is configured to perform speech reconstruction processing on the encoding result to obtain a reconstructed audio signal.

[0161] Optionally, the reconstruction module 702 is further configured to: use a decryptor to perform audio decoding processing on the encoding result to obtain a decoding result, wherein the audio decoding structure corresponding to the decoding result and the audio encoding structure corresponding to the encoding result are mirror images of each other; and use a domain inverter to perform domain inversion processing on the decoding result to obtain a reconstructed audio signal.

[0162] In the embodiment of the present disclosure, by obtaining the encoding result and then performing speech reconstruction processing on the encoding result, a reconstructed audio signal is obtained, thereby achieving the purpose of reliably restoring the original audio signal in a preset application scenario, thereby achieving the technical effect of improving the encoding utilization rate and encoding quality of the audio signal, and thus solving the technical problems of low encoding utilization rate and poor encoding quality when performing quantization encoding processing on audio signals in related technologies.

[0163] It should be noted that the acquisition module 701 and the reconstruction module 702 correspond to steps S41 to S42 in Example 2. The examples and application scenarios implemented by the two modules and the corresponding steps are the same, but are not limited to the contents disclosed in Example 2. It should be noted that the modules or units can be hardware components or software components stored in a memory and processed by one or more processors. The modules can also be part of the device and can be run in the computer terminal provided in Example 2.

[0164] It should be noted that the preferred implementation scheme involved in the above embodiments of the present disclosure is the same as the solution provided in Example 2, as well as the application scenario and implementation process, but is not limited to the solution provided in Example 2.

[0165] FIG8 is a result block diagram of an audio signal processing device according to Embodiment 3 of the present disclosure. As shown in FIG8 , the device includes:

[0166] An acquisition module 801 is configured to acquire an original audio signal;

[0167] a processing module 802 configured to perform domain conversion processing on the original audio signal using a target audio signal processing model to obtain a conversion result, perform audio encoding processing on the conversion result to obtain an encoding result, and perform speech reconstruction processing on the encoding result to obtain a reconstructed audio signal, wherein the conversion result is used to represent a time domain signal and a frequency domain signal in a plurality of representation forms obtained by converting the original audio signal, and the encoding result is used to restore the original audio signal in a preset application scenario;

[0168] The playing module 803 is configured to play the reconstructed audio signal.

[0169] Optionally, the audio signal processing device also includes: a reconstruction module 804, which is configured to use the initial audio signal processing model to perform audio reconstruction processing on the original training signal to obtain a reconstructed training signal; a determination module 805, which is configured to determine the target loss based on the original training signal and the reconstructed training signal, wherein the target loss includes at least one of the following: distance loss between the original training signal and the reconstructed training signal, frequency domain reconstruction loss, and discriminator adversarial loss; an update module 806, which is configured to update the model parameters of the initial audio signal processing model through the target loss to generate a target audio signal processing model.

[0170] In the embodiment of the present disclosure, by obtaining the original audio signal, and then using the target audio signal processing model to perform domain conversion processing on the original audio signal to obtain a conversion result, performing audio encoding processing on the conversion result to obtain an encoding result, and performing speech reconstruction processing on the encoding result to obtain a reconstructed audio signal, and finally playing the reconstructed audio signal, the technical effect of improving the encoding utilization rate and encoding quality of the audio signal is achieved, thereby solving the technical problems of low encoding utilization rate and poor encoding quality when performing quantization encoding processing on audio signals in related technologies.

[0171] It should be noted that the acquisition module 801, processing module 802, and playback module 803 correspond to steps S51 to S53 in Example 3. The examples and application scenarios implemented by the two modules and the corresponding steps are the same, but are not limited to the contents disclosed in Example 3. It should be noted that the above modules or units can be hardware components or software components stored in a memory and processed by one or more processors. The above modules can also be part of the device and can be run in the computer terminal provided in Example 1.

[0172] It should be noted that the preferred implementation scheme involved in the above embodiments of the present disclosure is the same as the solution provided in Example 3, as well as the application scenario and implementation process, but is not limited to the solution provided in Example 3.

[0173] Example 5

[0174] The embodiment of the present disclosure may provide a computer terminal, which may be any computer terminal device in a computer terminal group. Optionally, in this embodiment, the computer terminal may also be replaced by a terminal device such as a mobile terminal.

[0175] Optionally, in this embodiment, the computer terminal may be located in at least one network device among a plurality of network devices of a computer network.

[0176] In this embodiment, the above-mentioned computer terminal can execute the program code of the following steps in the audio signal processing method: obtaining an original audio signal; performing domain conversion processing on the original audio signal to obtain a conversion result, wherein the conversion result is used to represent time domain signals and frequency domain signals in multiple representation forms obtained by converting the original audio signal; performing audio encoding processing on the conversion result to obtain an encoding result, wherein the encoding result is used to restore the original audio signal in a preset application scenario.

[0177] In this embodiment, the above-mentioned computer terminal can execute the program code of the following steps in the audio signal processing method: obtaining the encoding result, wherein the encoding result is used to restore the original audio signal in a preset application scenario, the encoding result is obtained by performing audio encoding processing on the conversion result, the conversion result is obtained by performing domain conversion processing on the original audio signal, and the conversion result is used to represent time domain signals and frequency domain signals in various representation forms obtained by converting the original audio signal; performing speech reconstruction processing on the encoding result to obtain a reconstructed audio signal.

[0178] In this embodiment, the above-mentioned computer terminal can execute the program code of the following steps in the audio signal processing method: obtaining an original audio signal; using a target audio signal processing model to perform domain conversion processing on the original audio signal to obtain a conversion result, performing audio encoding processing on the conversion result to obtain an encoding result, and performing speech reconstruction processing on the encoding result to obtain a reconstructed audio signal, wherein the conversion result is used to represent time domain signals and frequency domain signals in multiple representation forms obtained by converting the original audio signal, and the encoding result is used to restore the original audio signal in a preset application scenario; and playing the reconstructed audio signal.

[0179] Alternatively, Figure 9 is a block diagram of a computer terminal according to Embodiment 5 of the present disclosure. As shown in the figure, the computer terminal 90 may include: one or more (only one is shown in the figure) processors 92, a memory 94, a storage controller, and a peripheral interface, wherein the peripheral interface is connected to a radio frequency module, an audio module, and a display.

[0180] The memory may be configured to store software programs and modules, such as program instructions / modules corresponding to the audio signal processing method and apparatus in the embodiments of the present disclosure. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby implementing the above-mentioned audio signal processing method. The memory may include a high-speed random access memory and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely located relative to the processor, and these remote memories may be connected to the computer terminal 90 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0181] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: obtain the original audio signal; perform domain conversion processing on the original audio signal to obtain a conversion result, wherein the conversion result is used to represent time domain signals and frequency domain signals in various representation forms obtained by converting the original audio signal; perform audio encoding processing on the conversion result to obtain an encoding result, wherein the encoding result is used to restore the original audio signal in a preset application scenario.

[0182] Optionally, the above-mentioned processor can also execute the program code of the following steps: using a domain converter to perform multiple dimensional domain conversion processing on the original audio signal to obtain a conversion result, wherein the conversion result includes some or all of the following conversion results: a first conversion result obtained by performing a short-time Fourier transform on the original audio signal; a second conversion result obtained by performing an amplitude-phase transformation on the first conversion result; and a third conversion result obtained by performing an amplitude-angle transformation on the first conversion result.

[0183] Optionally, the processor may also execute the program code of the following steps: performing audio encoding processing on the conversion result using an encoder to obtain an encoding result, wherein the encoding result is used to determine a hidden layer representation obtained by encoding time domain signals and frequency domain signals in multiple representation forms.

[0184] Optionally, the processor may further execute program code of the following steps: using a quantizer to perform layer-by-layer quantization processing on the encoding result to obtain a quantization result, wherein the quantization result is used to represent the discrete code corresponding to the original audio signal.

[0185] Optionally, the processor may further execute program code of the following steps: performing multi-layer downsampling convolution coding on frequency domain signals in various representation forms in the conversion result to obtain a coding result.

[0186] Optionally, the processor may further execute program code of the following steps: determining an initial codebook to be used according to a preset data selection method; in response to the activation times of some codewords in the initial codebook in the unit cluster data being lower than a preset threshold, updating some codewords to obtain a target codebook; and performing layer-by-layer residual quantization processing on the encoding result based on the target codebook to obtain a quantization result.

[0187] Optionally, the processor may further execute program code of the following steps: in response to the original audio signal containing semantic information, extracting semantic information from the original audio signal by performing semantic extraction processing on the encoding result.

[0188] Optionally, the processor may further execute program code of the following steps: based on the target codebook and a preset semantic augmentation method, performing layer-by-layer residual quantization processing on the encoding result and the semantic information to obtain a quantization result.

[0189] Optionally, the processor may also execute the program code of the following steps: performing audio decoding processing on the quantization result to obtain a decoding result, wherein the audio decoding structure corresponding to the decoding result and the audio encoding structure corresponding to the encoding result are mirror images of each other; performing domain inversion processing on the decoding result to obtain a reconstructed audio signal.

[0190] Optionally, the processor may also execute the program code of the following steps: responding to a control operation performed on a voice entry control, entering the original voice of the e-commerce service query; responding to a control operation performed on a query control, performing domain conversion processing on the original voice of the e-commerce service query to obtain an e-commerce service query conversion result, and performing audio encoding processing on the e-commerce service query conversion result to obtain an e-commerce service query encoding result; receiving e-commerce service query description information from a cloud service end, wherein the e-commerce service query description information is determined based on the reconstructed voice of the e-commerce service query, and the reconstructed voice of the e-commerce service query is obtained by performing domain inversion processing on the e-commerce service query encoding result; and displaying the e-commerce service query description information in a graphical user interface.

[0191] Optionally, the processor may also execute the program code of the following steps: obtaining an encoding result, wherein the encoding result is used to restore the original audio signal in a preset application scenario, the encoding result is obtained by performing audio encoding processing on the conversion result, the conversion result is obtained by performing domain conversion processing on the original audio signal, and the conversion result is used to represent time domain signals and frequency domain signals in various representation forms obtained by converting the original audio signal; performing speech reconstruction processing on the encoding result to obtain a reconstructed audio signal.

[0192] Optionally, the processor may also execute the program code of the following steps: performing audio decoding processing on the encoding result using a decoder to obtain a decoding result, wherein the audio decoding structure corresponding to the decoding result and the audio encoding structure corresponding to the encoding result are mirror images of each other; performing domain inversion processing on the decoding result using a domain inverter to obtain a reconstructed audio signal.

[0193] Optionally, the processor may also execute the program code of the following steps: obtaining an original audio signal; performing domain conversion processing on the original audio signal using a target audio signal processing model to obtain a conversion result, performing audio encoding processing on the conversion result to obtain an encoding result, and performing speech reconstruction processing on the encoding result to obtain a reconstructed audio signal, wherein the conversion result is used to represent time domain signals and frequency domain signals in various representation forms obtained by converting the original audio signal, and the encoding result is used to restore the original audio signal under a preset application scenario; and playing the reconstructed audio signal.

[0194] Optionally, the processor may also execute the program code of the following steps: performing audio reconstruction processing on the original training signal using the initial audio signal processing model to obtain a reconstructed training signal; determining a target loss based on the original training signal and the reconstructed training signal, wherein the target loss includes at least one of the following: distance loss between the original training signal and the reconstructed training signal, frequency domain reconstruction loss, and discriminator adversarial loss; and updating the model parameters of the initial audio signal processing model using the target loss to generate a target audio signal processing model.

[0195] By adopting the embodiment of the present disclosure, the original audio signal is obtained, and domain conversion processing is performed on the original audio signal to obtain a conversion result, and finally audio encoding processing is performed on the conversion result to obtain an encoding result, thereby achieving the purpose of reliably restoring the original audio signal in a preset application scenario, thereby achieving the technical effect of improving the encoding utilization rate and encoding quality of the audio signal, and further solving the technical problems of low encoding utilization rate and poor encoding quality when performing quantization encoding processing on audio signals in related technologies.

[0196] Those skilled in the art will appreciate that the structure shown in the figure is merely illustrative, and the computer terminal may also be a smartphone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile internet device (MID), a PAD, or other terminal device. FIG9 does not limit the structure of the above-mentioned electronic devices. For example, the computer terminal may include more or fewer components (such as a network interface, a display device, etc.) than shown in FIG9, or may have a configuration different from that shown in FIG9.

[0197] A person skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, which may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0198] Example 6

[0199] The embodiment of the present disclosure further provides a computer-readable storage medium. Optionally, in this embodiment, the computer-readable storage medium can be used to store the program code executed by the audio signal processing method provided in the first embodiment.

[0200] Optionally, in this embodiment, the above-mentioned storage medium may be located in any computer terminal in a computer terminal group in a computer network, or in any mobile terminal in a mobile terminal group.

[0201] Optionally, in this embodiment, the storage medium is configured to store program code for executing the following steps: obtaining an original audio signal; performing domain conversion processing on the original audio signal to obtain a conversion result, wherein the conversion result is used to represent time domain signals and frequency domain signals in multiple representation forms obtained by converting the original audio signal; performing audio encoding processing on the conversion result to obtain an encoding result, wherein the encoding result is used to restore the original audio signal in a preset application scenario.

[0202] Optionally, in this embodiment, the storage medium is configured to store program code for executing the following steps: using a domain converter to perform multiple dimensional domain conversion processing on the original audio signal to obtain a conversion result, wherein the conversion result includes some or all of the following conversion results: a first conversion result obtained by performing a short-time Fourier transform on the original audio signal; a second conversion result obtained by performing an amplitude-phase transform on the first conversion result; and a third conversion result obtained by performing an amplitude-angle transform on the first conversion result.

[0203] Optionally, in this embodiment, the storage medium is configured to store program code for executing the following steps: using an encoder to perform audio encoding processing on the conversion result to obtain an encoding result, wherein the encoding result is used to determine a hidden layer representation obtained by encoding time domain signals and frequency domain signals in multiple representation forms.

[0204] Optionally, in this embodiment, the storage medium is configured to store program code for executing the following steps: using a quantizer to perform layer-by-layer quantization processing on the encoding result to obtain a quantization result, wherein the quantization result is used to represent the discrete code corresponding to the original audio signal.

[0205] Optionally, in this embodiment, the storage medium is configured to store program codes for executing the following steps: performing multi-layer downsampling convolution coding on frequency domain signals in multiple representation forms in the conversion results to obtain coding results.

[0206] Optionally, in this embodiment, the storage medium is configured to store program code for executing the following steps: determining an initial codebook to be used according to a preset data selection method; in response to the activation number of some codewords in the initial codebook in the unit cluster data being lower than a preset threshold, updating some codewords to obtain a target codebook; and performing layer-by-layer residual quantization processing on the encoding result based on the target codebook to obtain a quantization result.

[0207] Optionally, in this embodiment, the storage medium is configured to store program code for executing the following steps: in response to the original audio signal containing semantic information, extracting semantic information from the original audio signal by performing semantic extraction processing on the encoding result.

[0208] Optionally, in this embodiment, the storage medium is configured to store program code for executing the following steps: based on the target codebook and a preset semantic augmentation method, performing layer-by-layer residual quantization processing on the encoding result and the semantic information to obtain a quantization result.

[0209] Optionally, in this embodiment, the storage medium is configured to store program code for executing the following steps: performing audio decoding processing on the quantization result to obtain a decoding result, wherein the audio decoding structure corresponding to the decoding result and the audio encoding structure corresponding to the encoding result are mirror images of each other; performing domain inversion processing on the decoding result to obtain a reconstructed audio signal.

[0210] Optionally, in this embodiment, the storage medium is configured to store program codes for executing the following steps: in response to a control operation performed on a voice entry control, recording the original voice of the e-commerce service query; in response to a control operation performed on a query control, performing domain conversion processing on the original voice of the e-commerce service query to obtain an e-commerce service query conversion result, and performing audio encoding processing on the e-commerce service query conversion result to obtain an e-commerce service query encoding result; receiving e-commerce service query description information from a cloud service end, wherein the e-commerce service query description information is determined based on the reconstructed voice of the e-commerce service query, and the reconstructed voice of the e-commerce service query is obtained by performing domain inversion processing on the e-commerce service query encoding result; and displaying the e-commerce service query description information in a graphical user interface.

[0211] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: obtaining an encoding result, wherein the encoding result is used to restore the original audio signal in a preset application scenario, the encoding result is obtained by performing audio encoding processing on the conversion result, the conversion result is obtained by performing domain conversion processing on the original audio signal, and the conversion result is used to represent time domain signals and frequency domain signals in various representation forms obtained by converting the original audio signal; performing speech reconstruction processing on the encoding result to obtain a reconstructed audio signal.

[0212] Optionally, in this embodiment, the storage medium is configured to store program codes for executing the following steps: performing audio decoding processing on the encoding result using a decoder to obtain a decoding result, wherein the audio decoding structure corresponding to the decoding result and the audio encoding structure corresponding to the encoding result are mirror images of each other; and performing domain inversion processing on the decoding result using a domain inverter to obtain a reconstructed audio signal.

[0213] Optionally, in this embodiment, the storage medium is configured to store program code for executing the following steps: obtaining an original audio signal; performing domain conversion processing on the original audio signal using a target audio signal processing model to obtain a conversion result, performing audio encoding processing on the conversion result to obtain an encoding result, and performing speech reconstruction processing on the encoding result to obtain a reconstructed audio signal, wherein the conversion result is used to represent time domain signals and frequency domain signals in multiple representation forms obtained by converting the original audio signal, and the encoding result is used to restore the original audio signal in a preset application scenario; and playing the reconstructed audio signal.

[0214] Optionally, in this embodiment, the storage medium is configured to store program code for executing the following steps: performing audio reconstruction processing on the original training signal using an initial audio signal processing model to obtain a reconstructed training signal; determining a target loss based on the original training signal and the reconstructed training signal, wherein the target loss includes at least one of the following: distance loss between the original training signal and the reconstructed training signal, frequency domain reconstruction loss, and discriminator adversarial loss; updating the model parameters of the initial audio signal processing model through the target loss to generate a target audio signal processing model.

[0215] By adopting the embodiment of the present disclosure, the original audio signal is obtained, and domain conversion processing is performed on the original audio signal to obtain a conversion result, and finally audio encoding processing is performed on the conversion result to obtain an encoding result, thereby achieving the purpose of reliably restoring the original audio signal in a preset application scenario, thereby achieving the technical effect of improving the encoding utilization rate and encoding quality of the audio signal, and further solving the technical problems of low encoding utilization rate and poor encoding quality when performing quantization encoding processing on audio signals in related technologies.

[0216] The serial numbers of the above-mentioned embodiments of the present disclosure are for description only and do not represent the advantages or disadvantages of the embodiments.

[0217] In the above embodiments of the present disclosure, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0218] In the several embodiments provided in the present disclosure, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0219] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0220] In addition, the functional units in the various embodiments of the present disclosure may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0221] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0222] The above is only a preferred embodiment of the present disclosure. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present disclosure. These improvements and modifications should also be regarded as within the scope of protection of the present disclosure.

Claims

1. An audio signal processing method, comprising: Get the original audio signal; Performing domain conversion processing on the original audio signal to obtain a conversion result, wherein the conversion result is used to represent a time domain signal and a frequency domain signal in a plurality of representation forms obtained by converting the original audio signal; The conversion result is subjected to audio coding processing to obtain a coding result, wherein the coding result is used to restore the original audio signal in a preset application scenario.

2. The audio signal processing method according to claim 1, wherein: The audio signal processing method is applied to an audio signal coding model, and the audio signal coding model includes: a domain converter, which performs domain conversion processing on the original audio signal, and obtains the conversion result including: The domain converter is used to perform multiple dimensional domain conversion processing on the original audio signal to obtain the conversion result, wherein the conversion result includes some or all of the following conversion results: A first conversion result obtained by performing short-time Fourier transform on the original audio signal; a second conversion result obtained by performing amplitude and phase transformation on the first conversion result; A third conversion result is obtained by performing amplitude-angle conversion on the first conversion result.

3. The audio signal processing method according to claim 2, wherein: The audio signal coding model further includes: an encoder, which performs audio coding processing on the conversion result, and obtains the coding result including: The encoder is used to perform audio encoding processing on the conversion result to obtain an encoding result, wherein the encoding result is used to determine a hidden layer representation obtained by encoding the time domain signals and frequency domain signals in the multiple representation forms.

4. The audio signal processing method according to claim 3, wherein: The audio signal coding model further includes: a quantizer, and the audio signal processing method further includes: The quantizer is used to perform layer-by-layer quantization processing on the encoding result to obtain a quantization result, wherein the quantization result is used to represent the discrete code corresponding to the original audio signal.

5. The audio signal processing method according to claim 3, wherein: Performing audio coding processing on the conversion result to obtain the coding result includes: Multi-layer downsampling convolution coding is performed on frequency domain signals in various representation forms in the conversion result to obtain the coding result.

6. The audio signal processing method according to claim 4, wherein: The encoding result is quantized layer by layer to obtain the quantized result, including: Determine the initial codebook to be used according to a preset data selection method; In response to the activation times of some codewords in the initial codebook in the unit cluster data being lower than a preset threshold, updating the some codewords to obtain a target codebook; The encoding result is subjected to layer-by-layer residual quantization processing based on the target codebook to obtain the quantization result.

7. The audio signal processing method according to claim 6, wherein: The audio signal processing method further includes: In response to the original audio signal containing semantic information, the semantic information is extracted from the original audio signal by performing semantic extraction processing on the encoding result.

8. The audio signal processing method according to claim 7, wherein: Performing layer-by-layer residual quantization processing on the encoding result based on the target codebook to obtain the quantization result includes: Based on the target codebook and a preset semantic augmentation method, the encoding result and the semantic information are subjected to layer-by-layer residual quantization processing to obtain the quantization result.

9. The audio signal processing method according to claim 4, wherein: The audio signal processing method further includes: Performing audio decoding processing on the quantization result to obtain a decoding result, wherein an audio decoding structure corresponding to the decoding result and an audio encoding structure corresponding to the encoding result are mirror images of each other; The decoding result is subjected to domain inversion processing to obtain a reconstructed audio signal.

10. The audio signal processing method according to claim 1, wherein: A graphical user interface is provided through a terminal device, and the content displayed by the graphical user interface at least partially includes an e-commerce service query scenario, and the audio signal processing method further includes: In response to the control operation performed on the voice input control, the original voice of the e-commerce service query is input; In response to a control operation performed on a query control, performing domain conversion processing on the e-commerce service query original voice to obtain an e-commerce service query conversion result, and performing audio encoding processing on the e-commerce service query conversion result to obtain an e-commerce service query encoding result; Receiving e-commerce service query description information from a cloud service end, wherein the e-commerce service query description information is determined based on the e-commerce service query reconstructed speech, and the e-commerce service query reconstructed speech is obtained by performing domain inversion processing on the e-commerce service query encoding result; The e-commerce service query description information is displayed in the graphical user interface.

11. A method for processing an audio signal, comprising: Obtaining an encoding result, wherein the encoding result is used to restore the original audio signal in a preset application scenario, the encoding result is obtained by performing audio encoding processing on the conversion result, the conversion result is obtained by performing domain conversion processing on the original audio signal, and the conversion result is used to represent time domain signals and frequency domain signals in multiple representation forms obtained by converting the original audio signal; The encoding result is subjected to speech reconstruction processing to obtain a reconstructed audio signal.

12. The audio signal processing method according to claim 11, wherein: The audio signal processing method is applied to an audio signal decoding model, the audio signal decoding model includes: a decoder and a domain inverter, and the encoding result is subjected to speech reconstruction processing to obtain the reconstructed audio signal including: The decoder is used to perform audio decoding processing on the encoding result to obtain a decoding result, wherein: The audio decoding structure corresponding to the decoding result and the audio encoding structure corresponding to the encoding result are mirror images of each other; The domain invertor is used to perform domain inversion processing on the decoding result to obtain the reconstructed audio signal.

13. A method for processing an audio signal, comprising: Get the original audio signal; The target audio signal processing model is used to perform domain conversion processing on the original audio signal to obtain a conversion result, the conversion result is subjected to audio encoding processing to obtain an encoding result, and the encoding result is subjected to speech reconstruction processing to obtain a reconstructed audio signal, wherein the conversion result is used to represent a time domain signal and a frequency domain signal in a plurality of representation forms obtained by converting the original audio signal, and the encoding result is used to restore the original audio signal in a preset application scenario; The reconstructed audio signal is played.

14. The audio signal processing method according to claim 13, wherein: The audio signal processing method further includes: Using the initial audio signal processing model to perform audio reconstruction processing on the original training signal to obtain a reconstructed training signal; Determining a target loss based on the original training signal and the reconstructed training signal, wherein the target loss includes at least one of the following: a distance loss between the original training signal and the reconstructed training signal, a frequency domain reconstruction loss, and a discriminator adversarial loss; The model parameters of the initial audio signal processing model are updated according to the target loss to generate the target audio signal processing model.

15. An electronic device, comprising: A memory storing an executable program; A processor is configured to run the program, wherein the program executes the audio signal processing method according to any one of claims 1 to 14 when running.

16. A computer-readable storage medium comprising a stored executable program, wherein: When the executable program is running, the device where the computer-readable storage medium is located is controlled to execute the audio signal processing method according to any one of claims 1 to 14.

Images