Audio signal processing method, electronic equipment and storage medium

Abstract

The invention discloses an audio signal processing method, electronic equipment and a storage medium, and relates to the field of large model technology and audio processing. The method comprises the following steps: acquiring an original audio signal; domain conversion processing is carried out on the original audio signal to obtain a conversion result, and the conversion result is used for representing time domain signals and frequency domain signals in various representation forms obtained by converting the original audio signal; and performing audio coding processing on the conversion result to obtain a coding result, the coding result being used for restoring the original audio signal in a preset application scene. The technical problems of low coding utilization rate and poor coding quality when quantization coding processing is performed on the audio signal in the prior art are solved.

Description

Technical Field

[0001] The present application relates to large model technology and audio processing fields, and specifically, to an audio signal processing method, electronic device and storage medium. Background Art

[0002] As one of the most commonly used data modalities, audio is widely present in daily life and plays an indispensable role. With the rise of large language models (LLM), a wave of multimodal unified modeling has been set off in the field of artificial intelligence. As an important data modality, audio has important research significance. In the related art, a variety of different audio quantization coding models have been developed to realize the quantization coding processing of audio signals, that is, to convert continuous audio signals into discrete digital signals, thereby reducing the amount of data required for storage or transmission while maintaining the quality of the original audio signal as much as possible. However, the audio quantization coding model in the related art mainly uses the time domain structure of the audio signal for encoding, which leads to low coding utilization and poor quantization coding quality.

[0003] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0004] The embodiments of the present application provide an audio signal processing method, an electronic device, and a storage medium to at least solve the technical problems of low coding utilization and poor coding quality when performing quantization coding processing on audio signals in the related art.

[0005] According to one aspect of an embodiment of the present application, there is provided an audio signal processing method, comprising: obtaining an original audio signal; performing domain conversion processing on the original audio signal to obtain a conversion result, wherein the conversion result is used to represent time domain signals and frequency domain signals in various representation forms obtained by converting the original audio signal; performing audio encoding processing on the conversion result to obtain an encoding result, wherein the encoding result is used to restore the original audio signal in a preset application scenario.

[0006] According to another aspect of an embodiment of the present application, there is also provided an audio signal processing method, including: obtaining an encoding result, wherein the encoding result is used to restore the original audio signal in a preset application scenario, the encoding result is obtained by performing audio encoding processing on the conversion result, the conversion result is obtained by performing domain conversion processing on the original audio signal, and the conversion result is used to represent time domain signals and frequency domain signals in various representation forms obtained by converting the original audio signal; performing speech reconstruction processing on the encoding result to obtain a reconstructed audio signal.

[0007] According to another aspect of an embodiment of the present application, an audio signal processing method is also provided, including: obtaining an original audio signal; performing domain conversion processing on the original audio signal using a target audio signal processing model to obtain a conversion result, performing audio encoding processing on the conversion result to obtain an encoding result, and performing speech reconstruction processing on the encoding result to obtain a reconstructed audio signal, wherein the conversion result is used to represent time domain signals and frequency domain signals in multiple representation forms obtained by converting the original audio signal, and the encoding result is used to restore the original audio signal under a preset application scenario; and playing the reconstructed audio signal.

[0008] According to another aspect of the embodiments of the present application, an electronic device is further provided, including: a memory storing an executable program; and a processor for running the program, wherein when the program is running, the audio signal processing method of any one of the embodiments of the present application is executed.

[0009] According to another aspect of an embodiment of the present application, a computer-readable storage medium is further provided, the computer-readable storage medium including a stored executable program, wherein when the executable program is running, the device where the computer-readable storage medium is located is controlled to execute any one of the audio signal processing methods in the embodiments of the present application.

[0010] In an embodiment of the present application, by acquiring an original audio signal, performing domain conversion processing on the original audio signal, obtaining a conversion result, and finally performing audio encoding processing on the conversion result to obtain an encoding result, the purpose of reliably restoring the original audio signal in a preset application scenario is achieved, thereby achieving the technical effect of improving the encoding utilization rate and encoding quality of the audio signal, and further solving the technical problems of low encoding utilization rate and poor encoding quality when performing quantization encoding processing on audio signals in related technologies.

[0011] It is easy to notice that the above general description and the following detailed description are only for the purpose of exemplifying and explaining the present application, and do not constitute a limitation of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0013] Figure 1 is a schematic diagram of an application scenario of an audio signal processing method according to Embodiment 1 of the present application;

[0014] Figure 2 is a flowchart of an audio signal processing method according to Embodiment 1 of the present application;

[0015] Figure 3is a schematic diagram of an audio signal processing method according to Embodiment 1 of the present application;

[0016] Figure 4 is a flowchart of an audio signal processing method according to Embodiment 2 of the present application;

[0017] Figure 5 is a flowchart of an audio signal processing method according to Embodiment 3 of the present application;

[0018] Figure 6 is a result block diagram of an audio signal processing device according to Embodiment 1 of the present application;

[0019] Figure 7 is a result block diagram of an audio signal processing device according to Embodiment 2 of the present application;

[0020] Figure 8 is a result block diagram of an audio signal processing device according to Embodiment 3 of the present application;

[0021] Fig. 9 It is a structural block diagram of a computer terminal according to an embodiment of the present application. DETAILED DESCRIPTION

[0022] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0023] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0024] The technical solution provided in this application is mainly implemented by large-scale model technology. The large model here refers to a deep learning model with large-scale model parameters, which can usually contain hundreds of millions, tens of billions, hundreds of billions, trillions or even more than 10 trillion model parameters. The large model can also be called a cornerstone model / foundation model (Foundation Model). The large model is pre-trained through large-scale unlabeled corpus to produce a pre-trained model with more than 100 million parameters. This model can adapt to a wide range of downstream tasks, and the model has good generalization ability, such as large-scale language model (Large Language Model, LLM), multi-modal pre-training model (multi-modal pre-training model), etc.

[0025] It should be noted that when the large model is actually used, the pre-trained model can be fine-tuned through a small number of samples so that the large model can be applied to different tasks. For example, the large model can be widely used in natural language processing (NLP), computer vision, speech processing and other fields, and can be specifically applied to computer vision tasks such as visual question answering (VQA), image description (IC), image generation, etc., and can also be widely used in text-based sentiment classification, text summary generation, machine translation and other natural language processing tasks. Therefore, the main application scenarios of the large model include but are not limited to digital assistants, intelligent robots, search, online education, office software, e-commerce, intelligent design, etc. In the embodiment of the present application, data processing by an audio signal processing model in voice communication and voice storage scenarios is used as an example for explanation.

[0026] First, some nouns or terms that appear in the description of the embodiments of the present application are subject to the following explanations:

[0027] Audio Codec: The process of converting a continuous audio signal sequence into a quantized code sequence.

[0028] Quantization encoding, token: a bounded non-negative integer representing the subscript of the quantized audio representation in the dictionary.

[0029] Token rate (TKR): The number of quantized codes required to quantize an audio signal of 1 second in length at a preset audio sampling rate.

[0030] Bitrate: The number of bits required to quantize and encode an audio signal of 1 second in length at a preset audio sampling rate.

[0031] The following four audio quantization coding models have been developed in the related art to realize the quantization coding processing of audio signals, including common open source audio quantization coding models such as encoder (Encodec), audio stream (SoundStream), and common open source audio quantization coding projects such as audio decoder (AudioDec) and audio codec (Descript-audio-codec). However, the audio quantization coding model in the related art generally directly models the original audio time domain signal, lacks the effective use of the frequency domain structure of the audio signal, and does not utilize the semantic information contained in the speech signal, but only encodes it from an acoustic perspective. The coding dictionary in the related art adopts a random initialization method. In the subsequent update process, even if some codewords are rarely activated, they will not be reassigned, resulting in a low codebook utilization rate. In addition, the training process of AudioDec is relatively cumbersome, and is specifically divided into two stages. The first stage trains the autoencoder, and the second stage is initialized with the first stage, and a vector quantizer is added for discrete encoding, and then the encoder and decoder are further trained. Encodec, SoundStream, and Descript-audio-codec have the same bit rate, but the time granularity is too fine, which leads to insufficient utilization of the encoding. As a result, the quality of the encoded speech is poor at the same bit rate.

[0032] In summary, the related art has the technical problems of low coding utilization and poor coding quality when performing quantization coding processing on audio signals. Currently, no effective solution has been proposed for the above problems.

[0033] Example 1

[0034] According to an embodiment of the present application, an audio signal processing method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0035] Considering the huge amount of model parameters of the large model and the limited computing resources of the mobile terminal, the above-mentioned audio signal processing method provided in the embodiment of the present application can be applied to Figure 1 The application scenarios shown are not limited to these. Figure 1In the application scenario shown, the large model is deployed in the server 10, and the server 10 can be connected to one or more client devices 20 through a local area network connection, a wide area network connection, an Internet connection, or other types of data networks. The client device 20 here may include but is not limited to: a smart phone, a tablet computer, a laptop computer, a PDA, a personal computer, a smart home device, a vehicle-mounted device, etc. The client device 20 can interact with the user through a graphical user interface to implement the call of the large model, thereby implementing the method provided in the embodiment of the present application.

[0036] In an embodiment of the present application, a system composed of a client device and a server can perform the following steps: the client device obtains the original audio signal and sends it to the server, the server performs domain conversion processing on the original audio signal to obtain a conversion result, wherein the conversion result is used to represent a time domain signal and a frequency domain signal in a variety of representation forms obtained by converting the original audio signal, and further, the server performs audio encoding processing on the conversion result to obtain an encoding result, wherein the encoding result is used to restore the original audio signal in a preset application scenario. It should be noted that when the operating resources of the client device can meet the deployment and operation conditions of the large model, the embodiment of the present application can be performed in the client device.

[0037] Under the above operating environment, this application provides Figure 2 The audio signal processing method shown. Figure 2 is a flow chart of an audio signal processing method according to Embodiment 1 of the present application. Figure 2 As shown, the method may include the following steps:

[0038] Step S21, obtaining an original audio signal;

[0039] Step S22, performing domain conversion processing on the original audio signal to obtain a conversion result, wherein the conversion result is used to represent a time domain signal and a frequency domain signal in a plurality of representation forms obtained by converting the original audio signal;

[0040] Step S23, performing audio encoding processing on the conversion result to obtain an encoding result, wherein the encoding result is used to restore the original audio signal in a preset application scenario.

[0041] The above-mentioned original audio signals include speech signals and other audio signals. Specifically, speech signals are signals generated by human voices, and speech signals contain semantic information and acoustic information. Semantic information refers to the meaning, implications and information conveyed by language, including vocabulary, grammar, semantics, logical relationships, etc. in speech. Semantic information can be understood through the structure and context of language, so as to convey the specific meaning and information that the speaker wants to express. Acoustic information refers to the physical characteristics and sound waveform of sound, including the frequency, amplitude, duration, pitch, volume, speech speed, etc. of sound. Acoustic information is understood by analyzing and processing the physical characteristics of sound, and can convey the sound quality and characteristics of sound. Other audio signals refer to all audio signals other than speech signals, including but not limited to music, acoustic events, such as knocking, applause, and door opening. These audio signals usually have no clear language features and only contain acoustic information, which can be used for tasks such as audio processing, audio analysis, and audio enhancement.

[0042] There are many ways to obtain the above-mentioned original audio signal, for example, using a professional recording device or a built-in microphone of a device such as a smart phone to record, placing the microphone near the sound source to be recorded, and then starting recording using recording software, so as to directly capture the original audio signal of the sound source. For another example, connect the output of an audio device (such as a musical instrument, computer output, etc.) to the audio input interface of a computer, and then use audio recording software to record. This method can capture the original audio signal output by the device. For another example, directly use audio editing software to open a preset audio file, and export the preset audio file as an original audio signal.

[0043] Domain conversion processing of the original audio signal can convert the original audio signal into the desired representation form, and the conversion results include but are not limited to the time domain, short-time frequency domain, amplitude phase spectrum, amplitude angle spectrum, etc. corresponding to the original audio data. Specifically, the time domain refers to the change of the audio signal in time, which can be represented by a waveform diagram. The time domain can represent the amplitude value of the original audio signal at different time points. The short-time frequency domain is to divide the original audio signal into several time windows, perform frequency domain analysis in each window, and obtain the spectrum information of the original audio signal in the window. This representation method can reveal the energy distribution of the original audio signal at different frequencies. The commonly used method is to use fast Fourier transform for frequency domain analysis. The amplitude phase spectrum represents the spectrum information of the original audio signal in the form of amplitude and phase, wherein the amplitude spectrum represents the energy size of the original audio signal at different frequencies, and the phase spectrum represents the phase information of the original audio signal at different frequencies. The amplitude angle spectrum represents the spectrum information of the original audio signal in the form of amplitude and angle, wherein the amplitude spectrum represents the energy size of the original audio signal at different frequencies, and the angle spectrum represents the phase information of the original audio signal at different frequencies. Compared with the amplitude-phase spectrum, the amplitude-angle spectrum pays more attention to the representation of phase information.

[0044] Exemplary methods for implementing domain conversion processing include but are not limited to Fourier transform, fast Fourier transform, discrete cosine transform, etc. These domain conversion methods can convert the original audio signal from the time domain to the frequency domain to obtain the spectrum information of the original audio signal, or convert the original audio signal from the frequency domain to the time domain to obtain the time domain waveform. In the frequency domain, the audio signal can be subjected to spectrum analysis, filtering, noise reduction, compression, etc., while in the time domain, the audio signal can be subjected to time domain filtering, time domain modification, etc. Domain conversion processing has important applications in audio signal processing, and can extract the characteristics of the original audio signal, improve the sound quality, and realize the functions of compressing the original audio signal.

[0045] Further, the conversion result is subjected to audio coding processing to obtain a coding result, and the coding result is used to restore the original audio signal in a preset application scenario. Specifically, audio quantization coding is a process of converting a continuous audio signal into a discrete digital signal. In the quantization coding process, the amplitude of the original audio signal can be discretized and mapped into a series of discrete values ​​to represent the amplitude of the original audio signal.

[0046] The above encoding results include multiple groups of discrete codes of equal length to represent the audio signal. The discrete codes are usually represented in binary form and stored or transmitted to the receiving end for decoding to restore the original audio signal. The audio encoding process can reduce the amount of data required for storage or transmission while maintaining the quality of the original audio signal as much as possible.

[0047] The above-mentioned preset application scenarios include but are not limited to voice communication, voice storage, language large models and other scenarios. Specifically, in the voice communication scenario, by performing audio coding processing on the original recorded voice signal, the voice signal can be compressed and encoded into a coding result and then transmitted. The receiving end can restore the coding result to the original recorded voice signal, thereby effectively reducing the bandwidth required for transmission and improving transmission efficiency. In the voice storage scenario, the recorded original voice signal, such as customer service communication records, is audio coded and stored quantized, and the coding result is restored to the original voice signal when retrieval and playback are required. In the language large model scenario, the voice signal is processed by audio coding to obtain discrete semantic tokens, which can be modeled uniformly with the text, thereby adding voice understanding and generation capabilities to the large model.

[0048] Based on the above steps S21 to S23, by acquiring the original audio signal and performing domain conversion processing on the original audio signal to obtain a conversion result, and finally performing audio encoding processing on the conversion result to obtain an encoding result, the purpose of reliably restoring the original audio signal in a preset application scenario is achieved, thereby achieving the technical effect of improving the encoding utilization rate and encoding quality of the audio signal, and further solving the technical problems of low encoding utilization rate and poor encoding quality when performing quantization encoding processing on the audio signal in the related technology.

[0049] The above-mentioned audio signal processing method provided in the embodiment of the present application can be applied to, but is not limited to, application scenarios involving voice communication, voice storage, and large voice models in the fields of e-commerce services, educational services, legal services, medical services, conference services, social network services, financial product services, logistics services, and navigation services, and the embodiment of the present application is not limited.

[0050] The audio signal processing method in the embodiment of the present application is further introduced below.

[0051] In an optional embodiment, the audio signal processing method is applied to an audio signal coding model, and the audio signal coding model includes: a domain converter, in step S22, the original audio signal is subjected to domain conversion processing, and the conversion result obtained includes:

[0052] The domain converter is used to perform multiple dimensional domain conversion processing on the original audio signal to obtain a conversion result, wherein the conversion result includes some or all of the following conversion results:

[0053] A first conversion result obtained by performing a short-time Fourier transform on the original audio signal;

[0054] A second conversion result obtained by performing amplitude and phase transformation on the first conversion result;

[0055] A third conversion result is obtained by performing amplitude-angle conversion on the first conversion result.

[0056] The domain converter in the above audio coding model can be used as a component of LLM to implement multiple dimensional domain conversion processing on the original audio signal to obtain a conversion result.

[0057] Specifically, by performing a short-time Fourier transform on the original audio signal, a first conversion result can be obtained. Short-time Fourier transform (STFT) is a method of performing time-frequency analysis on a signal, which can decompose the original audio signal x into a series of short-time segments whose spectra vary with time, that is, obtain the short-time frequency domain X.

[0058] X = STFT(x)

[0059] Specifically, the original audio signal is divided into multiple overlapping time windows. The signal in each time window is called a frame. The frame length is usually selected as a fixed value, such as 10 milliseconds. Then, each frame is windowed. Commonly used window functions include Hamming window, rectangular window, etc., and then each frame is Fourier transformed to convert the time domain signal into a frequency domain signal, thereby obtaining a spectrum representation of each frame, that is, a spectrogram. The first conversion result is the spectrogram of each frame, which can be used to analyze the energy distribution or frequency contour of the signal at different frequencies.

[0060] The amplitude-phase spectrum can be obtained by performing an amplitude-phase transformation on the first conversion result X, that is, the second conversion result X can be obtained. mag,pha :

[0061]

[0062] Among them, X i ,X r represent the real and imaginary parts of the complex spectrum respectively.

[0063] The first conversion result X is subjected to an amplitude-angle transformation to obtain an amplitude-angle spectrum, that is, the third conversion result X is obtained. mag,ang :

[0064] X mag,ang =log(|X|),angle(X i ,X r )

[0065] Based on the above optional embodiment, a domain converter is used to perform multi-dimensional domain conversion processing on the original audio signal to obtain a conversion result, thereby quickly extracting the spectral characteristics of the original audio signal and further improving the processing efficiency of the original audio signal.

[0066] In an optional embodiment, the audio signal coding model also includes: an encoder, which performs audio coding processing on the conversion result in step S23 to obtain the coding result, including: using the encoder to perform audio coding processing on the conversion result to obtain the coding result, wherein the coding result is used to determine the hidden layer representation obtained by encoding the time domain signal and the frequency domain signal in multiple representation forms.

[0067] The encoder in the above audio coding model can be used as a component of LLM to implement audio coding processing on the conversion result to obtain a coding result.

[0068] In an optional embodiment, the audio signal coding model also includes: a quantizer, and the audio signal processing method in the embodiment of the present application also includes: using a quantizer to perform layer-by-layer quantization processing on the coding result to obtain a quantization result, wherein the quantization result is used to represent the discrete code corresponding to the original audio signal.

[0069] The quantizer in the above audio coding model can be used as a component of LLM to implement layer-by-layer quantization processing of the coding result to obtain a quantized result.

[0070] Specifically, the conversion result is subjected to audio coding processing, so that a relatively compact hidden layer representation obtained by coding time domain signals and frequency domain signals in various representation forms can be obtained for audio processing, transmission or storage. The compact hidden layer representation is then quantized layer by layer to obtain multiple groups of discrete codes of equal length to represent the original audio signal.

[0071] Based on the above optional embodiment, an encoder is used to perform audio encoding processing on the conversion result to obtain an encoding result, and then the encoding result is quantized layer by layer to obtain a quantization result, thereby converting the original audio signal into a more compact and efficient representation, thereby reducing the amount of data and improving the transmission and storage efficiency of the audio signal.

[0072] In an optional embodiment, in step S23, audio coding is performed on the conversion result to obtain the coding result, which includes: performing multi-layer down-sampling convolution coding on frequency domain signals in multiple representation forms in the conversion result to obtain the coding result.

[0073] Specifically, for the audio data in the time domain form, the audio encoding process can refer to the audio encoding method in the related art, and the embodiments of the present application will not be repeated. For frequency domain signals in the form of short-time frequency domain, amplitude phase spectrum, amplitude angle spectrum, etc., multi-layer downsampling convolution encoding can be performed to obtain the encoding result. The specific encoder structure is shown in Table 1. In the encoder structure shown in Table 1, the network layer name (Layer Name) includes: domain transformation module (DomainTransformation, Domain Trans), pre-two-dimensional convolution (PreConv2D), encoder network block (EncBlock), two-dimensional convolution module 1 (Conv2D_1), two-dimensional convolution module 2 (Conv2D_2), down-sampled two-dimensional convolution (Conv2D_ds), dimension conversion module (Reshape), long short-term network layer (Long Short-TermMemory, LSTM), linear output layer (OutLinear). Table 1 also includes the input (Inputs), output (Outputs), network layer parameters (Kernel), and network layer stride (Stride) of the encoder structure. In Table 1, T is used to represent the length of the sequence after the domain conversion module, C_b represents the number of channels of the convolution module in the b-th encoder network block, and T b represents the length of the sequence representing the output of the convolutional module in the b-th encoder network block, F b represents the frequency dimension size of the convolutional module in the b-th encoder network block, S b T represents the stride size of the convolutional module in the b-th encoder network block. B represents the sequence length after B encoder network blocks, and D represents the vector dimension of the linear output layer output.

[0074] Table 1 Encoder structure

[0075]

[0076]

[0077] By stacking multiple layers of downsampling convolution Conv2D_ds in a large model, the temporal resolution can be halved, and the quantizer resolution can be doubled at the same bit rate, thereby obtaining better encoding quality and increasing the encoding rate.

[0078] In an optional embodiment, in step S232, the encoding result is quantized layer by layer to obtain the quantized result, including:

[0079] Step S2321, determining an initial codebook to be used according to a preset data selection method;

[0080] Step S2322, in response to the activation times of some codewords in the initial codebook in the unit cluster data being lower than a preset threshold, updating some codewords to obtain a target codebook;

[0081] Step S2323: Perform layer-by-layer residual quantization processing on the encoding result based on the target codebook to obtain a quantization result.

[0082] Specifically, the above-mentioned preset data selection method can be a K-means clustering method, which uses the K-means clustering method to cluster the first unit clustering data (mini-batch) in the codebook, so that the obtained cluster center is used as the initial codebook to be used, and then updated by exponential moving average (EMA). If the activation number of a codeword in a mini-batch is lower than the preset threshold, it is reassigned to obtain the target codebook, that is, the quantization codebook. Using the K-mean clustering result instead of a random number to initialize the codebook of the multi-layer quantizer, and reassigning the codewords with lower activation in the subsequent update process can effectively improve the utilization rate of the codebook.

[0083] Furthermore, the residual vector quantization (RVQ) is used to perform residual quantization on the coding result layer by layer based on the target codebook to obtain the quantization result Q n .

[0084]

[0085] Among them, Q n represents the output of the nth vector quantizer (VQ), and Q0 represents the input of RVQ.

[0086] RVQ quantizes the data into discrete codewords by dividing the input data into multiple sub-vectors and matching each sub-vector with a vector in a quantization codebook. RVQ differs from traditional vector quantization algorithms in that it uses a residual vector to represent the difference between the input data and the quantization codebook, which can effectively reduce the compression error and improve the compression ratio.

[0087] Based on the above optional embodiments, the initial codebook to be used is determined according to a preset data selection method, and then in response to the activation times of some codewords in the initial codebook in the unit clustering data being lower than a preset threshold, some codewords are updated to obtain a target codebook, and finally, the encoding result is subjected to layer-by-layer residual quantization processing based on the target codebook to obtain a quantization result, which can further improve the codebook utilization rate, and by reducing the time resolution, the resolution of the quantizer is correspondingly improved, thereby obtaining a higher encoding utilization rate.

[0088] In an optional embodiment, the audio signal processing method in the embodiment of the present application further includes: in response to the original audio signal containing semantic information, extracting semantic information from the original audio signal by performing semantic extraction processing on the encoding result.

[0089] Specifically, when the original audio signal is a speech signal containing semantic information, the encoding result is processed into semantic tokens to extract semantic information from the speech signal. This can reduce the content that needs to be modeled by the encoder. Only the acoustic information other than the semantic information in the speech signal needs to be focused on, such as timbre, pitch, etc., thereby improving the encoding efficiency through semantic decoupling.

[0090] In an optional embodiment, in step S2323, the encoding result is subjected to layer-by-layer residual quantization based on the target codebook to obtain the quantization result, including: based on the target codebook and a preset semantic augmentation method, the encoding result and the semantic information are subjected to layer-by-layer residual quantization to obtain the quantization result.

[0091] Specifically, in the embodiment of the present application, for the semantic information of the speech signal, the target codebook and the following three preset semantic augmentation methods can be used to perform layer-by-layer residual quantization processing on the encoding result and the semantic information to obtain the quantization result:

[0092] f cat (V a ,V s )=Concat(RVQ(V a ),V s )

[0093] f add (V a ,V s )=RVQ(V a )+V s

[0094] f res (V a ,V s )=RVQ(V a -V s )+V s

[0095] Among them, V a Represents the output of the encoder, V s Represents semantic information, including but not limited to semantically quantized tokens such as frame-aligned monophone categories.

[0096] Based on the above optional embodiment, based on the target codebook and the preset semantic augmentation method, the encoding result and the semantic information are subjected to layer-by-layer residual quantization processing to obtain a quantization result, which can simultaneously utilize the semantic information and acoustic information contained in the speech signal to further improve the quantization processing effect.

[0097] In an optional embodiment, the audio signal processing method in the embodiment of the present application further includes:

[0098] Step S24, performing audio decoding processing on the quantization result to obtain a decoding result, wherein the audio decoding structure corresponding to the decoding result and the audio encoding structure corresponding to the encoding result are mirror images of each other;

[0099] Step S25, performing domain inversion processing on the decoding result to obtain a reconstructed audio signal.

[0100] Specifically, the audio decoding structure and the audio encoding structure are mirror images of each other, and multiple groups of discrete codes in the quantization results are subjected to audio decoding processing, so as to obtain audio signals in different representation forms. Furthermore, the audio signals in different representation forms are subjected to domain inversion processing, so as to restore them to reconstructed audio signals that can be perceived by the human ear.

[0101] Based on the above optional embodiment, the quantization result is subjected to audio decoding processing to obtain a decoding result, and the decoding result is then subjected to domain inversion processing to obtain a reconstructed audio signal, so that the original audio signal can be reliably restored in a preset application scenario.

[0102] Optionally, a graphical user interface is provided by a terminal device, and the content displayed by the graphical user interface at least partially includes an e-commerce service query scenario. The audio signal processing method in the embodiment of the present application further includes:

[0103] Step S31, responding to the control operation performed on the voice recording control, recording the original voice of the e-commerce service query;

[0104] Step S32, in response to the control operation performed on the query control, performing domain conversion processing on the e-commerce service query original voice to obtain an e-commerce service query conversion result, and performing audio encoding processing on the e-commerce service query conversion result to obtain an e-commerce service query encoding result;

[0105] Step S33, receiving e-commerce service query description information from the cloud service end, wherein the e-commerce service query description information is determined based on the e-commerce service query reconstructed voice, and the e-commerce service query reconstructed voice is obtained by performing domain inversion processing on the e-commerce service query encoding result;

[0106] Step S34, displaying the e-commerce service query description information in the graphical user interface.

[0107] The above-mentioned voice input control allows users to interact with the computer through voice input, and can be used to set up voice recognition applications, intelligent assistants and voice control systems to provide a more natural and convenient input method, allowing users to interact with the computer without a keyboard or mouse. The above-mentioned control operations performed on the voice input control include but are not limited to single-click, long-press and other operations, so as to input the original voice of the e-commerce service query.

[0108] The above-mentioned query control can be a graphical button, check box, text box, drop-down menu, etc., or a non-graphical scroll bar, progress bar, etc. The user can interact with the query control to trigger domain conversion processing of the original voice of the e-commerce service query to obtain the e-commerce service query conversion result, and audio encoding processing of the e-commerce service query conversion result to obtain the e-commerce service query encoding result.

[0109] The e-commerce service query description information may be product introduction information, which can be determined based on the e-commerce service query reconstructed voice, where the e-commerce service query reconstructed voice is obtained by performing domain inversion processing on the e-commerce service query encoding result.

[0110] Based on the above optional embodiments, by responding to the control operation performed on the voice input control, the original voice of the e-commerce service query is recorded, and then responding to the control operation performed on the query control, the domain conversion processing is performed on the original voice of the e-commerce service query to obtain the e-commerce service query conversion result, and the audio encoding processing is performed on the e-commerce service query conversion result to obtain the e-commerce service query encoding result, and then the e-commerce service query description information is received from the cloud service end, and the e-commerce service query description information is displayed in the graphical user interface, which can improve the user's query experience and convenience.

[0111] Figure 3 is a schematic diagram of an audio signal processing method according to Embodiment 1 of the present application, such as Figure 3 As shown, after obtaining the original audio signal, the domain conversion module is used to perform domain conversion processing on the original audio signal to obtain time domain signals and frequency domain signals in various representation forms, and then these signals are input into the encoder for audio encoding processing to obtain the encoding result. If the original audio signal contains semantic information, the semantic information is extracted from the original audio signal by performing semantic extraction processing on the encoding result, and then the encoding result and the semantic information are subjected to layer-by-layer residual quantization processing based on the target codebook and the preset semantic augmentation method to obtain the quantization result. Finally, the quantization result is subjected to audio decoding processing to obtain the decoding result, wherein the audio decoding structure corresponding to the decoding result and the audio encoding structure corresponding to the encoding result are mirror structures of each other, and the decoding result is subjected to domain inversion processing to obtain the reconstructed audio signal.

[0112] The domain conversion module and domain inversion module in the embodiment of the present application can change the audio signal into a more structural short-time frequency domain, amplitude phase spectrum and amplitude angle spectrum, and other representation forms. In addition, the use of an encoder based on two-dimensional convolution for audio encoding processing can more effectively utilize the time-frequency structure of the audio signal. By changing the representation form of the audio signal and superimposing more downsampling convolution modules, the time resolution is reduced, and the resolution of the quantizer is correspondingly improved, thereby obtaining a higher coding utilization rate.

[0113] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0114] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0115] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, disk, CD), including several instructions for enabling a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0116] Example 2

[0117] Figure 4 FIG. 1 is a flowchart of an audio signal processing method according to Embodiment 2 of the present application. Figure 4 As shown, the method may include the following steps:

[0118] Step S41, obtaining an encoding result, wherein the encoding result is used to restore the original audio signal in a preset application scenario, the encoding result is obtained by performing audio encoding processing on the conversion result, the conversion result is obtained by performing domain conversion processing on the original audio signal, and the conversion result is used to represent a time domain signal and a frequency domain signal in a plurality of representation forms obtained by converting the original audio signal;

[0119] Step S42: Perform speech reconstruction processing on the encoding result to obtain a reconstructed audio signal.

[0120] Based on the above steps S41 to S42, by obtaining the encoding result and then performing speech reconstruction processing on the encoding result, a reconstructed audio signal is obtained, thereby achieving the purpose of reliably restoring the original audio signal in a preset application scenario, thereby achieving the technical effect of improving the encoding utilization rate and encoding quality of the audio signal, and thus solving the technical problems of low encoding utilization rate and poor encoding quality when performing quantization encoding processing on audio signals in related technologies.

[0121] The above-mentioned audio signal processing method provided in the embodiment of the present application can be applied to, but is not limited to, application scenarios involving voice communication, voice storage, and large voice models in the fields of e-commerce services, educational services, legal services, medical services, conference services, social network services, financial product services, logistics services, and navigation services, and the embodiment of the present application is not limited.

[0122] The audio signal processing method in the embodiment of the present application is further introduced below.

[0123] In an optional embodiment, the audio signal processing method is applied to an audio signal decoding model, the audio signal decoding model includes: a decoder and a domain inverter, in step S42, the encoding result is subjected to speech reconstruction processing, and the reconstructed audio signal includes:

[0124] Step S421, using a decoder to perform audio decoding processing on the encoding result to obtain a decoding result, wherein the audio decoding structure corresponding to the decoding result and the audio encoding structure corresponding to the encoding result are mirror images of each other;

[0125] Step S422: Use a domain invertor to perform domain inversion processing on the decoding result to obtain a reconstructed audio signal.

[0126] Based on the above optional embodiment, by performing audio decoding processing on the encoding result to obtain a decoding result, and then performing domain inversion processing on the decoding result to obtain a reconstructed audio signal, the original audio signal can be reliably restored in a preset application scenario.

[0127] For the parts not described in detail in the above embodiments of the present application, please refer to the relevant description of Embodiment 1 and will not be described in detail here.

[0128] Example 3

[0129] Figure 5 is a flow chart of an audio signal processing method according to Embodiment 3 of the present application. Figure 5 As shown, the method may include the following steps:

[0130] Step S51, obtaining an original audio signal;

[0131] Step S52, using the target audio signal processing model to perform domain conversion processing on the original audio signal to obtain a conversion result, performing audio encoding processing on the conversion result to obtain an encoding result, and performing speech reconstruction processing on the encoding result to obtain a reconstructed audio signal, wherein the conversion result is used to represent time domain signals and frequency domain signals in multiple representation forms obtained by converting the original audio signal, and the encoding result is used to restore the original audio signal in a preset application scenario;

[0132] Step S53: play the reconstructed audio signal.

[0133] Based on the above steps S51 to S53, by acquiring the original audio signal, and then using the target audio signal processing model to perform domain conversion processing on the original audio signal to obtain a conversion result, the conversion result is subjected to audio encoding processing to obtain an encoding result, and the encoding result is subjected to speech reconstruction processing to obtain a reconstructed audio signal, and finally the reconstructed audio signal is played, thereby achieving the technical effect of improving the encoding utilization rate and encoding quality of the audio signal, and thus solving the technical problems of low encoding utilization rate and poor encoding quality when performing quantization encoding processing on the audio signal in the related art.

[0134] The above-mentioned audio signal processing method provided in the embodiment of the present application can be applied to, but is not limited to, application scenarios involving voice communication, voice storage, and large voice models in the fields of e-commerce services, educational services, legal services, medical services, conference services, social network services, financial product services, logistics services, and navigation services, and the embodiment of the present application is not limited.

[0135] The audio signal processing method in the embodiment of the present application is further introduced below.

[0136] In an optional embodiment, the audio signal processing method in the embodiment of the present application further includes:

[0137] Step S521, using the initial audio signal processing model to perform audio reconstruction processing on the original training signal to obtain a reconstructed training signal;

[0138] Step S522, determining a target loss based on the original training signal and the reconstructed training signal, wherein the target loss includes at least one of the following: a distance loss between the original training signal and the reconstructed training signal, a frequency domain reconstruction loss, and a discriminator adversarial loss;

[0139] Step S523: Update the model parameters of the initial audio signal processing model by using the target loss to generate a target audio signal processing model.

[0140] Specifically, the target loss includes at least one of the following: the distance loss L between the original training signal and the reconstructed training signal t , frequency domain reconstruction loss L f , discriminator adversarial loss L adv / L feat , the quantization loss L of the similarity between the constrained quantization result and the corresponding encoding result cm ,Specifically, the calculation process of the target loss L is as follows:

[0141] L=λ t L t +λ f L f +λ adv L adv +λ feat L feat +λ cm L cm

[0142] Wherein, λ is the loss weight. For the frequency domain reconstruction loss, the present embodiment adds the reconstruction loss of the energy spectrum domain S, thereby increasing the attention of the target audio signal processing model to the mid-high frequency signals and improving the overall audio quality. The frequency domain reconstruction loss L f Obtained through the following calculation process:

[0143]

[0144] Among them, S i and M i Represents logarithmic compression power and Mel spectrum.

[0145] Discriminator adversarial loss L adv / L feat Obtained through the following calculation process:

[0146]

[0147]

[0148] Among them, D k,t represents the output of discriminator k at time step t.

[0149] The quantization loss L of the similarity between the constrained quantization result and the corresponding encoding result cm It can be obtained through the following calculation process:

[0150]

[0151] Wherein, V represents the input of RVQ.

[0152] Based on the above optional embodiments, by adding multiple losses and optimizing the multi-layer quantizer, the training process can be effectively streamlined and the training efficiency can be further improved.

[0153] For the parts not described in detail in the above embodiments of the present application, please refer to the relevant description of Embodiment 1 and will not be described in detail here.

[0154] Example 4

[0155] According to an embodiment of the present application, there is also provided an audio signal processing device for implementing the above audio signal processing method. Figure 6 is a result block diagram of an audio signal processing device according to Embodiment 1 of the present application, such as Figure 6 As shown, the device comprises:

[0156] An acquisition module 601 is used to acquire an original audio signal;

[0157] The conversion module 602 is used to perform domain conversion processing on the original audio signal to obtain a conversion result, wherein the conversion result is used to represent a time domain signal and a frequency domain signal in a plurality of representation forms obtained by converting the original audio signal;

[0158] The encoding module 603 is used to perform audio encoding processing on the conversion result to obtain an encoding result, wherein the encoding result is used to restore the original audio signal in a preset application scenario.

[0159] Optionally, the conversion module 602 is also used to: use a domain converter to perform multiple dimensional domain conversion processing on the original audio signal to obtain a conversion result, wherein the conversion result includes some or all of the following conversion results: a first conversion result obtained by performing a short-time Fourier transform on the original audio signal; a second conversion result obtained by performing an amplitude-phase transform on the first conversion result; and a third conversion result obtained by performing an amplitude-angle transform on the first conversion result.

[0160] Optionally, the encoding module 603 is further used to: use an encoder to perform audio encoding processing on the conversion result to obtain an encoding result, wherein the encoding result is used to determine a hidden layer representation obtained by encoding time domain signals and frequency domain signals in multiple representation forms.

[0161] Optionally, the audio signal processing apparatus further includes a quantization module 604, which is used to use a quantizer to perform layer-by-layer quantization processing on the encoding result to obtain a quantization result, wherein the quantization result is used to represent a discrete code corresponding to the original audio signal.

[0162] Optionally, the encoding module 603 is further configured to perform multi-layer down-sampling convolution encoding processing on frequency domain signals in various representation forms in the conversion result to obtain an encoding result.

[0163] Optionally, the quantization module 604 is also used to: determine the initial codebook to be used according to a preset data selection method; in response to the activation number of some codewords in the initial codebook in the unit clustering data being lower than a preset threshold, update some codewords to obtain a target codebook; and perform layer-by-layer residual quantization processing on the encoding result based on the target codebook to obtain a quantization result.

[0164] Optionally, the audio signal processing apparatus further includes: a processing module 605, configured to extract semantic information from the original audio signal by performing semantic extraction processing on the encoding result in response to the original audio signal containing semantic information.

[0165] Optionally, the encoding module 603 is further configured to: perform layer-by-layer residual quantization processing on the encoding result and the semantic information based on the target codebook and a preset semantic augmentation method to obtain a quantization result.

[0166] Optionally, the processing module 605 is also used to: perform audio decoding processing on the quantization result to obtain a decoding result, wherein the audio decoding structure corresponding to the decoding result and the audio encoding structure corresponding to the encoding result are mirror images of each other; and perform domain inversion processing on the decoding result to obtain a reconstructed audio signal.

[0167] Optionally, the audio signal processing device also includes: an input module 605, which is used to respond to the control operation performed on the voice input control and input the original voice of the e-commerce service query; the processing module 604 is also used to respond to the control operation performed on the query control, perform domain conversion processing on the original voice of the e-commerce service query to obtain an e-commerce service query conversion result, and perform audio encoding processing on the e-commerce service query conversion result to obtain an e-commerce service query encoding result; the audio signal processing device also includes: a receiving module 606, which is used to receive e-commerce service query description information from the cloud service end, wherein the e-commerce service query description information is determined based on the reconstructed voice of the e-commerce service query, and the reconstructed voice of the e-commerce service query is obtained by performing domain inversion processing on the e-commerce service query encoding result; a display module 607, which is used to display the e-commerce service query description information in a graphical user interface.

[0168] In an embodiment of the present application, by acquiring an original audio signal, performing domain conversion processing on the original audio signal, obtaining a conversion result, and finally performing audio encoding processing on the conversion result to obtain an encoding result, the purpose of reliably restoring the original audio signal in a preset application scenario is achieved, thereby achieving the technical effect of improving the encoding utilization rate and encoding quality of the audio signal, and further solving the technical problems of low encoding utilization rate and poor encoding quality when performing quantization encoding processing on audio signals in related technologies.

[0169] It should be noted that the acquisition module 601, the conversion module 602 and the encoding module 603 correspond to steps S21 to S23 in Example 1, and the three modules and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above-mentioned Example 2. It should be noted that the above-mentioned modules or units may be hardware components or software components stored in a memory and processed by one or more processors, and the above-mentioned modules may also be part of the device and run in the computer terminal provided in Example 1.

[0170] It should be noted that the preferred implementation scheme involved in the above embodiments of the present application is the same as the scheme provided in Example 1, as well as the application scenario and implementation process, but is not limited to the scheme provided in Example 1.

[0171] Figure 7 is a result block diagram of an audio signal processing device according to Embodiment 2 of the present application, such as Figure 7 As shown, the device comprises:

[0172] An acquisition module 701 is used to acquire an encoding result, wherein the encoding result is used to restore the original audio signal in a preset application scenario, the encoding result is obtained by performing audio encoding processing on the conversion result, the conversion result is obtained by performing domain conversion processing on the original audio signal, and the conversion result is used to represent a time domain signal and a frequency domain signal in a plurality of representation forms obtained by converting the original audio signal;

[0173] The reconstruction module 702 is used to perform speech reconstruction processing on the encoding result to obtain a reconstructed audio signal.

[0174] Optionally, the reconstruction module 702 is also used to: use a decryptor to perform audio decoding processing on the encoding result to obtain a decoding result, wherein the audio decoding structure corresponding to the decoding result and the audio encoding structure corresponding to the encoding result are mirror structures of each other; use a domain inverter to perform domain inversion processing on the decoding result to obtain a reconstructed audio signal.

[0175] In the embodiment of the present application, by obtaining the encoding result and then performing speech reconstruction processing on the encoding result, a reconstructed audio signal is obtained, thereby achieving the purpose of reliably restoring the original audio signal in a preset application scenario, thereby achieving the technical effect of improving the encoding utilization rate and encoding quality of the audio signal, and thus solving the technical problems of low encoding utilization rate and poor encoding quality when performing quantization encoding processing on audio signals in related technologies.

[0176] It should be noted that the acquisition module 701 and the reconstruction module 702 correspond to steps S41 to S42 in Example 2, and the examples and application scenarios implemented by the two modules and the corresponding steps are the same, but are not limited to the contents disclosed in the above-mentioned Example 2. It should be noted that the above-mentioned modules or units may be hardware components or software components stored in a memory and processed by one or more processors, and the above-mentioned modules may also be part of the device and run in the computer terminal provided in Example 2.

[0177] It should be noted that the preferred implementation scheme involved in the above embodiments of the present application is the same as the scheme provided in Example 2 as well as the application scenario and implementation process, but is not limited to the scheme provided in Example 2.

[0178] Figure 8 is a result block diagram of an audio signal processing device according to Embodiment 3 of the present application, such as Figure 8 As shown, the device comprises:

[0179] An acquisition module 801 is used to acquire an original audio signal;

[0180] A processing module 802 is used to perform domain conversion processing on the original audio signal using a target audio signal processing model to obtain a conversion result, perform audio encoding processing on the conversion result to obtain an encoding result, and perform speech reconstruction processing on the encoding result to obtain a reconstructed audio signal, wherein the conversion result is used to represent a time domain signal and a frequency domain signal in a plurality of representation forms obtained by converting the original audio signal, and the encoding result is used to restore the original audio signal in a preset application scenario;

[0181] The playing module 803 is used to play the reconstructed audio signal.

[0182] Optionally, the audio signal processing device also includes: a reconstruction module 804, which is used to use the initial audio signal processing model to perform audio reconstruction processing on the original training signal to obtain a reconstructed training signal; a determination module 805, which is used to determine the target loss based on the original training signal and the reconstructed training signal, wherein the target loss includes at least one of the following: distance loss between the original training signal and the reconstructed training signal, frequency domain reconstruction loss, and discriminator adversarial loss; an update module 806, which is used to update the model parameters of the initial audio signal processing model through the target loss to generate a target audio signal processing model.

[0183] In an embodiment of the present application, an original audio signal is obtained, and then a target audio signal processing model is used to perform domain conversion processing on the original audio signal to obtain a conversion result, the conversion result is subjected to audio encoding processing to obtain an encoding result, and the encoding result is subjected to speech reconstruction processing to obtain a reconstructed audio signal, and finally the reconstructed audio signal is played, thereby achieving the technical effect of improving the encoding utilization rate and encoding quality of the audio signal, thereby solving the technical problems of low encoding utilization rate and poor encoding quality when performing quantization encoding processing on the audio signal in the related art.

[0184] It should be noted that the acquisition module 801, the processing module 802 and the playback module 803 correspond to steps S51 to S53 in Example 3, and the examples and application scenarios implemented by the two modules and the corresponding steps are the same, but are not limited to the contents disclosed in the above-mentioned Example 3. It should be noted that the above-mentioned modules or units may be hardware components or software components stored in a memory and processed by one or more processors, and the above-mentioned modules may also be part of the device and run in the computer terminal provided in Example 1.

[0185] It should be noted that the preferred implementation scheme involved in the above embodiments of the present application is the same as the scheme provided in Example 3, as well as the application scenario and implementation process, but is not limited to the scheme provided in Example 3.

[0186] Example 5

[0187] The embodiment of the present application may provide a computer terminal, which may be any computer terminal device in a computer terminal group. Optionally, in this embodiment, the computer terminal may also be replaced by a terminal device such as a mobile terminal.

[0188] Optionally, in this embodiment, the computer terminal may be located in at least one network device among a plurality of network devices of the computer network.

[0189] In this embodiment, the above-mentioned computer terminal can execute the program code of the following steps in the audio signal processing method: obtaining an original audio signal; performing domain conversion processing on the original audio signal to obtain a conversion result, wherein the conversion result is used to represent time domain signals and frequency domain signals in various representation forms obtained by converting the original audio signal; performing audio encoding processing on the conversion result to obtain an encoding result, wherein the encoding result is used to restore the original audio signal in a preset application scenario.

[0190] In this embodiment, the above-mentioned computer terminal can execute the program code of the following steps in the audio signal processing method: obtaining the encoding result, wherein the encoding result is used to restore the original audio signal in a preset application scenario, the encoding result is obtained by performing audio encoding processing on the conversion result, the conversion result is obtained by performing domain conversion processing on the original audio signal, and the conversion result is used to represent time domain signals and frequency domain signals in various representation forms obtained by converting the original audio signal; performing speech reconstruction processing on the encoding result to obtain a reconstructed audio signal.

[0191] In this embodiment, the above-mentioned computer terminal can execute the program code of the following steps in the audio signal processing method: obtaining an original audio signal; using a target audio signal processing model to perform domain conversion processing on the original audio signal to obtain a conversion result, performing audio encoding processing on the conversion result to obtain an encoding result, and performing speech reconstruction processing on the encoding result to obtain a reconstructed audio signal, wherein the conversion result is used to represent time domain signals and frequency domain signals in various representation forms obtained by converting the original audio signal, and the encoding result is used to restore the original audio signal under a preset application scenario; and playing the reconstructed audio signal.

[0192] Optionally, Fig. 9 1 is a block diagram of a computer terminal according to an embodiment of the present application. As shown in the figure, the computer terminal 90 may include: one or more (only one is shown in the figure) processors 92, a memory 94, a storage controller, and a peripheral interface, wherein the peripheral interface is connected to a radio frequency module, an audio module, and a display.

[0193] Among them, the memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the audio signal processing method and device in the embodiment of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, realizing the above-mentioned audio signal processing method. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories can be connected to the computer terminal 90 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0194] The processor can call the information and application stored in the memory through the transmission device to execute the following steps: obtain the original audio signal; perform domain conversion processing on the original audio signal to obtain a conversion result, wherein the conversion result is used to represent time domain signals and frequency domain signals in various representation forms obtained by converting the original audio signal; perform audio encoding processing on the conversion result to obtain an encoding result, wherein the encoding result is used to restore the original audio signal in a preset application scenario.

[0195] Optionally, the processor may also execute the program code of the following steps: using a domain converter to perform multiple dimensional domain conversion processing on the original audio signal to obtain a conversion result, wherein the conversion result includes some or all of the following conversion results: a first conversion result obtained by performing a short-time Fourier transform on the original audio signal; a second conversion result obtained by performing an amplitude-phase transform on the first conversion result; and a third conversion result obtained by performing an amplitude-angle transform on the first conversion result.

[0196] Optionally, the processor may also execute the program code of the following steps: using an encoder to perform audio encoding processing on the conversion result to obtain an encoding result, wherein the encoding result is used to determine a hidden layer representation obtained by encoding time domain signals and frequency domain signals in multiple representation forms.

[0197] Optionally, the processor may also execute the program code of the following steps: using a quantizer to perform layer-by-layer quantization processing on the encoding result to obtain a quantization result, wherein the quantization result is used to represent the discrete code corresponding to the original audio signal.

[0198] Optionally, the processor may also execute the program code of the following steps: performing multi-layer down-sampling convolution coding processing on frequency domain signals in various representation forms in the conversion result to obtain a coding result.

[0199] Optionally, the processor may also execute the program code of the following steps: determining an initial codebook to be used according to a preset data selection method; in response to the activation times of some codewords in the initial codebook in the unit clustering data being lower than a preset threshold, updating some codewords to obtain a target codebook; and performing layer-by-layer residual quantization processing on the encoding result based on the target codebook to obtain a quantization result.

[0200] Optionally, the processor may further execute program code of the following steps: in response to the original audio signal containing semantic information, extracting semantic information from the original audio signal by performing semantic extraction processing on the encoding result.

[0201] Optionally, the processor may also execute the program code of the following steps: based on the target codebook and a preset semantic augmentation method, performing layer-by-layer residual quantization processing on the encoding result and the semantic information to obtain a quantization result.

[0202] Optionally, the processor may also execute the program code of the following steps: performing audio decoding processing on the quantization result to obtain a decoding result, wherein the audio decoding structure corresponding to the decoding result and the audio encoding structure corresponding to the encoding result are mirror images of each other; performing domain inversion processing on the decoding result to obtain a reconstructed audio signal.

[0203] Optionally, the processor may also execute the program code of the following steps: in response to a control operation performed on a voice entry control, input the original voice of an e-commerce service query; in response to a control operation performed on a query control, perform domain conversion processing on the original voice of the e-commerce service query to obtain an e-commerce service query conversion result, and perform audio encoding processing on the e-commerce service query conversion result to obtain an e-commerce service query encoding result; receive e-commerce service query description information from a cloud service end, wherein the e-commerce service query description information is determined based on the reconstructed voice of the e-commerce service query, and the reconstructed voice of the e-commerce service query is obtained by performing domain inversion processing on the e-commerce service query encoding result; and display the e-commerce service query description information in a graphical user interface.

[0204] Optionally, the processor may also execute the program code of the following steps: obtaining a coding result, wherein the coding result is used to restore the original audio signal in a preset application scenario, the coding result is obtained by performing audio coding processing on the conversion result, the conversion result is obtained by performing domain conversion processing on the original audio signal, and the conversion result is used to represent time domain signals and frequency domain signals in various representation forms obtained by converting the original audio signal; performing speech reconstruction processing on the coding result to obtain a reconstructed audio signal.

[0205] Optionally, the processor may also execute the program code of the following steps: using a decoder to perform audio decoding processing on the encoding result to obtain a decoding result, wherein the audio decoding structure corresponding to the decoding result and the audio encoding structure corresponding to the encoding result are mirror images of each other; using a domain inverter to perform domain inversion processing on the decoding result to obtain a reconstructed audio signal.

[0206] Optionally, the processor may also execute the program code of the following steps: obtaining an original audio signal; performing domain conversion processing on the original audio signal using a target audio signal processing model to obtain a conversion result, performing audio encoding processing on the conversion result to obtain an encoding result, and performing speech reconstruction processing on the encoding result to obtain a reconstructed audio signal, wherein the conversion result is used to represent time domain signals and frequency domain signals in various representation forms obtained by converting the original audio signal, and the encoding result is used to restore the original audio signal in a preset application scenario; and playing the reconstructed audio signal.

[0207] Optionally, the processor may also execute the program code of the following steps: performing audio reconstruction processing on the original training signal using the initial audio signal processing model to obtain a reconstructed training signal; determining a target loss based on the original training signal and the reconstructed training signal, wherein the target loss includes at least one of the following: a distance loss between the original training signal and the reconstructed training signal, a frequency domain reconstruction loss, and a discriminator adversarial loss; and updating the model parameters of the initial audio signal processing model using the target loss to generate a target audio signal processing model.

[0208] By adopting the embodiment of the present application, an original audio signal is obtained, and domain conversion processing is performed on the original audio signal to obtain a conversion result, and finally audio encoding processing is performed on the conversion result to obtain an encoding result, thereby achieving the purpose of reliably restoring the original audio signal in a preset application scenario, thereby achieving the technical effect of improving the encoding utilization rate and encoding quality of the audio signal, and further solving the technical problems of low encoding utilization rate and poor encoding quality when performing quantization encoding processing on audio signals in related technologies.

[0209] Those skilled in the art will appreciate that the structure shown in the figure is for illustration only, and the computer terminal may also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile Internet device (MID), a PAD, or other terminal devices. Fig. 9 It does not limit the structure of the above electronic device. For example, the computer terminal A may also include Fig. 9 More or fewer components (such as network interfaces, display devices, etc.) shown in, or having Fig. 9 Different configurations shown.

[0210] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0211] Example 6

[0212] The embodiment of the present application further provides a computer-readable storage medium. Optionally, in this embodiment, the computer-readable storage medium can be used to store the program code executed by the audio signal processing method provided in the first embodiment.

[0213] Optionally, in this embodiment, the above storage medium may be located in any computer terminal in a computer terminal group in a computer network, or in any mobile terminal in a mobile terminal group.

[0214] Optionally, in this embodiment, the storage medium is configured to store program codes for executing the following steps: obtaining an original audio signal; performing domain conversion processing on the original audio signal to obtain a conversion result, wherein the conversion result is used to represent time domain signals and frequency domain signals in multiple representation forms obtained by converting the original audio signal; performing audio encoding processing on the conversion result to obtain an encoding result, wherein the encoding result is used to restore the original audio signal in a preset application scenario.

[0215] Optionally, in this embodiment, the storage medium is configured to store program code for executing the following steps: using a domain converter to perform multiple dimensional domain conversion processing on the original audio signal to obtain a conversion result, wherein the conversion result includes some or all of the following conversion results: a first conversion result obtained by performing a short-time Fourier transform on the original audio signal; a second conversion result obtained by performing an amplitude-phase transform on the first conversion result; and a third conversion result obtained by performing an amplitude-angle transform on the first conversion result.

[0216] Optionally, in this embodiment, the storage medium is configured to store program code for executing the following steps: using an encoder to perform audio encoding processing on the conversion result to obtain an encoding result, wherein the encoding result is used to determine a hidden layer representation obtained by encoding time domain signals and frequency domain signals in multiple representation forms.

[0217] Optionally, in this embodiment, the storage medium is configured to store program codes for executing the following steps: using a quantizer to perform layer-by-layer quantization processing on the encoding result to obtain a quantization result, wherein the quantization result is used to represent a discrete code corresponding to the original audio signal.

[0218] Optionally, in this embodiment, the storage medium is configured to store program codes for executing the following steps: performing multi-layer down-sampling convolution coding processing on frequency domain signals in multiple representation forms in the conversion results to obtain coding results.

[0219] Optionally, in this embodiment, the storage medium is configured to store program codes for executing the following steps: determining an initial codebook to be used according to a preset data selection method; in response to the activation times of some codewords in the initial codebook in the unit clustering data being lower than a preset threshold, updating some codewords to obtain a target codebook; and performing layer-by-layer residual quantization processing on the encoding result based on the target codebook to obtain a quantization result.

[0220] Optionally, in this embodiment, the storage medium is configured to store program codes for executing the following steps: in response to the original audio signal containing semantic information, extracting semantic information from the original audio signal by performing semantic extraction processing on the encoding result.

[0221] Optionally, in this embodiment, the storage medium is configured to store program codes for executing the following steps: based on a target codebook and a preset semantic augmentation method, performing layer-by-layer residual quantization processing on the encoding result and the semantic information to obtain a quantization result.

[0222] Optionally, in this embodiment, the storage medium is configured to store program codes for executing the following steps: performing audio decoding processing on the quantization result to obtain a decoding result, wherein the audio decoding structure corresponding to the decoding result and the audio encoding structure corresponding to the encoding result are mirror images of each other; performing domain inversion processing on the decoding result to obtain a reconstructed audio signal.

[0223] Optionally, in this embodiment, the storage medium is configured to store program codes for executing the following steps: in response to a control operation performed on a voice entry control, entering an original voice of an e-commerce service query; in response to a control operation performed on a query control, performing domain conversion processing on the original voice of the e-commerce service query to obtain an e-commerce service query conversion result, and performing audio encoding processing on the e-commerce service query conversion result to obtain an e-commerce service query encoding result; receiving e-commerce service query description information from a cloud service end, wherein the e-commerce service query description information is determined based on the reconstructed voice of the e-commerce service query, and the reconstructed voice of the e-commerce service query is obtained by performing domain inversion processing on the e-commerce service query encoding result; and displaying the e-commerce service query description information in a graphical user interface.

[0224] Optionally, in this embodiment, the storage medium is configured to store program code for executing the following steps: obtaining an encoding result, wherein the encoding result is used to restore the original audio signal in a preset application scenario, the encoding result is obtained by performing audio encoding processing on the conversion result, the conversion result is obtained by performing domain conversion processing on the original audio signal, and the conversion result is used to represent time domain signals and frequency domain signals in multiple representation forms obtained by converting the original audio signal; performing speech reconstruction processing on the encoding result to obtain a reconstructed audio signal.

[0225] Optionally, in this embodiment, the storage medium is configured to store program codes for executing the following steps: using a decoder to perform audio decoding processing on the encoding result to obtain a decoding result, wherein the audio decoding structure corresponding to the decoding result and the audio encoding structure corresponding to the encoding result are mirror structures of each other; using a domain inverter to perform domain inversion processing on the decoding result to obtain a reconstructed audio signal.

[0226] Optionally, in this embodiment, the storage medium is configured to store program code for executing the following steps: obtaining an original audio signal; performing domain conversion processing on the original audio signal using a target audio signal processing model to obtain a conversion result, performing audio encoding processing on the conversion result to obtain an encoding result, and performing speech reconstruction processing on the encoding result to obtain a reconstructed audio signal, wherein the conversion result is used to represent time domain signals and frequency domain signals in multiple representation forms obtained by converting the original audio signal, and the encoding result is used to restore the original audio signal in a preset application scenario; and playing the reconstructed audio signal.

[0227] Optionally, in this embodiment, the storage medium is configured to store program codes for executing the following steps: performing audio reconstruction processing on the original training signal using an initial audio signal processing model to obtain a reconstructed training signal; determining a target loss based on the original training signal and the reconstructed training signal, wherein the target loss includes at least one of the following: a distance loss between the original training signal and the reconstructed training signal, a frequency domain reconstruction loss, and a discriminator adversarial loss; and updating the model parameters of the initial audio signal processing model using the target loss to generate a target audio signal processing model.

[0228] By adopting the embodiment of the present application, an original audio signal is obtained, and domain conversion processing is performed on the original audio signal to obtain a conversion result, and finally audio encoding processing is performed on the conversion result to obtain an encoding result, thereby achieving the purpose of reliably restoring the original audio signal in a preset application scenario, thereby achieving the technical effect of improving the encoding utilization rate and encoding quality of the audio signal, and further solving the technical problems of low encoding utilization rate and poor encoding quality when performing quantization encoding processing on audio signals in related technologies.

[0229] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0230] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0231] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0232] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0233] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0234] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0235] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. An audio signal processing method, characterized in that: include: Get the original audio signal; Performing domain conversion processing on the original audio signal to obtain a conversion result, wherein the conversion result is used to represent a time domain signal and a frequency domain signal in a plurality of representation forms obtained by converting the original audio signal; The conversion result is subjected to audio coding processing to obtain a coding result, wherein the coding result is used to restore the original audio signal in a preset application scenario.

2. The audio signal processing method according to claim 1, characterized in that: The audio signal processing method is applied to an audio signal coding model, and the audio signal coding model includes: a domain converter, which performs domain conversion processing on the original audio signal, and obtains the conversion result including: The domain converter is used to perform multiple dimensional domain conversion processing on the original audio signal to obtain the conversion result, wherein the conversion result includes some or all of the following conversion results: A first conversion result obtained by performing short-time Fourier transform on the original audio signal; a second conversion result obtained by performing amplitude and phase transformation on the first conversion result; A third conversion result is obtained by performing amplitude-angle conversion on the first conversion result.

3. The audio signal processing method according to claim 2, characterized in that: The audio signal coding model further includes: an encoder, which performs audio coding processing on the conversion result, and obtains the coding result including: The encoder is used to perform audio encoding processing on the conversion result to obtain an encoding result, wherein the encoding result is used to determine a hidden layer representation obtained by encoding the time domain signals and frequency domain signals in the multiple representation forms.

4. The audio signal processing method according to claim 3, characterized in that: The audio signal coding model further includes: a quantizer, and the audio signal processing method further includes: The quantizer is used to perform layer-by-layer quantization processing on the encoding result to obtain a quantization result, wherein the quantization result is used to represent the discrete code corresponding to the original audio signal.

5. The audio signal processing method according to claim 3, characterized in that: Performing audio coding processing on the conversion result to obtain the coding result includes: Multi-layer downsampling convolution coding is performed on frequency domain signals in various representation forms in the conversion result to obtain the coding result.

6. The audio signal processing method according to claim 4, characterized in that: The encoding result is quantized layer by layer to obtain the quantized result, including: Determine the initial codebook to be used according to a preset data selection method; In response to the activation times of some codewords in the initial codebook in the unit cluster data being lower than a preset threshold, updating the some codewords to obtain a target codebook; The encoding result is subjected to layer-by-layer residual quantization processing based on the target codebook to obtain the quantization result.

7. The audio signal processing method according to claim 6, characterized in that: The audio signal processing method further includes: In response to the original audio signal containing semantic information, the semantic information is extracted from the original audio signal by performing semantic extraction processing on the encoding result.

8. The audio signal processing method according to claim 7, characterized in that: Performing layer-by-layer residual quantization processing on the encoding result based on the target codebook to obtain the quantization result includes: Based on the target codebook and a preset semantic augmentation method, the encoding result and the semantic information are subjected to layer-by-layer residual quantization processing to obtain the quantization result.

9. The audio signal processing method according to claim 4, characterized in that: The audio signal processing method further includes: Performing audio decoding processing on the quantization result to obtain a decoding result, wherein an audio decoding structure corresponding to the decoding result and an audio encoding structure corresponding to the encoding result are mirror images of each other; The decoding result is subjected to domain inversion processing to obtain a reconstructed audio signal.

10. The audio signal processing method according to claim 1, characterized in that: A graphical user interface is provided through a terminal device, and the content displayed by the graphical user interface at least partially includes an e-commerce service query scenario, and the audio signal processing method further includes: In response to the control operation performed on the voice input control, the original voice of the e-commerce service query is input; In response to a control operation performed on a query control, performing domain conversion processing on the e-commerce service query original voice to obtain an e-commerce service query conversion result, and performing audio encoding processing on the e-commerce service query conversion result to obtain an e-commerce service query encoding result; Receiving e-commerce service query description information from a cloud service end, wherein the e-commerce service query description information is determined based on the e-commerce service query reconstructed speech, and the e-commerce service query reconstructed speech is obtained by performing domain inversion processing on the e-commerce service query encoding result; The e-commerce service query description information is displayed in the graphical user interface.

11. An audio signal processing method, characterized in that: include: Obtaining an encoding result, wherein the encoding result is used to restore the original audio signal in a preset application scenario, the encoding result is obtained by performing audio encoding processing on the conversion result, the conversion result is obtained by performing domain conversion processing on the original audio signal, and the conversion result is used to represent time domain signals and frequency domain signals in multiple representation forms obtained by converting the original audio signal; The encoding result is subjected to speech reconstruction processing to obtain a reconstructed audio signal.

12. The audio signal processing method according to claim 11, characterized in that: The audio signal processing method is applied to an audio signal decoding model, the audio signal decoding model includes: a decoder and a domain inverter, and the encoding result is subjected to speech reconstruction processing to obtain the reconstructed audio signal including: Performing audio decoding processing on the encoding result by using the decoder to obtain a decoding result, wherein the audio decoding structure corresponding to the decoding result and the audio encoding structure corresponding to the encoding result are mirror images of each other; The domain invertor is used to perform domain inversion processing on the decoding result to obtain the reconstructed audio signal.

13. An audio signal processing method, characterized in that: include: Get the original audio signal; The target audio signal processing model is used to perform domain conversion processing on the original audio signal to obtain a conversion result, the conversion result is subjected to audio encoding processing to obtain an encoding result, and the encoding result is subjected to speech reconstruction processing to obtain a reconstructed audio signal, wherein the conversion result is used to represent a time domain signal and a frequency domain signal in a plurality of representation forms obtained by converting the original audio signal, and the encoding result is used to restore the original audio signal in a preset application scenario; The reconstructed audio signal is played.

14. The audio signal processing method according to claim 13, characterized in that: The audio signal processing method further includes: Using the initial audio signal processing model to perform audio reconstruction processing on the original training signal to obtain a reconstructed training signal; Determining a target loss based on the original training signal and the reconstructed training signal, wherein the target loss includes at least one of the following: a distance loss between the original training signal and the reconstructed training signal, a frequency domain reconstruction loss, and a discriminator adversarial loss; The model parameters of the initial audio signal processing model are updated according to the target loss to generate the target audio signal processing model.

15. An electronic device, characterized in that: include: A memory storing an executable program; A processor is used to run the program, wherein the program executes the audio signal processing method according to any one of claims 1 to 14 when running.

16. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored executable program, wherein when the executable program is executed, the device where the computer-readable storage medium is located is controlled to execute the audio signal processing method according to any one of claims 1 to 14.

Images