Audio signal coding and decoding method and device based on neural network, equipment and medium

Abstract

The invention belongs to the technical field of signal processing, is suitable for the medical field and the financial field, and discloses an audio signal encoding and decoding method and device based on a neural network, equipment and a medium. The method comprises the following steps: respectively inputting an initial amplitude spectrum and an initial phase spectrum corresponding to an audio signal to be processed into an amplitude encoder and a phase encoder for parallel feature coding; inputting a target coding vector of the audio signal to be processed into a residual vector quantization module for quantization processing to obtain an amplitude implicit vector and a phase implicit vector of the audio signal to be processed, and respectively inputting the amplitude implicit vector and the phase implicit vector into an amplitude decoder and a phase decoder for parallel decoding, obtaining an initial amplitude spectrum and an initial phase spectrum; and performing inverse short-time Fourier transform processing on the initial amplitude spectrum and the initial phase spectrum of the audio signal to be processed to generate a reconstructed audio signal. According to the invention, the tone quality and efficiency of the audio codec at a low bit rate are effectively improved.

Description

Technical Field

[0001] The present invention belongs to the field of speech processing technology, is applicable to the medical field and the financial field, and in particular relates to a neural network-based audio signal encoding and decoding method, device, equipment and medium. Background Art

[0002] At present, audio compression technology has been widely used in many fields of daily life, especially in voice communication, online music streaming, and audio storage and transmission. For example, MP3, AAC, Opus and other codecs compress audio signals in the time domain or frequency domain through signal processing algorithms, thereby reducing the amount of data. In the financial field, for example, in financial customer service centers, a large number of voice calls need to be stored and processed. The use of audio compression technology can effectively reduce storage costs and transmission bandwidth and improve service efficiency. In the medical field, for example, in remote medical consultations, voice communication between doctors and patients needs to be transmitted over the network. Audio compression technology can ensure clear voice transmission, even in remote areas with poor network conditions, it can ensure smooth communication between doctors and patients.

[0003] Although these traditional audio codecs have achieved certain results in practical applications, they still have some shortcomings in terms of sound quality assurance, compression efficiency improvement, phase information processing and real-time performance. Especially under low bit rate conditions, it is difficult to effectively balance compression efficiency and sound quality restoration capabilities. Summary of the invention

[0004] The present invention provides a neural network-based audio signal encoding and decoding method, device, equipment and medium to solve the technical problem that existing audio codecs are difficult to effectively balance compression efficiency and sound quality restoration capabilities under low bit rate conditions.

[0005] In a first aspect, the present invention provides an audio signal encoding and decoding method based on a neural network, comprising:

[0006] Acquire an initial audio signal to be processed, and perform short-time Fourier transform processing on the acquired initial audio signal to be processed to obtain an initial amplitude spectrum and an initial phase spectrum corresponding to the audio signal to be processed;

[0007] Inputting the initial amplitude spectrum and the initial phase spectrum corresponding to the audio signal to be processed into an amplitude encoder and a phase encoder respectively for parallel feature encoding to obtain an amplitude encoding vector and a phase encoding vector corresponding to the audio signal to be processed;

[0008] Performing feature fusion on the amplitude coding vector and the phase coding vector corresponding to the audio signal to be processed to obtain a target coding vector of the audio signal to be processed;

[0009] Inputting the target coding vector of the audio signal to be processed into a residual vector quantization module for quantization processing to obtain an amplitude latent vector and a phase latent vector of the audio signal to be processed;

[0010] Inputting the amplitude latent vector and the phase latent vector of the audio signal to be processed into an amplitude decoder and a phase decoder for parallel decoding respectively, so as to obtain an initial amplitude spectrum and an initial phase spectrum of the audio signal to be processed;

[0011] An inverse short-time Fourier transform process is performed on the initial amplitude spectrum and the initial phase spectrum of the audio signal to be processed to generate a reconstructed audio signal.

[0012] In a second aspect, the present invention provides an audio signal encoding and decoding device based on a neural network, wherein the device is used to implement the audio signal encoding and decoding method based on a neural network as described in the first aspect, comprising:

[0013] An acquisition module is used to acquire an initial audio signal to be processed, and perform short-time Fourier transform processing on the acquired initial audio signal to be processed to obtain an initial amplitude spectrum and an initial phase spectrum corresponding to the audio signal to be processed;

[0014] An encoding module, used for inputting an initial amplitude spectrum and an initial phase spectrum corresponding to the audio signal to be processed into an amplitude encoder and a phase encoder respectively for parallel feature encoding, so as to obtain an amplitude encoding vector and a phase encoding vector corresponding to the audio signal to be processed;

[0015] A fusion module, used for performing feature fusion on the amplitude coding vector and the phase coding vector corresponding to the audio signal to be processed to obtain a target coding vector of the audio signal to be processed;

[0016] A quantization module, used for inputting the target coding vector of the audio signal to be processed into the residual vector quantization module for quantization processing to obtain the amplitude latent vector and phase latent vector of the audio signal to be processed;

[0017] A decoding module, used for inputting the amplitude latent vector and the phase latent vector of the audio signal to be processed into an amplitude decoder and a phase decoder for parallel decoding respectively, so as to obtain an initial amplitude spectrum and an initial phase spectrum of the audio signal to be processed;

[0018] The reconstruction module is used to perform inverse short-time Fourier transform processing on the initial amplitude spectrum and the initial phase spectrum of the audio signal to be processed to generate a reconstructed audio signal.

[0019] In a third aspect, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned neural network-based audio signal encoding and decoding method when executing the computer program.

[0020] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned neural network-based audio signal encoding and decoding method are implemented.

[0021] In the scheme implemented by the above-mentioned neural network-based audio signal encoding and decoding method, device, equipment and medium, the initial audio signal to be processed can be obtained through the client, and the obtained initial audio signal to be processed is short-time Fourier transform processed to obtain the initial amplitude spectrum and initial phase spectrum corresponding to the audio signal to be processed; the initial amplitude spectrum and initial phase spectrum corresponding to the audio signal to be processed are respectively input into the amplitude encoder and the phase encoder for parallel feature encoding to obtain the amplitude coding vector and phase coding vector corresponding to the audio signal to be processed; the amplitude coding vector and the phase coding vector corresponding to the audio signal to be processed are feature fused to obtain the target coding vector of the audio signal to be processed; the target coding vector of the audio signal to be processed is input into the residual vector quantization module for quantization processing to obtain the amplitude latent vector and phase latent vector of the audio signal to be processed; The amplitude latent vector and phase latent vector of the audio signal to be processed are respectively input into the amplitude decoder and the phase decoder for parallel decoding to obtain the initial amplitude spectrum and the initial phase spectrum of the audio signal to be processed; the initial amplitude spectrum and the initial phase spectrum of the audio signal to be processed are subjected to inverse short-time Fourier transform processing to generate a reconstructed audio signal. In the present invention, the initial amplitude spectrum and the initial phase spectrum corresponding to the audio signal to be processed are input into the amplitude encoder and the phase encoder in parallel for feature encoding; the target coding vector of the audio signal to be processed is then input into the residual vector quantization module for quantization processing; finally, the amplitude latent vector and the phase latent vector of the audio signal to be processed are respectively input into the amplitude decoder and the phase decoder for parallel decoding, thereby realizing efficient compression and high-fidelity decoding of the audio signal, and effectively improving the sound quality and efficiency of the audio codec at a low bit rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0023] Figure 1 is a schematic diagram of an application environment of an audio signal encoding and decoding method based on a neural network in an embodiment of the present invention;

[0024] Figure 2 is a flowchart of an audio signal encoding and decoding method based on a neural network in one embodiment of the present invention;

[0025] Figure 3 yes Figure 2 A schematic flow chart of a specific implementation of step S10;

[0026] Figure 4 yes Figure 2 A schematic flow chart of a specific implementation of step S20;

[0027] Figure 5 yes Figure 2 A schematic flow chart of a specific implementation of step S40;

[0028] Figure 6 yes Figure 2 A schematic flow chart of a specific implementation of step S50;

[0029] Figure 7 is a structural schematic diagram of an audio signal encoding and decoding device based on a neural network in one embodiment of the present invention;

[0030] Figure 8 is a schematic diagram of a structure of a computer device in one embodiment of the present invention;

[0031] Fig. 9 It is another structural schematic diagram of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0032] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0033] The neural network-based audio signal encoding and decoding method provided by the embodiment of the present invention can be applied in the following aspects: Figure 1 In the application environment, Figure 1It is a schematic diagram of an application environment of an audio signal encoding and decoding method based on a neural network in an embodiment of the present invention; wherein, the client communicates with the server through a network. The server can obtain the initial audio signal to be processed through the client, and perform short-time Fourier transform processing on the obtained initial audio signal to be processed to obtain an initial amplitude spectrum and an initial phase spectrum corresponding to the audio signal to be processed; the initial amplitude spectrum and the initial phase spectrum corresponding to the audio signal to be processed are respectively input into the amplitude encoder and the phase encoder for parallel feature encoding to obtain an amplitude coding vector and a phase coding vector corresponding to the audio signal to be processed; the amplitude coding vector and the phase coding vector corresponding to the audio signal to be processed are feature fused to obtain a target coding vector of the audio signal to be processed; the target coding vector of the audio signal to be processed is input into the residual vector quantization module for quantization processing to obtain an amplitude latent vector and a phase latent vector of the audio signal to be processed; the amplitude latent vector and the phase latent vector of the audio signal to be processed are The initial amplitude spectrum and initial phase spectrum of the audio signal to be processed are respectively input into the amplitude decoder and the phase decoder for parallel decoding to obtain the initial amplitude spectrum and the initial phase spectrum of the audio signal to be processed; the initial amplitude spectrum and the initial phase spectrum of the audio signal to be processed are subjected to inverse short-time Fourier transform processing to generate a reconstructed audio signal. In the present invention, the initial amplitude spectrum and the initial phase spectrum corresponding to the audio signal to be processed are input into the amplitude encoder and the phase encoder in parallel for feature encoding; the target coding vector of the audio signal to be processed is then input into the residual vector quantization module for quantization processing, and finally, the amplitude latent vector and the phase latent vector of the audio signal to be processed are respectively input into the amplitude decoder and the phase decoder for parallel decoding, thereby realizing efficient compression and high-fidelity decoding of the audio signal, and effectively improving the sound quality and efficiency of the audio codec at a low bit rate. The present invention is described in detail below through specific embodiments.

[0034] See also Figure 2 As shown, Figure 2 A flowchart of a neural network-based audio signal encoding and decoding method according to an embodiment of the present invention is provided. The neural network-based audio signal encoding and decoding method comprises the following steps:

[0035] S10: Acquire the initial audio signal to be processed, and perform short-time Fourier transform processing on the acquired initial audio signal to be processed, so as to obtain the initial amplitude spectrum and initial phase spectrum corresponding to the audio signal to be processed. For example, when analyzing voice records of financial transactions, it is first necessary to obtain the initial audio signal to be processed during the transaction. These audio signals may contain key information such as the conversation between the two parties to the transaction and transaction instructions. In telemedicine, it is necessary to obtain the initial audio signal to be processed containing information such as the description of the patient's condition. In order to further conduct in-depth analysis of these audio data, it is necessary to perform short-time Fourier transform processing on the acquired initial audio signal to be processed, so as to obtain the initial amplitude spectrum and initial phase spectrum corresponding to the audio signal to be processed. Specifically, Figure 3 As stated, Figure 3 yes Figure 2 A specific implementation flow diagram of step S10 in the embodiment of the present invention specifically includes the following steps S11-S14:

[0036] S11: Divide the acquired initial audio signal to be processed into a plurality of window audio signals, and select a window function for processing the window audio signal. Specifically, in the present invention, the audio signal is usually a long sequence data that is continuous in time, and it is difficult to effectively present the frequency characteristics of the signal that changes over time by directly performing Fourier transform on the entire audio signal. Therefore, first of all, the acquired initial audio signal to be processed must be divided into a plurality of relatively independent window audio signals. Secondly, it is necessary to select a corresponding window function according to a specific application scenario.

[0037] S12: Set the window length of each window audio signal and the window step size of each window audio signal. Specifically, the window length in the present invention refers to the time range covered by each window audio signal, and the window length needs to be reasonably set according to the actual time requirements. Secondly, the window step size specifies the size of the interval between two adjacent window audio signals, and the appropriate window step size needs to be determined based on the information integrity and computational cost of each window audio signal.

[0038] S13: According to the selected window function of the processing window audio signal, the window length of each window audio signal, and the window step length of each window audio signal, the audio signal of each window is subjected to windowing processing. Specifically, in the present invention, after the window function, window length and window step length are determined in steps S11 and S12, the audio signal of each window can be subjected to windowing processing according to these parameters, and by multiplying by a suitable window function, the edge transition of the window audio signal in the time domain can be made smoother, so that the spectrum information obtained after the subsequent short-time Fourier transform is more accurate and reliable, and more in line with the actual frequency characteristics of the original audio signal.

[0039] S14: Short-time Fourier transform is used to calculate the frequency spectrum information corresponding to the audio signal of each window after the windowing process, and the initial amplitude spectrum and initial phase spectrum corresponding to the audio signal to be processed are calculated based on the frequency spectrum information corresponding to the audio signal to be processed. In the present invention, short-time Fourier transform is used to calculate the audio signal of each window after the windowing process, and the window audio signal in the time domain is converted to the frequency domain to obtain the amplitude spectrum and phase spectrum corresponding to each window audio signal. Specifically, the initial amplitude spectrum and initial phase spectrum corresponding to the audio signal to be processed are calculated, including:

[0040] The calculation formula of the initial amplitude spectrum corresponding to the audio signal to be processed is:

[0041]

[0042] in, | Y(k) | represents the initial amplitude spectrum corresponding to the audio signal to be processed, Y(k) represents the frequency spectrum information corresponding to the audio signal to be processed, Re(Y(k)) represents the real part of Y(k), and Im(Y(k)) represents the imaginary part of Y(k);

[0043] The calculation formula of the initial phase spectrum corresponding to the audio signal to be processed is:

[0044]

[0045] in, Represents the initial phase spectrum corresponding to the audio signal to be processed.

[0046] S20: Inputting the initial amplitude spectrum and initial phase spectrum corresponding to the audio signal to be processed into an amplitude encoder and a phase encoder respectively for parallel feature encoding, and obtaining an amplitude encoding vector and a phase encoding vector corresponding to the audio signal to be processed. Specifically, Figure 4 As stated, Figure 4 yes Figure 2 A specific implementation flow diagram of step S20 in the embodiment of the present invention specifically includes the following steps S21-S24:

[0047] S21: Input the initial amplitude spectrum and phase spectrum corresponding to the audio signal to be processed into the convolution layer of the amplitude encoder and the phase encoder in parallel for feature extraction, and obtain the amplitude feature vector and phase feature vector corresponding to the audio signal to be processed. In the present invention, the convolution kernel of the convolution layer will capture the characteristic information in the amplitude spectrum and phase spectrum data in the local area. For example, for the amplitude spectrum, the local strength change characteristics of different frequency intervals will be extracted. For the phase spectrum, the local change trend of the phase can be extracted. Through multiple different convolution kernels and multiple convolution operations, more representative and abstract amplitude feature vectors and phase feature vectors can be extracted from the original amplitude spectrum and phase spectrum.

[0048] S22: Input the amplitude feature vector and phase feature vector corresponding to the audio signal to be processed into the normalization layer of the amplitude encoder and the phase encoder in parallel for normalization processing, and obtain the normalized amplitude feature vector and phase feature vector of the audio signal to be processed. In the present invention, the amplitude feature vector and the phase feature vector can be unified into a relatively standard value range through normalization processing, which helps to improve the stability of the subsequent processing process and accelerate the calculation convergence speed.

[0049] S23: The normalized amplitude feature vector and phase feature vector of the audio signal to be processed are input in parallel to the forward layer of the amplitude encoder and the phase encoder for nonlinear transformation processing to obtain the amplitude feature vector and the phase feature vector of the audio signal to be processed after nonlinear transformation. Specifically, in the present invention, through nonlinear transformation, for example, using an activation function, a nonlinear relationship can be generated between the elements in the feature vector, so that the actual complex changes of the audio signal can be better fitted, and deeper feature information can be excavated.

[0050] S24: Inputting the amplitude feature vector and phase feature vector of the audio signal to be processed after nonlinear transformation in parallel to the downsampling layer of the amplitude encoder and the phase encoder for downsampling processing, and obtaining the amplitude coding vector and phase coding vector corresponding to the audio signal to be processed. Specifically, in the present invention, by downsampling the amplitude feature vector and the phase feature vector after nonlinear transformation, the dimension of the feature vector is effectively reduced, while the most critical feature information is retained as much as possible.

[0051] S30: Feature fusion of the amplitude coding vector and the phase coding vector corresponding to the audio signal to be processed to obtain the target coding vector of the audio signal to be processed. Specifically, in the present invention, the amplitude coding vector and the phase coding vector corresponding to the audio signal to be processed can be feature spliced ​​and fused to obtain the target coding vector of the audio signal to be processed. For example, for the analysis of recordings of the customer service hotline of a financial institution, the target coding vector can comprehensively reflect the various features of the customer's voice, such as tone strength, intonation changes, etc., to help financial institutions more accurately analyze customer emotions, intentions, and needs, thereby optimizing service strategies.

[0052] S40: Input the target coding vector of the audio signal to be processed into the residual vector quantization module for quantization processing, and obtain the amplitude latent vector and phase latent vector of the audio signal to be processed. For example, in financial transaction voice monitoring, financial transaction audio often contains a large amount of sensitive information, such as transaction amount, transaction instructions, etc. Through quantization processing, key features can be effectively extracted to obtain the amplitude latent vector and phase latent vector of the audio signal to be processed. These latent vectors can be used for subsequent risk assessments. For example, by analyzing the sound characteristics, it can be determined whether the two parties to the transaction have abnormal emotions and whether it may cause transaction risks, provide risk warnings for financial institutions, and ensure the security and stability of transactions. Specifically, if Figure 5 As stated, Figure 5 yes Figure 2 A specific implementation flow diagram of step S40 in the embodiment of the present invention specifically includes the following steps S41-S43:

[0053] S41: cluster analysis is performed on the target feature vector of the audio signal to be processed to generate a codebook corresponding to the audio signal to be processed. Specifically, the present invention performs cluster analysis on the target feature vector of the audio signal to be processed to finally generate a codebook corresponding to the audio signal to be processed, and the generated codebook corresponding to the audio signal to be processed records the relevant information of each cluster, such as the center vector of each cluster and other information.

[0054] S42: Calculate the quantization codewords in the codebook corresponding to the generated audio signal to be processed, and calculate the residual feature vector of the audio signal to be processed according to the calculated quantization codewords of each codebook. Specifically, in the present invention, for each target feature vector, find the cluster to which each target feature vector belongs, then use the quantization codeword corresponding to the cluster to represent the target feature vector, and then calculate the difference between the target feature vector and the corresponding quantization codeword, and the calculated difference vector is the residual feature vector of the audio signal to be processed.

[0055] S43: extracting the amplitude latent vector and the phase latent vector of the audio signal to be processed according to the calculated residual feature vector of the audio signal to be processed. Specifically, the present invention helps to further mine higher-level feature information in the audio signal by extracting the amplitude latent vector and the phase latent vector, which can effectively reduce data redundancy and retain key amplitude information and phase information in the audio signal as much as possible.

[0056] S50: Inputting the amplitude latent vector and the phase latent vector of the audio signal to be processed into an amplitude decoder and a phase decoder for parallel decoding respectively, to obtain an initial amplitude spectrum and an initial phase spectrum of the audio signal to be processed. Figure 6 As stated, Figure 6 yes Figure 2 A specific implementation flow diagram of step S50 in the embodiment of the present invention specifically includes the following steps S51-S54:

[0057] S51: Input the amplitude latent vector and feature latent vector of the audio signal to be processed in parallel to the upsampling layer of the amplitude decoder and the phase decoder for upsampling processing, and obtain the amplitude coding vector and phase coding vector of the audio signal to be processed after upsampling. Specifically, the present invention can obtain the amplitude coding vector and phase coding vector of the audio signal to be processed after upsampling by upsampling the amplitude latent vector and feature latent vector of the audio signal to be processed. For example, the audio information contained in the original low-resolution or low-dimensional vector is not detailed enough. Through upsampling, it can obtain more detailed audio feature information, so that the subsequent further reconstructed audio signal is more accurate.

[0058] S52: Input the amplitude coding vector and phase coding vector after upsampling the audio signal to be processed into the normalization layer of the amplitude decoder and the phase decoder for normalization processing to obtain the normalized amplitude coding vector and phase coding vector of the audio signal to be processed. Specifically, in the present invention, the data distribution of the amplitude coding vector and the phase coding vector can be made more stable through normalization processing.

[0059] S53: Input the normalized amplitude coding vector and phase coding vector of the audio signal to be processed into the forward layer of the amplitude decoder and the phase decoder for nonlinear transformation processing, and obtain the amplitude coding vector and phase coding vector after nonlinear transformation of the audio signal to be processed. Specifically, the present invention uses an activation function to change the relationship between the elements in the amplitude coding vector and the phase coding vector through nonlinear transformation processing, so that the amplitude coding vector and the phase coding vector after nonlinear transformation can better fit the complex characteristics of the audio signal itself, and facilitate the subsequent reconstruction of high-quality audio-related spectrum information.

[0060] S54: Input the amplitude coding vector and phase coding vector after nonlinear transformation of the audio signal to be processed into the convolution layer of the amplitude decoder and the phase decoder for feature reconstruction, and obtain the initial amplitude spectrum and initial phase spectrum of the audio signal to be processed. Finally, input the amplitude coding vector and phase coding vector after nonlinear transformation into the convolution layer of the amplitude decoder and the phase decoder, and utilize the characteristics of the convolution layer to reconstruct the input vector, reintegrate the audio features contained in the amplitude coding vector and the phase coding vector obtained by the previous processing, and finally obtain the initial amplitude spectrum and initial phase spectrum of the audio signal to be processed.

[0061] S60: Perform inverse short-time Fourier transform processing on the initial amplitude spectrum and initial phase spectrum of the audio signal to be processed to generate a reconstructed audio signal. In the present invention, the frequency energy information and frequency phase information provided by the initial amplitude spectrum and initial phase spectrum of the audio signal to be processed are calculated by inverse short-time Fourier transform processing, and the form of the audio signal in the time domain is effectively restored by applying the calculation rules of inverse short-time Fourier transform to generate a reconstructed audio signal. For example, in the financial field, audio records of financial transactions are crucial, and any omission or distortion of information may lead to serious consequences. In this scenario, the use of the calculation rules of inverse short-time Fourier transform can effectively restore the form of the audio signal in the time domain to generate a reconstructed audio signal, and the reconstructed audio signal can be used for subsequent audits, dispute resolution, etc., to ensure the transparency and traceability of the financial transaction process.

[0062] In one embodiment of the present invention, the neural network-based audio signal encoding and decoding method further includes: calculating the signal-to-noise ratio of the reconstructed audio signal to be processed, and updating various parameter information in the encoder and decoder of the neural network according to the calculated signal-to-noise ratio. Specifically, in the present invention, by calculating the signal-to-noise ratio of the reconstructed audio signal, the quality of the reconstruction can be intuitively understood. If the signal-to-noise ratio is high, it means that the reconstructed audio signal is relatively pure and close to the original audio, indicating that the parameter settings of the current encoder and decoder are relatively reasonable; conversely, if the signal-to-noise ratio is low, it means that the reconstructed audio signal is not good, there are many noise problems, and the parameters need to be adjusted and optimized.

[0063] It can be seen that in the above scheme, the initial amplitude spectrum and initial phase spectrum corresponding to the audio signal to be processed are respectively input into the amplitude encoder and the phase encoder for parallel feature encoding; the target coding vector of the audio signal to be processed is then input into the residual vector quantization module for quantization processing; finally, the amplitude latent vector and phase latent vector of the audio signal to be processed are respectively input into the amplitude decoder and the phase decoder for parallel decoding, thereby achieving efficient compression and high-fidelity decoding of the audio signal, and effectively improving the sound quality and efficiency of the audio codec at low bit rates.

[0064] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0065] In one embodiment, a neural network-based audio signal encoding and decoding device is provided, and the neural network-based audio signal encoding and decoding device corresponds to the neural network-based audio signal encoding and decoding method in the above embodiment. Figure 7 As shown, Figure 7 7 is a schematic diagram of a structure of an audio signal encoding and decoding device based on a neural network in an embodiment of the present invention. The audio signal encoding and decoding device based on a neural network includes an acquisition module 71, an encoding module 72, a fusion module 73, a quantization module 74, a decoding module 75, and a reconstruction module 76. The functional modules are described in detail as follows:

[0066] An acquisition module 71 is used to acquire an initial audio signal to be processed, and perform short-time Fourier transform processing on the acquired initial audio signal to be processed to obtain an initial amplitude spectrum and an initial phase spectrum corresponding to the audio signal to be processed;

[0067] The encoding module 72 is used to input the initial amplitude spectrum and the initial phase spectrum corresponding to the audio signal to be processed into the amplitude encoder and the phase encoder respectively for parallel feature encoding, so as to obtain the amplitude encoding vector and the phase encoding vector corresponding to the audio signal to be processed;

[0068] A fusion module 73 is used to perform feature fusion on the amplitude coding vector and the phase coding vector corresponding to the audio signal to be processed to obtain a target coding vector of the audio signal to be processed;

[0069] A quantization module 74 is used to input the target coding vector of the audio signal to be processed into a residual vector quantization module for quantization processing to obtain an amplitude latent vector and a phase latent vector of the audio signal to be processed;

[0070] A decoding module 75, configured to input the amplitude latent vector and the phase latent vector of the audio signal to be processed into an amplitude decoder and a phase decoder for parallel decoding respectively, so as to obtain an initial amplitude spectrum and an initial phase spectrum of the audio signal to be processed;

[0071] The reconstruction module 76 is used to perform inverse short-time Fourier transform processing on the initial amplitude spectrum and the initial phase spectrum of the audio signal to be processed to generate a reconstructed audio signal.

[0072] In one embodiment, the acquisition module 71 is specifically configured to:

[0073] The obtained initial audio signal to be processed is divided into a plurality of window audio signals, and a window function for processing the window audio signals is selected;

[0074] Set the window length of each window audio signal and the window step size of each window audio signal;

[0075] Performing window processing on the audio signals in each window according to the selected window function for processing the audio signals in the window, the window length of the audio signals in each window, and the window step length of the audio signals in each window;

[0076] Short-time Fourier transform is used to calculate the frequency spectrum information corresponding to the audio signal of each window after the windowing process, and the initial amplitude spectrum and initial phase spectrum corresponding to the audio signal to be processed are calculated according to the calculated frequency spectrum information corresponding to the audio signal to be processed.

[0077] In one embodiment, the encoding module 72 is specifically configured to:

[0078] Inputting the initial amplitude spectrum and phase spectrum corresponding to the audio signal to be processed into the convolutional layers of the amplitude encoder and the phase encoder in parallel for feature extraction, so as to obtain the amplitude feature vector and the phase feature vector corresponding to the audio signal to be processed;

[0079] Inputting the amplitude feature vector and the phase feature vector corresponding to the audio signal to be processed into the normalization layer of the amplitude encoder and the phase encoder in parallel for normalization processing, so as to obtain the normalized amplitude feature vector and the phase feature vector of the audio signal to be processed;

[0080] Inputting the normalized amplitude feature vector and phase feature vector of the audio signal to be processed into the forward layer of the amplitude encoder and the phase encoder in parallel for nonlinear transformation processing, so as to obtain the amplitude feature vector and phase feature vector of the audio signal to be processed after nonlinear transformation;

[0081] The amplitude feature vector and phase feature vector of the audio signal to be processed after nonlinear transformation are input in parallel to the downsampling layer of the amplitude encoder and the phase encoder for downsampling processing to obtain the amplitude coding vector and the phase coding vector corresponding to the audio signal to be processed.

[0082] In one embodiment, the quantization module 74 is specifically configured to:

[0083] Performing cluster analysis on the target feature vector of the audio signal to be processed to generate a codebook corresponding to the audio signal to be processed;

[0084] Calculating the quantization codewords in the codebook corresponding to the generated audio signal to be processed, and calculating the residual feature vector of the audio signal to be processed according to the calculated quantization codewords of each codebook;

[0085] According to the calculated residual feature vector of the audio signal to be processed, an amplitude latent vector and a phase latent vector of the audio signal to be processed are extracted.

[0086] In one embodiment, the decoding module 75 is specifically configured to:

[0087] Inputting the amplitude latent vector and the feature latent vector of the audio signal to be processed into the upsampling layer of the amplitude decoder and the phase decoder in parallel for upsampling processing, so as to obtain the amplitude coding vector and the phase coding vector after the upsampling of the audio signal to be processed;

[0088] Inputting the upsampled amplitude coding vector and phase coding vector of the audio signal to be processed into the normalization layer of the amplitude decoder and the phase decoder for normalization processing to obtain the normalized amplitude coding vector and phase coding vector of the audio signal to be processed;

[0089] Inputting the normalized amplitude coding vector and phase coding vector of the audio signal to be processed into the forward layer of the amplitude decoder and the phase decoder for nonlinear transformation processing to obtain the amplitude coding vector and phase coding vector after nonlinear transformation of the audio signal to be processed;

[0090] The amplitude coding vector and the phase coding vector after the nonlinear transformation of the audio signal to be processed are input into the convolutional layers of the amplitude decoder and the phase decoder for feature reconstruction to obtain the initial amplitude spectrum and the initial phase spectrum of the audio signal to be processed.

[0091] In one embodiment, the acquisition module 71 is specifically configured to:

[0092] The calculation formula of the initial amplitude spectrum corresponding to the audio signal to be processed is:

[0093]

[0094] in, | Y(k) | represents the initial amplitude spectrum corresponding to the audio signal to be processed, Y(k) represents the frequency spectrum information corresponding to the audio signal to be processed, Re(Y(k)) represents the real part of Y(k), and Im(Y(k)) represents the imaginary part of Y(k);

[0095] The calculation formula of the initial phase spectrum corresponding to the audio signal to be processed is:

[0096]

[0097] in, Represents the initial phase spectrum corresponding to the audio signal to be processed.

[0098] In one embodiment, the neural network-based audio signal encoding and decoding device is further used for:

[0099] The signal-to-noise ratio of the reconstructed audio signal to be processed is calculated, and various parameter information in the encoder and decoder of the neural network are updated according to the calculated signal-to-noise ratio.

[0100] The present invention provides an audio signal encoding and decoding device based on a neural network, which realizes efficient compression and high-fidelity decoding of audio signals, and effectively improves the sound quality and efficiency of the audio codec at low bit rates by inputting the initial amplitude spectrum and initial phase spectrum corresponding to the audio signal to be processed into the amplitude encoder and the phase encoder respectively for parallel feature encoding; then inputting the target encoding vector of the audio signal to be processed into the residual vector quantization module for quantization processing; finally, inputting the amplitude latent vector and phase latent vector of the audio signal to be processed into the amplitude decoder and the phase decoder respectively for parallel decoding.

[0101] For the specific definition of the audio signal encoding and decoding device based on a neural network, please refer to the definition of the audio signal encoding and decoding method based on a neural network above, which will not be repeated here. Each module in the above-mentioned audio signal encoding and decoding device based on a neural network can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0102] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 8 As shown, Figure 8 1 is a schematic diagram of the structure of a computer device in one embodiment of the present invention. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external client via a network connection. When the computer program is executed by the processor, the functions or steps on the server side of a neural network-based audio signal encoding and decoding method are implemented.

[0103] In one embodiment, a computer device is provided. The computer device may be a client, and its internal structure diagram may be as follows: Fig. 9 As shown, Fig. 9: is another structural schematic diagram of a computer device in one embodiment of the present invention. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server via a network connection. When the computer program is executed by the processor, the functions or steps on the client side of a neural network-based audio signal encoding and decoding method are implemented.

[0104] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the following steps are implemented:

[0105] Acquire an initial audio signal to be processed, and perform short-time Fourier transform processing on the acquired initial audio signal to be processed to obtain an initial amplitude spectrum and an initial phase spectrum corresponding to the audio signal to be processed;

[0106] Inputting the initial amplitude spectrum and the initial phase spectrum corresponding to the audio signal to be processed into an amplitude encoder and a phase encoder respectively for parallel feature encoding to obtain an amplitude encoding vector and a phase encoding vector corresponding to the audio signal to be processed;

[0107] Performing feature fusion on the amplitude coding vector and the phase coding vector corresponding to the audio signal to be processed to obtain a target coding vector of the audio signal to be processed;

[0108] Inputting the target coding vector of the audio signal to be processed into a residual vector quantization module for quantization processing to obtain an amplitude latent vector and a phase latent vector of the audio signal to be processed;

[0109] Inputting the amplitude latent vector and the phase latent vector of the audio signal to be processed into an amplitude decoder and a phase decoder for parallel decoding respectively, so as to obtain an initial amplitude spectrum and an initial phase spectrum of the audio signal to be processed;

[0110] An inverse short-time Fourier transform process is performed on the initial amplitude spectrum and the initial phase spectrum of the audio signal to be processed to generate a reconstructed audio signal.

[0111] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0112] Acquire an initial audio signal to be processed, and perform short-time Fourier transform processing on the acquired initial audio signal to be processed to obtain an initial amplitude spectrum and an initial phase spectrum corresponding to the audio signal to be processed;

[0113] Inputting the initial amplitude spectrum and the initial phase spectrum corresponding to the audio signal to be processed into an amplitude encoder and a phase encoder respectively for parallel feature encoding to obtain an amplitude encoding vector and a phase encoding vector corresponding to the audio signal to be processed;

[0114] Performing feature fusion on the amplitude coding vector and the phase coding vector corresponding to the audio signal to be processed to obtain a target coding vector of the audio signal to be processed;

[0115] Inputting the target coding vector of the audio signal to be processed into a residual vector quantization module for quantization processing to obtain an amplitude latent vector and a phase latent vector of the audio signal to be processed;

[0116] Inputting the amplitude latent vector and the phase latent vector of the audio signal to be processed into an amplitude decoder and a phase decoder for parallel decoding respectively, so as to obtain an initial amplitude spectrum and an initial phase spectrum of the audio signal to be processed;

[0117] An inverse short-time Fourier transform process is performed on the initial amplitude spectrum and the initial phase spectrum of the audio signal to be processed to generate a reconstructed audio signal.

[0118] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can refer to the relevant descriptions on the server side and the client side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0119] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0120] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0121] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A method for encoding and decoding an audio signal based on a neural network, characterized in that: include: Acquire an initial audio signal to be processed, and perform short-time Fourier transform processing on the acquired initial audio signal to be processed to obtain an initial amplitude spectrum and an initial phase spectrum corresponding to the audio signal to be processed; Inputting the initial amplitude spectrum and the initial phase spectrum corresponding to the audio signal to be processed into an amplitude encoder and a phase encoder respectively for parallel feature encoding to obtain an amplitude encoding vector and a phase encoding vector corresponding to the audio signal to be processed; Performing feature fusion on the amplitude coding vector and the phase coding vector corresponding to the audio signal to be processed to obtain a target coding vector of the audio signal to be processed; Inputting the target coding vector of the audio signal to be processed into a residual vector quantization module for quantization processing to obtain an amplitude latent vector and a phase latent vector of the audio signal to be processed; Inputting the amplitude latent vector and the phase latent vector of the audio signal to be processed into an amplitude decoder and a phase decoder for parallel decoding respectively, so as to obtain an initial amplitude spectrum and an initial phase spectrum of the audio signal to be processed; An inverse short-time Fourier transform process is performed on the initial amplitude spectrum and the initial phase spectrum of the audio signal to be processed to generate a reconstructed audio signal.

2. The audio signal encoding and decoding method based on neural network according to claim 1, characterized in that: Performing short-time Fourier transform processing on the acquired initial audio signal to be processed to obtain an initial amplitude spectrum and an initial phase spectrum corresponding to the audio signal to be processed, including: The obtained initial audio signal to be processed is divided into a plurality of window audio signals, and a window function for processing the window audio signals is selected; Set the window length of each window audio signal and the window step size of each window audio signal; Performing window processing on the audio signals in each window according to the selected window function for processing the audio signals in the window, the window length of the audio signals in each window, and the window step length of the audio signals in each window; Short-time Fourier transform is used to calculate the frequency spectrum information corresponding to the audio signal of each window after the windowing process, and the initial amplitude spectrum and initial phase spectrum corresponding to the audio signal to be processed are calculated according to the calculated frequency spectrum information corresponding to the audio signal to be processed.

3. The audio signal encoding and decoding method based on neural network according to claim 1, characterized in that: The step of inputting the initial amplitude spectrum and the initial phase spectrum corresponding to the audio signal to be processed into an amplitude encoder and a phase encoder for parallel feature encoding respectively, and obtaining an amplitude coding vector and a phase coding vector corresponding to the audio signal to be processed, comprises: Inputting the initial amplitude spectrum and phase spectrum corresponding to the audio signal to be processed into the convolutional layers of the amplitude encoder and the phase encoder in parallel for feature extraction, so as to obtain the amplitude feature vector and the phase feature vector corresponding to the audio signal to be processed; Inputting the amplitude feature vector and the phase feature vector corresponding to the audio signal to be processed into the normalization layer of the amplitude encoder and the phase encoder in parallel for normalization processing, so as to obtain the normalized amplitude feature vector and the phase feature vector of the audio signal to be processed; Inputting the normalized amplitude feature vector and phase feature vector of the audio signal to be processed into the forward layer of the amplitude encoder and the phase encoder in parallel for nonlinear transformation processing, so as to obtain the amplitude feature vector and phase feature vector of the audio signal to be processed after nonlinear transformation; The amplitude feature vector and phase feature vector of the audio signal to be processed after nonlinear transformation are input in parallel to the downsampling layer of the amplitude encoder and the phase encoder for downsampling processing to obtain the amplitude coding vector and the phase coding vector corresponding to the audio signal to be processed.

4. The audio signal encoding and decoding method based on neural network according to claim 1, characterized in that: The step of inputting the target feature vector of the audio signal to be processed into a residual vector quantization module for quantization processing to obtain an amplitude latent vector and a phase latent vector of the audio signal to be processed includes: Performing cluster analysis on the target feature vector of the audio signal to be processed to generate a codebook corresponding to the audio signal to be processed; Calculating the quantization codewords in the codebook corresponding to the generated audio signal to be processed, and calculating the residual feature vector of the audio signal to be processed according to the calculated quantization codewords of each codebook; According to the calculated residual feature vector of the audio signal to be processed, an amplitude latent vector and a phase latent vector of the audio signal to be processed are extracted.

5. The audio signal encoding and decoding method based on neural network according to claim 1, characterized in that: The step of inputting the amplitude latent vector and the feature latent vector of the audio signal to be processed into an amplitude decoder and a phase decoder in parallel for decoding to obtain an initial amplitude spectrum and an initial phase spectrum of the audio signal to be processed includes: Inputting the amplitude latent vector and the feature latent vector of the audio signal to be processed into the upsampling layer of the amplitude decoder and the phase decoder in parallel for upsampling processing, so as to obtain the amplitude coding vector and the phase coding vector after the upsampling of the audio signal to be processed; Inputting the upsampled amplitude coding vector and phase coding vector of the audio signal to be processed into the normalization layer of the amplitude decoder and the phase decoder for normalization processing to obtain the normalized amplitude coding vector and phase coding vector of the audio signal to be processed; Inputting the normalized amplitude coding vector and phase coding vector of the audio signal to be processed into the forward layer of the amplitude decoder and the phase decoder for nonlinear transformation processing to obtain the amplitude coding vector and phase coding vector of the audio signal to be processed after nonlinear transformation; The amplitude coding vector and the phase coding vector after the nonlinear transformation of the audio signal to be processed are input into the convolutional layers of the amplitude decoder and the phase decoder for feature reconstruction to obtain the initial amplitude spectrum and the initial phase spectrum of the audio signal to be processed.

6. The method for encoding and decoding audio signals based on a neural network according to claim 2, characterized in that: The step of calculating the initial amplitude spectrum and the initial phase spectrum corresponding to the audio signal to be processed according to the calculated frequency spectrum information corresponding to the audio signal to be processed comprises: The calculation formula of the initial amplitude spectrum corresponding to the audio signal to be processed is: in, | Y(k) | represents the initial amplitude spectrum corresponding to the audio signal to be processed, Y(k) represents the frequency spectrum information corresponding to the audio signal to be processed, Re(Y(k)) represents the real part of Y(k), and Im(Y(k)) represents the imaginary part of Y(k); The calculation formula of the initial phase spectrum corresponding to the audio signal to be processed is: in, Represents the initial phase spectrum corresponding to the audio signal to be processed.

7. The audio signal encoding and decoding method based on neural network according to claim 1, characterized in that: The method further comprises: The signal-to-noise ratio of the reconstructed audio signal to be processed is calculated, and various parameter information in the encoder and decoder of the neural network are updated according to the calculated signal-to-noise ratio.

8. A neural network-based audio signal encoding and decoding device, the device being used to implement the neural network-based audio signal encoding and decoding method according to any one of claims 1 to 7, characterized in that: include: An acquisition module is used to acquire an initial audio signal to be processed, and perform short-time Fourier transform processing on the acquired initial audio signal to be processed to obtain an initial amplitude spectrum and an initial phase spectrum corresponding to the audio signal to be processed; An encoding module, used for inputting an initial amplitude spectrum and an initial phase spectrum corresponding to the audio signal to be processed into an amplitude encoder and a phase encoder respectively for parallel feature encoding, so as to obtain an amplitude encoding vector and a phase encoding vector corresponding to the audio signal to be processed; A fusion module, used for performing feature fusion on the amplitude coding vector and the phase coding vector corresponding to the audio signal to be processed to obtain a target coding vector of the audio signal to be processed; A quantization module, used for inputting the target coding vector of the audio signal to be processed into the residual vector quantization module for quantization processing to obtain the amplitude latent vector and phase latent vector of the audio signal to be processed; A decoding module, used for inputting the amplitude latent vector and the phase latent vector of the audio signal to be processed into an amplitude decoder and a phase decoder for parallel decoding respectively, so as to obtain an initial amplitude spectrum and an initial phase spectrum of the audio signal to be processed; The reconstruction module is used to perform inverse short-time Fourier transform processing on the initial amplitude spectrum and the initial phase spectrum of the audio signal to be processed to generate a reconstructed audio signal.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the neural network-based audio signal encoding and decoding method as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the neural network-based audio signal encoding and decoding method as described in any one of claims 1 to 7 are implemented.

Images