Audio signal coding and decoding method and apparatus based on neural network, device and medium

Abstract

The application belongs to the technical field of signal processing, and is suitable for the medical field and the financial field, and discloses an audio signal coding and decoding method, device, equipment and medium based on a neural network, the method comprising: inputting initial amplitude spectrum and initial phase spectrum corresponding to a to-be-processed audio signal into an amplitude encoder and a phase encoder respectively for parallel feature coding; inputting a target coding vector of the to-be-processed audio signal into a residual vector quantization module for quantization processing, obtaining an amplitude hidden vector and a phase hidden vector of the to-be-processed audio signal, and inputting the amplitude hidden vector and the phase hidden vector into an amplitude decoder and a phase decoder respectively for parallel decoding, obtaining the initial amplitude spectrum and the initial phase spectrum; and performing inverse short-time Fourier transform processing on the initial amplitude spectrum and the initial phase spectrum of the to-be-processed audio signal to generate a reconstructed audio signal. The application effectively improves the sound quality and efficiency of an audio codec under a low bit rate.

Description

Technical Field

[0001] The present invention belongs to the field of speech processing technology and is applicable to the medical and financial fields. In particular, it relates to a neural network-based audio signal encoding and decoding method, device, equipment and medium. Background Art

[0002] Audio compression technology is currently widely used in many areas of daily life, particularly in voice communications, online music streaming, and audio storage and transmission. For example, codecs such as MP3, AAC, and Opus use signal processing algorithms to compress audio signals in the time or frequency domain, thereby reducing data volume. In the financial sector, for example, in customer service centers, a large number of voice calls require storage and processing. The use of audio compression technology can effectively reduce storage costs and transmission bandwidth, improving service efficiency. In healthcare, for example, in telemedicine consultations, voice communications between doctors and patients must be transmitted over the network. Audio compression technology ensures clear voice transmission, enabling smooth communication between doctors and patients even in remote areas with poor network conditions.

[0003] Although these traditional audio codecs have achieved certain results in practical applications, they still have some shortcomings in terms of sound quality assurance, compression efficiency improvement, phase information processing, and real-time performance. Especially under low bit rate conditions, it is difficult to effectively balance compression efficiency and sound quality restoration capabilities. Summary of the Invention

[0004] The present invention provides a neural network-based audio signal encoding and decoding method, apparatus, device and medium to solve the technical problem that existing audio codecs are difficult to effectively balance compression efficiency and sound quality restoration capabilities under low bit rate conditions.

[0005] In a first aspect, the present invention provides an audio signal encoding and decoding method based on a neural network, comprising:

[0006] Acquire an initial audio signal to be processed, and perform short-time Fourier transform processing on the acquired initial audio signal to be processed to obtain an initial amplitude spectrum and an initial phase spectrum corresponding to the audio signal to be processed;

[0007] Inputting the initial amplitude spectrum and initial phase spectrum corresponding to the audio signal to be processed into an amplitude encoder and a phase encoder respectively for parallel feature encoding to obtain an amplitude coding vector and a phase coding vector corresponding to the audio signal to be processed;

[0008] Performing feature fusion on the amplitude coding vector and the phase coding vector corresponding to the audio signal to be processed to obtain a target coding vector of the audio signal to be processed;

[0009] Inputting the target coding vector of the audio signal to be processed into a residual vector quantization module for quantization processing to obtain an amplitude latent vector and a phase latent vector of the audio signal to be processed;

[0010] Inputting the amplitude latent vector and the phase latent vector of the audio signal to be processed into an amplitude decoder and a phase decoder respectively for parallel decoding to obtain an initial amplitude spectrum and an initial phase spectrum of the audio signal to be processed;

[0011] An inverse short-time Fourier transform process is performed on the initial amplitude spectrum and the initial phase spectrum of the audio signal to be processed to generate a reconstructed audio signal.

[0012] In a second aspect, the present invention provides a neural network-based audio signal encoding and decoding device, wherein the device is used to implement the neural network-based audio signal encoding and decoding method as described in the first aspect, comprising:

[0013] An acquisition module is used to acquire an initial audio signal to be processed, and perform short-time Fourier transform processing on the acquired initial audio signal to be processed to obtain an initial amplitude spectrum and an initial phase spectrum corresponding to the audio signal to be processed;

[0014] an encoding module, configured to input an initial amplitude spectrum and an initial phase spectrum corresponding to the audio signal to be processed into an amplitude encoder and a phase encoder, respectively, for parallel feature encoding to obtain an amplitude encoding vector and a phase encoding vector corresponding to the audio signal to be processed;

[0015] a fusion module, configured to perform feature fusion on the amplitude coding vector and the phase coding vector corresponding to the audio signal to be processed to obtain a target coding vector of the audio signal to be processed;

[0016] a quantization module, configured to input the target coding vector of the audio signal to be processed into a residual vector quantization module for quantization processing, thereby obtaining an amplitude latent vector and a phase latent vector of the audio signal to be processed;

[0017] a decoding module, configured to input the amplitude latent vector and the phase latent vector of the audio signal to be processed into an amplitude decoder and a phase decoder respectively for parallel decoding, so as to obtain an initial amplitude spectrum and an initial phase spectrum of the audio signal to be processed;

[0018] The reconstruction module is used to perform inverse short-time Fourier transform processing on the initial amplitude spectrum and the initial phase spectrum of the audio signal to be processed to generate a reconstructed audio signal.

[0019] In a third aspect, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the above-mentioned neural network-based audio signal encoding and decoding method are implemented.

[0020] In a fourth aspect, a computer-readable storage medium is provided, which stores a computer program. When the computer program is executed by a processor, the steps of the above-mentioned neural network-based audio signal encoding and decoding method are implemented.

[0021] In the scheme implemented by the above-mentioned neural network-based audio signal encoding and decoding method, device, equipment and medium, the initial audio signal to be processed can be obtained through the client, and the obtained initial audio signal to be processed is subjected to short-time Fourier transform processing to obtain the initial amplitude spectrum and initial phase spectrum corresponding to the audio signal to be processed; the initial amplitude spectrum and initial phase spectrum corresponding to the audio signal to be processed are respectively input into the amplitude encoder and the phase encoder for parallel feature encoding to obtain the amplitude coding vector and phase coding vector corresponding to the audio signal to be processed; the amplitude coding vector and the phase coding vector corresponding to the audio signal to be processed are feature fused to obtain the target coding vector of the audio signal to be processed; the target coding vector of the audio signal to be processed is input into the residual vector quantization module for quantization processing to obtain the amplitude latent vector and phase latent vector of the audio signal to be processed; the The amplitude latent vector and phase latent vector of the audio signal to be processed are respectively input into the amplitude decoder and the phase decoder for parallel decoding to obtain the initial amplitude spectrum and the initial phase spectrum of the audio signal to be processed; the initial amplitude spectrum and the initial phase spectrum of the audio signal to be processed are subjected to inverse short-time Fourier transform processing to generate a reconstructed audio signal. In the present invention, the initial amplitude spectrum and the initial phase spectrum corresponding to the audio signal to be processed are input into the amplitude encoder and the phase encoder in parallel for feature encoding; the target coding vector of the audio signal to be processed is then input into the residual vector quantization module for quantization processing; finally, the amplitude latent vector and the phase latent vector of the audio signal to be processed are respectively input into the amplitude decoder and the phase decoder for parallel decoding, thereby achieving efficient compression and high-fidelity decoding of the audio signal, and effectively improving the sound quality and efficiency of the audio codec at low bit rates. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0023] Figure 1 2 is a schematic diagram of an application environment of an audio signal encoding and decoding method based on a neural network according to an embodiment of the present invention;

[0024] Figure 2 1 is a flowchart of a neural network-based audio signal encoding and decoding method according to an embodiment of the present invention;

[0025] Figure 3 yes Figure 2 A schematic flow chart of a specific implementation of step S10;

[0026] Figure 4 yes Figure 2 A schematic flow chart of a specific implementation of step S20;

[0027] Figure 5 yes Figure 2 A schematic flow chart of a specific implementation of step S40;

[0028] Figure 6 yes Figure 2 A schematic flow chart of a specific implementation of step S50;

[0029] Figure 7 1 is a schematic structural diagram of an audio signal encoding and decoding device based on a neural network in one embodiment of the present invention;

[0030] Figure 8 is a structural diagram of a computer device in one embodiment of the present invention;

[0031] Figure 9 FIG. 2 is another structural diagram of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0032] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0033] The neural network-based audio signal encoding and decoding method provided by the embodiment of the present invention can be applied in the following fields: Figure 1 In the application environment, Figure 1This is a schematic diagram of an application environment of an audio signal encoding and decoding method based on a neural network in an embodiment of the present invention; wherein, the client communicates with the server through a network. The server can obtain the initial audio signal to be processed through the client, and perform short-time Fourier transform processing on the obtained initial audio signal to be processed to obtain the initial amplitude spectrum and initial phase spectrum corresponding to the audio signal to be processed; the initial amplitude spectrum and initial phase spectrum corresponding to the audio signal to be processed are respectively input into the amplitude encoder and the phase encoder for parallel feature encoding to obtain the amplitude coding vector and phase coding vector corresponding to the audio signal to be processed; the amplitude coding vector and the phase coding vector corresponding to the audio signal to be processed are feature fused to obtain the target coding vector of the audio signal to be processed; the target coding vector of the audio signal to be processed is input into the residual vector quantization module for quantization processing to obtain the amplitude latent vector and phase latent vector of the audio signal to be processed; the amplitude latent vector and phase latent vector of the audio signal to be processed are The initial amplitude spectrum and initial phase spectrum of the audio signal to be processed are respectively input into the amplitude decoder and the phase decoder for parallel decoding to obtain the initial amplitude spectrum and initial phase spectrum of the audio signal to be processed; the initial amplitude spectrum and initial phase spectrum of the audio signal to be processed are subjected to inverse short-time Fourier transform processing to generate a reconstructed audio signal. In the present invention, the initial amplitude spectrum and initial phase spectrum corresponding to the audio signal to be processed are input into the amplitude encoder and the phase encoder in parallel for feature encoding; the target coding vector of the audio signal to be processed is input into the residual vector quantization module for quantization processing, and finally, the amplitude latent vector and phase latent vector of the audio signal to be processed are respectively input into the amplitude decoder and the phase decoder for parallel decoding, thereby achieving efficient compression and high-fidelity decoding of the audio signal, and effectively improving the sound quality and efficiency of the audio codec at low bit rates. The present invention is described in detail below through specific embodiments.

[0034] See also Figure 2 As shown, Figure 2 A flowchart of a neural network-based audio signal encoding and decoding method according to an embodiment of the present invention is provided. The neural network-based audio signal encoding and decoding method includes the following steps:

[0035] S10: Acquire the initial audio signal to be processed, and perform short-time Fourier transform processing on the acquired initial audio signal to be processed to obtain the initial amplitude spectrum and initial phase spectrum corresponding to the audio signal to be processed. For example, when analyzing voice records of financial transactions, it is first necessary to obtain the initial audio signal to be processed during the transaction process. These audio signals may contain key information such as the conversation between the two parties to the transaction and transaction instructions. In telemedicine, it is necessary to obtain the initial audio signal to be processed containing information such as the description of the patient's condition. In order to further conduct in-depth analysis of these audio data, it is necessary to perform short-time Fourier transform processing on the acquired initial audio signal to be processed to obtain the initial amplitude spectrum and initial phase spectrum corresponding to the audio signal to be processed. Specifically, as Figure 3 As stated, Figure 3 yes Figure 2 A flow chart of a specific implementation of step S10 in FIG. 1 specifically includes the following steps S11-S14:

[0036] S11: Segment the acquired initial audio signal to be processed into multiple window audio signals, and select a window function for processing the window audio signals. Specifically, in the present invention, the audio signal is usually a long sequence of data that is continuous in time. Directly performing a Fourier transform on the entire audio signal is difficult to effectively present the frequency characteristics of the signal that change over time. Therefore, it is necessary to first segment the acquired initial audio signal to be processed into multiple relatively independent window audio signals. Secondly, it is necessary to select the corresponding window function based on the specific application scenario.

[0037] S12: Setting the window length and window step size of each window audio signal. Specifically, in the present invention, the window length refers to the time range covered by each window audio signal, and needs to be reasonably set according to actual time requirements. Secondly, the window step size specifies the size of the interval between two adjacent window audio signals. It is necessary to determine the appropriate window step size based on the information integrity and computational cost of each window audio signal.

[0038] S13: Windowing is performed on the audio signals in each window according to the selected window function of the processing window audio signal, the window length of each window audio signal, and the window step size of each window audio signal. Specifically, in the present invention, after determining the window function, window length, and window step size in steps S11 and S12, the audio signals in each window can be windowed according to these parameters. By multiplying the appropriate window function, the edge transition of the window audio signal in the time domain can be made smoother, thereby making the spectrum information obtained after the subsequent short-time Fourier transform more accurate and reliable, and more consistent with the actual frequency characteristics of the original audio signal.

[0039] S14: Using short-time Fourier transform to calculate the frequency spectrum information corresponding to the audio signal of each window after the windowing process, and calculating the initial amplitude spectrum and initial phase spectrum corresponding to the audio signal to be processed based on the calculated frequency spectrum information corresponding to the audio signal to be processed. In the present invention, the audio signal of each window after the windowing process is calculated using short-time Fourier transform, and the window audio signal in the time domain is converted to the frequency domain to obtain the amplitude spectrum and phase spectrum corresponding to each window audio signal. Specifically, calculating the initial amplitude spectrum and initial phase spectrum corresponding to the audio signal to be processed includes:

[0040] The calculation formula of the initial amplitude spectrum corresponding to the audio signal to be processed is:

[0041]

[0042] in, | Y(k) | represents the initial amplitude spectrum corresponding to the audio signal to be processed, Y(k) represents the frequency spectrum information corresponding to the audio signal to be processed, Re(Y(k)) represents the real part of Y(k), and Im(Y(k)) represents the imaginary part of Y(k);

[0043] The calculation formula of the initial phase spectrum corresponding to the audio signal to be processed is:

[0044]

[0045] in, Represents the initial phase spectrum corresponding to the audio signal to be processed.

[0046] S20: Inputting the initial amplitude spectrum and initial phase spectrum corresponding to the audio signal to be processed into the amplitude encoder and phase encoder respectively for parallel feature encoding, and obtaining the amplitude encoding vector and phase encoding vector corresponding to the audio signal to be processed. Specifically, Figure 4 As stated, Figure 4 yes Figure 2 A flow chart of a specific implementation of step S20 in FIG. 1 specifically includes the following steps S21-S24:

[0047] S21: The initial amplitude spectrum and phase spectrum corresponding to the audio signal to be processed are input in parallel to the convolution layer of the amplitude encoder and the phase encoder for feature extraction to obtain the amplitude feature vector and phase feature vector corresponding to the audio signal to be processed. In the present invention, the convolution kernel of the convolution layer will capture the feature information in the amplitude spectrum and phase spectrum data in the local area. For example, for the amplitude spectrum, the local strength change characteristics of different frequency intervals will be extracted. For the phase spectrum, the local change trend of the phase can be extracted. Through multiple different convolution kernels and multiple convolution operations, more representative and abstract amplitude feature vectors and phase feature vectors can be extracted from the original amplitude spectrum and phase spectrum.

[0048] S22: Inputting the amplitude eigenvector and phase eigenvector corresponding to the audio signal to be processed into the normalization layers of the amplitude encoder and phase encoder in parallel for normalization processing, thereby obtaining the normalized amplitude eigenvector and phase eigenvector of the audio signal to be processed. In the present invention, the normalization processing can unify the amplitude eigenvector and phase eigenvector to a relatively standard value range, which helps to improve the stability of subsequent processing and accelerate computational convergence.

[0049] S23: Inputting the normalized amplitude eigenvector and phase eigenvector of the audio signal to be processed into the forward layer of the amplitude encoder and phase encoder in parallel for nonlinear transformation processing to obtain the amplitude eigenvector and phase eigenvector of the audio signal to be processed after nonlinear transformation. Specifically, the present invention uses nonlinear transformation, for example, using an activation function, to generate nonlinear relationships between elements in the eigenvector, thereby better fitting the actual complex changes of the audio signal and mining deeper feature information.

[0050] S24: Inputting the amplitude eigenvector and phase eigenvector of the audio signal to be processed, which are obtained after nonlinear transformation, in parallel to the downsampling layers of the amplitude encoder and phase encoder for downsampling processing, thereby obtaining an amplitude encoding vector and a phase encoding vector corresponding to the audio signal to be processed. Specifically, in the present invention, by downsampling the amplitude eigenvector and phase eigenvector after nonlinear transformation, the dimension of the eigenvector is effectively reduced while retaining the most critical feature information as much as possible.

[0051] S30: Feature fusion of the amplitude coding vector and the phase coding vector corresponding to the audio signal to be processed is performed to obtain the target coding vector of the audio signal to be processed. Specifically, in the present invention, the amplitude coding vector and the phase coding vector corresponding to the audio signal to be processed can be feature spliced ​​and fused to obtain the target coding vector of the audio signal to be processed. For example, for the analysis of customer service hotline recordings of financial institutions, the target coding vector can comprehensively reflect the various features of the customer's voice, such as tone strength, intonation changes, etc., to help financial institutions more accurately analyze customer emotions, intentions and needs, thereby optimizing service strategies.

[0052] S40: Input the target coding vector of the audio signal to be processed into the residual vector quantization module for quantization processing to obtain the amplitude latent vector and phase latent vector of the audio signal to be processed. For example, in financial transaction voice monitoring, financial transaction audio often contains a large amount of sensitive information, such as transaction amount, transaction instructions, etc. Through quantization processing, key features can be effectively extracted to obtain the amplitude latent vector and phase latent vector of the audio signal to be processed. These latent vectors can be used for subsequent risk assessment, such as by analyzing the sound characteristics to determine whether the two parties to the transaction have abnormal emotions and whether it may cause transaction risks, provide risk warnings for financial institutions, and ensure the security and stability of transactions. Specifically, if Figure 5 As stated, Figure 5 yes Figure 2 A flow chart of a specific implementation of step S40 in FIG. 4 includes the following steps S41-S43:

[0053] S41: Performing cluster analysis on the target feature vectors of the audio signal to be processed to generate a codebook corresponding to the audio signal to be processed. Specifically, the present invention performs cluster analysis on the target feature vectors of the audio signal to be processed to ultimately generate a codebook corresponding to the audio signal to be processed. The generated codebook corresponding to the audio signal to be processed records relevant information of each cluster, such as the center vector of each cluster.

[0054] S42: Calculate the quantization codewords in the generated codebook corresponding to the audio signal to be processed, and calculate the residual eigenvector of the audio signal to be processed based on the calculated quantization codewords of each codebook. Specifically, in the present invention, for each target eigenvector, find the cluster to which each target eigenvector belongs, then represent the target eigenvector with the quantization codeword corresponding to the cluster, and then calculate the difference between the target eigenvector and the corresponding quantization codeword. The calculated difference vector is the residual eigenvector of the audio signal to be processed.

[0055] S43: Extracting the amplitude latent vector and phase latent vector of the audio signal to be processed based on the calculated residual feature vector of the audio signal to be processed. Specifically, by extracting the amplitude latent vector and phase latent vector, the present invention facilitates further mining higher-level feature information in the audio signal, effectively reducing data redundancy while preserving the key amplitude and phase information in the audio signal to the greatest extent possible.

[0056] S50: Input the amplitude latent vector and phase latent vector of the audio signal to be processed into the amplitude decoder and phase decoder respectively for parallel decoding to obtain the initial amplitude spectrum and initial phase spectrum of the audio signal to be processed. Figure 6 As stated, Figure 6 yes Figure 2 A flow chart of a specific implementation of step S50 in FIG. 5 specifically includes the following steps S51-S54:

[0057] S51: The amplitude latent vector and the feature latent vector of the audio signal to be processed are input in parallel to the upsampling layer of the amplitude decoder and the phase decoder for upsampling processing, so as to obtain the amplitude coding vector and the phase coding vector of the audio signal to be processed after upsampling. Specifically, the present invention can obtain the amplitude coding vector and the phase coding vector of the audio signal to be processed after upsampling by upsampling the amplitude latent vector and the feature latent vector of the audio signal to be processed. For example, the audio information contained in the original low-resolution or low-dimensional vector is not detailed enough, and through upsampling, it is possible to obtain more detailed audio feature information, so that the subsequent further reconstructed audio signal is more accurate.

[0058] S52: Inputting the upsampled amplitude code vector and phase code vector of the audio signal to be processed into the normalization layer of the amplitude decoder and the phase decoder for normalization processing to obtain the normalized amplitude code vector and phase code vector of the audio signal to be processed. Specifically, the normalization processing in the present invention can make the data distribution of the amplitude code vector and the phase code vector more stable.

[0059] S53: Inputting the normalized amplitude coding vector and phase coding vector of the audio signal to be processed into the forward layer of the amplitude decoder and the phase decoder for nonlinear transformation processing to obtain the amplitude coding vector and phase coding vector after nonlinear transformation of the audio signal to be processed. Specifically, the present invention uses an activation function to change the relationship between the elements in the amplitude coding vector and the phase coding vector through nonlinear transformation processing, so that the amplitude coding vector and phase coding vector after nonlinear transformation can better conform to the complex characteristics of the audio signal itself, facilitating the subsequent reconstruction of high-quality audio-related spectrum information.

[0060] S54: Input the amplitude coding vector and phase coding vector of the audio signal to be processed after nonlinear transformation into the convolutional layers of the amplitude decoder and phase decoder for feature reconstruction, thereby obtaining the initial amplitude spectrum and initial phase spectrum of the audio signal to be processed. Finally, input the amplitude coding vector and phase coding vector after nonlinear transformation into the convolutional layers of the amplitude decoder and phase decoder. Utilizing the characteristics of the convolutional layers, feature reconstruction is performed on the input vectors, and the audio features contained in the amplitude coding vector and phase coding vector obtained by the previous processing are reintegrated, ultimately obtaining the initial amplitude spectrum and initial phase spectrum of the audio signal to be processed.

[0061] S60: Perform inverse short-time Fourier transform processing on the initial amplitude spectrum and initial phase spectrum of the audio signal to be processed to generate a reconstructed audio signal. In the present invention, the frequency energy information and frequency phase information provided by the initial amplitude spectrum and initial phase spectrum of the audio signal to be processed are calculated by inverse short-time Fourier transform processing, and the form of the audio signal in the time domain is effectively restored by applying the calculation rules of inverse short-time Fourier transform to generate a reconstructed audio signal. For example, in the financial field, audio recording of financial transactions is of vital importance, and any omission or distortion of information may lead to serious consequences. In this scenario, the calculation rules of inverse short-time Fourier transform can be used to effectively restore the form of the audio signal in the time domain to generate a reconstructed audio signal. The reconstructed audio signal can be used for subsequent audits, dispute resolution, etc. to ensure the transparency and traceability of the financial transaction process.

[0062] In one embodiment of the present invention, the neural network-based audio signal encoding and decoding method further includes: calculating the signal-to-noise ratio of the reconstructed audio signal to be processed, and updating the various parameter information in the neural network encoder and decoder according to the calculated signal-to-noise ratio. Specifically, in the present invention, by calculating the signal-to-noise ratio of the reconstructed audio signal, the quality of the reconstruction can be intuitively understood. If the signal-to-noise ratio is high, it means that the reconstructed audio signal is relatively pure and close to the original audio, indicating that the parameter settings of the current encoder and decoder are relatively reasonable; conversely, if the signal-to-noise ratio is low, it means that the reconstructed audio signal is not good, there are problems such as more noise, and the parameters need to be adjusted and optimized.

[0063] It can be seen that in the above scheme, the initial amplitude spectrum and initial phase spectrum corresponding to the audio signal to be processed are respectively input into the amplitude encoder and the phase encoder for parallel feature encoding; the target coding vector of the audio signal to be processed is then input into the residual vector quantization module for quantization processing; finally, the amplitude latent vector and phase latent vector of the audio signal to be processed are respectively input into the amplitude decoder and the phase decoder for parallel decoding, thereby achieving efficient compression and high-fidelity decoding of the audio signal, effectively improving the sound quality and efficiency of the audio codec at low bit rates.

[0064] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0065] In one embodiment, a neural network-based audio signal encoding and decoding device is provided. The neural network-based audio signal encoding and decoding device corresponds to the neural network-based audio signal encoding and decoding method in the above embodiment. Figure 7 As shown, Figure 7 7 is a schematic diagram of the structure of an audio signal encoding and decoding device based on a neural network according to an embodiment of the present invention. The audio signal encoding and decoding device based on a neural network includes an acquisition module 71, an encoding module 72, a fusion module 73, a quantization module 74, a decoding module 75, and a reconstruction module 76. The functional modules are described in detail as follows:

[0066] An acquisition module 71 is configured to acquire an initial audio signal to be processed, and perform short-time Fourier transform processing on the acquired initial audio signal to be processed to obtain an initial amplitude spectrum and an initial phase spectrum corresponding to the audio signal to be processed;

[0067] An encoding module 72 is configured to input the initial amplitude spectrum and initial phase spectrum corresponding to the audio signal to be processed into an amplitude encoder and a phase encoder respectively for parallel feature encoding to obtain an amplitude encoding vector and a phase encoding vector corresponding to the audio signal to be processed;

[0068] A fusion module 73 is configured to perform feature fusion on the amplitude coding vector and the phase coding vector corresponding to the audio signal to be processed to obtain a target coding vector of the audio signal to be processed;

[0069] a quantization module 74 configured to input the target coding vector of the audio signal to be processed into a residual vector quantization module for quantization processing to obtain an amplitude latent vector and a phase latent vector of the audio signal to be processed;

[0070] A decoding module 75 is configured to input the amplitude latent vector and the phase latent vector of the audio signal to be processed into an amplitude decoder and a phase decoder for parallel decoding, respectively, to obtain an initial amplitude spectrum and an initial phase spectrum of the audio signal to be processed;

[0071] The reconstruction module 76 is configured to perform inverse short-time Fourier transform processing on the initial amplitude spectrum and the initial phase spectrum of the audio signal to be processed to generate a reconstructed audio signal.

[0072] In one embodiment, the acquisition module 71 is specifically configured to:

[0073] Splitting the obtained initial audio signal to be processed into a plurality of window audio signals, and selecting a window function for processing the window audio signals;

[0074] Set the window length and window step of each window audio signal;

[0075] Performing windowing processing on the audio signals in each window according to the selected window function for processing the audio signals in the window, the window length of the audio signals in each window, and the window step size of the audio signals in each window;

[0076] Short-time Fourier transform is used to calculate the frequency spectrum information corresponding to the audio signal of each window after windowing processing, and the initial amplitude spectrum and initial phase spectrum corresponding to the audio signal to be processed are calculated based on the calculated frequency spectrum information corresponding to the audio signal to be processed.

[0077] In one embodiment, the encoding module 72 is specifically configured to:

[0078] Inputting the initial amplitude spectrum and phase spectrum corresponding to the audio signal to be processed into the convolutional layers of the amplitude encoder and the phase encoder in parallel for feature extraction to obtain the amplitude feature vector and the phase feature vector corresponding to the audio signal to be processed;

[0079] Inputting the amplitude eigenvector and the phase eigenvector corresponding to the audio signal to be processed into the normalization layer of the amplitude encoder and the phase encoder in parallel for normalization processing to obtain the normalized amplitude eigenvector and phase eigenvector of the audio signal to be processed;

[0080] Inputting the normalized amplitude eigenvector and phase eigenvector of the audio signal to be processed into the forward layer of the amplitude encoder and the phase encoder in parallel for nonlinear transformation processing to obtain the amplitude eigenvector and phase eigenvector of the audio signal to be processed after nonlinear transformation;

[0081] The amplitude eigenvector and phase eigenvector of the audio signal to be processed after nonlinear transformation are input in parallel to the downsampling layer of the amplitude encoder and the phase encoder for downsampling processing to obtain the amplitude coding vector and the phase coding vector corresponding to the audio signal to be processed.

[0082] In one embodiment, the quantization module 74 is specifically configured to:

[0083] Performing cluster analysis on the target feature vector of the audio signal to be processed to generate a codebook corresponding to the audio signal to be processed;

[0084] Calculating the quantization codewords in the generated codebook corresponding to the audio signal to be processed, and calculating the residual eigenvector of the audio signal to be processed according to the calculated quantization codewords of each codebook;

[0085] According to the calculated residual eigenvector of the audio signal to be processed, an amplitude latent vector and a phase latent vector of the audio signal to be processed are extracted.

[0086] In one embodiment, the decoding module 75 is specifically configured to:

[0087] Inputting the amplitude latent vector and the feature latent vector of the audio signal to be processed into the upsampling layer of the amplitude decoder and the phase decoder in parallel for upsampling processing, thereby obtaining the amplitude coding vector and the phase coding vector of the upsampled audio signal to be processed;

[0088] Inputting the amplitude coding vector and phase coding vector of the upsampled audio signal to be processed into the normalization layer of the amplitude decoder and the phase decoder for normalization processing to obtain the normalized amplitude coding vector and phase coding vector of the audio signal to be processed;

[0089] Inputting the normalized amplitude coding vector and phase coding vector of the audio signal to be processed into the forward layer of the amplitude decoder and the phase decoder for nonlinear transformation processing to obtain the amplitude coding vector and phase coding vector after nonlinear transformation of the audio signal to be processed;

[0090] The amplitude coding vector and the phase coding vector after the nonlinear transformation of the audio signal to be processed are input into the convolution layer of the amplitude decoder and the phase decoder for feature reconstruction to obtain the initial amplitude spectrum and the initial phase spectrum of the audio signal to be processed.

[0091] In one embodiment, the acquisition module 71 is specifically configured to:

[0092] The calculation formula of the initial amplitude spectrum corresponding to the audio signal to be processed is:

[0093]

[0094] in, | Y(k) | represents the initial amplitude spectrum corresponding to the audio signal to be processed, Y(k) represents the frequency spectrum information corresponding to the audio signal to be processed, Re(Y(k)) represents the real part of Y(k), and Im(Y(k)) represents the imaginary part of Y(k);

[0095] The calculation formula of the initial phase spectrum corresponding to the audio signal to be processed is:

[0096]

[0097] in, Represents the initial phase spectrum corresponding to the audio signal to be processed.

[0098] In one embodiment, the neural network-based audio signal encoding and decoding device is further configured to:

[0099] The signal-to-noise ratio of the reconstructed audio signal to be processed is calculated, and various parameter information in the encoder and decoder of the neural network are updated according to the calculated signal-to-noise ratio.

[0100] The present invention provides an audio signal encoding and decoding device based on a neural network. The device inputs the initial amplitude spectrum and initial phase spectrum corresponding to the audio signal to be processed into the amplitude encoder and the phase encoder respectively for parallel feature encoding; then inputs the target coding vector of the audio signal to be processed into the residual vector quantization module for quantization processing; finally, inputs the amplitude latent vector and phase latent vector of the audio signal to be processed into the amplitude decoder and the phase decoder respectively for parallel decoding, thereby realizing efficient compression and high-fidelity decoding of the audio signal, and effectively improving the sound quality and efficiency of the audio codec at low bit rates.

[0101] For the specific definition of the neural network-based audio signal encoding and decoding device, please refer to the definition of the neural network-based audio signal encoding and decoding method above, and will not be repeated here. The various modules in the above-mentioned neural network-based audio signal encoding and decoding device can be implemented in whole or in part by software, hardware, and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0102] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 8 As shown, Figure 8 1 is a structural diagram of a computer device in one embodiment of the present invention. The computer device includes a processor, a memory, a network interface, and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external client via a network connection. When the computer program is executed by the processor, it implements the functions or steps on the server side of a neural network-based audio signal encoding and decoding method.

[0103] In one embodiment, a computer device is provided. The computer device may be a client, and its internal structure diagram may be as follows: Figure 9 As shown, Figure 91 is another structural schematic diagram of a computer device in one embodiment of the present invention. The computer device includes a processor, memory, a network interface, a display screen, and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server via a network connection. When executed by the processor, the computer program implements the functions or steps on the client side of a neural network-based audio signal encoding and decoding method.

[0104] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the following steps are performed:

[0105] Acquire an initial audio signal to be processed, and perform short-time Fourier transform processing on the acquired initial audio signal to be processed to obtain an initial amplitude spectrum and an initial phase spectrum corresponding to the audio signal to be processed;

[0106] Inputting the initial amplitude spectrum and initial phase spectrum corresponding to the audio signal to be processed into an amplitude encoder and a phase encoder respectively for parallel feature encoding to obtain an amplitude coding vector and a phase coding vector corresponding to the audio signal to be processed;

[0107] Performing feature fusion on the amplitude coding vector and the phase coding vector corresponding to the audio signal to be processed to obtain a target coding vector of the audio signal to be processed;

[0108] Inputting the target coding vector of the audio signal to be processed into a residual vector quantization module for quantization processing to obtain an amplitude latent vector and a phase latent vector of the audio signal to be processed;

[0109] Inputting the amplitude latent vector and the phase latent vector of the audio signal to be processed into an amplitude decoder and a phase decoder respectively for parallel decoding to obtain an initial amplitude spectrum and an initial phase spectrum of the audio signal to be processed;

[0110] An inverse short-time Fourier transform process is performed on the initial amplitude spectrum and the initial phase spectrum of the audio signal to be processed to generate a reconstructed audio signal.

[0111] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0112] Acquire an initial audio signal to be processed, and perform short-time Fourier transform processing on the acquired initial audio signal to be processed to obtain an initial amplitude spectrum and an initial phase spectrum corresponding to the audio signal to be processed;

[0113] Inputting the initial amplitude spectrum and initial phase spectrum corresponding to the audio signal to be processed into an amplitude encoder and a phase encoder respectively for parallel feature encoding to obtain an amplitude coding vector and a phase coding vector corresponding to the audio signal to be processed;

[0114] Performing feature fusion on the amplitude coding vector and the phase coding vector corresponding to the audio signal to be processed to obtain a target coding vector of the audio signal to be processed;

[0115] Inputting the target coding vector of the audio signal to be processed into a residual vector quantization module for quantization processing to obtain an amplitude latent vector and a phase latent vector of the audio signal to be processed;

[0116] Inputting the amplitude latent vector and the phase latent vector of the audio signal to be processed into an amplitude decoder and a phase decoder respectively for parallel decoding to obtain an initial amplitude spectrum and an initial phase spectrum of the audio signal to be processed;

[0117] An inverse short-time Fourier transform process is performed on the initial amplitude spectrum and the initial phase spectrum of the audio signal to be processed to generate a reconstructed audio signal.

[0118] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can be found in the relevant descriptions of the server side and the client side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0119] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0120] Those skilled in the art will clearly understand that for the sake of convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0121] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention, and should all be included in the scope of protection of the present invention.

Claims

1. A method for encoding and decoding audio signals based on a neural network, characterized in that: include: Acquire an initial audio signal to be processed, and perform short-time Fourier transform processing on the acquired initial audio signal to be processed to obtain an initial amplitude spectrum and an initial phase spectrum corresponding to the audio signal to be processed; Inputting the initial amplitude spectrum and initial phase spectrum corresponding to the audio signal to be processed into an amplitude encoder and a phase encoder respectively for parallel feature encoding to obtain an amplitude coding vector and a phase coding vector corresponding to the audio signal to be processed; Performing feature fusion on the amplitude coding vector and the phase coding vector corresponding to the audio signal to be processed to obtain a target coding vector of the audio signal to be processed; Inputting the target coding vector of the audio signal to be processed into a residual vector quantization module for quantization processing to obtain an amplitude latent vector and a phase latent vector of the audio signal to be processed; Inputting the amplitude latent vector and the phase latent vector of the audio signal to be processed into an amplitude decoder and a phase decoder respectively for parallel decoding to obtain an initial amplitude spectrum and an initial phase spectrum of the audio signal to be processed; Performing inverse short-time Fourier transform processing on the initial amplitude spectrum and the initial phase spectrum of the audio signal to be processed to generate a reconstructed audio signal; The step of inputting the initial amplitude spectrum and the initial phase spectrum corresponding to the audio signal to be processed into an amplitude encoder and a phase encoder respectively for parallel feature encoding to obtain an amplitude coding vector and a phase coding vector corresponding to the audio signal to be processed includes: Inputting the initial amplitude spectrum and phase spectrum corresponding to the audio signal to be processed into the convolutional layers of the amplitude encoder and the phase encoder in parallel for feature extraction to obtain the amplitude feature vector and the phase feature vector corresponding to the audio signal to be processed; Inputting the amplitude eigenvector and the phase eigenvector corresponding to the audio signal to be processed into the normalization layer of the amplitude encoder and the phase encoder in parallel for normalization processing to obtain the normalized amplitude eigenvector and phase eigenvector of the audio signal to be processed; Inputting the normalized amplitude eigenvector and phase eigenvector of the audio signal to be processed into the forward layer of the amplitude encoder and the phase encoder in parallel for nonlinear transformation processing to obtain the amplitude eigenvector and phase eigenvector of the audio signal to be processed after nonlinear transformation; The amplitude eigenvector and phase eigenvector of the audio signal to be processed after nonlinear transformation are input in parallel to the downsampling layer of the amplitude encoder and the phase encoder for downsampling processing to obtain the amplitude coding vector and the phase coding vector corresponding to the audio signal to be processed.

2. The audio signal encoding and decoding method based on a neural network according to claim 1, characterized in that: Performing short-time Fourier transform processing on the obtained initial audio signal to be processed to obtain an initial amplitude spectrum and an initial phase spectrum corresponding to the audio signal to be processed, including: Splitting the obtained initial audio signal to be processed into a plurality of window audio signals, and selecting a window function for processing the window audio signals; Set the window length and window step of each window audio signal; Performing windowing processing on the audio signals in each window according to the selected window function for processing the audio signals in the window, the window length of the audio signals in each window, and the window step size of the audio signals in each window; Short-time Fourier transform is used to calculate the frequency spectrum information corresponding to the audio signal of each window after windowing processing, and the initial amplitude spectrum and initial phase spectrum corresponding to the audio signal to be processed are calculated based on the calculated frequency spectrum information corresponding to the audio signal to be processed.

3. The audio signal encoding and decoding method based on a neural network according to claim 1, characterized in that: The step of inputting the target feature vector of the audio signal to be processed into a residual vector quantization module for quantization processing to obtain an amplitude latent vector and a phase latent vector of the audio signal to be processed includes: Performing cluster analysis on the target feature vector of the audio signal to be processed to generate a codebook corresponding to the audio signal to be processed; Calculating the quantization codewords in the generated codebook corresponding to the audio signal to be processed, and calculating the residual eigenvector of the audio signal to be processed according to the calculated quantization codewords of each codebook; According to the calculated residual eigenvector of the audio signal to be processed, an amplitude latent vector and a phase latent vector of the audio signal to be processed are extracted.

4. The audio signal encoding and decoding method based on neural network according to claim 1, characterized in that: The step of inputting the amplitude latent vector and the feature latent vector of the audio signal to be processed into an amplitude decoder and a phase decoder in parallel for decoding to obtain an initial amplitude spectrum and an initial phase spectrum of the audio signal to be processed includes: Inputting the amplitude latent vector and the feature latent vector of the audio signal to be processed into the upsampling layer of the amplitude decoder and the phase decoder in parallel for upsampling processing, thereby obtaining the amplitude coding vector and the phase coding vector of the upsampled audio signal to be processed; Inputting the amplitude coding vector and phase coding vector of the upsampled audio signal to be processed into the normalization layer of the amplitude decoder and the phase decoder for normalization processing to obtain the normalized amplitude coding vector and phase coding vector of the audio signal to be processed; Inputting the normalized amplitude coding vector and phase coding vector of the audio signal to be processed into the forward layer of the amplitude decoder and the phase decoder for nonlinear transformation processing to obtain the amplitude coding vector and phase coding vector after nonlinear transformation of the audio signal to be processed; The amplitude coding vector and the phase coding vector after the nonlinear transformation of the audio signal to be processed are input into the convolution layer of the amplitude decoder and the phase decoder for feature reconstruction to obtain the initial amplitude spectrum and the initial phase spectrum of the audio signal to be processed.

5. The audio signal encoding and decoding method based on neural network according to claim 2, characterized in that: The step of calculating the initial amplitude spectrum and the initial phase spectrum corresponding to the audio signal to be processed according to the calculated frequency spectrum information corresponding to the audio signal to be processed includes: The calculation formula of the initial amplitude spectrum corresponding to the audio signal to be processed is: Wherein, |Y(k)| represents the initial amplitude spectrum corresponding to the audio signal to be processed, Y(k) represents the frequency spectrum information corresponding to the audio signal to be processed, Re(Y(k)) represents the real part of Y(k), and Im(Y(k)) represents the imaginary part of Y(k); The calculation formula of the initial phase spectrum corresponding to the audio signal to be processed is: in, Represents the initial phase spectrum corresponding to the audio signal to be processed.

6. The audio signal encoding and decoding method based on neural network according to claim 1, characterized in that: The method further comprises: The signal-to-noise ratio of the reconstructed audio signal to be processed is calculated, and various parameter information in the encoder and decoder of the neural network are updated according to the calculated signal-to-noise ratio.

7. A neural network-based audio signal encoding and decoding device, the device being used to implement the neural network-based audio signal encoding and decoding method according to any one of claims 1 to 6, characterized in that: include: An acquisition module is used to acquire an initial audio signal to be processed, and perform short-time Fourier transform processing on the acquired initial audio signal to be processed to obtain an initial amplitude spectrum and an initial phase spectrum corresponding to the audio signal to be processed; an encoding module, configured to input an initial amplitude spectrum and an initial phase spectrum corresponding to the audio signal to be processed into an amplitude encoder and a phase encoder, respectively, for parallel feature encoding to obtain an amplitude encoding vector and a phase encoding vector corresponding to the audio signal to be processed; a fusion module, configured to perform feature fusion on the amplitude coding vector and the phase coding vector corresponding to the audio signal to be processed to obtain a target coding vector of the audio signal to be processed; a quantization module, configured to input the target coding vector of the audio signal to be processed into a residual vector quantization module for quantization processing, thereby obtaining an amplitude latent vector and a phase latent vector of the audio signal to be processed; a decoding module, configured to input the amplitude latent vector and the phase latent vector of the audio signal to be processed into an amplitude decoder and a phase decoder respectively for parallel decoding, so as to obtain an initial amplitude spectrum and an initial phase spectrum of the audio signal to be processed; The reconstruction module is used to perform inverse short-time Fourier transform processing on the initial amplitude spectrum and the initial phase spectrum of the audio signal to be processed to generate a reconstructed audio signal.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the neural network-based audio signal encoding and decoding method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the neural network-based audio signal encoding and decoding method according to any one of claims 1 to 6 are implemented.

Images