Audio processing method and device, equipment, storage medium and computer program product
By performing subband decomposition and feature processing on the audio signal, reducing the feature dimensions and performing quantitative encoding, the problem of low audio coding efficiency in the prior art is solved, and efficient audio data compression is achieved.
Patent Information
- Application Number
- CN202411348883.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2022-06-15
- Publication Date
- 2024-11-12
AI Technical Summary
In the prior art, it is difficult to effectively improve the audio encoding efficiency while ensuring audio quality.
By performing subband decomposition of the audio signal, low-frequency and high-frequency subband signals are extracted, feature extraction and analysis are performed separately, feature dimensions are reduced, and quantitative encoding and decoding are performed to synthesize audio signals to improve coding efficiency.
While ensuring audio quality, the audio encoding efficiency is significantly improved, the feature dimension is reduced, and more efficient data compression is achieved.
Smart Images

Figure CN118942470A_ABST
Abstract
Description
[0001] This application is a divisional application of the patent application with application number 202210681365.X, application date June 15, 2022, and invention name “audio processing method, device, equipment, storage medium and computer program product”. Technical Field
[0002] The present application relates to data processing technology, and in particular to an audio processing method, device, electronic device, computer-readable storage medium and computer program product. Background Art
[0003] Audio coding technology is a core technology in communication services including remote audio and video calls. To put it simply, speech coding technology is to use less network bandwidth resources to transmit as much voice information as possible. From the perspective of Shannon's information theory, speech coding is a kind of source coding. The purpose of source coding is to compress the amount of data we want to transmit as much as possible at the encoding end, remove the redundancy in the information, and at the same time be able to restore it losslessly (or nearly losslessly) at the decoding end.
[0004] However, there is no effective solution in the related technology on how to effectively improve the efficiency of audio coding while ensuring the audio quality. Summary of the invention
[0005] The embodiments of the present application provide an audio processing method, device, electronic device, computer-readable storage medium, and computer program product, which can improve audio encoding efficiency while ensuring audio quality.
[0006] The technical solution of the embodiment of the present application is implemented as follows:
[0007] The present application provides an audio processing method, including:
[0008] Performing sub-band decomposition processing on the audio signal to obtain a low-frequency sub-band signal and a high-frequency sub-band signal of the audio signal;
[0009] Performing feature extraction processing on the low-frequency sub-band signal to obtain low-frequency features of the low-frequency sub-band signal;
[0010] Performing high-frequency analysis processing on the high-frequency sub-band signal to obtain high-frequency features of the high-frequency sub-band signal;
[0011] Wherein, the feature dimension of the high-frequency feature is lower than the feature dimension of the low-frequency feature;
[0012] The low-frequency features are quantized and encoded to obtain a low-frequency code stream of the audio signal, and the high-frequency features are quantized and encoded to obtain a high-frequency code stream of the audio signal.
[0013] The present application provides an audio processing method, including:
[0014] Performing quantization decoding processing on the low-frequency code stream to obtain low-frequency features corresponding to the low-frequency code stream, and performing quantization decoding processing on the high-frequency code stream to obtain high-frequency features corresponding to the high-frequency code stream;
[0015] The low-frequency code stream and the high-frequency code stream are obtained by respectively encoding sub-band signals obtained after sub-band decomposition of the audio signal, and the feature dimension of the high-frequency feature is lower than the feature dimension of the low-frequency feature;
[0016] Performing feature reconstruction processing on the low-frequency feature to obtain a low-frequency sub-band signal corresponding to the low-frequency feature;
[0017] Performing high-frequency reconstruction processing on the high-frequency features to obtain high-frequency sub-band signals corresponding to the high-frequency features;
[0018] Sub-band synthesis processing is performed on the low-frequency sub-band signal and the high-frequency sub-band signal to obtain a synthesized audio signal corresponding to the low-frequency code stream and the high-frequency code stream.
[0019] The present application provides an audio processing device, including:
[0020] A decomposition module, used to perform sub-band decomposition processing on the audio signal to obtain a low-frequency sub-band signal and a high-frequency sub-band signal of the audio signal;
[0021] A feature extraction module, used for performing feature extraction processing on the low-frequency sub-band signal to obtain low-frequency features of the low-frequency sub-band signal;
[0022] A high frequency analysis module, used for performing high frequency analysis processing on the high frequency sub-band signal to obtain high frequency features of the high frequency sub-band signal;
[0023] Wherein, the feature dimension of the high-frequency feature is lower than the feature dimension of the low-frequency feature;
[0024] The encoding module is used to perform quantization encoding processing on the low-frequency features to obtain a low-frequency code stream of the audio signal, and to perform quantization encoding processing on the high-frequency features to obtain a high-frequency code stream of the audio signal.
[0025] The present application provides an audio processing device, including:
[0026] A decoding module, used to perform quantization decoding processing on a low-frequency code stream to obtain low-frequency features corresponding to the low-frequency code stream, and to perform quantization decoding processing on a high-frequency code stream to obtain high-frequency features corresponding to the high-frequency code stream;
[0027] The low-frequency code stream and the high-frequency code stream are obtained by respectively encoding sub-band signals obtained after sub-band decomposition of the audio signal, and the feature dimension of the high-frequency feature is lower than the feature dimension of the low-frequency feature;
[0028] A feature reconstruction module, used to perform feature reconstruction processing on the low-frequency feature to obtain a low-frequency sub-band signal corresponding to the low-frequency feature;
[0029] A high-frequency reconstruction module, used to perform high-frequency reconstruction processing on the high-frequency features to obtain a high-frequency sub-band signal corresponding to the high-frequency features;
[0030] The synthesis module is used to perform sub-band synthesis processing on the low-frequency sub-band signal and the high-frequency sub-band signal to obtain a synthesized audio signal corresponding to the low-frequency code stream and the high-frequency code stream.
[0031] An embodiment of the present application provides an electronic device for audio processing, the electronic device comprising:
[0032] A memory for storing executable instructions;
[0033] The processor is used to implement the audio processing method provided in the embodiment of the present application when executing the executable instructions stored in the memory.
[0034] An embodiment of the present application provides a computer-readable storage medium storing executable instructions for causing a processor to execute and implement the audio processing method provided in the embodiment of the present application.
[0035] An embodiment of the present application provides a computer program product, including a computer program or instructions, which, when executed by a processor, implements the audio processing method provided in the embodiment of the present application.
[0036] The embodiments of the present application have the following beneficial effects:
[0037] By decomposing the audio signal into a low-frequency sub-band signal and a high-frequency sub-band signal, since the low-frequency sub-band signal has a greater impact on audio coding than the high-frequency sub-band signal, differential signal processing is performed on the low-frequency sub-band signal and the high-frequency sub-band signal, so that the feature dimension of the high-frequency feature is lower than that of the low-frequency feature, and the low-frequency features and high-frequency features with reduced feature dimensions are quantized and encoded respectively, thereby improving the audio coding efficiency while ensuring the audio quality. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] Figure 1 is a schematic diagram of spectrum comparison at different bit rates provided in an embodiment of the present application;
[0039] Figure 2It is a schematic diagram of the architecture of the audio codec system provided in an embodiment of the present application;
[0040] Figure 3 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application;
[0041] Figure 4 is a flowchart of an audio processing method provided in an embodiment of the present application;
[0042] Figure 5 is a flowchart of an audio processing method provided in an embodiment of the present application;
[0043] Figure 6 is a schematic diagram of an end-to-end voice communication link provided by an embodiment of the present application;
[0044] Figure 7 It is a flowchart of a speech encoding and decoding method based on subband decomposition and neural network provided in an embodiment of the present application;
[0045] Figure 8 is a schematic diagram of a filter bank provided in an embodiment of the present application;
[0046] Fig. 9A is a schematic diagram of a common convolutional network provided in an embodiment of the present application;
[0047] Fig. 9B is a schematic diagram of a dilated convolutional network provided in an embodiment of the present application;
[0048] Fig.10 is a schematic diagram of frequency band expansion provided by an embodiment of the present application;
[0049] Fig.11 is a schematic diagram of a first neural network provided in an embodiment of the present application;
[0050] Fig.12 is a schematic diagram of a neural network structure for high frequency sub-band signals provided in an embodiment of the present application;
[0051] Fig.13 is a schematic diagram of a second neural network provided in an embodiment of the present application. DETAILED DESCRIPTION
[0052] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.
[0053] In the following description, the terms "first\second" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.
[0054] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.
[0055] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.
[0056] 1) Neural Network (NN): It is an algorithmic mathematical model that imitates the behavioral characteristics of animal neural networks and performs distributed parallel information processing. This network relies on the complexity of the system and adjusts the interconnected relationships between a large number of internal nodes to achieve the purpose of processing information.
[0057] 2) Deep Learning (DL): It is a new research direction in the field of Machine Learning (ML). Deep learning is to learn the inherent laws and representation levels of sample data. The information obtained in the learning process is very helpful for interpreting data such as text, images and sounds. Its ultimate goal is to enable machines to have analytical learning capabilities like humans and to recognize data such as text, images and sounds.
[0058] 3) Quantization: It refers to the process of approximating the continuous value (or a large number of discrete values) of a signal to a finite number (or a small number) of discrete values. Quantization includes vector quantization (VQ) and scalar quantization.
[0059] Among them, vector quantization is an effective lossy compression technology, and its theoretical basis is Shannon's rate-distortion theory. The basic principle of vector quantization is to replace the input vector with the index of the codeword that best matches the input vector in the code book for transmission and storage, while decoding only requires a simple table lookup operation. For example, a number of scalar data are combined into a vector space, and the vector space is divided into several small areas. When quantizing, the vectors that fall into the small areas are replaced by the corresponding indexes instead of the input vectors.
[0060] Scalar quantization is the quantization of scalars, that is, one-dimensional vector quantization, which divides the dynamic range into several small intervals, each of which has a representative value (i.e., index). When the input signal falls into a certain interval, the input signal is quantized into the representative value.
[0061] 4) Entropy coding: A lossless coding method that does not lose any information according to the entropy principle during the encoding process. It is also a key module in lossy coding and is located at the end of the encoder. Entropy coding includes Shannon coding, Huffman coding, Exp-Golomb coding, and arithmetic coding.
[0062] 5) Quadrature Mirror Filters (QMF): It is a filter pair that includes analysis and synthesis. The QMF analysis filter is used to decompose the sub-band signal to reduce the signal bandwidth so that each sub-band signal can be processed smoothly through its own channel; the QMF synthesis filter is used to synthesize the sub-band signals restored by the decoding end, for example, to reconstruct the original audio signal through zero-value interpolation and bandpass filtering.
[0063] Voice coding technology is to use less network bandwidth resources to transmit as much voice information as possible. The compression rate of voice codec can reach more than 10 times, that is, after the original 10MB of voice data is compressed by the encoder, only 1MB is needed to transmit, which greatly reduces the bandwidth resources consumed by transmitting information. For example, for a wideband voice signal with a sampling rate of 16000Hz, if a 16-bit sampling depth (the degree of precision of recording voice intensity during sampling) is used, the bit rate (the amount of data transmitted per unit time) of the uncompressed version is 256kbps; if voice coding technology is used, even if it is lossy coding, within the bit rate range of 10-20kbps, the quality of the reconstructed voice signal can be close to the uncompressed version, and even the auditory perception is no different. If a higher sampling rate service is required, such as 32000Hz ultra-wideband voice, the bit rate range must be at least 30kbps.
[0064] In the communication system, in order to ensure smooth communication, the industry deploys standard voice codec protocols, such as standards from international and domestic standard organizations such as ITU-T, 3GPP, IETF, AVS, CCSA, G.711, G.722, AMR series, EVS, OPUS and other standards. Figure 1A spectrum comparison diagram at different bit rates is given to demonstrate the relationship between compression bit rate and quality. Curve 101 is the spectrum curve of the original speech, that is, the signal without compression; Curve 102 is the spectrum curve of the OPUS encoder at a bit rate of 20kbps; Curve 103 is the spectrum curve of the OPUS encoder at a bit rate of 6kbps. Figure 1 It can be seen that as the encoding bit rate increases, the compressed signal is closer to the original signal.
[0065] In the related art, the principle of speech coding is roughly as follows: speech coding can directly encode the speech waveform samples, sample by sample; or, based on the principle of human vocalization, relevant low-dimensional features are extracted, the encoding end encodes the features, and the decoding end reconstructs the speech signal based on these parameters.
[0066] The above coding principles all come from voice signal modeling, that is, a compression method based on signal processing. In order to improve the coding efficiency while ensuring the voice quality relative to the compression method based on signal processing. The embodiments of the present application provide an audio processing method, a device, an electronic device, a computer-readable storage medium and a computer program product, which can improve the coding efficiency. The following describes an exemplary application of the electronic device provided by the embodiments of the present application. The electronic device provided by the embodiments of the present application can be implemented as a terminal device, or can be implemented as a server, or can be implemented by a terminal device and a server in collaboration. The following is an example of an electronic device implemented as a terminal device.
[0067] For example, see Figure 2 , Figure 2 It is a schematic diagram of the architecture of the audio encoding and decoding system 100 provided in an embodiment of the present application. The audio decoding system 100 includes: a server 200, a network 300, a terminal device 400 (i.e., an encoding end) and a terminal device 500 (i.e., a decoding end), wherein the network 300 can be a local area network, a wide area network, or a combination of the two.
[0068] In some embodiments, a client 410 is running on the terminal device 400, and the client 410 can be various types of clients, such as an instant messaging client, a web conference client, a live broadcast client, a browser, etc. In response to an audio collection instruction triggered by a sender (such as an initiator of a web conference, a host, an initiator of a voice call, etc.), the client 410 calls the microphone of the terminal device 400 to collect audio signals, and encodes the collected audio signals to obtain a bit stream.
[0069] For example, the client 410 calls the audio processing method provided in the embodiment of the present application to encode the collected audio signal, that is, the audio signal is subjected to sub-band decomposition processing to obtain a low-frequency sub-band signal and a high-frequency sub-band signal of the audio signal, the low-frequency sub-band signal is subjected to feature extraction processing to obtain a low-frequency feature of the low-frequency sub-band signal, the high-frequency sub-band signal is subjected to high-frequency analysis processing to obtain a high-frequency feature of the high-frequency sub-band signal, wherein the feature dimension of the high-frequency feature is lower than the feature dimension of the low-frequency feature, the low-frequency feature is subjected to quantization coding processing to obtain a low-frequency code stream of the audio signal, and the high-frequency feature is subjected to quantization coding processing to obtain a high-frequency code stream of the audio signal. The encoding end (i.e., the terminal device 400) performs differential signal processing on the low-frequency sub-band signal and the high-frequency sub-band signal, so that the feature dimension of the high-frequency feature is lower than the feature dimension of the low-frequency feature, and the low-frequency feature and the high-frequency feature with reduced feature dimension are quantized and encoded respectively, thereby improving the audio coding efficiency while ensuring the audio quality.
[0070] The client 410 can send the code stream (i.e., the low-frequency code stream and the high-frequency code stream) to the server 200 through the network 300, so that the server 200 sends the code stream to the terminal device 500 associated with the recipient (such as the participants, audience, recipient of the voice call, etc. of the online conference).
[0071] After receiving the code stream sent by the server 200, the client 510 (such as an instant messaging client, a web conference client, a live broadcast client, a browser, etc.) can decode the code stream to obtain an audio signal, thereby realizing audio communication.
[0072] For example, the client 510 calls the audio processing method provided in the embodiment of the present application to decode the received code stream, that is, performs quantization decoding processing on the low-frequency code stream to obtain the low-frequency features corresponding to the low-frequency code stream, and performs quantization decoding processing on the high-frequency code stream to obtain the high-frequency features corresponding to the high-frequency code stream, performs feature reconstruction processing on the low-frequency features to obtain the low-frequency sub-band signal corresponding to the low-frequency features, performs high-frequency reconstruction processing on the high-frequency features to obtain the high-frequency sub-band signal corresponding to the high-frequency features, performs sub-band synthesis processing on the low-frequency sub-band signal and the high-frequency sub-band signal to obtain the decoded audio signal.
[0073] In some embodiments, the embodiments of the present application can be implemented with the help of cloud technology (Cloud Technology). Cloud technology refers to a hosting technology that unifies a series of resources such as hardware, software, and network within a wide area network or a local area network to achieve data calculation, storage, processing, and sharing.
[0074] Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, and application technology based on the cloud computing business model. It can form a resource pool, which is used on demand and is flexible and convenient. Cloud computing technology will become an important support. The service interaction function between the above servers 200 can be realized through cloud technology.
[0075] For example, Figure 2 The server 200 shown in the figure can be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, as well as big data and artificial intelligence platforms. Figure 2 The terminal devices 400 and 500 shown in the figure can be smart phones, tablet computers, laptop computers, desktop computers, smart speakers, smart watches, vehicle-mounted terminals, etc., but are not limited thereto. The terminal devices (such as the terminal devices 400 and 500) and the server 200 can be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of the present application.
[0076] In some embodiments, the terminal device or server 200 can also implement the audio processing method provided in the embodiment of the present application by running a computer program. For example, the computer program can be a native program or software module in the operating system; it can be a native application (APP, Application), that is, a program that needs to be installed in the operating system to run, such as a live broadcast APP, a web conference APP, or an instant messaging APP, etc.; it can also be a small program, that is, a program that can be run only by downloading it to a browser environment; it can also be a small program that can be embedded in any APP. In short, the above-mentioned computer program can be an application, module or plug-in in any form.
[0077] In some embodiments, multiple servers may form a blockchain, and server 200 is a node on the blockchain. Information connections may exist between each node in the blockchain, and information may be transmitted between nodes through the above information connections. Among them, data related to the audio processing method provided in the embodiment of the present application (such as audio processing logic and code stream) may be stored on the blockchain.
[0078] See also Figure 3 , Figure 3 is a schematic diagram of the structure of an electronic device 500 provided in an embodiment of the present application, and takes the electronic device 500 as a terminal device as an example. Figure 3The electronic device 500 shown includes: at least one processor 520, a memory 550, at least one network interface 530 and a user interface 540. The various components in the electronic device 500 are coupled together via a bus system 550. It is understood that the bus system 550 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 550 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 550 is not described in detail. Figure 3 Various buses are labeled as bus system 550 .
[0079] The processor 520 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.
[0080] The user interface 540 includes one or more output devices 541 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 540 also includes one or more input devices 542, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.
[0081] The memory 550 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical drives, etc. The memory 550 may optionally include one or more storage devices that are physically remote from the processor 520.
[0082] The memory 550 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 550 described in the embodiments of the present application is intended to include any suitable type of memory.
[0083] In some embodiments, the memory 550 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.
[0084] Operating system 551, including system programs for processing various basic system services and performing hardware-related tasks, such as framework layer, core library layer, driver layer, etc., for implementing various basic services and processing hardware-based tasks;
[0085] A network communication module 552, used to reach other computing devices via one or more (wired or wireless) network interfaces 530, exemplary network interfaces 530 include: Bluetooth, Wireless Compatibility Authentication (WiFi), and Universal Serial Bus (USB);
[0086] a presentation module 553 for enabling presentation of information via one or more output devices 541 (e.g., display screen, speaker, etc.) associated with the user interface 540 (e.g., a user interface for operating peripheral devices and displaying content and information);
[0087] The input processing module 554 is used to detect one or more user inputs or interactions from one of the one or more input devices 542 and translate the detected inputs or interactions.
[0088] In some embodiments, the audio processing device provided in the embodiments of the present application can be implemented in software. Figure 3 An audio processing device 555 stored in a memory 550 is shown, which may be software in the form of a program or plug-in, including the following software modules: a decomposition module 5551, a feature extraction module 5552, a high-frequency analysis module 5553, an encoding module 5554, or a decoding module 5555, a feature reconstruction module 5556, a high-frequency reconstruction module 5557 and a synthesis module 5558, wherein the decomposition module 5551, the feature extraction module 5552, the high-frequency analysis module 5553, the encoding module 5554 are used to implement an audio encoding function, and the decoding module 5555, the feature reconstruction module 5556, the high-frequency reconstruction module 5557 and the synthesis module 5558 are used to implement an audio decoding function, and these modules are logical, and therefore can be arbitrarily combined or further split according to the functions implemented.
[0089] As mentioned above, the audio processing method provided in the embodiments of the present application can be implemented by various types of electronic devices. Figure 4 , Figure 4 The flowchart of the audio processing method provided in the embodiment of the present application is as follows: Figure 4 The steps shown are explained.
[0090] In step 101, sub-band decomposition processing is performed on an audio signal to obtain a low-frequency sub-band signal and a high-frequency sub-band signal of the audio signal.
[0091] As an example of obtaining an audio signal, the encoding end responds to an audio capture instruction triggered by a sender (such as the initiator of a web conference, a host, the initiator of a voice call, etc.), and calls the microphone of the terminal device of the encoding end to collect the audio signal to obtain the audio signal (also called the input signal).
[0092] After acquiring the audio signal, the audio signal is decomposed into low-frequency sub-band signals x through the QMF analysis filter. LB (n) and the high frequency subband signal x HB (n) Since the low-frequency sub-band signal has a greater impact on audio coding than the high-frequency sub-band signal, differential signal processing can be performed on the low-frequency sub-band signal and the high-frequency sub-band signal subsequently.
[0093] In some embodiments, sub-band decomposition processing is performed on an audio signal to obtain a low-frequency sub-band signal and a high-frequency sub-band signal of the audio signal, including: sampling the audio signal to obtain a sampled signal, wherein the sampled signal includes multiple sample points obtained by sampling; low-pass filtering the sampled signal to obtain a low-pass filtered signal; down-sampling the low-pass filtered signal to obtain a low-frequency sub-band signal of the audio signal; high-pass filtering the sampled signal to obtain a high-pass filtered signal; down-sampling the high-pass filtered signal to obtain a high-frequency sub-band signal of the audio signal.
[0094] It should be noted that the audio signal is a continuous analog signal, the sampling signal is a discrete digital signal, and the sampling point is a sampling value obtained by sampling the audio signal.
[0095] As an example, take an audio signal as an input signal with a sampling rate of Fs=32000 Hz as an example, sample the audio signal to obtain a sampled signal x(n) including 640 sample points. Call the analysis filter (2 channels) in the QMF filter bank, perform low-pass filtering on the sampled signal to obtain a low-pass filtered signal, perform high-pass filtering on the sampled signal to obtain a high-pass filtered signal, perform down-sampling on the low-pass filtered signal to obtain a low-frequency subband signal x of the audio signal. LB (n), downsample the high-pass filtered signal to obtain the high-frequency sub-band signal x of the audio signal HB (n). Low frequency subband signal x LB (n) and the high frequency subband signal x HB The effective bandwidth of (n) is 0-8kHz and 8-16kHz respectively, and the low-frequency subband signal x LB (n) and the high frequency subband signal x HB The number of sample points of (n) is 320.
[0096] It should be noted that the QMF filter bank is a filter pair that includes analysis and synthesis. For the QMF analysis filter, the input signal with a sampling rate of Fs can be decomposed into two signals with a sampling rate of Fs / 2, which represent the QMF low-pass signal and the QMF high-pass signal respectively. After the low-pass signal and high-pass signal are restored by the decoding end, they are synthesized by the QMF synthesis filter, and the reconstructed signal with a sampling rate of Fs corresponding to the input signal can be restored.
[0097] In step 102, feature extraction processing is performed on the low-frequency sub-band signal to obtain low-frequency features of the low-frequency sub-band signal.
[0098] For example, since the low-frequency sub-band signal has a greater impact on audio coding than the high-frequency sub-band signal, the low-frequency sub-band signal can be subjected to feature extraction processing through a neural network model to obtain the low-frequency features of the low-frequency sub-band signal, so as to minimize the feature dimension of the low-frequency features while ensuring the completeness of the low-frequency features. The embodiment of the present application is not limited to the structure of the neural network model. Among them, the dimension of the low-frequency features of the low-frequency sub-band signal is smaller than the dimension of the low-frequency sub-band signal.
[0099] In some embodiments, feature extraction processing is performed on the low-frequency sub-band signal to obtain low-frequency features of the low-frequency sub-band signal, including: convolution processing is performed on the low-frequency sub-band signal to obtain convolution features of the low-frequency sub-band signal; pooling processing is performed on the convolution features to obtain pooling features of the low-frequency sub-band signal; down-sampling processing is performed on the pooled features to obtain down-sampled features of the low-frequency sub-band signal; convolution processing is performed on the down-sampled features to obtain the low-frequency features of the low-frequency sub-band signal.
[0100] like Fig.11 As shown, based on the low-frequency subband signal x LB (n) Call the neural network model to generate a lower-dimensional feature vector F LB (n), that is, the low-frequency feature. First, the input low-frequency subband signal x is subjected to causal convolution LB (n) is convolved to obtain a 24×320 convolution feature. Then, the 24×320 convolution feature is pooled with a factor of 2 (i.e., preprocessed) to obtain a 24×160 pooled feature. Next, the 24×160 pooled feature is downsampled to obtain a 192×1 downsampled feature. Finally, the 192×1 downsampled feature is convolved again with a causal convolution to obtain a 56-dimensional feature vector F. LB (n).
[0101] In some embodiments, the downsampling process is implemented through multiple cascaded coding layers; the pooling features are downsampled to obtain the downsampled features of the low-frequency sub-band signal, including: downsampling the pooling features through the first coding layer in the multiple cascaded coding layers; outputting the downsampling result of the first coding layer to the subsequent cascaded coding layers, continuing the downsampling process and outputting the downsampling results through the subsequent cascaded coding layers until output to the last coding layer; and using the downsampling result output by the last coding layer as the downsampling feature of the low-frequency sub-band signal.
[0102] like Fig.11 As shown in the figure, three coding blocks (i.e., coding layers) with different downsampling factors (Down_factor) are cascaded to downsample the pooled features, that is, the 24×160 pooled features are first downsampled by the coding block with the downsampling factor (Down_factor=4) to obtain a 48×40 downsampled result, and then the 48×40 downsampled result is downsampled by the coding block with the downsampling factor (Down_factor=5) to obtain a 96×8 downsampled result, and finally the 96×8 downsampled result is downsampled by the coding block with the downsampling factor (Down_factor=8) to obtain a 192×1 downsampled feature. Taking the coding block (Down_factor=4) as an example, one or more dilated convolutions can be performed first, and pooling can be performed based on the Down_factor to achieve the downsampling effect.
[0103] It should be noted that after processing through one coding layer, the understanding of downsampling features is deepened. After learning through multiple coding layers, the downsampling features of low-frequency sub-band signals can be gradually and accurately learned. Through the cascaded coding layers, the downsampling features of low-frequency sub-band signals with progressive accuracy can be obtained.
[0104] In step 103, high frequency analysis processing is performed on the high frequency sub-band signal to obtain high frequency features of the high frequency sub-band signal.
[0105] Among them, the feature dimension of the high-frequency feature is lower than that of the low-frequency feature. Since the low-frequency sub-band signal has a greater impact on audio coding than the high-frequency sub-band signal, the low-frequency sub-band signal and the high-frequency sub-band signal are subjected to differentiated signal processing, so that the feature dimension of the high-frequency feature is lower than that of the low-frequency feature. Among them, the dimension of the low-frequency feature of the high-frequency sub-band signal is smaller than that of the high-frequency sub-band signal, and the high-frequency analysis processing is used to reduce the dimension of the high-frequency sub-band signal to achieve the function of data compression.
[0106] In some embodiments, high-frequency analysis processing is performed on the high-frequency sub-band signal to obtain high-frequency features of the high-frequency sub-band signal, including: calling a first neural network model, performing feature extraction processing on the high-frequency sub-band signal through the first neural network model, and obtaining high-frequency features of the high-frequency sub-band signal.
[0107] For example, a first neural network model similar to the neural network model for the low-frequency subband signal may be called for the high-frequency subband signal (eg Fig.12 As shown), the high-frequency sub-band signal is convolved through the first neural network model to obtain the convolution feature of the high-frequency sub-band signal; the convolution feature is pooled to obtain the pooled feature of the high-frequency sub-band signal; the pooled feature is down-sampled to obtain the down-sampled feature of the high-frequency sub-band signal; the down-sampled feature is convolved to obtain the low-frequency feature of the high-frequency sub-band signal.
[0108] It should be noted that, compared with the low-frequency sub-band signal, the high-frequency sub-band signal is less important to the quality. Therefore, the first neural network model ( Fig.12 ) does not need to be structured like Fig.11 The model structure shown is not as complex as Fig.11 The model structure shown, Fig.12 The model structure shown significantly reduces the number of channels.
[0109] In some embodiments, performing high frequency analysis processing on the high frequency sub-band signal to obtain the high frequency characteristics of the high frequency sub-band signal includes: performing frequency band extension processing on the high frequency sub-band signal to obtain the high frequency characteristics of the high frequency sub-band signal.
[0110] For example, since the high-frequency subband signal is relatively less important to quality than the low-frequency subband signal, the high-frequency subband signal can be compressed by another method, namely, band expansion (recovering a broadband speech signal from a band-limited narrowband speech signal) to quickly compress the high-frequency subband signal and extract the high-frequency features of the high-frequency subband signal.
[0111] In some embodiments, a high-frequency sub-band signal is subjected to frequency band expansion processing to obtain high-frequency characteristics of the high-frequency sub-band signal, including: performing frequency domain transform processing based on multiple sample points included in the high-frequency sub-band signal to obtain transform coefficients corresponding to the multiple sample points; dividing the transform coefficients corresponding to the multiple sample points into multiple sub-bands; performing mean processing on the transform coefficients included in each sub-band to obtain average energy corresponding to each sub-band, and using the average energy as the sub-band spectral envelope corresponding to each sub-band; and determining the sub-band spectral envelopes corresponding to the multiple sub-bands as the high-frequency characteristics of the high-frequency sub-band signal.
[0112] It should be noted that the frequency domain transformation method of the embodiment of the present application includes modified discrete cosine transform (MDCT, Modified Discrete Cosine Transform), discrete cosine transform (DCT, Discrete Cosine Transform), fast Fourier transform (FFT, Fast Fourier Transform), etc., and the embodiment of the present application is not limited to the method of frequency domain transformation. The mean processing of the embodiment of the present application includes arithmetic mean and geometric mean, and the embodiment of the present application is not limited to the method of mean processing.
[0113] In some embodiments, frequency domain transform processing is performed based on multiple sample points included in the high-frequency sub-band signal to obtain transform coefficients corresponding to the multiple sample points, including: obtaining a reference high-frequency sub-band signal of a reference audio signal, wherein the reference audio signal is an audio signal adjacent to the audio signal; based on the multiple sample points included in the reference high-frequency sub-band signal and the multiple sample points included in the high-frequency sub-band signal, discrete cosine transform processing is performed on the multiple sample points included in the high-frequency sub-band signal to obtain transform coefficients corresponding to the multiple sample points included in the high-frequency sub-band signal.
[0114] In some embodiments, the process of performing geometric mean processing on the transform coefficients included in each sub-band is as follows: determining the sum of squares of the transform coefficients corresponding to the sample points included in each sub-band; and determining the ratio of the sum of squares to the number of sample points included in the sub-band to obtain the average energy corresponding to each sub-band.
[0115] As an example, for a high frequency subband signal x comprising 320 points HB (n), call the modified discrete cosine transform (MDCT) to generate 320-point MDCT coefficients (i.e., the transform coefficients corresponding to the multiple sample points included in the high-frequency subband signal). Specifically, if there is a 50% overlap, the n+1th frame of high-frequency data (i.e., the reference audio signal) and the nth frame of high-frequency data (i.e., the audio signal) can be merged (joined), and the 640-point MDCT is calculated to obtain 320-point MDCT coefficients.
[0116] The 320-point MDCT coefficients are divided into N sub-bands (i.e., the transform coefficients corresponding to the multiple sample points are divided into multiple sub-bands). The sub-bands here are multiple adjacent MDCT coefficients that form a group. The 320-point MDCT coefficients can be divided into 8 sub-bands. For example, the 320 points can be evenly distributed, that is, the number of points included in each sub-band is the same. Of course, the embodiment of the present application cannot perform non-uniform division of the 320 points, such as the sub-bands that are biased towards low frequencies include fewer MDCT coefficients (higher frequency resolution), and the sub-bands that are biased towards high frequencies include more MDCT coefficients (lower frequency resolution).
[0117] According to Nyquist sampling theorem (to restore the original signal from the sampled signal without distortion, the sampling frequency should be greater than 2 times the highest frequency of the original signal. When the sampling frequency is less than 2 times the highest frequency of the spectrum, the spectrum of the signal is aliased. When the sampling frequency is greater than 2 times the highest frequency of the spectrum, the spectrum of the signal is not aliased), the 320-point MDCT coefficients above represent the spectrum of 8-16kHz. However, ultra-wideband voice communication does not necessarily require the spectrum to be 16kHz. For example, if the spectrum is set to 14kHz, only the MDCT coefficients of the first 240 points need to be considered. Correspondingly, the number of subbands can be controlled to 6.
[0118] For each subband, the average energy of all MDCT coefficients in the current subband is calculated (i.e., the transform coefficients included in each subband are averaged) as the subband spectrum envelope (the spectrum envelope is a smooth curve passing through the main peak points of the spectrum). For example, the MDCT coefficients included in the current subband are x(n), n = 1, 2, ..., 40, then the average energy Y is calculated by the geometric mean = ((x(1) 2 +x(2) 2 +…+x(40) 2 ) / 40). When the 320-point MDCT coefficients are divided into 8 sub-bands, 8 sub-band spectrum envelopes can be obtained. These 8 sub-band spectrum envelopes are the feature vectors F of the generated high-frequency sub-band signal. HB (n), i.e., high-frequency features.
[0119] In step 104, the low-frequency features are quantized and encoded to obtain a low-frequency code stream of the audio signal, and the high-frequency features are quantized and encoded to obtain a high-frequency code stream of the audio signal.
[0120] For example, differential signal processing is performed on the low-frequency subband signal and the high-frequency subband signal so that the feature dimension of the high-frequency feature is lower than that of the low-frequency feature. The low-frequency features and high-frequency features with reduced feature dimensions are quantized and encoded respectively, and the high-frequency code stream and the low-frequency code stream are transmitted to the decoding end. The high-frequency code stream and the low-frequency code stream are decoded by the decoding end to restore the audio signal, thereby improving the audio coding efficiency while ensuring the audio quality.
[0121] In some embodiments, low-frequency features are quantized and encoded to obtain a low-frequency code stream of an audio signal, including: quantizing the low-frequency features to obtain an index value of the low-frequency features; entropy encoding the index value of the low-frequency features to obtain a low-frequency code stream of the audio signal; quantizing and encoding the high-frequency features to obtain a high-frequency code stream of the audio signal, including: quantizing the high-frequency features to obtain an index value of the high-frequency features; entropy encoding the index value of the high-frequency features to obtain a high-frequency code stream of the audio signal.
[0122] For example, for the eigenvector F of the low-frequency subband signal LB (n) and the eigenvector F of the high-frequency subband signal HB (n), both scalar quantization (each component is quantized separately) and entropy coding methods can be performed. In addition, the embodiment of the present application does not limit the technical combination of vector quantization (multiple adjacent components are combined into a vector for joint quantization) and entropy coding, and the high-frequency code stream and the low-frequency code stream obtained by encoding are transmitted to the decoding end, and the high-frequency code stream and the low-frequency code stream are decoded by the decoding end.
[0123] As mentioned above, the audio processing method provided in the embodiments of the present application can be implemented by various types of electronic devices. Figure 5 , Figure 5 The flowchart of the audio processing method provided in the embodiment of the present application is as follows: Figure 5 The steps shown are explained.
[0124] In step 201, a low-frequency code stream is quantized and decoded to obtain low-frequency features corresponding to the low-frequency code stream, and a high-frequency code stream is quantized and decoded to obtain high-frequency features corresponding to the high-frequency code stream.
[0125] The low-frequency code stream and the high-frequency code stream are obtained by respectively encoding sub-band signals obtained after the audio signal is decomposed into sub-bands, and the feature dimension of the high-frequency feature is lower than that of the low-frequency feature.
[0126] For example, by Figure 4 After the audio processing method shown encodes the high-frequency code stream and the low-frequency code stream, the encoded high-frequency code stream and the low-frequency code stream are transmitted to the decoding end. After the decoding end receives the high-frequency code stream and the low-frequency code stream, it quantizes and decodes the low-frequency code stream to obtain the low-frequency features corresponding to the low-frequency code stream, and quantizes and decodes the high-frequency code stream to obtain the high-frequency features corresponding to the high-frequency code stream.
[0127] It should be noted that quantization decoding is the inverse process of quantization encoding. For the received bit stream, entropy decoding is first performed, and the estimated value F′ of the low-frequency feature vector is obtained by looking up the quantization table (i.e. inverse quantization, the quantization table is the mapping table generated by quantization during the encoding process). LB (n), that is, the low-frequency features corresponding to the low-frequency code stream, obtain the estimated value F′ of the high-frequency feature vector HB (n), that is, the high-frequency features corresponding to the high-frequency code stream. It should be noted that the process of decoding the received code stream by the decoding end is the inverse process of the encoding process by the encoding end. Therefore, the value generated in the decoding process is an estimate relative to the value in the encoding process. For example, the high-frequency features generated in the decoding process are an estimate relative to the high-frequency features in the encoding process.
[0128] For example, a low-frequency code stream is quantized and decoded to obtain low-frequency features corresponding to the low-frequency code stream, including: entropy decoding the low-frequency code stream to obtain an index value corresponding to the low-frequency code stream; inverse quantization is performed on the index value corresponding to the low-frequency code stream to obtain low-frequency features corresponding to the low-frequency code stream; a high-frequency code stream is quantized and decoded to obtain high-frequency features corresponding to the high-frequency code stream, including: entropy decoding the high-frequency code stream to obtain an index value corresponding to the high-frequency code stream; inverse quantization is performed on the index value corresponding to the high-frequency code stream to obtain high-frequency features corresponding to the high-frequency code stream.
[0129] In step 202, feature reconstruction processing is performed on the low-frequency features to obtain low-frequency sub-band signals corresponding to the low-frequency features.
[0130] For example, feature reconstruction is the inverse process of feature extraction, and feature reconstruction processing is performed on low-frequency features to obtain low-frequency sub-band signals (an estimated value) corresponding to the low-frequency features.
[0131] In some embodiments, feature reconstruction processing is performed on the low-frequency features to obtain a low-frequency sub-band signal corresponding to the low-frequency features, including: convolution processing on the low-frequency features to obtain convolution features of the low-frequency features; up-sampling processing on the convolution features to obtain up-sampled features of the low-frequency features; pooling processing on the up-sampled features to obtain pooled features of the low-frequency features; convolution processing on the pooled features to obtain a low-frequency sub-band signal corresponding to the low-frequency features.
[0132] like Fig.13 As shown, based on the low-frequency feature vector F′ LB (n), calling Fig.13 The neural network model shown generates the low-frequency subband signal x′ LB (n). Among them, Fig.13 The neural network model shown is similar to Fig.11The neural network model shown is similar, such as causal convolution, post-processing structure is similar to pre-processing. The structure of the decoding block is symmetrical with the encoding block on the encoding side: the encoding block on the encoding side first performs dilated convolution and then pooling to complete downsampling, while the decoding block on the decoding side first performs pooling to complete upsampling and then performs dilated convolution.
[0133] First, the low-frequency feature vector F′ of the input is transformed through causal convolution LB (n) is convolved to obtain a 192×1 convolution feature. Then, the 192×1 convolution feature is upsampled to obtain a 24×160 upsampled feature. Next, the 24×160 upsampled feature is pooled (i.e., post-processed) to obtain a 24×320 pooled feature. Finally, the pooled feature is convolved again through causal convolution to obtain a 320-dimensional low-frequency subband signal x′ LB (n).
[0134] In some embodiments, upsampling processing is implemented through multiple cascaded decoding layers; upsampling processing is performed on convolution features to obtain upsampled features of low-frequency features, including: upsampling processing is performed on convolution features through the first decoding layer in multiple cascaded decoding layers; the upsampling result of the first decoding layer is output to the subsequent cascaded decoding layers, and upsampling processing and upsampling result output are continued through the subsequent cascaded decoding layers until it is output to the last decoding layer; the upsampling result output by the last decoding layer is used as the upsampled feature of the low-frequency feature.
[0135] like Fig.13 As shown in the figure, three decoding blocks (i.e., decoding layers) with different upsampling factors (Up_factor) are cascaded to upsample the convolution features, i.e., the 192×1 convolution features are first upsampled by the decoding block with upsampling factor (Up_factor=8) to obtain a 96×8 upsampled result, and then the 96×8 upsampled result is upsampled by the decoding block with upsampling factor (Up_factor=5) to obtain a 48×40 upsampled result, and finally the 48×40 upsampled result is upsampled by the decoding block with upsampling factor (Up_factor=4) to obtain a 24×160 upsampled feature. Taking the decoding block (Up_factor=4) as an example, pooling can be performed based on Up_factor first, and then one or more dilated convolutions can be performed to achieve the upsampling effect.
[0136] It should be noted that after processing through a layer of decoding, the understanding of upsampling features is deepened. After learning through multiple layers of decoding, the upsampling features of low-frequency features can be gradually and accurately learned. Through the cascaded decoding layers, the upsampling features of low-frequency features with progressive accuracy can be obtained.
[0137] In step 203, high-frequency reconstruction processing is performed on the high-frequency features to obtain high-frequency sub-band signals corresponding to the high-frequency features.
[0138] For example, high-frequency reconstruction is the inverse process of high-frequency analysis. High-frequency features are upgraded through high-frequency reconstruction to achieve the function of data decompression.
[0139] In some embodiments, high-frequency reconstruction processing is performed on high-frequency features to obtain high-frequency sub-band signals corresponding to the high-frequency features, including: calling a second neural network model, performing feature reconstruction processing on the high-frequency features through the second neural network model, and obtaining high-frequency sub-band signals corresponding to the high-frequency features.
[0140] For example, when the encoding end performs feature extraction on the high-frequency sub-band signal to obtain the high-frequency features, the decoding end performs feature reconstruction processing on the high-frequency features to obtain the high-frequency sub-band signal corresponding to the high-frequency features.
[0141] In some embodiments, high-frequency reconstruction processing is performed on the high-frequency features to obtain high-frequency sub-band signals corresponding to the high-frequency features, including: performing inverse processing of frequency band expansion on the high-frequency features to obtain high-frequency sub-band signals corresponding to the high-frequency features.
[0142] For example, when the encoding end performs frequency band expansion on the high frequency sub-band signal to obtain high frequency features, the decoding end performs inverse processing of the frequency band expansion on the high frequency features to obtain a high frequency sub-band signal corresponding to the high frequency features.
[0143] In some embodiments, the high-frequency feature is subjected to inverse processing of frequency band expansion to obtain a high-frequency sub-band signal corresponding to the high-frequency feature, including: performing frequency domain transform processing based on multiple sample points included in the low-frequency sub-band signal to obtain transform coefficients corresponding to the multiple sample points; performing spectrum replication processing on the transform coefficients of the second half of the transform coefficients corresponding to the multiple sample points to obtain reference transform coefficients of the reference high-frequency sub-band signal; performing gain processing on the reference transform coefficients of the reference high-frequency sub-band signal based on the sub-band spectrum envelope corresponding to the high-frequency feature to obtain the gained reference transform coefficients; performing inverse frequency domain transform processing on the gained reference transform coefficients to obtain the high-frequency sub-band signal corresponding to the high-frequency feature.
[0144] It should be noted that the frequency domain transformation method of the embodiments of the present application includes modified discrete cosine transform (MDCT), discrete cosine transform (DCT), fast Fourier transform (FFT), etc. The embodiments of the present application are not limited to the frequency domain transformation method.
[0145] In some embodiments, based on the subband spectral envelope corresponding to the high-frequency feature, the reference transform coefficient of the reference high-frequency subband signal is gain processed to obtain the reference transform coefficient after gain, including: based on the subband spectral envelope corresponding to the high-frequency feature, the reference transform coefficient of the reference high-frequency subband signal is divided into multiple subbands; for any subband among the multiple subbands, the following processing is performed: determining a first average energy corresponding to the subband in the subband spectral envelope, and determining a second average energy corresponding to the subband; determining a gain factor based on the ratio of the first average energy to the second average energy; multiplying the gain factor by each reference transform coefficient included in the subband to obtain the reference transform coefficient after gain.
[0146] As an example, firstly, the low frequency subband signal x′ generated by the decoding end is LB (n) A 640-point MDCT transform similar to that at the encoding end is also performed to generate 320-point MDCT coefficients (i.e., MDCT coefficients of the low-frequency part), that is, frequency domain transform processing is performed based on multiple sample points included in the low-frequency sub-band signal to obtain transform coefficients corresponding to the multiple sample points.
[0147] Then, x′ LB (n) The 320-point MDCT coefficients generated are copied to generate the MDCT coefficients of the high-frequency part (i.e., the reference transform coefficients of the reference high-frequency subband signal). Referring to the basic characteristics of the speech signal, the low-frequency part has more harmonics and the high-frequency part has fewer harmonics. Therefore, in order to avoid simple copying, which causes the artificially generated MDCT spectrum of the high-frequency part to contain too many harmonics, the last 160 points of the 320-point MDCT coefficients that the low-frequency subband signal depends on can be used as the master, and the spectrum can be copied twice to generate the reference value of the 320-point MDCT coefficients of the reference high-frequency subband signal (i.e., the reference transform coefficients of the reference high-frequency subband signal), that is, the transform coefficients of the second half of the transform coefficients corresponding to the multiple sample points are subjected to spectrum copying processing to obtain the reference transform coefficients of the reference high-frequency subband signal.
[0148] Next, the 8 sub-band spectral envelopes obtained previously are called (i.e., the 8 sub-band spectral envelopes obtained by querying the quantization table, i.e., the sub-band spectral envelopes corresponding to the high-frequency features). These 8 sub-band spectral envelopes correspond to 8 high-frequency sub-bands, and the reference values of the MDCT coefficients of the generated 320-point reference high-frequency sub-band signal are divided into 8 reference high-frequency sub-bands (i.e., the reference transform coefficients of the reference high-frequency sub-band signal are divided into multiple sub-bands). In a band-by-band manner, based on a high-frequency sub-band and the corresponding reference high-frequency sub-band, the reference values of the MDCT coefficients of the generated 320-point reference high-frequency sub-band signal are gain controlled (multiplication in the frequency domain). For example, a gain factor is calculated based on the average energy of the high-frequency sub-band (i.e., the first average energy) and the average energy of the corresponding reference high-frequency sub-band (the second average energy), and the MDCT coefficient corresponding to each point in the corresponding reference high-frequency sub-band is multiplied by the gain factor to ensure that the energy of the high-frequency MDCT coefficients virtually generated by decoding is close to the original coefficient energy at the encoding end.
[0149] For example, assuming that the average energy of the high-frequency subband of the reference high-frequency subband signal generated by copying the past is Y_L, and the average energy of the high-frequency subband to be gain-controlled (i.e., the high-frequency subband corresponding to the subband spectrum envelope decoded based on the bitstream) is Y_H, then a gain factor a=sqrt(Y_H / Y_L) is calculated. With the gain factor a, each point in the reference high-frequency subband generated by copying is directly multiplied by a.
[0150] Finally, the inverse MDCT transform is called to generate the estimated value x′ of the high-frequency subband signal HB (n) (i.e., the high-frequency subband signal corresponding to the high-frequency feature). Perform an inverse MDCT transform on the 320-point MDCT coefficients after gain to generate 640-point estimated values. By overlapping, take the first 320 valid estimated values as x′ HB (n).
[0151] In step 204, sub-band synthesis processing is performed on the low-frequency sub-band signal and the high-frequency sub-band signal to obtain a synthesized audio signal corresponding to the low-frequency code stream and the high-frequency code stream.
[0152] For example, the subband synthesis process is the inverse process of the subband decomposition process. The decoding end performs subband synthesis processing on the low-frequency subband signal and the high-frequency subband signal to restore the audio signal, wherein the synthesized audio signal is the restored audio signal.
[0153] In some embodiments, sub-band synthesis processing is performed on the low-frequency sub-band signal and the high-frequency sub-band signal to obtain a synthesized audio signal corresponding to the low-frequency code stream and the high-frequency code stream, including: up-sampling processing on the low-frequency sub-band signal to obtain a low-pass filtered signal; up-sampling processing on the high-frequency sub-band signal to obtain a high-frequency filtered signal; and filtering synthesis processing is performed on the low-pass filtered signal and the high-frequency filtered signal to obtain a synthesized audio signal.
[0154] For example, after obtaining the low-frequency sub-band signal and the high-frequency sub-band signal, the low-frequency sub-band signal and the high-frequency sub-band signal are subjected to sub-band synthesis through a QMF synthesis filter to restore the audio signal.
[0155] The following is an explanation of an exemplary application of the embodiments of the present application in a practical application scenario.
[0156] The embodiments of the present application can be applied to various audio scenarios, such as voice calls, instant messaging, etc. The following is an example of a voice call:
[0157] In the related art, the principle of speech coding is roughly as follows: speech coding can directly encode the speech waveform samples, sample by sample; or, based on the principle of human vocalization, relevant low-dimensional features are extracted, the encoding end encodes the features, and the decoding end reconstructs the speech signal based on these parameters.
[0158] The above coding principles all come from speech signal modeling, that is, a compression method based on signal processing. In order to improve the coding efficiency while ensuring the voice quality relative to the compression method based on signal processing. The embodiment of the present application provides a speech coding and decoding method (i.e., an audio processing method) based on sub-band decomposition and neural network. Based on the characteristics of the speech signal, the speech signal of a specific sampling rate is decomposed into a low-frequency sub-band signal and a high-frequency sub-band signal. Different sub-band signals can be compressed using different data compression mechanisms. For the important part (low-frequency sub-band signal), after processing based on the neural network (NN, Neural Network) technology, a feature vector with a lower dimension than the input low-frequency sub-band signal can be obtained. For the relatively unimportant part (high-frequency sub-band signal), fewer bits are used for encoding.
[0159] The embodiments of the present application can be applied to Figure 6 The voice communication link shown takes the Voice over Internet Protocol (VoIP) conference system as an example, and deploys the voice codec technology involved in the embodiment of the present application in the encoding and decoding parts to solve the basic function of voice compression. The encoder is deployed on the uplink client 601, and the decoder is deployed on the downlink client 602. The voice is collected through the uplink client, and pre-processing enhancement, encoding and other processing are performed. The encoded code stream is transmitted to the downlink client 602 through the network, and decoding, enhancement and other processing are performed by the downlink client 602 to play back the decoded voice on the downlink client 602.
[0160] Considering forward compatibility (i.e., the new encoder is compatible with the existing encoder), a transcoder needs to be deployed in the background of the system (i.e., the server) to solve the problem of interoperability between the new encoder and the existing encoder. For example, if the sender (uplink client) is the new NN encoder and the receiver (downlink client) is the public switched telephone network (PSTN, Public Switched Telephone Network) (G.722). In the background, it is necessary to execute the NN decoder to generate the voice signal, and then call the G.722 encoder to generate a specific code stream to realize the transcoding function, so that the receiver can correctly decode based on the specific code stream.
[0161] Combine the following Figure 7 The speech encoding and decoding method based on subband decomposition and neural network provided in the embodiment of the present application is described as follows:
[0162] The following processing is performed on the encoding end:
[0163] For the input speech signal x(n) of the nth frame, use the analysis filter to decompose it into the low-frequency subband signal X LB (n) and the high frequency subband signal X HB (n).
[0164] For the low frequency subband signal X LB (n), call the first NN to obtain the low-dimensional feature vector F LB (n), eigenvector F LB The dimension of (n) is smaller than that of the low-frequency subband signal to reduce the amount of data. LB (n), call the dilated CNN to generate a lower-dimensional feature vector F LB (n). The embodiments of the present application do not limit other NN structures, such as autoencoder (Autoencoder), fully connected (FC, Full-Connection) network, long short-term memory (LSTM, Long Short-Term Memory) network, convolutional neural network (CNN, Convolutional Neural Network) + LSTM, etc.
[0165] For the high frequency subband signal X HB (n), considering that high frequencies are not as important to quality as low frequencies, the high frequency subband signal X HB (n) Other schemes can be used to extract the feature vector F HB(n). For example, based on the frequency band extension technology of speech signal analysis, the high-frequency sub-band signal can be generated with only 1-2kbps bit rate; the same NN structure as the low-frequency sub-band signal or a more streamlined network (for example, the output feature vector is smaller than the low-frequency feature vector F IB (n) smaller).
[0166] The eigenvector corresponding to the subband (i.e., F LB (n) and F HB (n)) performs vector quantization or scalar quantization, and performs entropy coding on the quantized index value, and the encoded code stream is transmitted to the decoding end.
[0167] The decoding end performs the following processing:
[0168] Decode the code stream received by the decoding end to obtain the estimated value F′ of the low-frequency feature vector LB (n) and the estimated value of the high-frequency eigenvector F′ HB (n).
[0169] For the low-frequency part, the estimated value F′ based on the low-frequency eigenvector LB (n) Call the second NN to generate the low-frequency subband signal estimate x′ LB (n).
[0170] For the high frequency part, the estimated value F′ based on the high frequency eigenvector HB (n) Call high-frequency reconstruction to generate the high-frequency subband signal estimate x′ HB (n).
[0171] Finally, QMF synthesis filtering is called to generate the reconstructed speech signal x′(n).
[0172] Before specifically introducing the speech encoding and decoding method based on subband decomposition and neural network provided in the embodiment of the present application, the QMF filter group, the void convolution network and the frequency band extension technology are first introduced.
[0173] The QMF filter bank is a filter pair that includes analysis and synthesis. For the QMF analysis filter, the input signal with a sampling rate of Fs can be decomposed into two signals with a sampling rate of Fs / 2, which represent the QMF low-pass signal and the QMF high-pass signal respectively. Figure 8 The spectral response of the low-pass part H_Low(z) and the high-pass part H_High(z) of the QMF filter shown in FIG. Based on the relevant theoretical knowledge of the QMF analysis filter bank, the correlation between the coefficients of the low-pass filter and the high-pass filter can be easily described, as shown in formula (1):
[0174] h Higgh (k) = -1k h Low (k) (1)
[0175] Among them, h Low (k) represents the coefficient of low-pass filtering, h High (k) represents the coefficient of high-pass filtering.
[0176] Similarly, according to QMF related theory, the QMF synthesis filter group can be described based on the QMF analysis filter group H_Low(z) and H_High(z), as shown in formula (2).
[0177] G Low (z) = H Low (z)
[0178] G High (z)=(-1)*H High (z) (2)
[0179] Among them, G Low (z) represents the restored low-pass signal, G High (z) represents the recovered high-pass signal.
[0180] The low-pass and high-pass signals recovered by the decoding end are synthesized by the QMF synthesis filter bank, so that the reconstructed signal with the sampling rate Fs corresponding to the input signal can be restored.
[0181] See also Fig. 9A and Fig. 9B , Fig. 9A is a schematic diagram of a common convolutional network provided in an embodiment of the present application, Fig. 9B Schematic diagram of a dilated convolutional network provided in an embodiment of the present application. Compared with ordinary convolutional networks, dilated convolution can increase the receptive field while keeping the size of the feature map unchanged, and can also avoid errors caused by upsampling and downsampling. Fig. 9A and Fig. 9B The convolution kernel size shown in is 3×3; however, Fig. 9A The receptive field of the ordinary convolution shown is only 3, while Fig. 9B The receptive field 902 of the dilated convolution shown reaches 5. That is, for a convolution kernel of size 3×3, Fig. 9A The receptive field of the ordinary convolution shown is 3, and the dilation rate (the number of intervals between points in the convolution kernel) is 1; Fig. 9B The dilated convolution shown has a receptive field of 5 and a dilation rate of 2.
[0182] The convolution kernel can also be used in similar Fig. 9A or Fig. 9BThe convolution kernel moves on the plane, which involves the concept of stride rate. For example, each time the convolution kernel shifts 1 grid, the corresponding stride rate is 1.
[0183] In addition, there is the concept of the number of convolution channels, which is the number of convolution kernel parameters used to perform convolution analysis. In theory, the more channels there are, the more comprehensive the signal analysis is and the higher the accuracy is; however, the higher the number of channels, the higher the complexity is. For example, a 1×320 tensor can use a 24-channel convolution operation, and the output is a 24×320 tensor.
[0184] It should be noted that the size of the dilated convolution kernel (for example, for speech signals, the size of the convolution kernel can be set to 1×3), the expansion rate, the shift rate and the number of channels can be defined according to actual application needs. The embodiments of the present application do not make specific limitations on this.
[0185] like Fig.10 The schematic diagram of frequency band extension (or frequency band replication) shown in the figure first reconstructs the broadband signal, then copies the broadband signal to the ultra-wideband signal, and finally performs shaping based on the ultra-wideband envelope. Fig.10 The frequency domain implementation scheme shown specifically includes:
[0186] 1) Implement a core layer encoding at a low sampling rate; 2) Select the low-frequency spectrum and copy it to the high frequency; 3) Perform gain control on the copied high-frequency spectrum based on the pre-recorded boundary information (describing the energy correlation between high and low frequencies, etc.). With only a bit rate of 1-2kbps, the sampling rate can be doubled.
[0187] The following is a detailed description of the speech encoding and decoding method based on subband decomposition and neural network provided in an embodiment of the present application.
[0188] In some embodiments, a speech signal with a sampling rate of Fs=32000 Hz is taken as an example (it should be noted that the method provided in the embodiment of the present application is also applicable to scenarios with other sampling rates, including but not limited to: 8000 Hz, 32000 Hz, and 48000 Hz). At the same time, assuming that the frame length is set to 20 ms, therefore, for Fs=32000 Hz, it is equivalent to containing 640 sample points per frame.
[0189] Reference below Figure 7 The flowchart shown in FIG. 1 describes the encoding end and the decoding end in detail.
[0190] The process on the encoding side is as follows:
[0191] 1. Generate input signal.
[0192] For a speech signal with a sampling rate of Fs=32000 Hz, the input signal of the nth frame includes 640 sample points, which are recorded as input signal x(n).
[0193] 2. QMF signal decomposition.
[0194] Calling the QMF analysis filter (2-channel QMF) for downsampling, we can obtain two sub-band signals, namely the low-frequency sub-band signal X LB (n) and the high frequency subband signal X HB (n). Low frequency subband signal X LB (n) and the high frequency subband signal X HB The effective bandwidth of (n) is 0-8kHz and 8-16kHz respectively, and the low-frequency subband signal X LB (n) and the high frequency subband signal X HB The number of sample points of (n) is 320.
[0195] 3. The low frequency subband signal X LB (n) Input the first NN for data compression.
[0196] Based on the low-frequency subband signal x LB (n), call the first NN to generate a lower dimensional feature vector F LB (n). It should be noted that X LB The dimension of (n) is 320, and F LB The dimension of (n) is 56. From the perspective of data volume, the first NN plays the role of "dimensionality reduction" and realizes the function of data compression.
[0197] See also Fig.11 The network structure diagram of the first NN is shown in FIG. 1 , and the process of data compression by the first NN is described in detail below:
[0198] First, a 24-channel causal convolution is called to expand the input tensor (i.e., vector) to a 24×320 tensor.
[0199] Then, the 24×320 tensor is preprocessed. For example, a pooling operation with a factor of 2 is performed on the 24×320 tensor, and the activation function may be ReLU to generate a 24×160 tensor.
[0200] Next, cascade three coding blocks with different downsampling factors (Down_factor). Taking the coding block (Down_factor=4) as an example, one or more dilated convolutions can be performed first, and the size of each convolution kernel is fixed to 1×3 and the shift rate (Stride Rate) is 1. In addition, the dilation rate (Dilation Rate) of one or more dilated convolutions can be set according to demand, such as 3. Of course, the embodiment of the present application does not limit the dilation rate of different dilated convolution settings. Then, the Down_factor of the three coding blocks is set to 4, 5, and 8, respectively, which is equivalent to setting pooling factors of different sizes, which plays a role in downsampling. Finally, the number of channels of the three coding blocks is set to 48, 96, and 192, respectively. Therefore, after three coding blocks, the 24×160 tensor is converted into 48×40, 96×8, and 192×1 tensors respectively.
[0201] Finally, after similar preprocessing of the 192×1 tensor, a 56-dimensional feature vector F can be output. LB (n).
[0202] 4. For the high frequency sub-band signal x HB (n) Perform high frequency analysis.
[0203] The purpose of high-frequency analysis is to extract the high-frequency subband signal x HB (n) to generate a lower-dimensional feature vector F HB (n).
[0204] In some embodiments, another similar NN structure may be introduced with reference to step 3 to generate a low-dimensional feature vector. Compared with the low-frequency sub-band signal, the high-frequency sub-band signal is less important to the quality. Therefore, the NN structure for the high-frequency sub-band signal does not need to be as complex as the first NN. Fig.12 The NN structure for the high frequency sub-band signal is shown. The NN structure is similar to the structure of the first NN, but the number of channels is greatly reduced compared with the structure of the first NN.
[0205] However, for the high frequency subband signal, although Fig.12 The NN structure shown in the figure reduces the data volume of the high-frequency sub-band signal a lot (from 320 dimensions to 8 dimensions), but the model complexity of the NN structure is not optimal. Therefore, the embodiment of the present application proposes another method to compress the high-frequency sub-band signal, namely, band expansion (recovering the broadband speech signal from the band-limited narrowband speech signal). The application of band expansion in the embodiment of the present application is specifically described below:
[0206] For the high frequency subband signal x including 320 points HB(n), call the modified discrete cosine transform (MDCT) to generate 320-point MDCT coefficients. Specifically, if there is a 50% overlap, the high-frequency data of the n+1th frame can be merged (joined) with the high-frequency data of the nth frame, and the 640-point MDCT is calculated to obtain 320-point MDCT coefficients.
[0207] The MDCT coefficients of 320 points are divided into N sub-bands, where the sub-bands are groups of adjacent multiple MDCT coefficients, and the MDCT coefficients of 320 points can be divided into 8 sub-bands. For example, the 320 points can be evenly distributed, that is, the number of points included in each sub-band is consistent. Of course, the embodiment of the present application cannot divide the 320 points non-uniformly, such as the MDCT coefficients included in the sub-bands with low frequencies are less (the frequency resolution is higher), and the MDCT coefficients included in the sub-bands with high frequencies are more (the frequency resolution is lower).
[0208] According to Nyquist sampling theorem (to restore the original signal from the sampled signal without distortion, the sampling frequency should be greater than 2 times the highest frequency of the original signal. When the sampling frequency is less than 2 times the highest frequency of the spectrum, the spectrum of the signal is aliased. When the sampling frequency is greater than 2 times the highest frequency of the spectrum, the spectrum of the signal is not aliased), the 320-point MDCT coefficients above represent the spectrum of 8-16kHz. However, ultra-wideband voice communication does not necessarily require the spectrum to be 16kHz. For example, if the spectrum is set to 14kHz, only the MDCT coefficients of the first 240 points need to be considered. Correspondingly, the number of subbands can be controlled to 6.
[0209] For each subband, the average energy of all MDCT coefficients in the current subband is calculated as the subband spectrum envelope (the spectrum envelope is a smooth curve passing through the main peak points of the spectrum). For example, the MDCT coefficients included in the current subband are x(n), n = 1, 2, ..., 40, then the average energy Y = ((x(1) 2 +x(2) 2 +…+x(40) 2 ) / 40). When the 320-point MDCT coefficients are divided into 8 sub-bands, 8 sub-band spectrum envelopes can be obtained. These 8 sub-band spectrum envelopes are the feature vectors F of the generated high-frequency sub-band signal. HB (n).
[0210] In summary, using any of the above two methods (NN structure and frequency band expansion), the 320-dimensional high-frequency subband signal can be output as an 8-dimensional feature vector. Therefore, only a small amount of data is needed to represent the high-frequency information, and the coding efficiency is significantly improved.
[0211] 5. Quantization coding.
[0212] For the eigenvector F of the low-frequency subband signal LB (n) and the eigenvector F of the high-frequency subband signal HB (n), both scalar quantization (each component is quantized separately) and entropy coding methods can be performed. In addition, the embodiments of the present application do not limit the technical combination of vector quantization (multiple adjacent components are combined into a vector for joint quantization) and entropy coding.
[0213] After quantizing and encoding the feature vector, the corresponding bit stream can be generated. According to experiments, high-quality compression of 32kHz ultra-wideband signals can be achieved at a bit rate of 6-10kbps.
[0214] The process of decoding is as follows:
[0215] 1. Quantization decoding.
[0216] Quantization decoding is the inverse process of quantization encoding. For the received bit stream, entropy decoding is first performed, and the estimated value F′ of the low-frequency feature vector is obtained by looking up the quantization table. LB (n) and the estimated value of the high-frequency eigenvector F′ HB (n).
[0217] 2. The estimated value F′ of the low-frequency eigenvector LB (n) Input the second NN.
[0218] The estimated value F′ based on the low-frequency eigenvector LB (n), calling Fig.13 The second NN shown generates an estimate x′ of the low-frequency subband signal LB (n).
[0219] The second NN is similar to the first NN, such as causal convolution and post-processing structure in the second NN is similar to pre-processing in the first NN. The structure of the decoding block is symmetrical to the encoding block at the encoding end: the encoding block at the encoding end first performs dilated convolution and then pooling to complete downsampling, while the decoding block at the decoding end first performs pooling to complete upsampling and then performs dilated convolution.
[0220] 3. Estimation of high-frequency eigenvector F′ HB (n) Perform high-frequency reconstruction.
[0221] Similar to the high frequency analysis at the encoding end, the high frequency reconstruction in the embodiment of the present application includes two schemes.
[0222] The first implementation of high-frequency reconstruction corresponds to the first implementation of high-frequency analysis at the encoding end (corresponding to Fig.12 The NN structure shown in Figure 1). The estimated value F′ based on the high-frequency feature vector HB (n), call the deep neural network to generate the estimated value x′ of the high-frequency subband signalHB (n).
[0223] The structure of a deep neural network and the first implementation of high-frequency analysis ( Fig.12 ), such as causal convolution, the post-processing structure is similar to the pre-processing in the first implementation of high-frequency analysis.
[0224] The structure of the decoding block is symmetrical to that of the encoding block on the encoding side: the encoding block on the encoding side first performs dilated convolution and then performs pooling to complete downsampling, while the decoding block on the decoding side first performs pooling to complete upsampling and then performs dilated convolution.
[0225] The second implementation of high-frequency reconstruction corresponds to the first implementation of high-frequency analysis at the encoding end (corresponding to the frequency band extension technology). Based on the 8 sub-band spectral envelopes decoded from the bitstream, that is, the estimated value F′ of the high-frequency feature vector HB (n) Perform the following operations:
[0226] First, the estimated value x′ of the low-frequency subband signal generated by the decoding end LB (n), a 640-point MDCT transform similar to that at the encoding end is also performed to generate 320-point MDCT coefficients (i.e., MDCT coefficients of the low-frequency part).
[0227] Then, x′ LB (n) The 320-point MDCT coefficients generated are copied to generate the MDCT coefficients of the high-frequency part. Referring to the basic characteristics of the speech signal, the low-frequency part has more harmonics and the high-frequency part has fewer harmonics. Therefore, in order to avoid simple copying, which causes the artificially generated MDCT spectrum of the high-frequency part to contain too many harmonics, the last 160 points of the 320-point MDCT coefficients that the low-frequency subband depends on can be used as the master, and the spectrum can be copied twice to generate the reference value of the 320-point MDCT coefficients of the high-frequency subband signal.
[0228] Next, the 8 sub-band spectral envelopes obtained previously (i.e., the 8 sub-band spectral envelopes obtained by querying the quantization table) are called. These 8 sub-band spectral envelopes correspond to 8 high-frequency sub-bands, and the reference values of the MDCT coefficients of the generated 320-point high-frequency sub-band signals are divided into 8 reference high-frequency sub-bands. In a band-by-band manner, based on a high-frequency sub-band and the corresponding reference high-frequency sub-band, the reference values of the MDCT coefficients of the generated 320-point high-frequency sub-band signals are gain controlled (multiplication in the frequency domain). For example, the gain factor is calculated based on the average energy of the high-frequency sub-band and the average energy of the corresponding reference high-frequency sub-band, and the MDCT coefficient corresponding to each point in the corresponding reference high-frequency sub-band is multiplied by the gain factor to ensure that the energy of the high-frequency MDCT coefficients virtually generated by decoding is close to the original coefficient energy at the encoding end.
[0229] For example, assuming that the average energy of the high-frequency sub-band of the high-frequency sub-band signal generated by copying the past is Y_L, and the average energy of the high-frequency sub-band to be gain-controlled (i.e., the high-frequency sub-band corresponding to the sub-band spectrum envelope decoded based on the bitstream) is Y_H, then a gain factor a=sqrt(Y_H / Y_L) is calculated. After the gain factor a is obtained, each point in the high-frequency sub-band generated by copying is directly multiplied by a.
[0230] Finally, the inverse MDCT transform is called to generate the estimated value x′ of the high-frequency subband signal HB (n) Perform an inverse MDCT transform on the 320-point MDCT coefficients after gain to generate 640-point estimated values. By overlapping, take the first 320 effective estimated values as x′ HB (n).
[0231] 4. Synthesis Filter
[0232] At the decoding end, the estimated value x′ of the low-frequency subband signal is obtained LB (n) and the estimated value x′ of the high frequency subband signal HB After that, we only need to upsample and call the QMF synthesis filter to generate a 640-point reconstructed signal x′(n).
[0233] The embodiment of the present application can collect data and jointly train the relevant networks of the encoding end and the decoding end to obtain the optimal parameters. The user only needs to prepare the data and set the corresponding network structure. After the training is completed in the background, the trained model can be put into use.
[0234] In summary, the speech encoding and decoding method based on subband decomposition and neural network provided in the embodiment of the present application significantly improves the coding efficiency compared with the signal processing scheme while ensuring audio quality and acceptable complexity through the organic combination of signal decomposition, signal processing technology and deep neural network.
[0235] So far, the audio processing method provided by the embodiment of the present application has been described in combination with the exemplary application and implementation of the terminal device provided by the embodiment of the present application. The embodiment of the present application also provides an audio processing device. In actual applications, each functional module in the audio processing device can be implemented by hardware resources of an electronic device (such as a terminal device, a server or a server cluster), such as computing resources such as a processor, communication resources (such as for supporting various communication methods such as optical cables and cellular), and memory. Figure 3An audio processing device 555 stored in a memory 550 is shown, which may be software in the form of programs and plug-ins, for example, software modules designed in programming languages such as C / C++ and Java, application software designed in programming languages such as C / C++ and Java, or dedicated software modules in large software systems, application programming interfaces, plug-ins, cloud services, and other implementation methods. Different implementation methods are described below with examples.
[0236] The audio processing device 555 includes a series of modules, including a decomposition module 5551, a feature extraction module 5552, a high frequency analysis module 5553, and an encoding module 5554. The following further describes the scheme for implementing audio encoding by cooperating among the modules in the audio processing device 555 provided in the embodiment of the present application.
[0237] A decomposition module is used to perform sub-band decomposition processing on the audio signal to obtain a low-frequency sub-band signal and a high-frequency sub-band signal of the audio signal; a feature extraction module is used to perform feature extraction processing on the low-frequency sub-band signal to obtain the low-frequency features of the low-frequency sub-band signal; a high-frequency analysis module is used to perform high-frequency analysis processing on the high-frequency sub-band signal to obtain the high-frequency features of the high-frequency sub-band signal; wherein the feature dimension of the high-frequency feature is lower than the feature dimension of the low-frequency feature; an encoding module is used to perform quantization encoding processing on the low-frequency feature to obtain the low-frequency code stream of the audio signal, and perform quantization encoding processing on the high-frequency feature to obtain the high-frequency code stream of the audio signal.
[0238] In some embodiments, the decomposition module is further used to perform sampling processing on the audio signal to obtain a sampled signal, wherein the sampled signal includes multiple sample points obtained by sampling; perform low-pass filtering processing on the sampled signal to obtain a low-pass filtered signal; down-sample the low-pass filtered signal to obtain a low-frequency sub-band signal of the audio signal; perform high-pass filtering processing on the sampled signal to obtain a high-pass filtered signal; down-sample the high-pass filtered signal to obtain a high-frequency sub-band signal of the audio signal.
[0239] In some embodiments, the feature extraction module is also used to perform convolution processing on the low-frequency sub-band signal to obtain the convolution feature of the low-frequency sub-band signal; perform pooling processing on the convolution feature to obtain the pooling feature of the low-frequency sub-band signal; perform down-sampling processing on the pooling feature to obtain the down-sampled feature of the low-frequency sub-band signal; perform convolution processing on the down-sampled feature to obtain the low-frequency feature of the low-frequency sub-band signal.
[0240] In some embodiments, the downsampling process is implemented through multiple cascaded coding layers; the feature extraction module is also used to downsample the pooled features through the first coding layer among the multiple cascaded coding layers; the downsampling result of the first coding layer is output to the subsequent cascaded coding layers, and the downsampling process and downsampling result output are continued through the subsequent cascaded coding layers until it is output to the last coding layer; the downsampling result output by the last coding layer is used as the downsampling feature of the low-frequency sub-band signal.
[0241] In some embodiments, the high-frequency analysis module is also used to call the first neural network model, perform feature extraction processing on the high-frequency sub-band signal through the first neural network model to obtain the high-frequency characteristics of the high-frequency sub-band signal; or, perform frequency band expansion processing on the high-frequency sub-band signal to obtain the high-frequency characteristics of the high-frequency sub-band signal.
[0242] In some embodiments, the high-frequency analysis module is also used to perform frequency domain transform processing based on multiple sample points included in the high-frequency sub-band signal to obtain transform coefficients corresponding to the multiple sample points respectively; divide the transform coefficients corresponding to the multiple sample points into multiple sub-bands; perform mean processing on the transform coefficients included in each of the sub-bands to obtain the average energy corresponding to each of the sub-bands, and use the average energy as the sub-band spectral envelope corresponding to each of the sub-bands; determine the sub-band spectral envelopes corresponding to the multiple sub-bands as the high-frequency characteristics of the high-frequency sub-band signal.
[0243] In some embodiments, the high-frequency analysis module is also used to obtain a reference high-frequency sub-band signal of a reference audio signal, wherein the reference audio signal is an audio signal adjacent to the audio signal; based on a plurality of sample points included in the reference high-frequency sub-band signal and a plurality of sample points included in the high-frequency sub-band signal, discrete cosine transform processing is performed on the plurality of sample points included in the high-frequency sub-band signal to obtain transform coefficients corresponding to the plurality of sample points included in the high-frequency sub-band signal.
[0244] In some embodiments, the high frequency analysis module is further used to determine the sum of squares of transform coefficients corresponding to the sample points included in each of the sub-bands; and the ratio of the sum of squares to the number of sample points included in the sub-band is determined to obtain the average energy corresponding to each of the sub-bands.
[0245] In some embodiments, the encoding module is also used to quantize the low-frequency features to obtain the index value of the low-frequency features; perform entropy coding on the index value of the low-frequency features to obtain the low-frequency code stream of the audio signal; the quantization coding of the high-frequency features to obtain the high-frequency code stream of the audio signal includes: quantizing the high-frequency features to obtain the index value of the high-frequency features; performing entropy coding on the index value of the high-frequency features to obtain the high-frequency code stream of the audio signal.
[0246] The audio processing device 555 includes a series of modules, including a decoding module 5555, a feature reconstruction module 5556, a high frequency reconstruction module 5557 and a synthesis module 5558. The following further describes the scheme of implementing audio decoding by cooperating with each module in the audio processing device 555 provided in the embodiment of the present application.
[0247] A decoding module, used to perform quantization decoding processing on a low-frequency code stream to obtain low-frequency features corresponding to the low-frequency code stream, and to perform quantization decoding processing on a high-frequency code stream to obtain high-frequency features corresponding to the high-frequency code stream;
[0248] The low-frequency code stream and the high-frequency code stream are obtained by respectively encoding sub-band signals obtained after sub-band decomposition of the audio signal, and the feature dimension of the high-frequency feature is lower than the feature dimension of the low-frequency feature;
[0249] A feature reconstruction module, used to perform feature reconstruction processing on the low-frequency feature to obtain a low-frequency sub-band signal corresponding to the low-frequency feature;
[0250] A high-frequency reconstruction module, used to perform high-frequency reconstruction processing on the high-frequency features to obtain a high-frequency sub-band signal corresponding to the high-frequency features;
[0251] The synthesis module is used to perform sub-band synthesis processing on the low-frequency sub-band signal and the high-frequency sub-band signal to obtain a synthesized audio signal corresponding to the low-frequency code stream and the high-frequency code stream.
[0252] In some embodiments, the feature reconstruction module is also used to perform convolution processing on the low-frequency features to obtain convolution features of the low-frequency features; perform upsampling processing on the convolution features to obtain upsampled features of the low-frequency features; perform pooling processing on the upsampled features to obtain pooled features of the low-frequency features; perform convolution processing on the pooled features to obtain a low-frequency sub-band signal corresponding to the low-frequency features.
[0253] In some embodiments, the upsampling process is implemented through multiple cascaded decoding layers; the feature reconstruction module is also used to perform upsampling on the convolutional features through the first decoding layer among the multiple cascaded decoding layers; the upsampling result of the first decoding layer is output to the subsequent cascaded decoding layers, and the upsampling process and upsampling result output are continued through the subsequent cascaded decoding layers until it is output to the last decoding layer; the upsampling result output by the last decoding layer is used as the upsampling feature of the low-frequency feature.
[0254] In some embodiments, the high-frequency reconstruction module is also used to call a second neural network model to perform feature reconstruction processing on the high-frequency features through the second neural network model to obtain a high-frequency sub-band signal corresponding to the high-frequency features; or, to perform inverse processing of frequency band expansion on the high-frequency features to obtain a high-frequency sub-band signal corresponding to the high-frequency features.
[0255] In some embodiments, the high-frequency reconstruction module is also used to perform frequency domain transform processing based on multiple sample points included in the low-frequency sub-band signal to obtain transform coefficients corresponding to the multiple sample points respectively; perform spectrum replication processing on the transform coefficients of the second half of the transform coefficients corresponding to the multiple sample points respectively to obtain reference transform coefficients of the reference high-frequency sub-band signal; based on the sub-band spectrum envelope corresponding to the high-frequency feature, perform gain processing on the reference transform coefficients of the reference high-frequency sub-band signal to obtain the reference transform coefficients after gain; perform inverse frequency domain transform processing on the reference transform coefficients after gain to obtain the high-frequency sub-band signal corresponding to the high-frequency feature.
[0256] In some embodiments, the high-frequency reconstruction module is also used to divide the reference transformation coefficient of the reference high-frequency sub-band signal into multiple sub-bands based on the sub-band spectral envelope corresponding to the high-frequency feature; perform the following processing for any of the multiple sub-bands: determine the first average energy corresponding to the sub-band in the sub-band spectral envelope, and determine the second average energy corresponding to the sub-band; determine a gain factor based on the ratio of the first average energy to the second average energy; multiply the gain factor by each of the reference transformation coefficients included in the sub-band to obtain the reference transformation coefficient after gain.
[0257] In some embodiments, the decoding module is also used to perform entropy decoding on the low-frequency code stream to obtain an index value corresponding to the low-frequency code stream; perform inverse quantization on the index value corresponding to the low-frequency code stream to obtain a low-frequency feature corresponding to the low-frequency code stream; perform entropy decoding on the high-frequency code stream to obtain an index value corresponding to the high-frequency code stream; perform inverse quantization on the index value corresponding to the high-frequency code stream to obtain a high-frequency feature corresponding to the high-frequency code stream.
[0258] In some embodiments, the synthesis module is also used to upsample the low-frequency subband signal to obtain a low-pass filtered signal; upsample the high-frequency subband signal to obtain a high-frequency filtered signal; and perform filtering and synthesis processing on the low-pass filtered signal and the high-frequency filtered signal to obtain the synthesized audio signal.
[0259] The embodiment of the present application provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the electronic device performs the audio processing method described above in the embodiment of the present application.
[0260] The present application embodiment provides a computer-readable storage medium storing executable instructions, wherein the executable instructions are stored. When the executable instructions are executed by a processor, the processor will execute the audio processing method provided by the present application embodiment, for example, Figure 4-5 The audio processing method is shown.
[0261] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface storage, optical disk, or CD-ROM; or it may be various devices including one or any combination of the above memories.
[0262] In some embodiments, executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.
[0263] As an example, executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).
[0264] By way of example, executable instructions may be deployed to be executed on one computing device, or on multiple computing devices located at one site, or on multiple computing devices distributed across multiple sites and interconnected by a communication network.
[0265] It is understandable that in the embodiments of the present application, related data such as user information is involved. When the embodiments of the present application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.
[0266] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.
Claims
1. An audio processing method, characterized in that: The method comprises: Obtaining a low-frequency sub-band signal and a high-frequency sub-band signal of an audio signal; Performing feature extraction processing on the low-frequency sub-band signal through a first neural network model to obtain low-frequency features of the low-frequency sub-band signal; Performing frequency band expansion processing on the high frequency sub-band signal to obtain high frequency features of the high frequency sub-band signal; The low-frequency features are quantized and encoded to obtain a low-frequency code stream of the audio signal, and the high-frequency features are quantized and encoded to obtain a high-frequency code stream of the audio signal.
2. The method according to claim 1, characterized in that The step of performing feature extraction processing on the low-frequency sub-band signal by using the first neural network model to obtain the low-frequency features of the low-frequency sub-band signal includes: The following processing is performed by the first neural network model: Performing convolution processing on the low-frequency sub-band signal to obtain convolution features of the low-frequency sub-band signal; Performing pooling processing on the convolutional features to obtain pooling features of the low-frequency sub-band signal; Downsampling the pooled features to obtain downsampled features of the low-frequency subband signal; The down-sampled features are convolved to obtain low-frequency features of the low-frequency sub-band signal.
3. The method according to claim 2, characterized in that The downsampling process is implemented by a plurality of cascaded coding layers; The downsampling process is performed on the pooled features to obtain the downsampling features of the low-frequency sub-band signal, including: Downsampling the pooled features through a first coding layer of the multiple cascaded coding layers; Outputting the downsampling result of the first coding layer to a subsequent cascaded coding layer, and continuing to perform downsampling processing and output downsampling results through the subsequent cascaded coding layers until outputting to the last coding layer; The down-sampling result output by the last coding layer is used as the down-sampling feature of the low-frequency sub-band signal.
4. The method according to claim 1, characterized in that: The step of performing frequency band extension processing on the high frequency sub-band signal to obtain the high frequency characteristics of the high frequency sub-band signal includes: Performing frequency domain transformation processing based on multiple sample points included in the high frequency sub-band signal to obtain transformation coefficients corresponding to the multiple sample points respectively; Dividing the transform coefficients corresponding to the plurality of sample points into a plurality of sub-bands; Performing mean processing on the transform coefficients included in each of the sub-bands to obtain average energy corresponding to each of the sub-bands, and using the average energy as the sub-band spectrum envelope corresponding to each of the sub-bands; Subband spectrum envelopes respectively corresponding to the multiple subbands are determined as high-frequency features of the high-frequency subband signal.
5. The method according to claim 4, characterized in that The performing frequency domain transform processing based on the multiple sample points included in the high frequency sub-band signal to obtain transform coefficients respectively corresponding to the multiple sample points includes: Acquire a reference high frequency subband signal of a reference audio signal, wherein the reference audio signal is an audio signal adjacent to the audio signal; Based on the multiple sample points included in the reference high-frequency sub-band signal and the multiple sample points included in the high-frequency sub-band signal, discrete cosine transform processing is performed on the multiple sample points included in the high-frequency sub-band signal to obtain transform coefficients corresponding to the multiple sample points included in the high-frequency sub-band signal.
6. The method according to claim 4, characterized in that The performing mean processing on the transform coefficients included in each of the sub-bands to obtain the average energy corresponding to each of the sub-bands includes: Determine the sum of squares of transform coefficients corresponding to the sample points included in each of the sub-bands; The ratio of the square sum to the number of sample points included in the sub-band is determined to obtain the average energy corresponding to each of the sub-bands.
7. An audio processing method, characterized in that: The method comprises: Obtaining a low-frequency sub-band signal and a high-frequency sub-band signal of an audio signal; Performing feature extraction processing on the low-frequency sub-band signal through a first neural network model to obtain low-frequency features of the low-frequency sub-band signal; Performing frequency band expansion processing on the high frequency sub-band signal through a second neural network model to obtain high frequency features of the high frequency sub-band signal; The low-frequency features are quantized and encoded to obtain a low-frequency code stream of the audio signal, and the high-frequency features are quantized and encoded to obtain a high-frequency code stream of the audio signal.
8. The method according to claim 7, characterized in that The step of performing frequency band extension processing on the high frequency sub-band signal through the second neural network model to obtain high frequency features of the high frequency sub-band signal includes: The following processing is performed by the second neural network model: Performing convolution processing on the high frequency sub-band signal to obtain convolution features of the high frequency sub-band signal; Performing pooling processing on the convolutional features to obtain pooling features of the high-frequency sub-band signal; Downsampling the pooled features to obtain downsampled features of the high frequency subband signal; The down-sampled features are convolved to obtain high-frequency features of the high-frequency sub-band signal.
9. An audio processing method, characterized in that: The method comprises: Performing quantization decoding processing on the low-frequency code stream to obtain low-frequency features corresponding to the low-frequency code stream, and performing quantization decoding processing on the high-frequency code stream to obtain high-frequency features corresponding to the high-frequency code stream; The low-frequency code stream and the high-frequency code stream are obtained by respectively encoding sub-band signals of the audio signal; Performing feature reconstruction processing on the low-frequency features through a third neural network model to obtain a low-frequency sub-band signal corresponding to the low-frequency features; Performing inverse processing of frequency band expansion on the high-frequency feature to obtain a high-frequency sub-band signal corresponding to the high-frequency feature; A synthesized audio signal is obtained based on the low frequency sub-band signal and the high frequency sub-band signal.
10. The method according to claim 9, characterized in that The step of performing feature reconstruction processing on the low-frequency feature through the third neural network model to obtain a low-frequency sub-band signal corresponding to the low-frequency feature includes: The following processing is performed by the third neural network model: Performing convolution processing on the low-frequency features to obtain convolution features of the low-frequency features; Performing upsampling processing on the convolution feature to obtain upsampled features of the low-frequency feature; Performing a de-pooling process on the up-sampled features to obtain de-pooled features of the low-frequency features; The de-pooled features are convolved to obtain low-frequency sub-band signals corresponding to the low-frequency features.
11. The method according to claim 10, characterized in that The upsampling process is implemented by a plurality of cascaded decoding layers; The upsampling process is performed on the convolution feature to obtain the upsampled feature of the low-frequency feature, including: Upsampling the convolutional features through a first decoding layer in the plurality of cascaded decoding layers; Outputting the upsampling result of the first decoding layer to the subsequent cascaded decoding layers, and continuing the upsampling process and outputting the upsampling result through the subsequent cascaded decoding layers until outputting to the last decoding layer; The upsampling result output by the last decoding layer is used as the upsampling feature of the low-frequency feature.
12. The method according to claim 11, characterized in that The inverse processing of frequency band expansion on the high frequency feature to obtain a high frequency sub-band signal corresponding to the high frequency feature includes: Performing frequency domain transform processing based on multiple sample points included in the low-frequency sub-band signal to obtain transform coefficients corresponding to the multiple sample points respectively; Performing spectrum replication processing on the transform coefficients of the second half of the transform coefficients corresponding to the plurality of sample points, respectively, to obtain reference transform coefficients of the reference high frequency subband signal; Based on the sub-band spectrum envelope corresponding to the high-frequency feature, performing gain processing on the reference transform coefficient of the reference high-frequency sub-band signal to obtain the reference transform coefficient after gain; The reference transform coefficient after gain is subjected to inverse frequency domain transform processing to obtain a high-frequency sub-band signal corresponding to the high-frequency feature.
13. The method according to claim 12, characterized in that The step of performing gain processing on the reference transform coefficient of the reference high frequency subband signal based on the subband spectrum envelope corresponding to the high frequency feature to obtain the reference transform coefficient after gain includes: Based on the sub-band spectrum envelope corresponding to the high-frequency feature, dividing the reference transform coefficients of the reference high-frequency sub-band signal into a plurality of sub-bands; The following processing is performed for any of the plurality of sub-bands: Determining a first average energy corresponding to the sub-band in the sub-band spectral envelope, and determining a second average energy corresponding to the sub-band; determining a gain factor based on a ratio of the first average energy to the second average energy; The gain factor is multiplied by each of the reference transform coefficients included in the subband to obtain the reference transform coefficient after gain.
14. An audio processing method, characterized in that: The method comprises: Performing quantization decoding processing on the low-frequency code stream to obtain low-frequency features corresponding to the low-frequency code stream, and performing quantization decoding processing on the high-frequency code stream to obtain high-frequency features corresponding to the high-frequency code stream; The low-frequency code stream and the high-frequency code stream are obtained by respectively encoding sub-band signals of the audio signal; Performing feature reconstruction processing on the low-frequency features through a third neural network model to obtain a low-frequency sub-band signal corresponding to the low-frequency features; Performing feature reconstruction processing on the high-frequency features through a fourth neural network model to obtain a high-frequency sub-band signal corresponding to the high-frequency features; A synthesized audio signal is obtained based on the low frequency sub-band signal and the high frequency sub-band signal.
15. The method according to claim 14, characterized in that The step of performing feature reconstruction processing on the high-frequency feature through the fourth neural network model to obtain a high-frequency sub-band signal corresponding to the high-frequency feature includes: The following processing is performed by the fourth neural network model: Performing convolution processing on the high-frequency features to obtain convolution features of the high-frequency features; Performing up-sampling processing on the convolution feature to obtain up-sampled features of the high-frequency feature; Performing a de-pooling process on the up-sampled features to obtain de-pooled features of the high-frequency features; The de-pooled features are convolved to obtain high-frequency sub-band signals corresponding to the high-frequency features.
16. An audio processing system, characterized in that: The system comprises: A first terminal and a second terminal; wherein, The first terminal is used to obtain a low-frequency subband signal and a high-frequency subband signal of an audio signal; perform feature extraction processing on the low-frequency subband signal through a first neural network model to obtain a low-frequency feature of the low-frequency subband signal; perform frequency band expansion processing on the high-frequency subband signal to obtain a high-frequency feature of the high-frequency subband signal; perform quantization encoding processing on the low-frequency feature to obtain a low-frequency code stream of the audio signal, and perform quantization encoding processing on the high-frequency feature to obtain a high-frequency code stream of the audio signal, and send the low-frequency code stream and the high-frequency code stream to the second terminal; The second terminal is used to perform quantization decoding processing on the low-frequency code stream sent by the first terminal to obtain a low-frequency feature estimation value, and to perform quantization decoding processing on the high-frequency code stream sent by the first terminal to obtain a high-frequency feature estimation value; perform feature reconstruction processing on the low-frequency feature estimation value through a third neural network model to obtain a low-frequency sub-band signal estimation value; perform inverse processing of frequency band expansion on the high-frequency feature estimation value to obtain a high-frequency sub-band signal estimation value; and obtain a synthetic audio signal based on the low-frequency sub-band signal estimation value and the high-frequency sub-band signal estimation value.
17. An audio processing system, characterized in that: The system comprises: A first terminal and a second terminal; wherein, The first terminal is used to obtain a low-frequency subband signal and a high-frequency subband signal of an audio signal; perform feature extraction processing on the low-frequency subband signal through a first neural network model to obtain a low-frequency feature of the low-frequency subband signal; perform frequency band expansion processing on the high-frequency subband signal through a second neural network model to obtain a high-frequency feature of the high-frequency subband signal; perform quantization encoding processing on the low-frequency feature to obtain a low-frequency code stream of the audio signal, and perform quantization encoding processing on the high-frequency feature to obtain a high-frequency code stream of the audio signal, and send the low-frequency code stream and the high-frequency code stream to the second terminal; The second terminal is used to perform quantization decoding processing on the low-frequency code stream sent by the first terminal to obtain a low-frequency feature estimation value, and to perform quantization decoding processing on the high-frequency code stream sent by the first terminal to obtain a high-frequency feature estimation value; perform feature reconstruction processing on the low-frequency feature estimation value through a third neural network model to obtain a low-frequency sub-band signal estimation value; perform feature reconstruction processing on the high-frequency feature estimation value through a fourth neural network model to obtain a high-frequency sub-band signal estimation value; and obtain a synthetic audio signal based on the low-frequency sub-band signal estimation value and the high-frequency sub-band signal estimation value.
18. An electronic device, characterized in that: The electronic device comprises: A memory for storing executable instructions; A processor, configured to implement the audio processing method according to any one of claims 1 to 15 when executing the executable instructions stored in the memory.
19. A computer-readable storage medium, characterized in that: Executable instructions are stored, which are used to implement the audio processing method described in any one of claims 1 to 15 when executed by a processor.
20. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the audio processing method according to any one of claims 1 to 15 is implemented.
