Audio frequency processing method, device, electronic device, computer-readable storage medium and computer program product
By filtering and transforming audio signals to extract spectral envelope and flatness information, the method addresses the inefficiencies in coding high frequency signals, resulting in improved audio quality and reduced noise in decoded audio.
Patent Information
- Application Number
- JP2024524602
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2022-06-15
- Filing Date
- 2023-04-27
- Publication Date
- 2024-11-06
- Estimated Expiration
- 2043-04-27
AI Technical Summary
Existing audio frequency coding technologies struggle to efficiently code and decode high frequency signals with low code rates, leading to inaccuracies and noise in the reconstructed audio, particularly due to the human ear's sensitivity to low frequency signals, which results in the neglect of high frequency parts affecting subjective hearing sensation.
The method involves filtering audio frequency signals into low and high frequency components, performing frequency domain transformations, extracting spectral envelope and flatness information, and applying quantization coding to improve the coding efficiency of high frequency parts by incorporating spectral flatness and envelope information, thereby enhancing the quality of decoded audio.
This approach improves the quality of decoded audio frequencies by accurately coding and decoding high frequency signals, reducing noise and enhancing the overall sound quality, especially in low code rate scenarios.
Smart Images

Figure 2024541008000001_ABST
Abstract
Description
[Technical field]
[0001] Cross-references to related applications This application is filed based on a Chinese patent application having application number 202210681060.9 and filing date June 15, 2022, and seeks priority from the Chinese patent application, the entire contents of which are hereby incorporated by reference into this application.
[0002] The present application relates to audio frequency processing technology, and in particular to an audio frequency processing method, device, electronic device, computer readable storage medium and computer program product. [Background technology]
[0003] In the coding scene of ultra-wideband audio frequencies, according to the hearing mechanism and psychoacoustic model of the human ear, users are usually more sensitive to the low frequency part of the signal than the high frequency part of the signal. In the coding and decoding process, more code rate is allocated to the low frequency part of the signal compared to the high frequency part of the signal. However, it does not mean that the high frequency part is abandoned, and the loss of the high frequency part will affect the subjective hearing sensation.
[0004] Therefore, in the coding scene of ultra-wideband audio frequency, it is necessary to code and decode high-frequency signals. However, there is no effective solution in the related art for the technical problem of how to achieve highly efficient coding and decoding of high-frequency signals in the case of an extremely low code rate. Summary of the Invention [Problem to be solved by the invention]
[0005] The present embodiments provide an audio frequency processing method, device, electronic device, computer-readable storage medium and computer program product, and can improve the quality of the audio frequency obtained by subsequent decoding by coding spectral flatness information during coding to improve the perfection of the coding of the high frequency part. [Means for solving the problem]
[0006] The technical solution in the embodiment of the present application is realized as follows.
[0007] In accordance with an embodiment of the present invention, there is provided a method for audio frequency processing, the method being performed by an electronic device, comprising: filtering the audio frequency signal to obtain a low frequency signal and a high frequency signal; coding the low frequency signal to obtain a code stream of the low frequency signal; frequency domain transform processing the low frequency signal to obtain a low frequency spectrum, and frequency domain transform processing the high frequency signal to obtain a high frequency spectrum; performing a spectral envelope extraction process on the low frequency spectrum and the high frequency spectrum to obtain spectral envelope information of the audio frequency signal, and performing a spectral flatness extraction process on the high frequency spectrum to obtain spectral flatness information of the high frequency spectrum; quantizing and coding the spectral flatness information of the high-frequency spectrum and the spectral envelope information of the voice frequency signal to obtain a band extension code stream of the voice frequency signal, and forming a coding code stream of the voice frequency signal by the band extension code stream and the code stream of the low-frequency signal.
[0008] An embodiment of the present invention provides an audio frequency processing apparatus, comprising: a band splitting module arranged to filter the audio frequency signal to obtain a low frequency signal and a high frequency signal; a coding module arranged to code-process the low-frequency signal to obtain a codestream of the low-frequency signal; a frequency domain transform module arranged to frequency domain transform the low frequency signal to obtain a low frequency spectrum and to frequency domain transform the high frequency signal to obtain a high frequency spectrum; an extraction module configured to perform a spectral envelope extraction process on the low-frequency spectrum and the high-frequency spectrum to obtain spectral envelope information of the audio frequency signal, and to perform a spectral flatness extraction process on the high-frequency spectrum to obtain spectral flatness information of the high-frequency spectrum; a quantization module configured to quantize and code the spectral flatness information of the high-frequency spectrum and the spectral envelope information of the audio frequency signal to obtain a band extension code stream of the audio frequency signal, and to form a coding code stream of the audio frequency signal by the band extension code stream and the code stream of the low-frequency signal.
[0009] An embodiment of the present invention provides a method for audio frequency processing, the method being performed by an electronic device, comprising: decomposing the coding codestream to obtain a bandwidth extension codestream and a codestream of the low frequency signal; decoding the code stream of the low-frequency signal to obtain a low-frequency signal, and performing a frequency domain transform on the low-frequency signal to obtain a low-frequency spectrum of the low-frequency signal; inverse quantizing the bandwidth extension codestream to obtain spectral flatness information and spectral envelope information; performing a high-frequency spectrum reconstruction process based on the spectral flatness information, the spectral envelope information and the low-frequency spectrum to obtain a high-frequency spectrum; and time-domain transforming the high-frequency spectrum to obtain a high-frequency signal, and synthesizing the low-frequency signal and the high-frequency signal to obtain an audio frequency signal corresponding to the coding code stream.
[0010] An embodiment of the present invention provides an audio frequency processing apparatus, comprising: a decomposition module arranged to decompose the coding codestream to obtain a bandwidth extension codestream and a codestream of the low frequency signal; a core module configured to decode the code stream of the low-frequency signal to obtain a low-frequency signal, and to frequency-domain transform the low-frequency signal to obtain a low-frequency spectrum of the low-frequency signal; a dequantization module configured to dequantize the bandwidth extension codestream to obtain spectral flatness information and spectral envelope information; a reconstruction module configured to perform a high-frequency spectrum reconstruction process based on the spectral flatness information, the spectral envelope information and the low-frequency spectrum to obtain a high-frequency spectrum; a time domain transform module configured to time domain transform the high frequency spectrum to obtain a high frequency signal, and to synthesize the low frequency signal and the high frequency signal to obtain an audio frequency signal corresponding to the coding code stream.
[0011] An embodiment of the present application provides an electronic device, a memory for storing computer executable instructions; and a processor for implementing the audio frequency processing method provided by the present embodiment when the computer-executable instructions stored in the memory are executed.
[0012] An embodiment of the present invention provides a computer-readable storage medium having stored thereon computer-executable instructions which, when executed by a processor, implement the audio frequency processing method provided by the embodiment of the present invention.
[0013] An embodiment of the present application provides a computer program product including computer executable instructions, which, when executed by a processor, implements the audio frequency processing method provided by the embodiment of the present application.
[0014] The present embodiment has the following beneficial effects:
[0015] The audio frequency signal is filtered to obtain a low frequency signal with a low frequency and a high frequency signal with a high frequency, the low frequency signal is coded to obtain a code stream of the low frequency signal, the spectrum envelope information of the audio frequency signal and the spectrum flatness information of the high frequency signal are extracted from the low frequency spectrum of the low frequency signal and the high frequency spectrum of the high frequency signal, the spectrum flatness information and the spectrum envelope information are quantized and coded to obtain a band extension code stream of the audio frequency signal, and the coding code stream of the audio frequency signal is combined with the code stream of the low frequency signal to form a coding code stream of the audio frequency signal, so that the effective coding of the high frequency signal can be realized through the spectrum envelope information and the spectrum flatness information, and the spectrum flatness information helps to recover the high frequency signal and has a supplementary effect on the spectrum envelope information, so that the perfection of the coding of the high frequency part can be improved, and the quality of the audio frequency obtained in the subsequent decoding process can be improved. [Brief description of the drawings]
[0016] [Figure 1] FIG. 1 is a structural schematic diagram of an audio frequency processing system provided by an embodiment of the present application; [Figure 2A-2B] FIG. 2 is a structural schematic diagram of an electronic device provided by an embodiment of the present application. [Figure 3A-3D] FIG. 2 is a schematic flow diagram of an audio frequency processing method provided by an embodiment of the present application. [Figure 4] FIG. 2 is a schematic diagram of band extension coding in the audio frequency processing method provided by an embodiment of the present application; [Diagram 5]FIG. 2 is a schematic diagram of a band extension decoding in the audio frequency processing method provided by an embodiment of the present application; [Figure 6] FIG. 2 is a schematic diagram of a band extension decoding in the audio frequency processing method provided by an embodiment of the present application; [Figure 7] 1 is a schematic diagram of coding in the audio frequency processing method provided by an embodiment of the present application; [Figure 8] FIG. 2 is a schematic diagram of decoding in the audio frequency processing method provided by an embodiment of the present application. [Figure 9] FIG. 1 is a schematic diagram of a spectrum provided by an example of the present application. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0017] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be described in more detail below in combination with the accompanying drawings, but the described embodiments should not be regarded as limitations on the present application, and all other embodiments obtained by those skilled in the art without performing creative work shall all fall within the scope of protection of the present application.
[0018] In the following description, references to a "set of embodiments" describe a subset of all possible embodiments, but it should be understood that the "set of embodiments" can be the same or different subsets of all possible embodiments, and can be combined with each other unless inconsistent.
[0019] In the following description, the terms "first / second / third" referred to are merely for distinguishing between similar objects and do not indicate a particular order of arrangement of the objects, and "first / second / third" can be used interchangeably with a particular order or order where permitted, and it should be understood that the embodiments of the present application described herein can also be implemented in an order other than that illustrated or described herein.
[0020] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art. The terms used herein are only for the purpose of describing the embodiments of the present application, and are not intended to limit the present application.
[0021] Before describing the embodiments of the present application in more detail, the nouns and terms referred to in the embodiments of the present application will be explained, and the following interpretations will be applied to the nouns and terms referred to in the embodiments of the present application.
[0022] 1) Bandwidth Extension (BWE): Also known as band duplication, it is a typical technology in the field of voice frequency coding. Bandwidth extension technology is a type of parameter coding technology, which can effectively expand the bandwidth at the receiving terminal through bandwidth expansion, thereby improving the quality of voice frequency signals (also called "audio signals"), allowing users to intuitively perceive more vivid timbre, louder volume, and better voice clarity.
[0023] 2) Quadrature Mirror Filters (QMF): Includes an analysis-synthesis filter pair. The analysis filters are used to decompose subband signals and reduce the signal bandwidth so that each subband signal can be smoothly processed through the channel. The synthesis filters are used to synthesize each subband signal recovered at the decoding terminal, and reconstruct the original voice frequency signal through methods such as zero-value interpolation and band-pass filtering.
[0024] 4) Modified Discrete Cosine Transform (MDCT): A type of linear lapped orthogonal transform. It uses time-domain aliasing cancellation technology and includes a 50% time-domain overlapping window. It effectively overcomes the edge effect in the overlapping window without degrading the coding performance, thereby effectively eliminating the periodic noise generated by the edge effect.
[0025] 5) Spectral Band Replication (SBR): A technology for improving the source coding system. It is achieved by reducing the spectral bandwidth at the coding terminal and replicating the corresponding audio frequencies at the decoding terminal, which can significantly reduce the coding bit rate while maintaining the same sound quality perception.
[0026] 7) Neural Networks (NN): A mathematical algorithm model that mimics the behavioral characteristics of animal neural networks and performs distributed and parallel information processing. This type of network relies on the complexity of the system and achieves the purpose of information processing by adjusting the interconnection relationships between a large number of internal nodes.
[0027] In the coding scenario of ultra-wideband audio frequency, according to the hearing mechanism and psychoacoustic model of the human ear, users are usually more sensitive to the low frequency part of the signal than the high frequency part of the signal. In the coding and decoding process, more code rate is allocated to the low frequency part of the signal compared to the high frequency part of the signal. However, it does not mean that the high frequency part is abandoned, and the loss of the high frequency part will affect the subjective hearing sensation. Therefore, it is necessary to code and decode high frequency signals in the coding scenario of ultra-wideband audio frequency.
[0028] In the related art, a parameterized representation can be implemented for a high-frequency signal, and the high-frequency part of the audio frequency signal is reconstructed in a decoding terminal according to the parameters and the corresponding low-frequency part of the audio frequency signal. In the related art, when performing parameterized coding for a high-frequency signal, only the spectral envelope information of the high-frequency signal is taken into consideration, and coding with stronger characteristics cannot be performed for the high-frequency signal. Therefore, the applicant finds that, in implementing the embodiments of the present application, for the bandwidth extension scheme of the related art using a non-AI audio codec, there is a certain error in the result obtained by decoding, and the error is caused to a relatively large extent by the coding and decoding of the high-frequency signal being not very accurate, and for the bandwidth extension scheme of the related art using an AI audio codec, a model of the low-frequency signal is constructed through a neural network, and the error associated with the result obtained after code transmission is significantly different from the error of the result obtained by the non-AI audio codec, and the result obtained by decoding has an even larger error, that is, the error caused by the coding and decoding of the high-frequency signal being not very accurate is more prominent, and there is obvious noise in the high-frequency part reconstructed in the decoding terminal.
[0029] The present embodiment provides a voice frequency processing method, device, electronic device, computer readable storage medium and computer program product, and improves the quality of the voice frequency obtained by decoding in the subsequent process by coding spectral flatness information during coding to improve the perfection of coding of the high frequency part. An exemplary application of the electronic device provided by the present embodiment will be described below. The electronic device provided by the present embodiment can be implemented as a terminal, a server, or a terminal and a server in cooperation. The following description will be given by taking as an example a case where the terminal and the server implement the voice frequency processing method provided by the present embodiment in cooperation.
[0030] Referring to FIG. 1, FIG. 1 is a structural schematic diagram of a voice frequency decoding system 100 provided by an embodiment of the present application. In order to realize the support of voice applications, as shown in FIG. 1, the voice frequency decoding system 100 includes a server 200, a network 300, a first terminal 400 (i.e., a coding terminal) and a second terminal 500 (i.e., a decoding terminal), and the network 300 can be a local area network, or a wide area network, or a combination of both.
[0031] In a series of embodiments, the client 410 is operated on the first terminal 400, and the client 410 can be various types of clients, for example, an instant messaging client, a network conference client, a live broadcast client, a browser, etc. In response to a voice frequency collection command issued by a sender (for example, a caller of a network conference, a main caster, a speaker of a voice call, etc.), the client 410 activates a microphone provided in the first terminal 400 to collect a voice frequency signal, and filters the collected voice frequency signal to obtain a low frequency signal and a high frequency signal. The frequency of the low-frequency signal is lower than that of the high-frequency signal, the low-frequency signal is coded to obtain a code stream of the low-frequency signal, the low-frequency signal is frequency domain converted to obtain a low-frequency spectrum, the high-frequency signal is frequency domain converted to obtain a high-frequency spectrum, the low-frequency spectrum and the high-frequency spectrum are spectral envelope extracted to obtain spectral envelope information of the voice frequency signal, the high-frequency spectrum is spectral flatness extracted to obtain spectral flatness information of the high-frequency spectrum, the spectral flatness information of the high-frequency spectrum and the spectral envelope information of the voice frequency signal are quantized and coded to obtain a band extension code stream of the voice frequency signal, and the band extension code stream and the code stream of the low-frequency signal form a coding code stream of the voice frequency signal. Then, the client 410 transmits the coding code stream to the server 200 via the network 300, and the server 200 can transmit the code stream to a second terminal 500 associated with a receiving side (e.g., a participant of a network conference, an audience, a recipient of a voice call, etc.).After receiving the coding code stream sent by the server 200, the client 510 (e.g., an instant messaging client, a network conference client, a live broadcast client, a browser, etc.) performs decomposition processing of the coding code stream to obtain a bandwidth extension code stream and the code stream of the low-frequency signal, performs decoding processing of the code stream of the low-frequency signal to obtain a low-frequency signal, performs frequency domain transformation processing of the low-frequency signal to obtain a low-frequency spectrum of the low-frequency signal, performs inverse quantization processing of the bandwidth extension code stream to obtain spectrum flatness information and spectrum envelope information, performs reconstruction processing of the high-frequency spectrum according to the spectrum flatness information, the spectrum envelope information and the low-frequency spectrum to obtain a high-frequency spectrum, the frequency of the high-frequency spectrum is higher than the frequency of the low-frequency spectrum, performs time domain transformation processing of the high-frequency spectrum to obtain a high-frequency signal, and performs synthesis processing of the low-frequency signal and the high-frequency signal to obtain an audio frequency signal corresponding to the coding code stream.
[0032] The voice frequency processing method provided in the embodiments of the present application can be widely applied to various different types of voice call application scenarios, such as voice calls made through instant messaging clients, voice calls in game applications, and voice calls in network conference clients.
[0033] Take the network conference scene as an example. Network conference is an important part of online business. In the network conference, the voice collecting device (e.g., microphone) of the network conference participant needs to collect the voice signal of the speaker and then transmit the collected voice signal to other participants of the network conference. This process involves the transmission and playback of voice signals among multiple participants. In this scenario, the voice frequency processing method provided in the embodiment of the present application can be applied to code and decode the voice signal of the network conference, so as to make the coding and decoding of high-frequency signals in the voice signal more efficient and accurate, and improve the quality of voice calls in the network conference.
[0034] In another series of embodiments, the present embodiment can be realized through Cloud Technology, which refers to a hosting technology that unifies system sources such as hardware, software, and networks within a wide area network or a local area network to realize calculation, storage, processing, and sharing of data.
[0035] Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, and application technology that are applied based on the cloud computing business model, and can form a resource pool, which can be used as needed, and is flexible and convenient. Cloud computing technology will be an important support. The service interoperability between the above servers 200 can be realized through cloud technology.
[0036] Exemplarily, the server 200 shown in FIG. 1 may be an independent physical server, a server group or a distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud memory, network services, cloud communications, middleware services, domain services, security services, CDN, and big data, artificial intelligence platforms. The first terminal 400 and the second terminal 500 shown in FIG. 1 may be, but are not limited to, a smartphone, a tablet PC, a notebook computer, a desktop computer, a smart speaker, a smart watch, a voice recognition device, a smart home appliance, an in-vehicle terminal, an aircraft, etc. The terminals (e.g., the first terminal 400 and the second terminal 500) and the server 200 may be directly or indirectly connected by wired or wireless communication, and are not limited in the present embodiment.
[0037] In a series of embodiments, the terminal (e.g., the second terminal 500) or the server 200 can further operate a computer program to realize the audio frequency processing method provided by the embodiments of the present application. For example, the computer program can be a native program or software module in an operating system, a native application program (APP, Application), that is, a program that can only be operated after being implemented in an operating system, such as a live streaming APP, a network conference APP, or an instant messaging APP, or a mini program, that is, a program that can be operated after being downloaded into a browser environment. In summary, the computer program can be any type of application program, module, or plug-in.
[0038] Referring to Fig. 2A, Fig. 2A is a structural schematic diagram of an electronic device provided by an embodiment of the present application, and a first terminal 400 shown in Fig. 2A includes at least one processor 410, a memory 450, at least one network interface 420 and a user interface 430. Each component of the first terminal is coupled together through a bus system 440. As can be understood, the bus system 440 is used to realize the connection communication between these components. The bus system 440 includes a data bus, and further includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, all the various buses are referred to as the bus system 440 in Fig. 2A.
[0039] The processor 410 may be an integrated circuit chip having signal processing capabilities, such as a general purpose processor, a digital signal processor (DSP), or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc., where the general purpose processor may be a microprocessor or any conventional processor, etc.
[0040] The user interface 430 includes one or more output devices 431, such as one or more speakers and / or one or more visual displays capable of presenting media content, and one or more input devices 432, such as user interface members that support user input, such as a keyboard, mouse, microphone, touch panel, camera, and other input buttons and controls.
[0041] Memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid state memory, hard disk drives, optical disk drives, etc. Memory 450 may include one or more storage devices whose physical location is remote from processor 410.
[0042] The memory 450 may include volatile or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be read only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 450 described in the present embodiments is intended to include any suitable type of memory.
[0043] In one set of embodiments, memory 450 can store data to support various operations, examples of which include programs, modules and data structures, or a subset or superset, illustratively described below.
[0044] The operating system 451 includes system programs for processing various basic system services and executing hardware-related tasks, such as a framework layer, a core base layer, and a drive layer, and is used to realize various basic operations and process hardware-based tasks.
[0045] The network communications module 452 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 420, example network interfaces 420 include Bluetooth, Wireless Fidelity (WiFi), and Universal Serial Bus (USB).
[0046] The presentation module 453 is used to make information presentable via output devices 431 (e.g., displays, speakers, etc.) associated with one or more user interfaces 430 (e.g., user interfaces for operating peripherals and displaying content, information).
[0047] The input processing module 454 is used to detect one or more user inputs or interactions from one of the one or more input devices 432 and to translate the detected inputs or interactions.
[0048] In a series of embodiments, the device provided in the present embodiment can be realized by adopting a software method, and Fig. 2A shows an audio frequency processing device 455 stored in memory 450, which can be software in the form of program, plug-in, etc., and includes the following software modules: band division module 4551, core module 4552, frequency domain transformation module 4553, extraction module 4554 and quantization module 4555, and these modules are logical, so they can be arbitrarily combined or further decomposed according to the functions to be realized. The following text will explain the functions of each module.
[0049] Referring to Fig. 2B, Fig. 2B is a structural schematic diagram of an electronic device provided by an embodiment of the present application, and the second terminal 500 shown in Fig. 2B includes at least one processor 510, a memory 550, at least one network interface 520, and a user interface 530. Each component of the second terminal 500 is coupled together via a bus system 540. The user interface 530 includes one or more output devices 531 that can present media content. The user interface 530 further includes one or more input devices 532. The memory 550 further includes an operating system 551, a network communication module 552, a presentation module 553, and an input processing module 554.
[0050] In a series of embodiments, the device provided in the present embodiment can be realized by adopting software method, Fig. 2B shows an audio frequency processing device 555 stored in memory 550, which can be software in the form of program or plug-in, etc., and includes the following software modules: decomposition module 5551, decoding module 5552, inverse quantization module 5553, reconstruction module 5554 and time domain transform module 5555, and these modules are logical, so they can be arbitrarily combined or further decomposed according to the functions to be realized. The following text will explain the function of each module.
[0051] The following describes the voice frequency decoding method provided by the embodiment of the present application from the perspective of the interaction between a first terminal device (ie, a coding terminal), a server, and a second terminal device (ie, a decoding terminal).
[0052] Referring to FIG. 3A, FIG. 3A is a flow diagram of an audio frequency processing method provided by an embodiment of the present application, which describes the steps shown in FIG. 3A in combination, and the steps shown in FIG. 3A are performed by a first terminal device (i.e., a coding terminal).
[0053] It should be noted here that the steps performed by the terminal device are specifically performed by the client that runs on the terminal device, and for the sake of simplicity, the present application does not make a specific distinction between the terminal device and the client that runs on the terminal device.It should also be noted that the voice frequency decoding method provided in the present application embodiment can be performed by various types of computer programs that run on the terminal device, but is not limited to the client that the terminal device runs, and can also be the operating system, software module, script and mini-program mentioned in the above text, so the example of the client in the following text should not be considered as a restriction on the present application embodiment.
[0054] In step 101, the audio frequency signal is filtered to obtain a low frequency signal and a high frequency signal.
[0055] For example, the frequency of the low-frequency signal is lower than that of the high-frequency signal, the high-frequency signal refers to a signal with a frequency exceeding 3 megahertz, and the low-frequency signal refers to a signal with a frequency between 30 kilohertz and 300 kilohertz. The audio frequency signal is an ultra-wideband signal with a sampling rate of 32 kilohertz (kHz), and the 32 kHz sampling rate means sampling 32,000 times per second to obtain 32,000 sampling points. The audio frequency signal is divided into frames according to a frame length of 640 points, with 640 sampling points as one frame, the frame length of each frame being 640 points, and the time length of each frame being 0.02 seconds. The audio frequency signal is filtered through a group of QMF band division filters to obtain a high-frequency part (high-frequency signal) with a frame length of 320 points and a low-frequency part (low-frequency signal) with a frame length of 320 points.
[0056] In step 102, the low-frequency signal is subjected to a coding process to obtain a codestream of the low-frequency signal.
[0057] In a series of embodiments, in step 102, the low-frequency signal is coded to obtain a code stream of the low-frequency signal, which can be realized by the following technical solution: the low-frequency signal is subjected to a feature extraction process to obtain a first feature of the low-frequency signal, and the first feature is subjected to a quantization coding process to obtain a code stream of the low-frequency signal of the audio frequency signal.
[0058] For example, the coding process is used to compress the low-frequency signal and retain the information carried in the low-frequency signal, and the coding process can be a traditional coding technique or a coding process based on deep learning technique, for example, the coding process for the low-frequency signal can be realized through the Penguins voice engine based on deep learning, and a feature extraction process is performed on the low-frequency signal through a neural network model to obtain a feature vector (first feature), and the data dimension of the feature vector is smaller than the data dimension of the low-frequency signal. A vector quantization or scalar quantization process is performed on the feature vector corresponding to the low-frequency signal to obtain an index value, and an entropy coding process is performed on the index value to obtain a code stream of the low-frequency signal.
[0059] For example, the entire dynamic range of the feature vector of a low-frequency signal is divided into multiple intervals, each interval has a representative value, and when the feature vector that falls within the interval is replaced with this representative value during quantization, the feature vector at that time is one-dimensional, so it is called standard quantization, and the extension and extension of scalar quantization is seen in vector quantization. A vector is composed of multiple scalar data, and in vector quantization, quantization is performed on the vector, which divides the vector space into multiple small regions, each small region searches for one representative vector, and the vector that falls within the small region during quantization is replaced by this representative vector.
[0060] In the present embodiment, a feature vector (first feature) that is significantly smaller than the dimension of the original signal is generated through a neural network model, and then the effect of low code rate coding can be achieved through techniques such as entropy coding.
[0061] In step 103, the low-frequency signal is frequency domain transformed to obtain a low-frequency spectrum, and the high-frequency signal is frequency domain transformed to obtain a high-frequency spectrum.
[0062] For example, the frequency domain transform process can be an MDCT process, in which the MDCT process is performed on a high-frequency signal to obtain a high-frequency spectrum (including a plurality of spectrum coefficients); or the DCT process is performed on a high-frequency signal to obtain a high-frequency spectrum (including a plurality of spectrum coefficients); the MDCT process is performed on a low-frequency signal to obtain a low-frequency spectrum (including a plurality of spectrum coefficients); or the DCT process is performed on a low-frequency signal to obtain a low-frequency spectrum (including a plurality of spectrum coefficients).
[0063] In step 104, the low-frequency spectrum and the high-frequency spectrum are subjected to a spectral envelope extraction process to obtain the spectral envelope information of the audio frequency signal, and the high-frequency spectrum is subjected to a flatness extraction process to obtain the spectral flatness information of the high-frequency spectrum.
[0064] In the following, we will first explain the origin of the spectrum. The audio frequency signal is a time domain signal, and the horizontal axis of the sequence signal represents time, and the vertical axis represents amplitude. For example, if the amplitude (which can indicate the volume) of each audio frequency frame is different, it cannot embody the change rule with the change of frequency over time. Therefore, it is necessary to perform frequency domain analysis (for example, Fourier transform) on the time domain signal to obtain a spectrogram, which can display the spectrum range of the audio frequency signal. Specifically, the time domain signal can be decomposed into a DC component (that is, a constant) and the sum of multiple sine wave signals, and each sine wave component has its own frequency and amplitude value. Therefore, by plotting the amplitude values of the above several sine wave signals on the corresponding frequency with the frequency value on the horizontal axis and the amplitude value on the vertical axis, it becomes a signal amplitude value distribution map, that is, the spectrogram shown in Figure 9.
[0065] As an example, the frequency range shown in the spectrum in Fig. 9 is 0 Hz to 10000 Hz, and it can be seen from Fig. 9 that the amplitude rises or falls with the change in frequency, and the curve formed by connecting the highest points (peaks in the spectrogram) before each amplitude drop in the spectrogram is called the spectrum envelope. The spectrum envelope extraction process is, in other words, to extract information indicated by the low-frequency spectrum envelope from the low-frequency spectrum, that is, low-frequency spectrum envelope information, and to extract information indicated by the high-frequency spectrum envelope from the high-frequency spectrum, that is, high-frequency envelope information. The low-frequency spectrum envelope here is the curve formed by connecting the highest points before each amplitude drop in the spectrogram corresponding to the low-frequency section in Fig. 9, and the high-frequency spectrum envelope here is the curve formed by connecting the highest points before each amplitude drop in the spectrogram corresponding to the high-frequency section in Fig. 9.
[0066] In a series of embodiments, performing spectral envelope extraction processing on the low-frequency spectrum and the high-frequency spectrum to obtain spectral envelope information of the audio frequency signal in step 104 can be realized by the following technical solution: performing spectral envelope extraction processing on the low-frequency spectrum to obtain low-frequency spectral envelope information of the low-frequency spectrum, performing spectral envelope extraction processing on the high-frequency spectrum to obtain high-frequency spectral envelope information of the high-frequency spectrum, and constructing the spectral envelope information of the audio frequency signal using the low-frequency spectral envelope information and the high-frequency spectral envelope information.
[0067] According to the embodiment of the present application, by extracting the spectral envelope information, coding of the energy of low frequency signals and high frequency signals is realized during band extension coding, and by improving the effectiveness of coding, a better recovery effect can be obtained during the subsequent decoding process.
[0068] In the following, a specific embodiment will be introduced in which a low-frequency spectrum is subjected to a spectral envelope extraction process to obtain low-frequency spectral envelope information of the low-frequency spectrum, and a specific embodiment will be introduced in which a high-frequency spectrum is subjected to a spectral envelope extraction process to obtain high-frequency spectral envelope information of the high-frequency spectrum.
[0069] In a series of embodiments, the above-mentioned high-frequency spectrum is subjected to a spectral envelope extraction process to obtain high-frequency spectral envelope information of the high-frequency spectrum, which can be realized by the following technical solution: obtaining second fusion configuration data of the high-frequency spectrum, the second fusion configuration data including the spectral line ordinals of each second spectral line combination, and for each second spectral line combination, extracting a spectral coefficient corresponding to each spectral line ordinal of the second spectral line combination from the high-frequency spectrum; squaring the spectral coefficients of each spectral line ordinal to obtain second squared spectral coefficients of each spectral line ordinal; if the number of spectral line ordinals of the second spectral line combination is more than one, adding the second squared spectral coefficients of the multiple spectral line ordinals to obtain a second summation result; logarithmizing the second summation result to obtain second fusion spectral envelope information corresponding to the second spectral line combination; and generating high-frequency spectral envelope information based on the second fusion spectral envelope information of at least one second spectral line combination.
[0070] For example, the second fusion configuration data is stored in the terminal locally or in the server in the form of a data table, so that the first terminal device can easily read it directly from the terminal locally or obtain it from the server. An example of the second fusion configuration data is specifically shown in Table 1. From Table 1, it can be read that the second fusion configuration data includes four second spectral line combinations, the spectral line ordinal of the first second spectral line combination is 0-19, the spectral line ordinal of the second second spectral line combination is 20-54, the spectral line ordinal of the third second spectral line combination is 55-89, and the spectral line ordinal of the second second spectral line combination is 90-130, and each spectral line has its own spectral coefficient.
[0071] [Table 1]
[0072] For example, the spectral coefficients of the high-frequency spectrum are fused according to Table 1 to extract the spectral envelope information of each second spectral line combination in the high-frequency spectrum as shown in formula (1).
[0073]
number
[0074] In the formula, m k denotes the spectral coefficient of the spectrum whose spectral line ordinal number is k in the high frequency spectrum (obtained through MDCT transformation processing), and i denotes the spectral envelope ordinal number (the ordinal number of the second spectral line combination). For example, when i is 1,
[0075]
number
[0076] m0, m1, …, m 19 are squared and the squared results are added together, and Spec_env(i) represents the second fused spectral envelope information of the second spectral line combination whose spectral envelope ordinal number is i.
[0077] In the following, the second spectral line combination with the spectral envelope ordinal number 1 will be taken as an example for detailed description. It can be seen from Table 1 that the quantity of the second spectral line combination is 4. For the second spectral line combination A with the spectral envelope ordinal number 1, the spectral coefficients of each spectral line ordinal number corresponding to the second spectral line combination A are extracted from the high frequency spectrum, that is, the spectral coefficient m0 of the spectrum with the spectral line ordinal number 0 to the spectral coefficient m1 of the spectrum with the spectral line ordinal number 19 are extracted. 19 Then, the spectral coefficients of each spectral line ordinal are squared to obtain the second squared spectral coefficient m k 2 , e.g., m0 2 , m1 2 , …, m 19 2 Since the number of the spectral line ordinals of the second spectral line combination A is 20, if the second spectral line ordinals are multiple, the second squared spectral coefficients of the multiple spectral line ordinals are added to obtain the second sum result
[0078]
number
[0079] corresponds to adding 20 second spectral squaring coefficients and logarithmizing the second addition result to obtain second fused spectral envelope information Spec_env(1) corresponding to the second spectral line combination. If the spectral line ordinal number of the second spectral line combination B is 1, the second squared spectral coefficient corresponding to the only spectral line ordinal number is logarithmized to obtain second fused spectral envelope information corresponding to the second spectral line combination. If the number of second spectral line combinations is multiple, the high-frequency spectral envelope information is composed of the second fused spectral envelope information Spec_env(i) of the multiple second spectral line combinations. If the number of second spectral line combinations is 1, the second fused spectral envelope information Spec_env(i) of the second spectral line combination is used as the high-frequency spectral envelope information.
[0080] In the embodiment of the present application, when the spectrum envelope information fusion process based on the second fusion configuration data is performed on the spectrum of the high-frequency signal, the second fusion configuration data is used to indicate which spectral lines need to be fused, which is based on the theoretical basis of the critical band in the psychoacoustic model, and is obtained by comprehensively considering the quality of BWE and the code rate in the specific experiment. The critical band is a result obtained by an experiment based on psychoacoustic, and specifically reflects the conversion between the physical mechanical stimulation and the neural electrical stimulation in the cochlea of the human inner ear. For a certain frequency and other frequencies within a certain range nearby it, and for the pure tone voice frequency signal, the neural electrical stimulation converted by the human ear is consistent, which indicates that there is no need to use an excessively large code rate to achieve an excessively high frequency domain resolution. Through multiple experimental measurements, it has been found that when the energy envelope selection range of the high-frequency part is the second fusion configuration data, a relatively good code rate and voice frequency quality can be achieved.
[0081] In a series of embodiments, performing the spectral envelope extraction process on the above-mentioned low-frequency spectrum to obtain low-frequency spectral envelope information of the low-frequency spectrum is realized by the following technical solution: obtaining first fusion configuration data of the low-frequency spectrum, the first fusion configuration data including the spectral line ordinals of each first spectral line combination; for each first spectral line combination, extracting a spectral coefficient corresponding to each spectral line ordinal of the first spectral line combination from the low-frequency spectrum; squaring the spectral coefficient of each spectral line ordinal to obtain a first squared spectral coefficient of each spectral line ordinal; if the number of spectral line ordinals of the first spectral line combination is more than one, adding the first squared spectral coefficients of the multiple spectral line ordinals to obtain a first sum; logarithmizing the first sum result to obtain first fused spectral envelope information corresponding to the first spectral line combination; and generating low-frequency spectral envelope information based on the first fused spectral envelope information of at least one first spectral line combination.
[0082] For example, the first fusion configuration data can be stored locally in the terminal or stored in a server in the form of a data table, and can be directly read by the first terminal device locally in the terminal or obtained from the server. An example of the first fusion configuration data is specifically shown in Table 2. From Table 2, it can be read that the first fusion configuration data includes one first spectral line combination, and the spectral line ordinal number of the first spectral line combination No. 1 is 80 to 150, and each spectral line has its own spectral coefficient.
[0083] [Table 2]
[0084] For example, the spectral coefficients of the low-frequency spectrum are fused according to Table 2 to extract the spectral envelope information of each first spectral line combination in the low-frequency spectrum as shown in Formula (2).
[0085]
number
[0086] In the formula, m k denotes the spectral coefficient of the spectrum whose spectral line ordinal number is k in the low-frequency spectrum (obtained through MDCT transformation processing), and i denotes the spectral envelope ordinal number (the ordinal number of the first spectral line combination). For example, when i is 1,
[0087]
number
[0088] m 80 , m 81 , …, m 149 are squared and the squared results are added together, and Spec_env(i) represents the first fused spectral envelope information of the first spectral line combination whose spectral envelope ordinal number is i.
[0089] In the following, the first spectral line combination with the spectral envelope ordinal number 1 will be taken as an example for detailed description. It can be seen from Table 2 that the quantity of the first spectral line combination is 1. For the first spectral line combination A with the spectral envelope ordinal number 1, the spectral coefficients of each spectral line ordinal number corresponding to the first spectral line combination A are extracted from the low frequency spectrum, that is, the spectral coefficient m 80 The spectral coefficient m of the spectrum with the spectral line ordinal number 150 from 150 The first squared spectral coefficient mk 2 , e.g., m 80 2 , m 81 2 , …, m 150 2 Since the number of the spectral line ordinals of the first spectral line combination A is 71, if the number of the spectral line ordinals of the first spectral line combination A is multiple, the first squared spectral coefficients of the multiple spectral line ordinals are added to obtain the first sum result
[0090]
number
[0091] corresponds to adding 71 first spectral squaring coefficients and logarithmizing the first addition result to obtain first fused spectral envelope information Spec_env(1) corresponding to the first spectral line combination. If the number of spectral line ordinals of the first spectral line combination B is 1, the first squared spectral coefficient corresponding to the only spectral line ordinal is logarithmized to obtain first fused spectral envelope information corresponding to the first spectral line combination. If the number of first spectral line combinations is multiple, the first fused spectral envelope information Spec_env(i) of the multiple first spectral line combinations constitutes the low-frequency spectral envelope information. Alternatively, if the number of first spectral line combinations is 1, the first fused spectral envelope information Spec_env(i) of the first spectral line combination is used as the low-frequency spectral envelope information.
[0092] In the embodiment of the present application, when performing a spectral envelope information fusion process based on the first fusion configuration data on the spectrum of a low-frequency signal, the first fusion configuration data is used to indicate what spectral lines need to be fused, which is obtained through statistical and measurement experiments; when the coding of the low-frequency signal is AI ultra-wideband voice coding, AI ultra-wideband voice coding has relatively strong modeling ability for voice and has noise reduction ability, so it is necessary to introduce variables to measure and predict the noise reduction results, and the energy envelope of the low-frequency part can be used as a predictive variable; and through statistics and measurements on a large-scale data set, it is found that when the energy envelope selection range of the low-frequency part is the first fusion configuration data, a relatively good code rate and voice frequency quality can be achieved.
[0093] In a series of embodiments, performing the spectral flatness extraction process on the high-frequency spectrum in step 104 to obtain the spectral flatness information of the high-frequency spectrum can be realized by the following technical solution: obtaining third fusion configuration data of the high-frequency spectrum, the third fusion configuration data including the spectral line ordinals of each third spectral line combination; obtaining a geometric mean value of the third spectral line combination for each third spectral line combination, and obtaining an arithmetic mean value of the third spectral line combination; comparing the geometric mean value of the third spectral line combination with the arithmetic mean value of the third spectral line combination as the spectral flatness information of the third spectral line combination; and generating the spectral flatness information of the high-frequency spectrum based on the spectral flatness information of at least one third spectral line combination.
[0094] For example, the third fusion configuration data can be stored locally in the terminal or stored in a server in the form of a data table, and can be directly read by the first terminal device locally in the terminal or obtained from the server. An example of the third fusion configuration data is specifically shown in Table 3. From Table 3, it can be read that the third fusion configuration data includes two third spectral line combinations, the spectral line ordinal number of the first third spectral line combination is 0-39, and the spectral line ordinal number of the second third spectral line combination is 40-80, and each spectral line has its own spectral coefficient.
[0095] [Table 3]
[0096] For example, the spectral coefficients of the high-frequency spectrum are fused according to Table 3 to extract the spectral flatness information of each third spectral line combination in the high-frequency spectrum as shown in Formula (3).
[0097]
number
[0098] In the formula, nume(i) and demo(i) respectively denote the geometric mean value and the arithmetic mean value of the i-th third spectral line combination in the high frequency spectrum, and the spectral flatness information Flatness(i) denotes a comparison value between the geometric mean value and the arithmetic mean value of the i-th third spectral line combination, where i denotes the ordinal number of the third spectral line combination.
[0099] In the following, the third spectral line combination with the ordinal number 1 will be taken as an example for detailed description. It can be seen from Table 3 that the quantity of the third spectral line combination is 2. For the third spectral line combination A with the first number shown in Table 3, the spectral coefficients of each spectral line ordinal number corresponding to the third spectral line combination A are extracted from the high frequency spectrum, that is, the spectral coefficient m0 of the spectrum with the spectral line ordinal number 0 to the spectral coefficient m1 of the spectrum with the spectral line ordinal number 39 are extracted. 39 The spectral coefficients m0 of the spectrum with the spectral line ordinal number 0 to m 39 Based on this, the arithmetic mean value and the geometric mean value of the third spectral line combination A are determined, and the comparison value between the geometric mean value and the arithmetic mean value is taken as the spectral flatness information of the third spectral line combination A. If there are a plurality of third spectral line combinations, the spectral flatness information Flatness(i) of the plurality of third spectral line combinations constitutes the spectral flatness information of the high-frequency spectrum, or if there is only one third spectral line combination, the spectral flatness information Flatness(i) of the third spectral line combination is taken as the spectral flatness information of the high-frequency spectrum.
[0100] The high frequency spectral flatness fusion in Table 3 is theoretically based on the critical band in the psychoacoustic model, and the specific experiment is obtained by comprehensively considering the quality of BWE and the code rate. The critical band is the result obtained by the experiment based on psychoacoustic, specifically reflecting the conversion between the physical mechanical stimulation and the neural electrical stimulation in the cochlea of the human inner ear. For a certain frequency and other frequencies in a certain range nearby, and for the pure tone voice frequency signal, the neural electrical stimulation converted by the human ear is consistent, which means that there is no need to use excessively high code rates to achieve excessively high frequency domain resolution. Through the statistics and measurements of large data sets, it has been found that when the high frequency spectral flatness fusion selection range is the third fusion configuration result, it can achieve a relatively good code rate and voice frequency quality.
[0101] In a series of embodiments, obtaining the geometric mean value of the third spectral line combination can be achieved by the following technical solution: extracting spectral coefficients corresponding to each spectral line ordinal of the third spectral line combination from the high frequency spectrum, squaring the spectral coefficients of each spectral line ordinal to obtain third squared spectral coefficients for each spectral line ordinal; if the third spectral line combination has more than one spectral line ordinal, multiplying the third squared spectral coefficients of the multiple spectral line ordinal numbers to obtain a first multiplication result; and performing a root-setting process on the first multiplication result according to the number of spectral line ordinal numbers to obtain the geometric mean value corresponding to the third spectral line combination.
[0102] As an example, the calculation process of the geometric mean value can be referred to formula (4).
[0103]
number
[0104] In the above example, for the third spectral line combination A (i is 1) with an ordinal number of 1, the spectral coefficients corresponding to each spectral line ordinal of the third spectral line combination A are extracted from the high frequency spectrum, that is, the spectral coefficient m0 of the spectrum with the spectral line ordinal number 0 to the spectral coefficient m 39 The spectral coefficients of each spectral line ordinal are then squared to obtain the third squared spectral coefficient m k 2 , e.g., m0 2 , m1 2 etc., and if the number of spectral line ordinals of the third spectral line combination is multiple, multiply the third squared spectral coefficients of the multiple spectral line ordinals to obtain the first multiplication result
[0105]
number
[0106] This is equivalent to exponentiating 40 third spectral squaring coefficients, and a root setting process is performed on the first multiplication result based on the quantity of the spectral line ordinals (i.e., equivalent to raising to the 40th power) to obtain the geometric mean value num(1) corresponding to the third spectral line combination.
[0107] In a series of embodiments, obtaining the arithmetic mean value of the third spectral line combination can be achieved by the following technical solution: extracting spectral coefficients corresponding to each spectral line ordinal of the third spectral line combination from the high frequency spectrum, squaring the spectral coefficients of each spectral line ordinal to obtain third squared spectral coefficients for each spectral line ordinal; if the third spectral line combination has more than one spectral line ordinal, adding the third squared spectral coefficients of the multiple spectral line ordinal numbers to obtain a third sum; and averaging the third sum according to the number of spectral line ordinal numbers to obtain the arithmetic mean value corresponding to the third spectral line combination.
[0108] As an example, the calculation process of the geometric mean value can be referred to formula (5).
[0109]
number
[0110] Based on the above example, for the third spectral line combination with ordinal number 1 (i is 1), the spectral coefficients corresponding to each spectral line ordinal of the third spectral line combination A are extracted from the high frequency spectrum, that is, the spectral coefficient m0 of the spectrum with the spectral line ordinal number 0 to the spectral coefficient m 39 The spectral coefficients of each spectral line ordinal are then squared to obtain the third squared spectral coefficient m k 2 , e.g., m0 2 , m1 2 etc., and if the number of spectral line ordinals of the third spectral line combination is multiple, the third squared spectral coefficients of the multiple spectral line ordinals are added to obtain the third sum result
[0111]
number
[0112] This corresponds to adding 40 third spectral squaring coefficients, and the third sum is averaged based on the quantity of the spectral line ordinals (i.e., divided by 40) to obtain the arithmetic mean demo(1) corresponding to the third spectral line combination.
[0113] In step 105, the spectral flatness information of the high-frequency spectrum and the spectral envelope information of the voice frequency signal are quantized and coded to obtain a band extension code stream of the voice frequency signal, and a coding code stream of the voice frequency signal is obtained by combining the band extension code stream with the code stream of the low-frequency signal.
[0114] In a series of embodiments, the quantization coding of the spectral flatness information of the high frequency spectrum and the spectral envelope information of the audio frequency signal in step 105 to obtain a band extension code stream of the audio frequency signal can be realized by the following technical solution: obtain a quantization table of the spectral flatness information and a quantization table of the spectral envelope information, quantize the spectral flatness information of the high frequency spectrum according to the quantization table of the spectral flatness information to obtain a quantization result of the spectral flatness, quantize the spectral envelope information of the audio frequency signal according to the quantization table of the spectral envelope information to obtain a quantization result of the spectral envelope, and configure a band extension code stream of the audio frequency signal according to the quantization result of the spectral flatness and the quantization result of the spectral envelope.
[0115] In a series of embodiments, obtaining the above-mentioned quantization table of spectrum flatness information and the quantization table of spectrum envelope information can be achieved by the following technical solutions: obtaining a plurality of voice sample signals; filtering each of the voice sample signals to obtain a low-frequency sample signal and a high-frequency sample signal, the frequency of the low-frequency sample signal being lower than that of the high-frequency sample signal; frequency domain transforming the low-frequency sample signal to obtain a low-frequency sample spectrum; frequency domain transforming the high-frequency sample signal to obtain a high-frequency sample spectrum; filtering the low-frequency sample signal and the high-frequency sample spectrum; This can be realized by the technical solution of performing the following processes: performing spectral envelope extraction processing to obtain spectral envelope information of the voice sample signals, performing spectral flatness extraction processing on the high frequency spectrum to obtain spectral flatness information of the voice sample signals; performing clustering processing on the spectral flatness information of the multiple voice sample signals to obtain a plurality of spectral flatness clustering centers, and building a quantization table of the spectral flatness information based on the multiple spectral flatness clustering centers; performing clustering processing on the spectral envelope information of the multiple voice sample signals to obtain a plurality of spectral envelope clustering centers, and building a quantization table of the spectral envelope information based on the multiple spectral envelope clustering centers.
[0116] For example, after obtaining a plurality of voice sample signals, the process of filtering the voice sample signals to obtain the spectral envelope information and the spectral flatness information of the voice sample signals can refer to the specific embodiment of step 104.
[0117] As an example, the quantization table used for quantizing the spectral flatness information is as shown in Table 4. The clustering centers of each spectral flatness are embodied in Table 4, which correspond to the four clustering centers obtained by the clustering process. In the subsequent quantization process, the spectral flatness information A is quantized to become the clustering center with the smallest numerical difference from A among the four clustering centers.
[0118] [Table 4]
[0119] As an example, a quantization table used for the spectral envelope information of the high frequency portion is as shown in Table 5. Table 5 embodies the clustering centers of the spectral envelope obtained by clustering based on the first subband and the second subband of the high frequency portion of the sample data, which correspond to 31 clustering centers obtained by the clustering process. In the subsequent quantization process, the spectral envelope information A of the first subband and the second subband of the high frequency portion is quantized so that it becomes the clustering center with the smallest numerical difference from A among the 31 clustering centers.
[0120] [Table 5]
[0121] As an example, the quantization table used for the spectral envelope information of the high frequency part is further shown in Table 6. Table 6 embodies the clustering centers of the spectral envelope obtained by clustering based on the third and fourth subbands of the high frequency part of the sample data, which correspond to eight clustering centers obtained by the clustering process. In the subsequent quantization process, the spectral envelope information A of the third and fourth subbands of the high frequency part is quantized so that it becomes the clustering center with the smallest numerical difference from A among the eight clustering centers.
[0122] [Table 6]
[0123] As an example, the spectral envelope quantization table for the low frequency part is as shown in Table 7. Table 7 embodies the clustering centers of the spectral envelope obtained by clustering based on the low frequency part of the sample data, which correspond to eight clustering centers obtained by the clustering process. In the subsequent quantization process, the spectral envelope information A of the low frequency part is quantized to become the clustering center with the smallest numerical difference from A among the eight clustering centers.
[0124] [Table 7]
[0125] The generation process of Tables 4 to 7 was obtained by experimental statistics, and a large amount of voice frequency files were clustered according to the above flow, and a large amount of statistical distribution based on the voice frequency distribution was finally obtained. Considering the code rate and the quality of the voice frequency comprehensively, clustering quantization was performed on the statistical distribution, and Tables 4 to 7 were finally generated.
[0126] By using the quantization coding method, the spectral flatness information and the spectral envelope information can be effectively compressed and represented, so as to reduce the data amount of the spectral flatness information and the spectral envelope information, thereby avoiding excessive occupation of communication resources, and effectively improving communication efficiency.
[0127] The voice frequency processing method provided by the present embodiment can perform effective coding by combining spectral envelope information and spectral flatness information for high frequency parts without giving more code rate than the related art, and can perform effective characterization with low complexity for high frequency parts, so that a more true and natural voice frequency signal can be restored in the subsequent decoding process. In particular, when the encoder is an encoder based on a neural network model, the low frequency signal is modeled through a neural network, and there is a significant difference between the error associated with the result obtained after code transmission and the error of the result obtained by a non-AI audio codec, and there is an even larger error in the result obtained by decoding, that is, the error associated with the coding and decoding of high frequency signals being not very accurate becomes more significant, and there is significant noise in the high frequency part reconstructed at the decoding terminal using the band extension scheme of the related art, but by applying the voice frequency processing method provided by the present embodiment, the accurate high frequency part can be reconstructed to restore a more true and natural voice frequency signal.
[0128] Please refer to FIG. 3B, which is a flow diagram of the audio frequency processing method provided in the embodiment of the present application, and describes the steps shown in FIG. 3B in combination.
[0129] In step 201, the coding codestream is decomposed to obtain a bandwidth extension codestream and a codestream of the low frequency signal.
[0130] For example, refer to Fig. 8, which is a schematic diagram of decoding in the audio frequency processing method provided by the embodiment of the present application, where a decoding terminal decomposes a received coding code stream into a BWE code stream and the code stream of the low frequency signal, the code stream of the low frequency signal is restored by an AI ultra-wideband audio decoder, the low frequency signal and the BWE code stream are restored by a BWE decoder provided by the embodiment of the present application to restore a high frequency code stream, the time domain of the high frequency code stream is converted into a high frequency signal, and an ultra-wideband signal is generated from the high frequency signal and the low frequency signal by a synthesis filter group.
[0131] In step 202, the code stream of the low-frequency signal is decoded to obtain a low-frequency signal, and the low-frequency signal is subjected to a frequency domain transform to obtain a low-frequency spectrum of the low-frequency signal.
[0132] For example, refer to FIG. 5. FIG. 5 is a schematic diagram of band extension decoding in the audio frequency processing method provided by the embodiment of the present application. The low-frequency signal in FIG. 5 is the low-frequency signal obtained by decoding. The low-frequency signal obtained by decoding is subjected to frequency domain transformation processing to obtain the low-frequency spectrum of the low-frequency signal. The frequency domain transformation processing can be MDCT processing or DCT processing.
[0133] In step 203, the bandwidth extension codestream is dequantized to obtain spectral flatness information and spectral envelope information.
[0134] Since the bandwidth extension code stream is obtained by subjecting the spectral flatness information and the spectral envelope information to a quantization coding process, the spectral flatness information and the spectral envelope information can be obtained by a dequantization decoding process.
[0135] In step 204, a high-frequency spectrum reconstruction process is performed based on the spectral flatness information, the spectral envelope information and the low-frequency spectrum to obtain a high-frequency spectrum.
[0136] In a series of embodiments, refer to FIG. 3C, which is a schematic flow diagram of an audio frequency processing method provided in an embodiment of the present application. In step 204, performing a high-frequency spectrum reconstruction process based on the spectral flatness information, the spectral envelope information and the low-frequency spectrum to obtain a high-frequency spectrum can be realized by steps 2041 to 2044 shown in FIG. 3C.
[0137] In step 2041, the low-frequency spectrum is subjected to a spectral flatness extraction process to obtain low-frequency spectral flatness information of the low-frequency spectrum, and subband spectral flatness information of each low-frequency subband in the low-frequency spectrum is extracted from the low-frequency spectral flatness information.
[0138] In a series of embodiments, performing the spectral flatness extraction process on the low-frequency spectrum in step 2041 to obtain low-frequency spectral flatness information of the low-frequency spectrum can be realized by the following technical solution: obtaining fourth fusion configuration data of the low-frequency spectrum, the fourth fusion configuration data including the spectral line ordinals of each fourth spectral line combination, obtaining the geometric mean value of the fourth spectral line combination for each fourth spectral line combination, and obtaining the arithmetic mean value of the fourth spectral line combination; obtaining the comparison value between the geometric mean value of the low-frequency spectrum and the arithmetic mean value of the low-frequency spectrum as the spectral flatness information of the fourth spectral line combination; and generating the spectral flatness information of the low-frequency spectrum based on the spectral flatness information of at least one fourth spectral line combination.
[0139] In a series of embodiments, obtaining the geometric mean value of the above-mentioned fourth spectral line combination can be achieved by the following technical solution: extracting spectral coefficients corresponding to each spectral line ordinal of the fourth spectral line combination from the low-frequency spectrum, squaring the spectral coefficients of each spectral line ordinal to obtain fourth squared spectral coefficients of each spectral line ordinal; if the quantity of the spectral line ordinal of the fourth spectral line combination is more than one, multiplying the fourth squared spectral coefficients of the multiple spectral line ordinal numbers to obtain a second multiplication result, and then root-setting the second multiplication result according to the quantity of the spectral line ordinal to obtain the geometric mean value corresponding to the fourth spectral line combination.
[0140] In a series of embodiments, obtaining the arithmetic mean value of the fourth spectral line combination can be achieved by the following technical solution: extracting spectral coefficients corresponding to each spectral line ordinal of the fourth spectral line combination from the low frequency spectrum, squaring the spectral coefficients of each spectral line ordinal to obtain fourth squared spectral coefficients for each spectral line ordinal; if the fourth spectral line combination has a plurality of spectral line ordinals, adding the fourth squared spectral coefficients of the plurality of spectral line ordinals to obtain a fourth summation result, and averaging the fourth summation result according to the number of spectral line ordinals to obtain the arithmetic mean value corresponding to the fourth spectral line combination.
[0141] The embodiment of determining the low-frequency spectral flatness information of the low-frequency spectrum in step 2041 can refer to the real-time method of extracting the spectral flatness information of the high-frequency spectrum in step 104, and the only difference is that the processing object is changed from the high-frequency spectrum to the low-frequency spectrum, so the fourth spectral line combination used is also different from the third spectral line combination.
[0142] In step 2042, subband spectral flatness information corresponding to each high-frequency subband of the high-frequency spectrum is extracted from the spectral flatness information, and subband spectral envelope information corresponding to each high-frequency subband of the high-frequency spectrum is extracted from the spectral envelope information.
[0143] In step 2043, for each high-frequency subband in the high-frequency spectrum, determine the spectral flatness numerical difference between the subband flatness information of each low-frequency subband in the low-frequency spectrum and the subband spectral flatness information of the high-frequency subband, and determine the low-frequency subband with the smallest spectral flatness numerical difference as the target spectrum.
[0144] In step 2044, an amplitude value adjustment process is performed on a target spectrum corresponding to each high-frequency subband according to the subband spectral envelope information corresponding to each high-frequency subband of the high-frequency spectrum and the spectral flatness numerical difference corresponding to each high-frequency subband, and the adjustment results corresponding to the multiple high-frequency subbands are spliced into the high-frequency spectrum.
[0145] In a series of embodiments, in step 2044, the amplitude value adjustment process is performed on the target spectrum corresponding to each high frequency subband according to the subband spectral envelope information corresponding to each high frequency subband of the high frequency spectrum and the spectral flatness numerical difference corresponding to each high frequency subband. This can be realized by the following technical solution: for the target spectrum corresponding to each high frequency subband, a process of determining a white noise suitable for the spectral flatness numerical difference of the high frequency subband, and adding the suitable white noise to the target spectrum to obtain a composite target spectrum; a process of determining spectral envelope information of the composite target spectrum, and determining the spectral envelope numerical difference between the spectral envelope information of the composite target spectrum and the spectral envelope information of the high frequency subband; and a process of adjusting the amplitude value of the composite target spectrum based on the spectral envelope numerical difference.
[0146] For example, a specific restoration process can be as shown in Figure 6, first, the low frequency spectrum is subjected to spectral flatness analysis and calculation to obtain the spectral flatness of the low frequency part, the calculation process can refer to formulas (7) to (9), then according to the spectral flatness information of the high frequency part, the low frequency part closest to each high frequency subband is selected as the target spectrum, then according to the difference of the spectral flatness information and the spectral envelope information, the target spectrum is energy fine-tuned, finally, a plurality of subbands in the high frequency part are spliced to be a perfect high frequency spectrum, and adjusted by a gradient filter, a perfect high frequency spectrum is obtained. The high frequency spectrum is subjected to the time-frequency inverse transform of MDCT to obtain a high frequency signal, and the restored high frequency signal and the low frequency signal decoded by the decoder are input to a group of orthogonal mirror image mixing filters for synthesis and filtering, and a super-wideband voice signal is obtained.
[0147] In the voice frequency processing method provided in the embodiment of the present application, according to the spectrum, spectral envelope information and spectral flatness information of the high frequency part of the restored low frequency signal in the decoding terminal, joint processing is carried out to reconstruct the high frequency spectrum, and the decoding terminal controls the error, so as to avoid the voice encoder (especially the very low code rate voice encoder based on NN modeling) from amplifying the coding error in the low frequency part in the high frequency part, thereby greatly improving the sound quality of the decoding.
[0148] In step 205, the high frequency spectrum is subjected to a time domain transform process to obtain a high frequency signal, and the low frequency signal and the high frequency signal are subjected to a synthesis process to obtain a voice frequency signal corresponding to the coding code stream.
[0149] For example, a time-frequency inverse transform of MDCT is performed on a high-frequency spectrum to obtain a high-frequency signal, and the high-frequency signal obtained by the recovery and the low-frequency signal obtained by decoding by a decoder are input to a group of quadrature mirror image mixing filters for synthesis filtering to obtain a voice frequency signal.
[0150] Please refer to Fig. 3D, which is a flow diagram of the audio frequency processing method provided by the embodiment of the present application, in which the complete coding and decoding process is shown.
[0151] In step 301, a coding terminal filters a voice frequency signal to obtain a low frequency signal and a high frequency signal.
[0152] In step 302, a coding terminal performs coding processing on a low-frequency signal to obtain a codestream of the low-frequency signal.
[0153] In step 303, the coding terminal performs frequency domain transform processing on the low-frequency signal to obtain a low-frequency spectrum, and performs frequency domain transform processing on the high-frequency signal to obtain a high-frequency spectrum.
[0154] In step 304, the coding terminal performs a spectral envelope extraction process on the low-frequency spectrum and the high-frequency spectrum to obtain the spectral envelope information of the voice frequency signal, and performs a spectral flatness extraction process on the high-frequency spectrum to obtain the spectral flatness information of the high-frequency spectrum.
[0155] In step 305, the coding terminal performs quantization coding processing on the spectral flatness information of the high-frequency spectrum and the spectral envelope information of the voice-frequency signal to obtain a band extension code stream of the voice-frequency signal, and forms a coding code stream of the voice-frequency signal by the band extension code stream and the code stream of the low-frequency signal.
[0156] In step 306, the coding terminal transmits the coding code stream to the decoding terminal.
[0157] In step 307, the decoding terminal decomposes the coding codestream to obtain a bandwidth extension codestream and a codestream of the low-frequency signal.
[0158] In step 308, a decoding terminal decodes the code stream of the low-frequency signal to obtain a low-frequency signal, and performs a frequency domain transform process on the low-frequency signal to obtain a low-frequency spectrum of the low-frequency signal.
[0159] In step 309, the decoding terminal performs inverse quantization processing on the bandwidth extension codestream to obtain spectral flatness information and spectral envelope information.
[0160] In step 310, the decoding terminal performs a reconstruction process of the high-frequency spectrum according to the spectral flatness information, the spectral envelope information and the low-frequency spectrum to obtain the high-frequency spectrum.
[0161] In step 311, the decoding terminal performs a time domain transform process on the high frequency spectrum to obtain a high frequency signal, and performs a synthesis process on the low frequency signal and the high frequency signal to obtain a voice frequency signal corresponding to the coding code stream.
[0162] The voice frequency signal is filtered to obtain a low frequency signal with a low frequency and a high frequency signal with a high frequency, the low frequency signal is coded to obtain a code stream of the low frequency signal, the spectrum envelope information of the voice frequency signal and the spectrum flatness information of the high frequency signal are extracted from the low frequency spectrum of the low frequency signal and the high frequency spectrum of the high frequency signal, the spectrum flatness information and the spectrum envelope information are quantized and coded to obtain a band extension code stream of the voice frequency signal, and the code stream of the low frequency signal is combined to form a coding code stream of the voice frequency signal, so that the spectrum envelope information and the spectrum flatness information are used to effectively code the high frequency signal, and the quality of the voice frequency obtained in the subsequent decoding process can be improved by jointly processing the spectrum, the spectrum envelope information and the spectrum flatness information of the high frequency part restored in the decoding terminal to reconstruct the high frequency spectrum.
[0163] The following describes an exemplary application of the present embodiment in one practical application scenario.
[0164] In a series of embodiments, a client is operated in the first terminal, and the client can be various types of clients, for example, an instant messaging client, a network conference client, a live broadcast client, a browser, etc. In response to a voice frequency collection command issued by a sender (for example, a caller of a network conference, a main caster, a speaker of a voice call, etc.), the client activates a microphone provided in the first terminal to collect voice frequency signals, and filters the collected voice frequency signals to obtain low frequency signals and high frequency signals. The frequency of the low-frequency signal is lower than that of the high-frequency signal, the low-frequency signal is coded to obtain a code stream of the low-frequency signal, the low-frequency signal is frequency domain converted to obtain a low-frequency spectrum, the high-frequency signal is frequency domain converted to obtain a high-frequency spectrum, the low-frequency spectrum and the high-frequency spectrum are spectral envelope extracted to obtain spectral envelope information of the voice frequency signal, the high-frequency spectrum is spectral flatness extracted to obtain spectral flatness information of the high-frequency spectrum, the spectral flatness information of the high-frequency spectrum and the spectral envelope information of the voice frequency signal are quantized and coded to obtain a band extension code stream of the voice frequency signal, and the band extension code stream and the code stream of the low-frequency signal form a coding code stream of the voice frequency signal.Then, the client transmits the coding code stream to a server via a network, and the server can transmit the code stream to a second terminal associated with a receiving side (e.g., a participant of a network conference, an audience, a recipient of a voice call, etc.).After receiving the coding code stream sent by the server, a client (such as an instant messaging client, a network conference client, a live streaming client, a browser, etc.) performs decomposition processing on the coding code stream to obtain a bandwidth extension code stream and the code stream of the low-frequency signal, performs decoding processing on the code stream of the low-frequency signal to obtain a low-frequency signal, performs frequency domain transformation processing on the low-frequency signal to obtain a low-frequency spectrum of the low-frequency signal, performs inverse quantization processing on the bandwidth extension code stream to obtain spectrum flatness information and spectrum envelope information, performs reconstruction processing on the high-frequency spectrum according to the spectrum flatness information, the spectrum envelope information and the low-frequency spectrum to obtain a high-frequency spectrum, the frequency of the high-frequency spectrum is higher than the frequency of the low-frequency spectrum, performs time domain transformation processing on the high-frequency spectrum to obtain a high-frequency signal, and performs synthesis processing on the low-frequency signal and the high-frequency signal to obtain an audio frequency signal corresponding to the coding code stream.
[0165] In the voice frequency processing method provided in the embodiment of the present application, a coding terminal compresses low-frequency parts in a voice frequency signal to obtain a code stream of the low-frequency signal, and at the same time, implements a bandwidth expansion scheme based on spectral flatness information to realize ultra-wideband voice code transmission under a very low code rate.
[0166] Referring to FIG. 4, FIG. 4 is a schematic diagram of the band extension coding in the audio frequency processing method provided by the embodiment of the present application, the input signal is an ultra-wideband signal with a sampling rate of 32 kilohertz (kHz), the 32 kHz sampling rate means sampling 32,000 times per second to obtain 32,000 sampling points, the input signal is divided into frames according to a frame length of 640 points, and 640 sampling points are divided into one frame, the frame length of each frame is 640 points, and the frame time length of each frame is 0.02 seconds, and the signal is processed through a group of QMF band division filters to obtain a high-frequency part with a frame length of 320 points and a low-frequency part with a frame length of 320 points, which are referred to as high-frequency signal (high-frequency signal) and low-frequency signal (low-frequency signal) respectively in the following text.
[0167] Based on the sampling points of the frame length 320 and the sampling points of the frame shift 160, the high-frequency signal and the low-frequency signal are respectively subjected to MDCT time-frequency transformation to obtain the corresponding high-frequency spectrum (high-frequency spectrum) and low-frequency spectrum (low-frequency spectrum), where the sampling points of the frame shift 160 refer to the time interval corresponding to the sampling points where the time difference between the start positions of two adjacent frames is 160.
[0168] The low-frequency spectrum and the high-frequency spectrum are fused according to the corresponding spectrum envelope fusion table to extract the spectrum envelope information, and the formula used for extracting the spectrum envelope information is as shown in Formula (6).
[0169]
number
[0170] In the formula, m k denotes the k-th spectral coefficient of the MDCT transformation result, and i denotes the ordinal number of the spectral envelope. For example, when i is 1, m0, m1, ..., m 19 are squared respectively, and the squared results are added together.
[0171] The spectral envelope fusion tables for the high frequency part and the low frequency part used in the present embodiment are shown in Tables 8 and 9, respectively.
[0172] First, referring to Table 8 in combination, Table 8 shows the fusion process for four sets of MDCT transformation results, in which the 0th to 19th spectral coefficients of the MDCT transformation results are fused according to formula (6), which corresponds to fusing the 0th to 19th spectral lines, the 20th to 54th spectral coefficients of the MDCT transformation results are fused according to formula (6), the 55th to 89th spectral coefficients of the MDCT transformation results are fused according to formula (6), and the 90th to 130th spectral coefficients of the MDCT transformation results are fused according to formula (6).
[0173] [Table 8]
[0174] The high frequency spectrum envelope fusion Table 8 is based on the theoretical basis of the critical band in the psychoacoustic model, and is obtained by comprehensively considering the quality of BWE and the code rate in specific experiments. The critical band is the result obtained by the experiment based on psychoacoustic, specifically reflecting the conversion between the physical mechanical stimulation and the neural electrical stimulation in the cochlea of the human inner ear, and for a certain frequency and other frequencies in a certain range nearby and the pure tone sound frequency signal, the neural electrical stimulation converted by the human ear is consistent, which means that there is no need to use excessively high code rates to achieve excessively high frequency domain resolution. Based on multiple experimental measurements, and using code rate and BWE quality as the evaluation indexes of the experimental results, the data shown in Table 8 was obtained.
[0175] Further, referring to Table 9 in combination, Table 9 shows the fusion process for one set of MDCT transform results, and the fusion process is performed on the 80th to 150th spectral coefficients of the MDCT transform results based on formula (6).
[0176] [Table 9]
[0177] Table 9 of the spectrum envelope fusion of the low frequency part is also obtained by statistical and measurement experiments. If the encoder used for the low frequency part is an AI ultra-wideband voice encoder, the AI ultra-wideband voice encoder has a relatively strong modeling ability for voice and has a noise reduction ability, so it is necessary to introduce variables to measure and predict its noise reduction effect. The energy envelope of the low frequency part can be used as a predictive variable, and through statistics and measurements on a large data set, it has been found that when the energy envelope selection range of the low frequency part is the data shown in Table 9, a relatively accurate and stable predicted value can be obtained, and the complexity and code rate are also acceptable. Taking into consideration factors such as calculation accuracy, stability, complexity and code rate, the data shown in Table 9 was selected as the spectrum envelope fusion table for the low frequency part.
[0178] The high frequency spectrum is fused according to the corresponding spectrum flatness fusion table to extract the spectrum flatness information, and the calculation for extracting the spectrum flatness information can refer to formula (7) to formula (9).
[0179]
number
[0180] In the formula, m kindicates the k-th spectral coefficient of the MDCT transformation result, nume(i) and demo(i) respectively indicate the geometric mean value and arithmetic mean value of each spectral line in the MDCT transformation result, and the spectral flatness information Flatness(i) is a comparison value between the geometric mean value and the arithmetic mean value. The spectral flatness information reflects whether the voice frequency corresponding to the corresponding step spectrum is closer to white noise or is a pure tone signal closer to a single frequency. i indicates the ordinal number of the spectral flatness information, for example, when i is 1, m0, m1, ..., m 39 are squared, and the spectral flatness information is determined based on the squared results.
[0181] The high frequency spectrum extracts the spectral flatness information based on the spectral flatness fusion table of the high frequency part, and the result is as shown in Table 10.
[0182] [Table 10]
[0183] The high frequency spectrum flatness fusion table 10 is based on the critical band in the psychoacoustic model as a theoretical basis, and is obtained by comprehensively considering the quality of BWE and the code rate in the specific experiment. The critical band is the result obtained by the experiment based on psychoacoustic, specifically reflecting the conversion between the physical mechanical stimulation and the neural electrical stimulation in the cochlea of the human inner ear, and for a certain frequency and other frequencies in a certain range nearby, and the pure tone sound frequency signal, the neural electrical stimulation converted by the human ear is consistent, which means that there is no need to use excessively high code rates to achieve excessively high frequency domain resolution. Based on multiple experimental measurements, and using code rate and BWE quality as the evaluation index of the experimental results, the data shown in table 10 was obtained.
[0184] The spectral envelope information and the spectral flatness information are quantized and coded according to the corresponding quantization tables to form the BWE code stream. The quantization table used for quantizing the spectral flatness information is shown in Table 11.
[0185] [Table 11]
[0186] The generation process of Table 11 is obtained by statistical experimentation. The spectral flatness of a large number of audio frequency files is calculated according to the above flow, and a statistical distribution based on a large number of audio frequency distributions is finally obtained. The statistical distribution is clustered and quantized by comprehensively considering the code rate and the quality of the audio frequency, and Table 11 is finally generated. The generation methods of the spectral envelope quantization table 12 of the first and second subbands of the high frequency part, the spectral envelope quantization table 13 of the third and fourth subbands of the high frequency part, and the spectral envelope quantization table 14 of the low frequency part are similar to the generation method of Table 11. The specific results of all quantization tables are correlated with the experimental statistics, and the dimensions of the quantization tables can be flexibly adjusted according to specific application scenarios.
[0187] The spectral envelope quantization table for the first and second subbands in the high frequency part is as shown in Table 12.
[0188] [Table 12]
[0189] The spectral envelope quantization table for the third and fourth subbands in the high frequency part is as shown in Table 13.
[0190] [Table 13]
[0191] The spectral envelope quantization table for the low frequency part is as shown in Table 14.
[0192] [Table 14]
[0193] 5 is a schematic diagram of the band extension decoding in the voice frequency processing method provided by the embodiment of the present application, in which the decoding terminal recovers the ultra-wideband voice frequency signal after receiving the BWE code stream and the low frequency signal. After receiving the BWE code stream, the decoding terminal recovers the spectrum envelope information and the spectrum flatness information through the decoding and inverse quantization module. The low frequency spectrum is obtained by performing MDCT time-frequency transformation on the low frequency time domain signal. According to the low frequency spectrum, the high frequency spectrum envelope information and the high frequency spectrum flatness information, the high frequency spectrum is recovered. The specific restoration process is shown in Figure 6. The restoration process is as follows: first, the low-frequency spectrum is subjected to flatness analysis and calculation to obtain the spectral flatness of the low-frequency part, and the calculation process can refer to formulas (7) to (9). Then, according to the spectral flatness information of the high-frequency part, the low-frequency part closest to each high-frequency subband is selected as the target spectrum; then, according to the difference of the spectral flatness information and the spectral envelope information, the target spectrum is energy fine-tuned; finally, the multiple subbands in the high-frequency part are spliced to be a perfect high-frequency spectrum, and adjusted by a gradient filter, and a perfect high-frequency spectrum is obtained. The high-frequency spectrum is subjected to the time-frequency inverse transform of MDCT to obtain a high-frequency signal; the restored high-frequency signal and the low-frequency signal decoded by the decoder are input into a group of orthogonal mirror image mixing filters for synthesis and filtering, and an ultra-wideband voice signal is obtained.
[0194] The voice frequency processing method provided in the embodiment of the present application performs joint decision and adjustment according to the spectrum of the low-frequency signal restored by the decoding terminal, the original spectrum envelope information contained in the BWE side information, and the high-frequency spectrum flatness information to reconstruct the high-frequency spectrum, thereby minimizing the amplification of the coding error in the low-frequency part in the high-frequency part by the ultra-low code rate voice encoder (especially the ultra-low code rate voice encoder based on NN modeling), and thus the sound quality of the decoding is greatly improved.
[0195] Referring to Fig. 7, Fig. 7 is a schematic diagram of the coding in the voice frequency processing method provided by the embodiment of the present application, in the coding terminal, the ultra-wideband signal passes through a group of analysis filters to obtain a high frequency part and a low frequency part, and the AI ultra-wideband voice encoder codes and compresses the low frequency part to obtain the code stream of the low frequency signal. The low frequency part and the high frequency part are simultaneously input to the BWE encoder provided by the embodiment of the present application, which generates a BWE code stream and implements the final code stream together with the code stream of the low frequency signal.
[0196] Referring to Fig. 8, Fig. 8 is a schematic diagram of the decoding in the audio frequency processing method provided by the embodiment of the present application, in which a decoding terminal decomposes a received coding code stream into a BWE code stream and the code stream of the low frequency signal, the code stream of the low frequency signal is passed through an AI ultra-wideband audio decoder to recover the low frequency signal, the low frequency signal and the BWE code stream are passed through the BWE decoder of the embodiment of the present application to recover the high frequency code stream, the high frequency code stream is time domain transformed into a high frequency signal, and the high frequency signal and the low frequency signal are passed through a synthesis filter group to generate an ultra-wideband signal.
[0197] The bandwidth extension technology in the voice frequency processing method provided by the present embodiment can be combined with an AI ultra-wideband voice encoder to realize ultra-wideband voice coding with a very low code rate. The present embodiment adds spectrum flatness information and spectrum envelope information as BWE side information, so that the wideband signal of the AI ultra-wideband voice decoder is extended to an ultra-wideband signal under very low complexity, and the error control for the low-frequency file generated by the voice encoder based on the NN modeling, which is the AI ultra-wideband voice encoder, is added to the decoder side, so that the impact of low-frequency quantization noise on the reconstructed high-frequency signal during bandwidth extension is reduced compared with other bandwidth extension methods in the related art.
[0198] In the following, the exemplary structure of the audio frequency processing device 455 provided in the present embodiment will be described as a software module. In a series of embodiments, as shown in Fig. 2A, the software modules in the audio frequency processing device 455 stored in the memory 450 include a band splitting module 4551 arranged to filter an audio frequency signal to obtain a low frequency signal and a high frequency signal, where the frequency of the low frequency signal is lower than that of the high frequency signal, a coding module 4552 arranged to code the low frequency signal to obtain a code stream of the low frequency signal, and a frequency domain transform module 4553 arranged to frequency domain transform the low frequency signal to obtain a low frequency spectrum and frequency domain transform the high frequency signal to obtain a high frequency spectrum. The audio signal processing device may include a module 4553, an extraction module 4554 arranged to perform spectral envelope extraction processing on the low-frequency spectrum and the high-frequency spectrum to obtain spectral envelope information of the audio frequency signal, and perform spectral flatness extraction processing on the high-frequency spectrum to obtain spectral flatness information of the high-frequency spectrum, and a quantization module 4555 arranged to perform quantization coding processing on the spectral flatness information of the high-frequency spectrum and the spectral envelope information of the audio frequency signal to obtain a band extension code stream of the audio frequency signal, and to construct a coding code stream of the audio frequency signal by the band extension code stream and the code stream of the low-frequency signal.
[0199] In one series of embodiments, the extraction module 4554 is further configured to perform a spectral envelope extraction process on the low frequency spectrum to obtain low frequency spectral envelope information of the low frequency spectrum, perform a spectral envelope extraction process on the high frequency spectrum to obtain high frequency spectral envelope information of the high frequency spectrum, and construct spectral envelope information of the audio frequency signal using the low frequency spectral envelope information and the high frequency spectral envelope information.
[0200] In one embodiment, the extraction module 4554 is further configured to: obtain first fused configuration data of the low-frequency spectrum, the first fused configuration data including spectral line ordinals of each of the first spectral line combinations; and for each of the first spectral line combinations, extract spectral coefficients from the low-frequency spectrum corresponding to each spectral line ordinal of the first spectral line combination; square the spectral coefficients of each spectral line ordinal to obtain first squared spectral coefficients for each spectral line ordinal; if the number of spectral line ordinals of the first spectral line combination is more than one, add the first squared spectral coefficients of the multiple spectral line ordinals to obtain a first sum; logarithmize the first sum to obtain first fused spectral envelope information corresponding to the first spectral line combination; and generate the low-frequency spectral envelope information based on the first fused spectral envelope information of the at least one first spectral line combination.
[0201] In one set of embodiments, the extraction module 4554 is further configured to obtain second fused configuration data of the high frequency spectrum, the second fused configuration data including the spectral line ordinals of each second spectral line combination, and for each second spectral line combination, extract a spectral coefficient corresponding to each spectral line ordinal of the second spectral line combination from the high frequency spectrum; square the spectral coefficient of each spectral line ordinal to obtain a second squared spectral coefficient for each spectral line ordinal; if the quantity of the spectral line ordinals of the second spectral line combination is more than one, add the second squared spectral coefficients of the multiple spectral line ordinals to obtain a second summation result; logarithmize the second summation result to obtain a second fused spectral envelope information corresponding to the second spectral line combination; and generate the high frequency spectral envelope information based on the second fused spectral envelope information of the at least one second spectral line combination.
[0202] In one set of embodiments, the extraction module 4554 is further configured to obtain third fusion configuration data of the high frequency spectrum, the third fusion configuration data including the spectral line ordinals of each third spectral line combination, and for each third spectral line combination, obtain a geometric mean value of the third spectral line combination and obtain an arithmetic mean value of the third spectral line combination; obtain a comparison value between the geometric mean value of the third spectral line combination and the arithmetic mean value of the third spectral line combination as the spectral flatness information of the third spectral line combination; and generate spectral flatness information of the high frequency spectrum based on the spectral flatness information of the at least one third spectral line combination.
[0203] In one embodiment, the extraction module 4554 is further configured to obtain third fused configuration data of the high frequency spectrum, the third fused configuration data including the spectral line ordinals of each third spectral line combination, and for each third spectral line combination, extract a spectral coefficient corresponding to each spectral line ordinal of the third spectral line combination from the high frequency spectrum; square the spectral coefficient of each spectral line ordinal to obtain a third squared spectral coefficient for each spectral line ordinal; if the quantity of the spectral line ordinals of the third spectral line combination is multiple, multiply the third squared spectral coefficients of the multiple spectral line ordinals to obtain a first multiplication result; root the first multiplication result based on the quantity of the spectral line ordinals to obtain a geometric mean value corresponding to the third spectral line combination; and configure the geometric mean value of the third spectral line combination as the geometric mean value of the multiple third spectral line combinations.
[0204] In one embodiment, the extraction module 4554 is further configured to obtain third fused configuration data of the high frequency spectrum, the third fused configuration data including the spectral line ordinals of each third spectral line combination, and for each third spectral line combination, extract a spectral coefficient from the high frequency spectrum corresponding to each spectral line ordinal of the third spectral line combination; square the spectral coefficient of each spectral line ordinal to obtain a third squared spectral coefficient for each spectral line ordinal; if the quantity of the spectral line ordinals of the third spectral line combination is multiple, add the third squared spectral coefficients of the multiple spectral line ordinals to obtain a third sum; average the third sum according to the quantity of the spectral line ordinals to obtain an arithmetic mean value corresponding to the third spectral line combination; and configure the arithmetic mean value of the third spectral line combination as the arithmetic mean value of the multiple third spectral line combinations.
[0205] In a series of embodiments, the quantization module 4555 is further configured to obtain a quantization table of spectral flatness information and a quantization table of spectral envelope information, quantize the spectral flatness information of the high-frequency spectrum according to the quantization table of the spectral flatness information to obtain a quantization result of the spectral flatness, quantize the spectral envelope information of the audio frequency signal according to the quantization table of the spectral envelope information to obtain a quantization result of the spectral envelope, and construct a band extension code stream of the audio frequency signal according to the quantization result of the spectral flatness and the quantization result of the spectral envelope.
[0206] In a series of embodiments, the quantization module 4555 further includes the following processes: obtaining a plurality of audio sample signals; for each audio sample signal, filtering the audio sample signal to obtain a low frequency sample signal and a high frequency sample signal of the audio sample signal, where the frequency of the low frequency sample signal is lower than the frequency of the high frequency sample signal; frequency domain transforming the low frequency sample signal to obtain a low frequency sample spectrum, and frequency domain transforming the high frequency sample signal to obtain a high frequency sample spectrum; extracting a spectrum envelope of the low frequency sample spectrum and the high frequency sample spectrum to obtain spectral envelope information of the audio sample signal, and extracting a spectrum flatness of the high frequency spectrum to obtain the spectral flatness of the audio sample signal. a process of clustering the spectral flatness information of the plurality of voice sample signals to obtain a plurality of spectral flatness clustering centers and a spectral flatness corresponding to each of the spectral flatness clustering centers, and constructing a quantization table of the spectral flatness information based on the spectral flatness information of the plurality of spectral flatness clustering centers and each of the spectral flatness clustering centers; and a process of clustering the spectral envelope information of the plurality of voice sample signals to obtain a plurality of spectral envelope clustering centers and a spectral envelope information corresponding to each of the spectral envelope clustering centers, and constructing a quantization table of the spectral envelope information based on the plurality of spectral envelope clustering centers and the spectral envelope information corresponding to each of the spectral envelope clustering centers.
[0207] In one set of embodiments, the coding module 4552 is further configured to: filter the audio frequency signal to obtain a low frequency signal and a high frequency signal of the audio frequency signal, where the frequency of the low frequency signal is lower than the frequency of the high frequency signal; perform a feature extraction process on the low frequency signal to obtain a first feature of the low frequency signal; perform a high frequency analysis process on the high frequency signal to obtain a second feature of the high frequency signal, where a feature dimension of the second feature is lower than the feature dimension of the first feature; and perform a quantization coding process on the first feature and the second feature to obtain a code stream of the low frequency signal of the audio frequency signal.
[0208] In the following, an exemplary structure of the audio frequency processing device 555 provided in the present embodiment will be described as a software module. In a series of embodiments, as shown in Fig. 2B, the software modules in the audio frequency processing device 555 stored in the memory 550 include a decomposition module 5551 arranged to decompose the coding code stream to obtain a band extension code stream and the code stream of the low frequency signal, a decoding module 5552 arranged to decode the code stream of the low frequency signal to obtain a low frequency signal, and to perform a frequency domain transform process on the low frequency signal to obtain a low frequency spectrum of the low frequency signal, and a decode module 5553 arranged to dequantize the band extension code stream to obtain spectral flatness information and spectral envelope information. The reconstruction module 5554 may include a dequantization module 5553 configured to obtain interleaved information based on the spectral flatness information, the spectral envelope information and the low frequency spectrum, a reconstruction module 5554 configured to perform a reconstruction process of the high frequency spectrum based on the spectral flatness information, the spectral envelope information and the low frequency spectrum to obtain a high frequency spectrum, where the frequency of the high frequency spectrum is higher than the frequency of the low frequency spectrum, and a time domain transform module 5555 configured to perform a time domain transform process of the high frequency spectrum to obtain a high frequency signal, and to synthesize the low frequency signal and the high frequency signal to obtain an audio frequency signal corresponding to the coding code stream.
[0209] In a series of embodiments, the reconstruction module 5554 is further configured to perform a spectral flatness extraction process on the low frequency spectrum to obtain low frequency spectral flatness information of the low frequency spectrum, extract subband spectral flatness information corresponding to each high frequency subband of the high frequency spectrum from the spectral flatness information, extract subband spectral envelope information corresponding to each high frequency subband of the high frequency spectrum from the spectral envelope information, determine a spectral flatness numerical value difference between the subband spectral flatness information of each low frequency subband in the low frequency spectrum and the subband spectral flatness information of the high frequency subband for each high frequency subband of the high frequency spectrum, determine the low frequency subband with the smallest spectral flatness numerical value difference as the target spectrum, and perform an amplitude value adjustment process on the target spectrum corresponding to each high frequency subband according to the subband envelope information corresponding to each high frequency subband of the high frequency spectrum and the spectral flatness numerical value difference corresponding to each high frequency subband, and splice the adjustment results corresponding to the multiple high frequency subbands into the high frequency spectrum.
[0210] In a series of embodiments, the reconstruction module 5554 is further configured to perform the following processes for a target spectrum corresponding to each high frequency subband: determining suitable white noise for the spectral flatness numerical difference of the high frequency subband, and adding suitable white noise to the target spectrum to obtain a composite target spectrum; determining spectral envelope information of the composite target spectrum, and determining a spectral envelope numerical difference between the spectral envelope information of the composite target spectrum and the spectral envelope information of the high frequency subband; and adjusting the amplitude value of the composite target spectrum based on the spectral envelope numerical difference.
[0211] In one set of embodiments, the reconstruction module 5554 is further configured to obtain a geometric mean value of the low frequency spectrum, obtain an arithmetic mean value of the low frequency spectrum, and obtain a comparison value between the geometric mean value of the low frequency spectrum and the arithmetic mean value of the low frequency spectrum as low frequency spectrum flatness information.
[0212] In one embodiment, the reconstruction module 5554 is further configured to obtain fourth fused configuration data of the low frequency spectrum, the fourth fused configuration data including the spectral line ordinals of each fourth spectral line combination, and configured to perform, for each fourth spectral line combination, the following operations: extracting from the low frequency spectrum a spectral coefficient corresponding to each spectral line ordinal of the fourth spectral line combination; squaring the spectral coefficients of each spectral line ordinal to obtain a fourth squared spectral coefficient for each spectral line ordinal; if the quantity of the spectral line ordinals of the fourth spectral line combination is more than one, multiplying the fourth squared spectral coefficients of the plurality of spectral line ordinals to obtain a second multiplication result; rooting the second multiplication result based on the quantity of the spectral line ordinals to obtain a geometric mean value corresponding to the fourth spectral line combination; and constructing a geometric mean value of the low frequency spectrum by the geometric mean value of the plurality of fourth spectral line combinations.
[0213] In one set of embodiments, the reconstruction module 5554 is further configured to obtain fourth fused configuration data of the low frequency spectrum, the fourth fused configuration data including the spectral line ordinals of each fourth spectral line combination, and is configured to perform, for each fourth spectral line combination, the following operations: extracting from the low frequency spectrum a spectral coefficient corresponding to each spectral line ordinal of the fourth spectral line combination; squaring the spectral coefficients of each spectral line ordinal to obtain a fourth squared spectral coefficient for each spectral line ordinal; if the quantity of the spectral line ordinals of the fourth spectral line combination is more than one, adding the fourth squared spectral coefficients of the multiple spectral line ordinals to obtain a fourth summation result; averaging the fourth summation result based on the quantity of the spectral line ordinals to obtain an arithmetic mean value corresponding to the fourth spectral line combination; and constructing an arithmetic mean value of the low frequency spectrum by the arithmetic mean value of the multiple fourth spectral line combinations.
[0214] In an embodiment of the present application, a computer program product is provided, the computer program product includes computer-executable instructions, the computer-executable instructions are stored in a computer-readable storage medium, a processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium, and the processor executes the computer-executable instructions to cause the electronic device to perform the above-mentioned audio frequency processing method of the embodiment of the present application.
[0215] In an embodiment of the present application, a computer-readable storage medium having executable instructions stored thereon is provided, and the computer-executable instructions are stored thereon, and when the computer-executable instructions are executed by a processor, the processor is induced to perform an audio frequency processing method provided by an embodiment of the present application, for example, the audio frequency processing method shown in Figures 3A to 3D.
[0216] In one set of embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface storage device, optical disk or CD-ROM, or may be any device including one or any combination of the above memories.
[0217] In one set of embodiments, the computer executable instructions may take the form of a program, software, software module, script or code, written in any type of programming language (compiled or interpreted, or declarative or procedural), and may be arranged in any form, including as an independent program or as a module, component, subprogram or other unit suitable for use within a computing environment.
[0218] By way of example, the computer executable instructions are not necessarily stored in files corresponding to a file system, but may be files stored in other programs or data, such as within one or more scripts of a HyperText Markup Language (HTML) document, stored in a single file specific to the program being discussed, or stored in multiple common files (e.g., stored in one or more module, subprogram or code portion files).
[0219] By way of example, the computer-executable instructions may be arranged to execute on one electronic device, or may be executed on multiple electronic devices located at one location, or may be executed on multiple electronic devices distributed across multiple locations and interconnected by a communications network.
[0220] In summary, according to the embodiment of the present application, a voice frequency signal is filtered to obtain a low frequency signal with a low frequency and a high frequency signal with a high frequency, the low frequency signal is coded to obtain a code stream of the low frequency signal, the spectrum envelope information of the voice frequency signal and the spectrum flatness information of the high frequency signal are extracted from the low frequency spectrum of the low frequency signal and the high frequency spectrum of the high frequency signal, the spectrum flatness information and the spectrum envelope information are quantized and coded to obtain a band extension code stream of the voice frequency signal, and the code stream of the low frequency signal is combined to form a coding code stream of the voice frequency signal, so that the effective coding of the high frequency signal can be realized through the spectrum envelope information and the spectrum flatness information, the perfection of the coding of the high frequency part is improved, and the quality of the voice frequency obtained in the subsequent decoding process can be improved.
[0221] The above description is merely an embodiment of the present application and does not limit the scope of protection of the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and scope of the present application are intended to be included within the scope of protection of the present application.
Claims
1. 1. A method of audio frequency processing performed by an electronic device, comprising: filtering the audio frequency signal to obtain a low frequency signal and a high frequency signal; performing a first coding process on the low-frequency signal to obtain a code stream of the low-frequency signal; frequency domain transform processing the low frequency signal to obtain a low frequency spectrum, and frequency domain transform processing the high frequency signal to obtain a high frequency spectrum; performing a spectral envelope extraction process on the low frequency spectrum and the high frequency spectrum to obtain spectral envelope information of the audio frequency signal, and performing a spectral flatness extraction process on the high frequency spectrum to obtain spectral flatness information of the high frequency spectrum; performing quantization coding processing on the spectral flatness information of the high frequency spectrum and the spectral envelope information of the voice frequency signal to obtain a band extension code stream of the voice frequency signal, and forming a coding code stream of the voice frequency signal by the band extension code stream and the code stream of the low frequency signal.
2. The step of performing a spectral envelope extraction process on the low frequency spectrum and the high frequency spectrum to obtain spectral envelope information of the audio frequency signal includes: performing a spectral envelope extraction process on the low-frequency spectrum to obtain low-frequency spectral envelope information of the low-frequency spectrum; performing a spectrum envelope extraction process on the high frequency spectrum to obtain high frequency spectrum envelope information of the high frequency spectrum; and constituting a spectral envelope information of the audio frequency signal with the low frequency spectral envelope information and the high frequency spectral envelope information.
3. The step of performing a spectral envelope extraction process on the low-frequency spectrum to obtain low-frequency spectral envelope information of the low-frequency spectrum includes: obtaining first fusion configuration data of the low frequency spectrum, the first fusion configuration data including a spectral line ordinal number of each first spectral line combination; For each of the first spectral line combinations, extracting spectral coefficients from the low frequency spectrum corresponding to each of the spectral line ordinals of the first spectral line combination; squaring the spectral coefficients of each of said spectral line ordinals to obtain first squared spectral coefficients of each of said spectral line ordinals; if the number of spectral line ordinals of the first spectral line combination is more than one, adding the first squared spectral coefficients of the plurality of spectral line ordinals to obtain a first summation result; a process of logarithmizing the first sum result to obtain first fused spectral envelope information corresponding to the first spectral line combination; and generating the low frequency spectral envelope information based on a first fused spectral envelope information of at least one of the first spectral line combinations.
4. The step of performing a spectrum envelope extraction process on the high frequency spectrum to obtain high frequency spectrum envelope information of the high frequency spectrum includes: obtaining second fused configuration data of the high frequency spectrum, the second fused configuration data including a spectral line ordinal number of each second spectral line combination; For each of the second spectral line combinations, extracting spectral coefficients from the high frequency spectrum corresponding to each of the spectral line ordinals of the second spectral line combination; squaring the spectral coefficients of each of said spectral line ordinals to obtain second squared spectral coefficients of each of said spectral line ordinals; if the number of the spectral line ordinals of the second spectral line combination is more than one, adding the second squared spectral coefficients of the plurality of the spectral line ordinals to obtain a second addition result; a process of logarithmizing the second sum result to obtain second fused spectral envelope information corresponding to the second spectral line combination; and generating the high frequency spectral envelope information based on a second fused spectral envelope information of at least one of the second spectral line combinations.
5. The step of performing a spectral flatness extraction process on the high frequency spectrum to obtain spectral flatness information of the high frequency spectrum includes: obtaining third fused configuration data of the high frequency spectrum, the third fused configuration data including a spectral line ordinal number of each third spectral line combination; For each said third spectral line combination, obtaining a geometric mean value of the third spectral line combination and obtaining an arithmetic mean value of the third spectral line combination; a process of determining a comparison value between the geometric mean value of the third spectral line combination and the arithmetic mean value of the third spectral line combination as spectral flatness information of the third spectral line combination; and generating spectral flatness information of the high frequency spectrum based on spectral flatness information of at least one of the third spectral line combinations.
6. Obtaining the geometric mean value of the third spectral line combination comprises: extracting spectral coefficients from the high frequency spectrum corresponding to each spectral line ordinal of the third spectral line combination; squaring the spectral coefficients of each of said spectral line ordinals to obtain third squared spectral coefficients of each of said spectral line ordinals; When the number of the spectral line ordinals of the third spectral line combination is more than one, multiplying the third squared spectral coefficients of the plurality of the spectral line ordinals to obtain a first multiplication result; 6. The method of claim 5, further comprising: root-setting the first multiplication result based on a quantity of the spectral line ordinals to obtain a geometric mean value corresponding to the third spectral line combination.
7. Obtaining an arithmetic mean value of the third spectral line combination comprises: extracting spectral coefficients from the high frequency spectrum corresponding to each spectral line ordinal of the third spectral line combination; squaring the spectral coefficients of each of said spectral line ordinals to obtain third squared spectral coefficients of each of said spectral line ordinals; if the number of the spectral line ordinals of the third spectral line combination is more than one, add up the third squared spectral coefficients of the plurality of the spectral line ordinals to obtain a third addition result; 6. The method of claim 5, further comprising averaging the third summation based on a quantity of the spectral line ordinals to obtain an arithmetic mean value corresponding to the third spectral line combination.
8. quantizing and coding the spectral flatness information of the high frequency spectrum and the spectral envelope information of the audio frequency signal to obtain a band extension code stream of the audio frequency signal; obtaining a quantization table of the spectral flatness information and a quantization table of the spectral envelope information; quantizing the spectral flatness information of the high frequency spectrum according to a quantization table of the spectral flatness information to obtain a quantization result of the spectral flatness; quantizing the spectral envelope information of the audio frequency signal according to a quantization table of the spectral envelope information to obtain a quantization result of the spectral envelope; and constructing a bandwidth extension codestream of the voice frequency signal from the quantized result of the spectral flatness and the quantized result of the spectral envelope.
9. Obtaining the quantization table of the spectral flatness information and the quantization table of the spectral envelope information A plurality of audio sample signals is obtained, and for each of the audio sample signals: filtering the audio sample signal to obtain low and high frequency sample signals of the audio sample signal, the low frequency sample signal having a lower frequency than the high frequency sample signal; frequency domain transform processing the low frequency sampled signals to obtain a low frequency sampled spectrum, and frequency domain transform processing the high frequency sampled signals to obtain a high frequency sampled spectrum; performing a spectral envelope extraction process on the low frequency sample spectrum and the high frequency sample spectrum to obtain spectral envelope information of the audio sample signal, and performing a spectral flatness extraction process on the high frequency spectrum to obtain spectral flatness information of the audio sample signal; clustering the spectral flatness information of the plurality of audio sample signals to obtain a plurality of spectral flatness clustering centers, and constructing a quantization table of the spectral flatness information based on the plurality of spectral flatness clustering centers; The method according to claim 8, further comprising: performing a process of clustering the spectral envelope information of the plurality of the audio sample signals to obtain a plurality of spectral envelope clustering centers; and constructing a quantization table of the spectral envelope information based on the plurality of spectral envelope clustering centers.
10. 1. A method of audio frequency processing performed by an electronic device, comprising: decomposing the coding codestream to obtain a bandwidth extension codestream and a codestream of a low frequency signal; decoding the code stream of the low-frequency signal to obtain a low-frequency signal, and performing a frequency domain transform on the low-frequency signal to obtain a low-frequency spectrum of the low-frequency signal; inverse quantizing the bandwidth extension codestream to obtain spectral flatness information and spectral envelope information; performing a high-frequency spectrum reconstruction process based on the spectral flatness information, the spectral envelope information and the low-frequency spectrum to obtain a high-frequency spectrum; performing a time domain transform process on the high frequency spectrum to obtain a high frequency signal, and performing a synthesis process on the low frequency signal and the high frequency signal to obtain a voice frequency signal corresponding to the coding code stream.
11. performing a high-frequency spectrum reconstruction process based on the spectral flatness information, the spectral envelope information, and the low-frequency spectrum to obtain a high-frequency spectrum, performing a spectral flatness extraction process on the low-frequency spectrum to obtain low-frequency spectral flatness information of the low-frequency spectrum, and extracting subband spectral flatness information of each low-frequency subband in the low-frequency spectrum from the low-frequency spectral flatness information; extracting subband spectral flatness information corresponding to each high-frequency subband of the high-frequency spectrum from the spectral flatness information, and extracting subband spectral envelope information corresponding to each high-frequency subband of the high-frequency spectrum from the spectral envelope information; For each high-frequency subband of the high-frequency spectrum, determine a spectral flatness value difference between the subband spectral flatness information of each low-frequency subband in the low-frequency spectrum and the subband spectral flatness information of the high-frequency subband, and determine the low-frequency subband with the smallest spectral flatness value difference as a target spectrum; 11. The method according to claim 10, further comprising: performing an amplitude value adjustment process on a target spectrum corresponding to each of the high frequency subbands according to subband spectral envelope information corresponding to each of the high frequency subbands of the high frequency spectrum and a spectral flatness numerical difference corresponding to each of the high frequency subbands; and splicing the adjustment results corresponding to the plurality of high frequency subbands to the high frequency spectrum.
12. According to the subband spectral envelope information corresponding to each high frequency subband of the high frequency spectrum and the spectral flatness value difference corresponding to each high frequency subband, performing an amplitude value adjustment process on the target spectrum corresponding to each high frequency subband, For each of the target spectra corresponding to the high frequency subbands, determining a suitable white noise for the spectral flatness value difference of the high frequency subband, and adding the suitable white noise to the target spectrum to obtain a composite target spectrum; determining spectral envelope information of the composite target spectrum, and determining a spectral envelope value difference between the spectral envelope information of the composite target spectrum and the spectral envelope information of the high frequency subband; and performing an amplitude value adjustment on amplitude values of the composite target spectrum based on the spectral envelope value difference.
13. a band splitting module arranged to filter the audio frequency signal to obtain a low frequency signal and a high frequency signal; a coding module arranged to code-process the low-frequency signal to obtain a codestream of the low-frequency signal; a frequency domain transform module arranged to frequency domain transform the low frequency signal to obtain a low frequency spectrum and to frequency domain transform the high frequency signal to obtain a high frequency spectrum; an extraction module configured to perform a spectral envelope extraction process on the low-frequency spectrum and the high-frequency spectrum to obtain spectral envelope information of the audio frequency signal, and to perform a spectral flatness extraction process on the high-frequency spectrum to obtain spectral flatness information of the high-frequency spectrum; a quantization module configured to quantize and code the spectral flatness information of the high-frequency spectrum and the spectral envelope information of the audio frequency signal to obtain a band extension code stream of the audio frequency signal, and to form a coding code stream of the audio frequency signal by the band extension code stream and the code stream of the low-frequency signal.
14. a decomposition module arranged to decompose the coding codestream to obtain a bandwidth extension codestream and a codestream of a low frequency signal; a decoding module configured to decode the code stream of the low-frequency signal to obtain a low-frequency signal, and to frequency-domain transform the low-frequency signal to obtain a low-frequency spectrum of the low-frequency signal; a dequantization module configured to dequantize the bandwidth extension codestream to obtain spectral flatness information and spectral envelope information; a reconstruction module configured to perform a reconstruction process of a high-frequency spectrum based on the spectral flatness information, the spectral envelope information and the low-frequency spectrum to obtain a high-frequency spectrum; a time domain transform module arranged to time domain transform the high frequency spectrum to obtain a high frequency signal, and to synthesize the low frequency signal and the high frequency signal to obtain an audio frequency signal corresponding to the coding code stream.
15. a memory for storing computer executable instructions; and a processor for implementing the audio frequency processing method of any one of claims 1 to 9 or 10 to 12 when executing computer executable instructions stored in the memory.
16. A computer program causing a computer to implement the audio frequency processing method according to any one of claims 1 to 9 or 10 to 12.
Citation Information
Patent Citations
Method for filling opening son (sub) tape using encoding index as well as method for generating encoding index
CN101281748A
Vector quantization device
JP1998154938A
Method for searching element distribution, vector quantizing method, pattern recognizing method, sound recognizing method, sound recognizing device and recording medium recording program for deciding recognition result
JP2000261321A
Method and equipment for regenerating signal, method and equipment for recording signal, and method and equipment for generating sign row
JP2003337599A
Method and device for encoding digital data
JP2005195983A
