Audio decoding method, audio encoding method, and electronic device

Abstract

The application provides an audio decoding method, an audio encoding method and an electronic device, and relates to the technical field of audio. The application determines a cutoff frequency for distinguishing unvoiced and voiced signals in a speech signal frame at an encoding end, and then reconstructs the speech signal according to the cutoff frequency of the speech signal frame and a harmonic model at a decoding end. In this way, each frame of the speech signal reconstructed by the decoding end contains both the voiced part and the unvoiced part, the smoothness and continuity of the speech signal are improved, and the listening experience of the speech is improved. Especially in the scene of an extremely low code rate, the continuity of the speech can be obviously improved.

Description

Technical Field

[0001] This application relates to the field of multimedia technology, and in particular to an audio decoding method, an audio encoding method, and an electronic device. Background Technology

[0002] To better utilize limited bandwidth for voice signal transmission, the encoding end typically encodes the voice signal first, and then transmits the encoded bitstream to the decoding end. The decoding end decodes the received bitstream to obtain the decoded audio signal.

[0003] Currently, when encoding speech signals, the encoder can segment the speech signal into frames, then encode these frames and add a voiced / unvoiced indicator parameter to distinguish between unvoiced and voiced signals. When decoding these frames, the decoder uses different methods based on the voiced / unvoiced indicator parameter. If the parameter indicates the frame is unvoiced, the decoder uses noise-type excitation. If the parameter indicates it is voiced, the decoder uses pulse-type excitation to reconstruct the corresponding speech signal. However, this reconstructed signal results in each frame being either unvoiced or voiced, leading to poor continuity. Summary of the Invention

[0004] Based on the above research, this application provides an audio decoding method, an audio encoding method, and an electronic device, which can enable the decoded speech signal to simultaneously contain voiced and unvoiced components, thereby improving the smoothness and continuity of the speech signal and enhancing the listening experience of the speech.

[0005] To achieve the above objectives, this application adopts the following technical solution:

[0006] In a first aspect, an audio decoding method is provided, the method comprising: receiving an audio bitstream; decoding the audio bitstream to obtain the cutoff frequency of the current frame, the cutoff frequency being used to distinguish the frequency bands of unvoiced and voiced sounds in the current frame; and reconstructing the speech signal using a harmonic model based on the cutoff frequency of the current frame.

[0007] The solution provided in the first aspect above distinguishes the frequency bands of unvoiced and voiced sounds in the current frame by using the cutoff frequency. This allows the harmonic model to reconstruct both unvoiced and voiced signals in the current frame, so that the decoded and reconstructed speech signal of the current frame can simultaneously contain both voiced and unvoiced components. This improves the smoothness and continuity of the speech signal and enhances the listening experience of the speech.

[0008] As an example, a specific implementation of decoding an audio stream can be to dequantize the audio stream to obtain the cutoff frequency, fundamental frequency, speech feature parameters, and gain of the current frame.

[0009] As one possible implementation, the speech signal is reconstructed using a harmonic model based on the cutoff frequency of the current frame. This includes: determining a first harmonic order and a second harmonic order based on the cutoff frequency and the fundamental frequency of the current frame; and reconstructing the speech signal using a harmonic model based on the first and second harmonic orders. The first harmonic order is the harmonic order corresponding to voiced sounds in the current frame, and the second harmonic order is the harmonic order corresponding to unvoiced sounds in the current frame. Thus, by determining the harmonic orders corresponding to voiced sounds and unvoiced sounds in the current frame based on the cutoff frequency and the fundamental frequency, the harmonic model can reconstruct both the voiced and unvoiced components of the speech signal in the current frame. This results in the reconstructed speech signal of the current frame including both voiced and unvoiced components, improving the smoothness and continuity of the speech signal and enhancing its listening experience.

[0010] As an example, the first device can obtain the first harmonic order and the second harmonic order using the following formula, based on the cutoff frequency and the fundamental frequency of the current frame:

[0011] M = ω cut / ω0

[0012] M max =2π*4000 / ω0

[0013] M 清 ∈[M+1,M max ]

[0014] M 浊音 ∈[1,M]

[0015] Where M is the cutoff frequency ω cut A multiple of the fundamental frequency ω0. M max This refers to the maximum permissible harmonic order within 4000Hz, given a fundamental frequency ω0. M 浊音 M represents the first harmonic order. 清 It is the second harmonic order.

[0016] As one possible implementation, decoding the audio bitstream to obtain the cutoff frequency of the current frame includes: decoding the audio bitstream to obtain the cutoff frequency of a target frame, where the target frame refers to the frame that has an encoded cutoff frequency and is closest to the current frame; and interpolating the cutoff frequency of the current frame based on the cutoff frequency of the target frame. Thus, by obtaining the cutoff frequency of the speech signal frame through interpolation, the number of encoded bits can be reduced, thereby reducing the amount of data transmitted and achieving an extremely low bitrate transmission effect.

[0017] As one possible implementation, when reconstructing the speech signal using a harmonic model, a random phase is added to the unvoiced segment of the current frame. Thus, adding a random phase to the unvoiced portion can blur the spectrum of the unvoiced portion, thereby improving the naturalness of the unvoiced sound.

[0018] As one possible implementation, the method further includes: acquiring speech feature parameters of the current frame; determining the amplitude coefficients of each harmonic in the current frame based on the speech feature parameters; and reconstructing the speech signal using a harmonic model based on the cutoff frequency of the current frame, including: reconstructing the speech signal using a harmonic model based on the amplitude coefficients of each harmonic in the current frame and the cutoff frequency of the current frame. Thus, reconstructing the speech signal using a harmonic model based on the amplitude coefficients and cutoff frequency allows the reconstructed speech signal of the current frame to include both voiced and unvoiced components, improving the smoothness and continuity of the speech signal and enhancing the listening experience.

[0019] As an example, the specific implementation of the first device reconstructing the speech signal using a harmonic model can be achieved through the following formula:

[0020]

[0021] Where, θ m (n) represents the random phase of the nth frame. m is the actual harmonic order. When m∈[1,M], m is the harmonic order of the voiced component in the current frame. When m∈[M+1,M], m is the harmonic order of the voiced component in the current frame. max In the case of , m is the harmonic order of the unvoiced portion in the current frame. ω0(n) is the fundamental frequency of the nth frame. x(n) is the speech signal reconstructed through the harmonic model. R indicates taking the real part. This indicates that the voiced portion of the signal in the nth frame is reconstructed. This indicates that the signal of the unvoiced portion in the nth frame is reconstructed.

[0022] As one possible implementation, the speech signal is obtained by processing the signal reconstructed from the harmonic model using the gain of the current frame. This avoids signal distortion and optimizes the auditory experience.

[0023] Secondly, an audio encoding method is provided, the method comprising: acquiring the current frame of a speech signal; and sending an audio bitstream, the audio bitstream including the cutoff frequency of the current frame, the cutoff frequency being used to distinguish between unvoiced and voiced frequencies in the current frame.

[0024] The audio encoding method provided in the second aspect above, by encoding the cutoff frequency of the current frame, enables each frame of speech signal reconstructed by the decoding end to simultaneously contain both voiced and unvoiced components, thereby improving the smoothness and continuity of the speech signal and enhancing the listening experience of the speech.

[0025] As one possible implementation, before sending the audio bitstream, the method further includes: determining the cutoff frequency of the current frame; and quantizing based on the cutoff frequency to obtain the audio bitstream. Thus, by determining the cutoff frequency of the frequency band used to distinguish between voiced and unvoiced sounds in the current frame, each frame of speech signal reconstructed by the decoder can simultaneously contain both voiced and unvoiced components.

[0026] As one possible implementation, determining the cutoff frequency of the current frame includes: determining the signal energy distribution of the current frame based on its spectrum; determining the autocorrelation function of the current frame based on its signal energy distribution; and determining the cutoff frequency of the current frame based on its autocorrelation function. Thus, by determining the cutoff frequency of the frequency band used to distinguish between voiced and unvoiced sounds in the current frame, the speech signal of the current frame reconstructed by the decoder can simultaneously include both voiced and unvoiced components.

[0027] As an example, the cutoff frequency of the current frame can be obtained using the following formula:

[0028]

[0029] r z (τ)=IDET[Z 2 (ω)]

[0030] Z(ω)=10 lg|X(ω)|-igε(ω)

[0031] Where, ω cut Let X(ω) be the cutoff frequency, and X(ω) be the spectrum of the current frame. ε(ω) is the spectral envelope of the spectrum X(ω). Z(ω) represents the spectrum after removing the spectral envelope. 2 (ω) represents the signal energy distribution of the current frame. z (τ) represents the autocorrelation function of the current frame. ω0 is the fundamental frequency of the current frame, and i is the adjustment factor, i∈[-a,a], where a can take values ​​according to requirements, for example, i∈[-5,5]. The current frame has a delay of The maximum correlation coefficient at r. Z (0) is the correlation coefficient of the current frame when the delay is 0 (τ=0).

[0032] As one possible implementation, the method further includes: determining the gain and speech feature parameters of the current frame; and performing quantization based on the cutoff frequency, including: quantizing the cutoff frequency, fundamental frequency, gain, and speech feature parameters of the current frame. Thus, by quantizing the cutoff frequency, fundamental frequency, gain, and speech feature parameters of the current frame, the number of coded bits can be reduced, thereby reducing the amount of data transmitted.

[0033] As one possible implementation, quantizing the cutoff frequency, fundamental frequency, gain, and speech feature parameters of the current frame includes: performing a first quantization on the fundamental frequency, gain, and speech feature parameters of the current frame, and performing a second quantization on the cutoff frequency of the current frame, wherein the first quantization and the second quantization are different. This reduces the impact of quantization on the accuracy of the cutoff frequency, while also reducing the number of bits used in audio encoding, reducing the amount of data transmitted, and achieving extremely low bitrate transmission.

[0034] As one possible implementation, determining the cutoff frequency of the current frame includes: determining the cutoff frequency of the current frame when the interval between the current frame and the frame whose cutoff frequency was previously determined is M frames, where M is an integer greater than or equal to 1. This reduces the number of encoded bits, decreases the amount of data transmitted, and achieves extremely low bitrate transmission.

[0035] Thirdly, an audio encoding / decoding system is provided, including a first device and a second device, wherein the first device is used to perform the method of any possible implementation of the first aspect, and the second device is used to perform the method of any possible implementation of the second aspect.

[0036] In one possible implementation, if the first device is a terminal, the first device is also used to receive an audio stream sent by a third device. The third device can be a server.

[0037] In one possible implementation, if the first device is a server, it is also used to send the voice signal obtained from the decoded audio stream to a third device. The third device can be a terminal.

[0038] The beneficial effects of the solution provided in the third aspect can be found in the description of any of the embodiments in the first and second aspects, and will not be repeated here.

[0039] Fourthly, an electronic device is provided, comprising: a memory for storing a computer program; a transceiver for receiving or transmitting radio signals; and a processor for executing the computer program, such that the electronic device implements a method as described in any possible implementation of the first aspect or any possible implementation of the second aspect.

[0040] Fifthly, a computer-readable storage medium is provided, on which computer program code is stored, wherein when the computer program code is executed by a processing circuit, it implements a method as described in any possible implementation of the first aspect or any possible implementation of the second aspect.

[0041] Sixthly, a computer program product is provided for running on a computer to implement the method as described in any possible implementation of the first aspect or the method as described in any possible implementation of the second aspect. Attached Figure Description

[0042] Figure 1 This is a schematic diagram of audio decoding in conventional techniques;

[0043] Figure 2 This is one of the schematic diagrams of the system architecture to which the audio encoding and decoding method provided in the embodiments of this application is applicable;

[0044] Figure 3 A second schematic diagram of the system architecture applicable to the audio encoding and decoding method provided in the embodiments of this application;

[0045] Figure 4 The third schematic diagram of the system architecture applicable to the audio encoding and decoding method provided in the embodiments of this application;

[0046] Figure 5 The fourth schematic diagram of the system architecture applicable to the audio encoding and decoding method provided in the embodiments of this application;

[0047] Figure 6 A schematic diagram of the structure of the first device provided in an embodiment of this application;

[0048] Figure 7 This is a schematic diagram of the functional modules of the audio encoding device and audio decoding device provided in the embodiments of this application;

[0049] Figure 8 A schematic diagram illustrating the principle of audio encoding and decoding provided in the embodiments of this application;

[0050] Figure 9 This is a timing diagram of an audio encoding / decoding method provided in an embodiment of this application. Detailed Implementation

[0051] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0052] It should be noted that the collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution of this application all comply with the provisions of relevant laws and regulations and do not violate public order and good morals. For example, in the technical solution of this application, the processing of user personal information is carried out with the user's authorization, which will be stated uniformly here and will not be repeated below.

[0053] In the description of the embodiments of this application, unless otherwise stated, " / " means "or", for example, A / B can mean A or B; "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. In addition, in the description of the embodiments of this application, "multiple" means two or more.

[0054] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this embodiment, unless otherwise stated, "a plurality of" means two or more.

[0055] Furthermore, in the embodiments of this application, words such as "exemplarily" and "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design scheme described as "exemplary" in this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of the term "exemplary" is intended to present concepts in a concrete manner.

[0056] Currently, satellite communication is of great significance for personnel rescue in remote, network-free environments (such as off-road activities and disaster sites). Some terminal devices can already support sending voice signals via the BeiDou satellite for emergency assistance.

[0057] When transmitting voice signals via satellite communication, the encoding end typically encodes the voice signal first, and then transmits the encoded bitstream to the decoding end via satellite communication. The decoding end decodes the received bitstream to obtain the decoded voice signal.

[0058] Currently, when encoding speech signals, the encoder can segment the speech signal into frames, obtain speech signal frames, and then encode these frames, adding a voiced / unvoiced marker parameter to distinguish whether a frame contains voiced or unvoiced sounds. When decoding speech signal frames, the decoder uses different methods based on the voiced / unvoiced marker parameter. For example... Figure 1As shown, when the voiced / unvoiced tone indicator parameter indicates that the speech signal frame is unvoiced, the decoder uses noise-type excitation to decode the speech signal frame; when the voiced / unvoiced tone indicator parameter indicates that the speech signal frame is voiced, it uses pulse-type excitation to decode the speech signal frame, thereby reconstructing the speech signal corresponding to the speech signal frame. However, each reconstructed speech signal frame obtained in this way is either unvoiced or voiced, resulting in poor continuity of the reconstructed speech signal. Especially in scenarios with extremely low bit rates such as satellite communication, this will significantly affect the continuity of the reconstructed speech signal.

[0059] In scenarios with extremely low bitrates, neural networks are currently used for speech encoding and decoding. However, neural networks are large in scale and are not suitable for deployment in devices with weak processor capabilities, such as wearable devices.

[0060] Based on the above research, this application provides an audio encoding and decoding method. The encoding end determines a cutoff frequency to distinguish between voiced and unvoiced sounds in a speech signal frame, and then the decoding end reconstructs the speech signal based on the cutoff frequency and harmonic model of the speech signal frame. This allows the reconstructed speech signal to simultaneously include both voiced and unvoiced components, improving the smoothness and continuity of the speech signal and enhancing its listening experience. Especially in scenarios with extremely low bitrates, it can significantly improve the continuity of speech.

[0061] Furthermore, the method of reconstructing the speech signal by using the cutoff frequency of the speech signal frame and the harmonic model in the embodiments of this application is also applicable to devices with weaker processor capabilities.

[0062] As an example, the audio encoding and decoding method provided in this application embodiment can be applied to voice communication scenarios with low channel rates, such as satellite voice SMS and satellite voice calls.

[0063] Please see Figure 2 , Figure 2 This is one of the schematic diagrams of a system architecture applicable to the audio encoding and decoding method provided in the embodiments of this application. The system architecture may include a first device 210 and a second device 220. A communication connection can be established between the first device 210 and the second device 220 for information exchange, such as voice information exchange.

[0064] As an example, the first device 210 can be a decoder for decoding the speech signal. The second device 220 can be an encoder for encoding the speech signal. In some examples, the first device 210 can also be an encoder, and the second device 220 can also be a decoder.

[0065] The following explanation uses the first device 210 as the decoding end and the second device 220 as the encoding end as an example.

[0066] In this embodiment, the first device 210 and the second device 220 can be different terminals. These terminals can be mobile phones, tablets, watches, computers with wireless transceiver capabilities, virtual reality (VR) terminal devices, augmented reality (AR) terminal devices, wireless terminals in industrial control, wireless terminals in self-driving, wireless terminals in remote medical care, wireless terminals in smart grids, wireless terminals in transportation safety, wireless terminals in smart cities, wireless terminals in smart homes, vehicle-mounted terminals, RSUs with terminal functions, etc. These terminals can also be vehicle-mounted modules, vehicle-mounted components, vehicle-mounted chips, or vehicle-mounted units built into a vehicle as one or more components or units. The vehicle can implement the audio encoding / decoding method provided in this application through the built-in vehicle-mounted modules, vehicle-mounted components, vehicle-mounted chips, or vehicle-mounted units.

[0067] The following uses a satellite voice SMS scenario as an example to specifically describe the system architecture to which the audio encoding and decoding method provided in this application is applicable.

[0068] Please refer to the following: Figure 3 The following explanation uses a second device 220 (watch A) and a first device 210 (mobile phone) as an example. Figure 3 As shown, in the scenario of satellite voice short messages, the system architecture can also include a satellite, a satellite short message receiving station, and a server.

[0069] Among these, the satellite can be a communication satellite, such as the BeiDou satellite system. Satellites can also serve as relay stations in radio communications, relaying or reflecting radio signals.

[0070] Satellite short message receiving stations are used to receive radio signals transmitted by satellites, such as audio streams encoded from voice signals, and to forward the received audio streams. Long-distance communication can be achieved through satellites and satellite short message receiving stations. As an example, in the case of a BeiDou satellite, the satellite short message receiving station is also a BeiDou short message receiving station.

[0071] exist Figure 3In the architecture shown, watch A may include a voice module, a connectivity service module, and a satellite service module. The voice module is used for encoding and decoding voice signals. As an example, the voice module can be a low-bitrate voice module, capable of encoding and decoding voice signals in low-bitrate scenarios.

[0072] The Connectivity Service module leverages cloud technology to provide cloud communication services, supporting cross-platform and cross-device audio and video calls and data transmission. For example, the Connectivity Service module could be a Connectivity Voice Messaging module, which can automatically transfer voice signals received by the watch to other devices.

[0073] The satellite service module is used to provide communication services with satellites. For example, the satellite service module can be a BeiDou regional short message communication (RSMC) module, which is used to communicate with BeiDou satellites.

[0074] As an example, the watch's Connectivity service module can also be combined with a satellite service module to provide users with more reliable communication. Users can enable satellite communication through the Connectivity service module. When satellite communication is enabled, the satellite service module can send messages (such as audio streams encoded from voice signals) to the satellite.

[0075] exist Figure 3 In the architecture shown, the server is used for data forwarding. For example, the server can receive messages (such as audio streams) sent by satellite short message receiving stations and forward them.

[0076] As an example, the server can also provide Changlian Cloud services. Changlian Cloud is an intelligent communication platform built on cloud computing technology, supporting high-quality audio and video calls with hundreds of millions of concurrent connections. Through intelligent traffic scheduling, it can provide stable low-latency communication services.

[0077] exist Figure 3 In the architecture shown, the mobile phone may include a voice module, an SMS service module, and a connectivity service module. The SMS service module provides SMS receiving and sending services; for example, it can receive messages (such as audio streams) sent by the SMS gateway, or send messages (such as audio streams) to the SMS gateway.

[0078] The functions of the voice module and the seamless service module can be referred to in the above description of the voice module and the seamless service module in Watch A, and will not be repeated here.

[0079] In some examples, the mobile phone may also include a satellite service module, which enables communication between the mobile phone and satellites.

[0080] based on Figure 3 The architecture shown illustrates that, with satellite communication enabled on Watch A, the user can input voice signals via Watch A, such as through the Connect service module within Watch A. Upon receiving the voice signal, Watch A encodes the voice signal using its voice module to obtain an audio stream, which is then forwarded to the satellite via the satellite service module. The satellite, upon receiving the audio stream, forwards it to a satellite short message receiving station. The satellite short message receiving station then sends the audio stream to the server. The server, upon receiving the audio stream, can forward it to the mobile phone. For mobile phones with Connect service enabled, the server can directly send the audio stream to the phone via Connect cloud service, such as to the Connect service module within the phone. For mobile phones without Connect service enabled, the server can send the audio stream to the phone via an SMS gateway, such as to the SMS service module within the phone. Upon receiving the audio stream, the mobile phone can decode it using its voice module to enable voice playback.

[0081] In some examples, Figure 3 The system architecture shown can also include other terminals, and these other terminals have a seamless connectivity service module. After the mobile phone decodes the audio stream, it can directly transfer the decoded voice signal to other terminals through the seamless connectivity service module. Taking Watch B as an example, after the mobile phone decodes the audio stream, it can use the seamless connectivity service provided by the seamless connectivity service module to transfer the decoded voice signal to Watch B via Bluetooth.

[0082] The above embodiments are illustrated using the decoding end, i.e., the first device 210, as an example of a terminal. In other embodiments of this application, the first device 210 can also be a server as described in the above embodiments, i.e., the decoding end can be a server. In this case, after decoding the audio stream, the server can forward the decoded voice signal to the terminal. Thus, through server decoding, voice playback can be achieved even for terminals without satellite communication capabilities.

[0083] The server can be a distributed server cluster consisting of multiple servers or a single server; there are no specific restrictions.

[0084] The following uses satellite voice SMS as an example, combined with Figure 3 The architecture shown is explained below. Figure 4 As shown, the server may also include a voice module, which can decode the audio stream.

[0085] based on Figure 4The architecture shown allows users to input voice signals via the Connect service module in Watch A. After receiving the voice signal, Watch A encodes it using its voice module to obtain an audio stream, which is then forwarded to the satellite via the satellite service module. Upon receiving the audio stream, the satellite forwards it to a satellite short message receiving station. The satellite short message receiving station then sends the audio stream to the server. Upon receiving the audio stream, the server decodes it using its voice module to obtain the decoded voice signal. The server can then send the voice signal to a mobile phone. For mobile phones with Connect service enabled, the server can directly send the voice signal to the phone via Connect cloud service, such as to the Connect service module within the phone. For mobile phones without Connect service enabled, the server can send the voice signal to the phone via an SMS gateway, such as to the SMS service module within the phone. Once the mobile phone receives the voice signal, it can play back the voice message.

[0086] In some embodiments of this application, in the scenario of satellite voice communication, the first device 210 can also be a gateway device, such as a satellite ground gateway. Please refer to... Figure 5 The following explanation uses mobile phone A as the second device and satellite ground gateway as the first device. Figure 5 As shown, in the scenario of satellite voice calls, the system architecture can also include a satellite, a terrestrial communication network, and a mobile phone B.

[0087] The satellite can be a communication satellite, such as the BeiDou satellite system. For a detailed description of the satellite, please refer to [link / reference needed]. Figure 3 The examples shown are not elaborated upon here.

[0088] Terrestrial communication networks refer to networks that forward messages sent by satellites to terminals.

[0089] As an example, a terrestrial communication network may include a satellite core network, a cellular core network, and a cellular access network. The satellite core network is used for forwarding and routing data packets between the satellite ground gateway and the cellular core network. The cellular core network is used for forwarding and routing data packets based on the user's location and communication needs. The cellular access network is used to connect terminals to the cellular core network, providing an interface between the terminal and the network.

[0090] Mobile phone B is used to receive voice calls initiated by mobile phone A.

[0091] exist Figure 5 In the architecture shown, mobile phone A may include a satellite service module, a voice module, and a satellite calling module. The descriptions of the satellite service module and the voice module can be found in [reference needed]. Figure 3 The examples shown are not elaborated upon here.

[0092] The satellite communication module is used to enable the satellite communication function. When the satellite communication function is enabled, the satellite service module can send communication messages (such as audio streams obtained based on voice signal encoding) to the satellite.

[0093] exist Figure 5 In the architecture shown, the terrestrial satellite gateway may include a voice module. The terrestrial satellite gateway can encode and decode voice signals through the voice module.

[0094] based on Figure 5 The architecture shown allows a user to initiate a satellite call via mobile phone A, such as a voice call initiated through the satellite calling module in mobile phone A. After initiating a voice call, mobile phone A encodes the user's input voice signal through its voice module to obtain an audio stream, which is then forwarded to the satellite via the satellite service module. Upon receiving the audio stream, the satellite forwards it to the satellite ground gateway. The satellite ground gateway decodes the audio stream through its voice module to obtain the voice signal. After obtaining the voice signal, the satellite ground gateway sends it to the satellite core network. The satellite core network routes the voice signal to the cellular core network. The cellular core network then routes the voice signal to the cellular access network, which in turn routes it to mobile phone B. Thus, mobile phone B can receive the voice signal from mobile phone A, enabling voice calls between mobile phones A and B.

[0095] The audio encoding and decoding method provided in this application embodiment will be specifically described below, taking the first device and the second device as different terminals. For ease of understanding, the structure of the first device will be described first.

[0096] Please see Figure 6 , Figure 6 This is a schematic diagram of the structure of a first device provided in an embodiment of this application. Figure 6 As shown, the first device 600 may include a processor 610, a memory 620, a mobile communication module 630, a wireless communication module 640, antenna 1 and antenna 2, a voice module 650, a speaker 660, a receiver 670, a microphone 680, etc.

[0097] Processor 610 may include one or more processing units, such as: application processor (AP), modem processor, GPU, image signal processor (ISP), controller, memory, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0098] The controller can serve as the central nervous system and command center of the first device 600. The controller can generate operation control signals based on the instruction opcode and timing signals to control the fetching and execution of instructions.

[0099] The processor 610 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 610 is a cache memory. This memory can store instructions or data that the processor 610 has just used or that are used repeatedly. If the processor 610 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 610, and thus improves the efficiency of the system.

[0100] The memory 620 can be used to store computer executable program code, which includes instructions. The processor 610 executes various functional applications and data processing of the first device 600 by running the instructions stored in the memory 620. The memory 620 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback, image playback, etc.), etc. The data storage area may store data created during the use of the first device 600 (such as audio data, phonebook, etc.). Furthermore, the memory 620 may include Double Data Rate Synchronous Dynamic Random Access Memory (DDR SDRAM), high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0101] As an example, the audio encoding and decoding method provided in this application embodiment can be pre-loaded into the memory 620 in the form of a computer program product, and the processor 610 can execute the instructions in the memory to implement the steps of the audio encoding and decoding method provided in this application embodiment.

[0102] The wireless communication function of the first device 600 can be implemented through antenna 1, antenna 2, mobile communication module 630, wireless communication module 640, modem processor, and baseband processor.

[0103] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the first device 600 can be used to cover one or more communication frequency bands. Different antennas can also be multiplexed to improve antenna utilization. For example, antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with a tuning switch.

[0104] The mobile communication module 630 can provide solutions for wireless communication, including 2G / 3G / 4G / 5G, applied to the first device 600. The mobile communication module 630 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 630 can receive electromagnetic waves via antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation. The mobile communication module 630 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via antenna 1. In some embodiments, at least some functional modules of the mobile communication module 630 may be housed in the processor 610. In some embodiments, at least some functional modules of the mobile communication module 630 and at least some modules of the processor 610 may be housed in the same device.

[0105] The modem processor may include a modulator and a demodulator. The modulator modulates the low-frequency baseband signal to be transmitted into a mid-to-high frequency signal. The demodulator demodulates the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After processing by the baseband processor, the low-frequency baseband signal is transmitted to the application processor. The application processor outputs an audio signal through an audio device (not limited to speaker 660, receiver 670, etc.). In some embodiments, the modem processor may be a separate device. In other embodiments, the modem processor may be independent of the processor 610 and may be housed in the same device as the mobile communication module 630 or other functional modules.

[0106] The wireless communication module 640 can provide solutions for wireless communication applications on the first device 600, including wireless local area networks (WLANs) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module 640 can be one or more devices integrating at least one communication processing module. The wireless communication module 640 receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to processor 610. The wireless communication module 640 can also receive signals to be transmitted from processor 610, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2.

[0107] In some embodiments, antenna 1 of the first device 600 is coupled to mobile communication module 630, and antenna 2 is coupled to wireless communication module 640, so that the first device 600 can communicate with networks and other devices through wireless communication technology.

[0108] The wireless communication technologies may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technologies. The GNSS may include Global Positioning System (GPS), Global Navigation Satellite System (GLONASS), BeiDou Navigation Satellite System (BDS), Quasi-Zenith Satellite System (QZSS), and / or Satellite Based Augmentation Systems (SBAS).

[0109] The first device 600 can implement audio functions through a voice module 650, a speaker 660, a receiver 670, a microphone 680, and an application processor. Examples include voice calls and voice text messages.

[0110] The voice module 650 can be used to encode and decode voice signals. In some embodiments, the voice module 650 may be located in the processor 110, or some functional modules of the voice module 650 may be located in the processor 610.

[0111] The speaker 660, also known as a "loudspeaker," is used to convert electrical signals into voice signals. The first device 600 can listen to music or make hands-free calls through the speaker 660.

[0112] The receiver 670, also known as the "earpiece," is used to convert electrical signals into voice signals. When the first device 600 is receiving a telephone call or voice message, the receiver 670 can be brought close to the listener's ear to receive the voice message.

[0113] Microphone 680, also known as a "microphone" or "voice transducer," is used to convert speech signals into electrical signals. When making a phone call or sending a voice message, the user can speak by bringing their mouth close to microphone 680, inputting the speech signal into microphone 680. The first device 600 may have at least one microphone 680. In some embodiments, the first device 600 may have two microphones 680, which, in addition to acquiring speech signals, can also perform noise reduction. In other embodiments, the first device 600 may also have three, four, or more microphones 680, which can acquire speech signals, reduce noise, identify the sound source, and perform directional recording, etc.

[0114] As an example, in a decoding scenario, the first device 600 can decode the audio stream through the voice module 650 after receiving the audio stream, and play the voice signal through the receiver 670 or the speaker 660.

[0115] As an example, in an encoding scenario, the first device 600 can receive the user's voice signal through the microphone 680 and encode the user's voice signal through the voice module 650 to obtain an audio bitstream.

[0116] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the first device 600. In other embodiments of this application, the first device 600 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0117] The first device provided in this application embodiment can run an operating system (OS). This operating system can be various operating systems used in the industry, such as an operating system based on OpenHarmony, like HarmonyOS; or other operating systems such as Android. TM iOS TM Mobile operating system; it can also be various open-source operating systems or their derivatives, such as Linux OS. TM This includes other embedded operating systems, as well as future new operating systems, such as AI operating systems based on artificial intelligence. An operating system is a set of interconnected system software programs that manage and control terminal operation, utilize and run hardware and software resources, and provide public services to organize user interaction. In a terminal, the operating system connects downwards to the physical devices at the hardware layer and upwards to provide a runtime environment for application software.

[0118] An operating system typically includes: a kernel layer, a system service layer, a framework layer, and an application layer.

[0119] The application layer includes applications, which can include system applications and third-party applications.

[0120] The framework layer provides application programming interfaces (APIs) and programming frameworks for applications in the application layer.

[0121] The system service layer includes the core capabilities of the system and provides services to applications through the framework layer.

[0122] The kernel layer is the layer between hardware and software. It can include hardware drivers and the operating system kernel. In addition to providing hardware drivers, the kernel layer also supports functions such as memory management and system process management.

[0123] Understandably, due to the diverse types and forms of terminals, and the wide range of application scenarios, the operating systems used on these terminals may differ depending on their form and function, application scenarios, and user needs. Therefore, the basic functions implemented by the first device provided in this application can be implemented using either a general-purpose operating system or a dedicated operating system; no specific limitation is imposed.

[0124] It should be noted that the structure of the second device can be referred to the structural description of the first device, and will not be repeated here.

[0125] The functional modules for implementing the audio encoding and decoding method provided in the embodiments of this application will be described below.

[0126] Please refer to the following: Figure 7 , Figure 7 This is a functional block diagram of the audio encoding device and audio decoding device provided in the embodiments of this application. The audio encoding device 710 can be applied to... Figure 2 The second device shown, the audio decoding device 720, can be applied to... Figure 2 The first device shown. (As shown) Figure 7 As shown, the audio encoding device 710 may include a framing module 711, a speech feature parameter extraction module 712, a fundamental frequency parameter estimation module 713, a gain parameter estimation module 714, a cutoff frequency estimation module 715, a quantization module 716, and a data transmission module 717. It is understood that the functional modules included in the audio encoding device 710 can be deployed in software form within the operating system of the first device, such as within the various layers of the operating system of the first device. Different functional modules can be deployed in different layers or in the same layer. The specific layer to which they are deployed can be set according to actual needs, and this embodiment does not impose specific limitations here.

[0127] The framing module 711 is used to perform framing processing on the received voice signal to obtain voice signal frames.

[0128] The speech feature parameter extraction module 712 is used to extract speech feature parameters of speech signal frames. For example, the speech feature parameter extraction module can be used to extract linear predictive coding (LPC) coefficients of speech signal frames and convert LPC coefficients into line spectral frequencies (LSF) coefficients.

[0129] The fundamental frequency parameter estimation module 713 is used to estimate the fundamental frequency angular frequency (hereinafter referred to as "fundamental frequency") of the speech signal frame.

[0130] The gain parameter estimation module 714 is used to estimate the signal gain (hereinafter referred to as "gain") of the speech signal frame.

[0131] The cutoff frequency estimation module 715 is used to estimate the cutoff frequency in the speech signal frame. The cutoff frequency refers to the voiced cutoff frequency, used to distinguish between unvoiced and voiced components in the speech signal frame. Frequency bands below the cutoff frequency in the speech signal frame can be considered the frequency bands corresponding to voiced components, and frequency bands above the cutoff frequency can be considered the frequency bands corresponding to unvoiced components.

[0132] The quantization module 716 is used to quantize the speech feature parameters, fundamental frequency, gain, cutoff frequency, and other parameters of the speech signal frame to obtain the audio bitstream. By quantizing the speech feature parameters, fundamental frequency, gain, cutoff frequency, and other parameters of the speech signal frame, the amount of data in the audio bitstream can be reduced, and the transmission efficiency can be improved.

[0133] The data transmission module 717 is used to send the audio stream to the first device.

[0134] like Figure 7 As shown, the audio decoding device 720 may include a data receiving module 721, a dequantization module 722, a harmonic order determination module 723, a harmonic amplitude determination module 724, a speech generation module 725, and a speech playback module 726. It is understood that the functional modules included in the audio decoding device 720 can be deployed in software form within the operating system of the second device, such as within the various layers of the operating system of the second device. Different functional modules can be deployed in different layers or in the same layer. The specific layer to which they are deployed can be set according to actual needs, and this embodiment does not impose specific limitations here.

[0135] The data receiving module 721 is used to receive the audio stream sent by the second device.

[0136] The dequantization module 722 is used to dequantize the audio bitstream to obtain parameters such as speech feature parameters, fundamental frequency, gain, and cutoff frequency of the speech signal frame.

[0137] The harmonic order determination module 723 is used to determine the harmonic order of the speech signal frame based on the cutoff frequency and the fundamental frequency, including the harmonic order corresponding to voiced sounds in the speech signal frame and the harmonic order corresponding to unvoiced sounds in the speech signal frame.

[0138] The harmonic amplitude determination module 724 is used to determine the amplitude coefficients of each harmonic of the speech signal frame based on the speech feature parameters.

[0139] The speech generation module 725 is used to reconstruct speech signals using a harmonic model and process the reconstructed speech signals using gain. The harmonic order, amplitude coefficient, and fundamental frequency are parameters in the harmonic model.

[0140] The voice playback module 726 is used to control playback devices (such as receivers, speakers, etc.) to play voice signals.

[0141] based on Figure 2 , Figure 7 The architecture shown is as follows: Figure 8 As shown, the second device can receive the user's voice signal, perform frame segmentation on the voice signal, and calculate parameters such as voice feature parameters, fundamental frequency, gain, and cutoff frequency of the current voice signal frame. Then, it performs quantization processing on the voice feature parameters, fundamental frequency, gain, and cutoff frequency of the current frame to obtain the audio bitstream. After obtaining the audio bitstream, the second device can send the audio bitstream to the first device.

[0142] After receiving the audio bitstream, the first device can dequantize it to obtain parameters such as speech feature parameters, fundamental frequency, gain, and cutoff frequency of the current frame. Based on the cutoff frequency and fundamental frequency, the harmonic orders of unvoiced and voiced sounds in the speech signal frame are determined. Based on the speech feature parameters, the amplitude coefficients of each harmonic in the speech signal frame are determined. After obtaining the amplitude coefficients and harmonic orders, the first device can reconstruct the speech signal using a harmonic model, perform gain processing on the reconstructed speech signal, and then output the gain-processed speech signal. In this way, each frame of the reconstructed speech signal can simultaneously contain both voiced and unvoiced components, improving the smoothness and continuity of the speech signal and enhancing the listening experience. Especially in scenarios with extremely low bitrates, it can significantly improve the continuity of speech.

[0143] The following will combine Figures 2 to 7 The architecture of this application is described in detail below, outlining the audio encoding and decoding methods provided in the embodiments. Please refer to the relevant documentation. Figure 9 , Figure 9 This is a timing diagram illustrating an audio encoding / decoding method provided in an embodiment of this application. Figure 9 As shown, the audio encoding and decoding process provided in this application embodiment may include S901 to S907.

[0144] S901, the second device receives the voice signal and obtains the current frame of the voice signal.

[0145] The second device can receive the voice signal input by the user, perform frame-segmentation processing on the voice signal, and obtain multiple voice signal frames.

[0146] In the embodiments of this application, the current frame refers to the audio signal frame currently being processed. It is understood that the embodiments of this application can encode multiple audio signal frames obtained through framing processing. The encoding of the audio signal frame will be described below using the current frame as an example. The encoding of other audio signal frames can be referred to the encoding description of the current frame, and will not be repeated below.

[0147] S902, the second device determines the base frequency, gain, voice feature parameters, and cutoff frequency of the current frame.

[0148] In this embodiment, the fundamental frequency is the most basic periodic vibration frequency in a speech signal, reflecting the vibration characteristics of the vocal cords. As an example, the second device can determine the fundamental frequency of the current frame using an autocorrelation method, a harmonic model-based method, or a combination of both; no specific limitation is imposed.

[0149] In this embodiment, the signal gain of the speech signal frame can be used to adjust the amplitude level of the speech signal so that the output speech can maintain a stable audible loudness. As an example, the second device can determine the gain of the current frame through spectral subtraction, automatic gain control (AGC), etc., without any specific limitation.

[0150] In this embodiment, the speech feature parameters can be LPC coefficients. LPC coefficients are important parameters used in speech signal processing to describe the characteristics of speech signals. As an example, the second device can determine the LPC coefficients using an LPC algorithm. The LPC algorithm can be, but is not limited to, autocorrelation LPC algorithms, adaptive LPC algorithms, code-excited LPC algorithms, multi-pulse LPC algorithms, etc., and no specific limitation is imposed.

[0151] To improve the efficiency and quality of speech signal processing, as an example, the second device can convert the LPC coefficients into LSF coefficients after obtaining them. Therefore, in this embodiment, the speech feature parameters can also be LSF coefficients. The process of converting LPC coefficients to LSF coefficients can refer to conventional techniques and will not be elaborated here.

[0152] In this embodiment, LSF coefficients have better quantization characteristics and stability compared to LPC coefficients. By converting LPC coefficients to LSF coefficients, quantization accuracy and stability can be improved during quantization encoding. Furthermore, since LSF coefficients have a strictly monotonically increasing characteristic, fewer bits are needed for quantization encoding compared to LPC coefficients, thereby reducing the encoding bit rate, decreasing the amount of data transmitted, and improving transmission efficiency.

[0153] In this embodiment, the cutoff frequency of the current frame is related to the signal energy distribution of the current frame. For example, the second device can obtain the signal energy distribution of the current frame based on its spectrum. Then, an inverse Fourier transform is performed on the signal energy distribution of the current frame to obtain its autocorrelation function. The cutoff frequency of the current frame is determined based on its autocorrelation function.

[0154] As an example, the second device can obtain the cutoff frequency of the current frame using formula (1).

[0155]

[0156] r z (τ)=IDFT[Z 23 (ω)] (1)

[0157] Z(ω)=10 lg|X(ω)|-lgε(ω)

[0158] Where, ω cut Let X(ω) be the cutoff frequency, and X(ω) be the spectrum of the speech signal in the current frame. ε(ω) is the spectral envelope of the spectrum X(ω). Z(ω) represents the spectrum after removing the spectral envelope. 2 (ω) represents the signal energy distribution of the current frame. z (τ) represents the autocorrelation function of the current frame. ω0 is the fundamental frequency of the current frame, and i is the adjustment factor, i∈[-a,a], where a can take values ​​according to requirements, for example, i∈[-5,5]. The current frame has a delay of The maximum correlation coefficient at r. Z (0) is the correlation coefficient of the current frame when the delay is 0 (τ=0). (This is achieved through...) The proportion of voiced sound in the current frame can be determined. After obtaining the proportion of voiced sound in the current frame, the cutoff frequency can be obtained by combining it with the sampling frequency (π) of the current frame.

[0159] To further conserve the bits occupied by the cutoff frequency during encoding, reduce the amount of data transmitted, and achieve an extremely low bit rate transmission effect, in this embodiment of the application, the second device may also calculate the cutoff frequency only for a portion of the voice signal frames.

[0160] For example, the second device can determine the cutoff frequency of the current frame when the interval between the current frame and the frame where the cutoff frequency was previously determined is M frames, where M is an integer greater than or equal to 1.

[0161] It should be noted that after the second device performs frame segmentation on the audio signal, it can sort each audio signal frame, such as by time order, to obtain the frame order of each audio signal frame. After obtaining the frame order of each audio signal frame, the second device can determine the audio signal frame for which the cutoff frequency needs to be calculated based on the frame order and the value of M.

[0162] Taking M=2 as an example, when M=2, the second device can calculate the cutoff frequency every two frames. For example, after calculating the cutoff frequency for the first frame (the audio signal frame with frame number 1), the second device can calculate the cutoff frequency for the fourth frame (the audio signal frame with frame number 4), the seventh frame (the audio signal frame with frame number 7), and so on, by calculating the cutoff frequency every two frames.

[0163] The embodiments of this application can reduce the number of encoded bits and the amount of data transmitted by calculating the cutoff frequency of a portion of the voice signal frames.

[0164] S903, the second device quantizes the cutoff frequency, fundamental frequency, gain, and speech feature parameters of the current frame.

[0165] As an example, the second device may perform quantization processing on the cutoff frequency, fundamental frequency, gain, and speech feature parameters of the current frame in ways including but not limited to uniform quantization, affine quantization, and vector quantization.

[0166] Taking vector quantization as an example, the second device quantizes the cutoff frequency, fundamental frequency, gain, and speech feature parameters of the current frame. Specifically, the second device constructs a feature vector from the cutoff frequency, fundamental frequency, gain, and speech feature parameters of the current frame. Then, it searches the codebook for the codeword that best matches this feature vector and uses the index value of the best-matching codeword as the encoding result. This completes the quantization of the cutoff frequency, fundamental frequency, gain, and speech feature parameters of the current frame.

[0167] To further reduce the amount of data transmitted, in this embodiment of the application, the second device may use a multi-frame joint vector quantization processing method to quantize the cutoff frequency, fundamental frequency, gain and speech feature parameters of the current frame.

[0168] Among them, the multi-frame joint vector quantization method can be, but is not limited to, multi-frame joint residual vector quantization (RVQ), multi-frame joint adaptive vector quantization, multi-frame joint multi-level vector quantization, etc.

[0169] The following explanation uses a 6-frame joint RVQ quantization method as an example. The second device can determine the cutoff frequency, fundamental frequency, gain, and speech feature parameters of each speech signal frame. After obtaining the cutoff frequency, fundamental frequency, gain, and speech feature parameters of the six speech signal frames, the second device can use RVQ quantization to quantize these six speech signal frames together. This reduces the number of bits used in audio encoding and reduces the amount of data transmitted.

[0170] Understandably, the number of frames for joint quantization can be set according to requirements, and this application embodiment does not impose any limitations. By quantizing the cutoff frequency, fundamental frequency, gain, and speech feature parameters of the current frame, this application embodiment can reduce the number of encoded bits, reduce the amount of data transmitted, and achieve an extremely low bit rate transmission effect.

[0171] Considering that quantizing parameters such as the cutoff frequency, fundamental frequency, gain, and speech feature parameters of the current frame together has a significant impact on the accuracy of the cutoff frequency, and that a decrease in cutoff frequency accuracy will affect the accuracy of distinguishing between unvoiced and voiced components in the speech signal frame, thus affecting the smoothness of the reconstructed speech signal. Therefore, in this embodiment, different quantization methods can be used for the cutoff frequency of the current frame to reduce the impact of quantization on the accuracy of the cutoff frequency.

[0172] For example, the second device performs a first quantization on the fundamental frequency, gain, and speech feature parameters of the current frame, and a second quantization on the cutoff frequency of the current frame. The first quantization and the second quantization can be different.

[0173] For example, the first quantization can be a multi-frame joint RVQ quantization method, and the second quantization can be uniform quantization (such as 3-bit quantization).

[0174] Understandably, in the embodiments of this application, the second device completes the encoding of the current frame by quantizing the cutoff frequency, fundamental frequency, gain, and speech feature parameters of the current frame.

[0175] Understandably, when the second device calculates the cutoff frequency only for a portion of the speech signal frames, for speech signal frames for which the cutoff frequency is not calculated, the second device does not need to quantize the cutoff frequency of the speech signal frame, but only quantizes the fundamental frequency, gain, and speech feature parameters of the speech signal frame. In the embodiments of this application, by performing a first quantization on the fundamental frequency, gain, and speech feature parameters of the current frame, and a second quantization on the cutoff frequency of the current frame, the impact of quantization on the accuracy of the cutoff frequency can be reduced. On the other hand, by quantizing the cutoff frequency, fundamental frequency, gain, and speech feature parameters of the current frame, the number of encoded bits can also be reduced, thereby reducing the amount of data transmitted.

[0176] S904, the second device sends an audio stream to the first device.

[0177] In this embodiment, after the second device encodes the multiple audio signal frames obtained from the framing process, it can encapsulate the encoded data (such as quantization results) of each audio signal frame. Then, the encapsulated encoded data of each audio signal frame is sorted in chronological order to form a continuous audio stream, which is then sent to the first device.

[0178] When the second device employs a multi-frame joint vector quantization method, it can further encapsulate the encoded data of the joint frames, then sort the encapsulated encoded data of each joint frame in chronological order to form a continuous audio bitstream, which is then sent to the first device. A joint frame comprises multiple jointly quantized frames, and the encoded data of the joint frame is the quantization result of the multi-frame joint quantization. Taking a 6-frame joint RVQ quantization method as an example, when the second device uses a 6-frame joint RVQ quantization method, each joint frame comprises 6 jointly quantized frames, and the encoded data of each joint frame is the quantization result of the 6-frame joint quantization.

[0179] It should be noted that when the second device uses different quantization methods, if the first quantization is a multi-frame joint quantization method and the second quantization is a single-frame quantization method, then the second device can encapsulate the coded data of multiple frames. Taking a 6-frame joint RVQ quantization method as an example, and a 3-bit quantization method as the second quantization method, the second device can encapsulate the 6-frame coded data. The 6-frame coded data includes the 6-frame joint RVQ quantization result and the 3-bit quantization result corresponding to each of the 6 frames. If there is a speech signal frame in the 6 frames for which the cutoff frequency has not been calculated, then the 3-bit quantization result corresponding to that frame is empty.

[0180] Understandably, if both the first and second quantization methods are single-frame quantization, the second device can encapsulate the encoded data of each speech signal frame. The encoded data of each speech signal frame includes the quantization result of the first quantization and the quantization result of the second quantization corresponding to that speech signal frame. If the cutoff frequency of the speech signal frame is not calculated, the encoded data of the speech signal frame will not include the quantization result of the second quantization, or the quantization result of the second quantization will be empty.

[0181] In satellite communication scenarios, a second device can send an audio stream to the satellite, which then forwards the audio stream to the first device. For details, please refer to... Figure 3 , Figure 4 , Figure 5 The examples shown are not elaborated upon here.

[0182] S905, the first device receives the audio bitstream, decodes the audio bitstream, and obtains the cutoff frequency, fundamental frequency, voice feature parameters, and gain of the current frame.

[0183] The first device, after receiving the audio bitstream, can dequantize the audio bitstream to obtain the cutoff frequency, fundamental frequency, speech feature parameters, and gain of the current frame.

[0184] For example, after receiving the audio bitstream, the first device can obtain the index value of the current frame in the audio bitstream, and then search for the codeword corresponding to the index value in the same codebook as the second device. The feature vector is reconstructed through the codeword, thereby obtaining the cutoff frequency, fundamental frequency, speech feature parameters and gain of the current frame.

[0185] It should be noted that if the second device only calculates the cutoff frequency for a portion of the voice signal frames, the current frame may not have a cutoff frequency encoded. If the current frame does not have a cutoff frequency encoded, the first device can determine the cutoff frequency of the current frame through interpolation.

[0186] For example, the second device can decode the audio stream to obtain the cutoff frequency of the target frame. The target frame refers to the frame that has an encoded cutoff frequency and is closest to the current frame. The cutoff frequency of the current frame is obtained by interpolation based on the cutoff frequency of the target frame.

[0187] The following explanation uses the example of the second device calculating the cutoff frequency every two frames. When the second device calculates the cutoff frequency every two frames, in the audio stream received by the first device, the first frame is encoded with the cutoff frequency, the second and third frames are not encoded with the cutoff frequency, the fourth frame is encoded with the cutoff frequency, the fifth and sixth frames are not encoded with the cutoff frequency, and the seventh frame is encoded with the cutoff frequency.

[0188] If the current frame is the second frame, and the frame closest to the current frame that has a cutoff frequency is the first frame, then the target frame is the first frame. The first device can then interpolate the cutoff frequency of the second frame based on the cutoff frequency of the first frame.

[0189] If the current frame is the third frame, and the frame closest to the current frame that has a cutoff frequency is the fourth frame, then the target frame is the fourth frame. The first device can then interpolate the cutoff frequency of the third frame based on the cutoff frequency of the fourth frame.

[0190] Similarly, the first device can obtain the cutoff frequency of frames in the audio bitstream that are not encoded with a cutoff frequency through interpolation.

[0191] The interpolation method can be, but is not limited to, polynomial interpolation, spline interpolation, linear interpolation, etc., and there are no specific restrictions.

[0192] Understandably, if the current frame encoding has a cutoff frequency, the first device uses the cutoff frequency encoded by the current frame, without needing to interpolate to obtain the cutoff frequency of the current frame.

[0193] This application embodiment calculates the cutoff frequency of some speech signal frames during the encoding stage and obtains the cutoff frequency of the speech signal frames through interpolation during the decoding stage, thereby reducing the number of encoded bits and reducing the amount of data transmitted, achieving an extremely low bit rate transmission effect.

[0194] S906, the first device determines the first harmonic order and the second harmonic order based on the cutoff frequency and the fundamental frequency of the current frame, and determines the amplitude coefficient of each harmonic in the current frame based on the speech feature parameters of the current frame.

[0195] Wherein, the first harmonic order is the harmonic order corresponding to the voiced tone in the current frame, that is, the first harmonic order is the harmonic order corresponding to the voiced tone in the current frame reconstructed by the harmonic model. The second harmonic order is the harmonic order corresponding to the unvoiced tone in the current frame, that is, the second harmonic order is the harmonic order corresponding to the unvoiced tone in the current frame reconstructed by the harmonic model.

[0196] As an example, the first device can obtain the first harmonic order and the second harmonic order using formula (2) based on the cutoff frequency and the fundamental frequency of the current frame.

[0197] M = ω cut / ω0

[0198] M max =2π*4000 / ω0

[0199] M 清 ∈[M+1,M max (2)

[0200] M 浊音 ∈[1,M]

[0201] Where M is the cutoff frequency ω cut A multiple of the fundamental frequency ω0. M max It is the maximum permissible harmonic order within 4000Hz, given a fundamental frequency ω0. M 浊音 M represents the harmonic order of the voiced component in the current frame, specifically the first harmonic order. 清 This is the harmonic order of the unvoiced portion in the current frame, i.e., the second harmonic order.

[0202] This application's embodiments reconstruct the harmonic orders of the unvoiced and voiced components in the current frame using the cutoff frequency and the fundamental frequency of the current frame. Therefore, when reconstructing the speech signal using a harmonic model, the model can reconstruct the speech signal based on the harmonic orders of the unvoiced components and the voiced components, ensuring that the reconstructed speech signal includes both unvoiced and voiced components. This improves the smoothness and continuity of the speech signal, enhancing the listening experience. Especially in extremely low bitrate scenarios, it can significantly improve the continuity of speech.

[0203] In this embodiment, when reconstructing the speech signal based on the harmonic model, the amplitude coefficients of each harmonic in the current frame are also required. Therefore, the first device can also determine the amplitude coefficients of each harmonic in the current frame based on the speech feature parameters of the current frame.

[0204] As an example, when the speech feature parameter is the LPC coefficient, the first device can calculate the amplitude coefficient of each harmonic based on the LPC coefficient using formula (3).

[0205]

[0206] A m (n)=α t A (m,n) +(1-α t A (m,n-1) m = 1, 2, ..., M max

[0207] Among them, A (m,n) α represents the amplitude of the m-th harmonic in the n-th frame. t α represents the interpolation coefficients. i Here are the LPC coefficients. ω represents phase information. ω0(n) is the fundamental frequency of the nth frame. mω0(n) represents the mth harmonic of the nth frame. L is the number of LPC coefficients calculated for each frame. A m (n) represents the amplitude coefficient of the mth harmonic in the nth frame.

[0208] As an example, when the speech feature parameters are LSF coefficients, the LSF coefficients can be converted into LPC coefficients first, and then the amplitude coefficients of each harmonic can be obtained by formula (3).

[0209] S908, the first device reconstructs the speech signal using a harmonic model.

[0210] In the embodiments of this application, after obtaining the first harmonic order, the second harmonic order, and the amplitude coefficients of each harmonic, the first device can reconstruct the speech signal through the harmonic model.

[0211] For example, the first device can reconstruct the speech signal using formula (4) based on the first harmonic order, the second harmonic order, and the amplitude coefficient of each harmonic.

[0212]

[0213] Where m is the actual harmonic order. When m∈[1,M], m is the harmonic order of the voiced component in the current frame. When m∈[M+1,M], m is the harmonic order of the voiced component in the current frame. max In the case of ], m is the harmonic order of the unvoiced portion in the current frame. ω0(n) is the fundamental frequency of the nth frame, and x(n) is the speech signal reconstructed through the harmonic model. R indicates taking the real part. This indicates the reconstruction of the voiced portion of the speech signal in the nth frame. This indicates that the speech signal of the unvoiced portion in the nth frame is reconstructed.

[0214] To enhance the naturalness of unvoiced speech, in this embodiment, when reconstructing the speech signal using a harmonic model, a random phase can be added to the unvoiced audio segment of the current frame. Based on this, in this embodiment, after obtaining the first harmonic order, the second harmonic order, and the amplitude coefficients of each harmonic, the first device can also reconstruct the speech signal using the harmonic model shown in formula (5).

[0215]

[0216] Where, θ m (n) represents the random phase of the nth frame.

[0217] This application embodiment improves the naturalness of the unvoiced sound by adding a random phase to the unvoiced sound portion, which can blur the spectrum of the unvoiced sound portion.

[0218] To avoid signal distortion and optimize the auditory experience, in this embodiment, the second device can further process the signal reconstructed from the harmonic model using gain. Based on this, in this embodiment, the second device can also reconstruct the speech signal according to the gain and the harmonic model.

[0219] For example, the second device can reconstruct the speech signal using formula (6) based on the gain and harmonic model.

[0220]

[0221] Where g(n) is the gain of the nth frame.

[0222] Understandably, if the first device is a terminal, it can play back the voice signal after reconstructing it. If the first device is a server, it can forward the voice signal to the terminal after reconstructing it. If the first device is a terrestrial satellite gateway, it can route the voice signal to the terminal after reconstructing it. For details, please refer to... Figure 3 , Figure 4 , Figure 5 The examples shown are not elaborated upon here.

[0223] This application's embodiments reconstruct the harmonic orders of the unvoiced and voiced components in each frame by using the cutoff frequency. Thus, when reconstructing the speech signal using a harmonic model based on the harmonic orders of the unvoiced and voiced components, each reconstructed frame of speech signal can simultaneously contain both voiced and unvoiced components, improving the smoothness and continuity of the speech signal and enhancing its listening experience. This is especially noticeable in scenarios with extremely low bit rates, significantly improving speech continuity.

[0224] Based on the same inventive concept, this application provides an electronic device, which can be the first device or the second device described above. Specifically, the electronic device may include a transceiver, a memory, a processor, and one or more computer programs. The transceiver, memory, and processor can be connected via one or more communication buses. The one or more computer programs are stored in the memory and configured to be executed by the processor, and when executed by the processor, they implement the above-described functions. Figure 8 The functions or steps in the method shown.

[0225] Furthermore, embodiments of this application may also provide an audio encoding / decoding system, including a first device and a second device, wherein the first device is used to perform... Figure 9 In the method shown, S905 to S907, the second device is used to perform... Figure 9 S901 to S904 in the method shown.

[0226] For this audio codec system, if the first device is a terminal, the first device is also used to receive the audio bitstream sent by the third device (such as a server). If the first device is a server, the first device is also used to send the voice signal obtained by decoding the audio bitstream to the third device (such as the terminal).

[0227] Similarly, if the first device is a ground satellite gateway, it can also be used to send the voice signal obtained from the decoded audio stream to a third device (such as a terminal).

[0228] This application also provides a computer-readable storage medium storing computer program instructions, which, when executed by a processing circuit, implement the above-described functionality. Figure 9 The functions or steps in the method shown.

[0229] Furthermore, embodiments of this application may also provide a chip system, which includes a processing circuit and a storage medium. The storage medium stores computer program instructions that, when executed by the processing circuit, implement the above-mentioned functions. Figure 9 The functions or steps in the method shown.

[0230] Furthermore, embodiments of this application may also provide a computer program product containing instructions, which, when run on a computer, causes the computer to perform the aforementioned... Figure 9 The functions or steps in the method shown.

[0231] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, the specific working processes of the chip system, electronic device, computer-readable storage medium, and computer program product containing instructions described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0232] It is understood that the steps of the methods or algorithms described in conjunction with the embodiments of this application can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in random access memory, flash memory, erasable programmable read-only memory, electrically erasable programmable read-only memory, registers, hard disks, portable hard disks, read-only optical disks, or any other form of storage medium. An exemplary embodiment involves a storage medium coupled to a processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and storage medium can reside in an application-specific integrated circuit (ASIC). Alternatively, the ASIC can reside within an electronic device. Of course, the processor and storage medium can also exist as discrete components within an electronic device.

[0233] In an alternative approach, when implemented in software, it can be implemented entirely or partially as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are implemented. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., digital video disk (DVD)), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0234] The above description is merely a specific implementation of the embodiments of this application, but the protection scope of the embodiments of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the embodiments of this application should be covered within the protection scope of the embodiments of this application. Therefore, the protection scope of the embodiments of this application should be determined by the protection scope of the claims.

Claims

1. An audio decoding method, characterized in that, The method includes: Receive audio stream; Decode the audio stream to obtain the cutoff frequency of the current frame. The cutoff frequency is used to distinguish the frequency bands of unvoiced and voiced sounds in the current frame. The speech signal is reconstructed using a harmonic model based on the cutoff frequency of the current frame.

2. The method according to claim 1, characterized in that, The step of reconstructing the speech signal using a harmonic model based on the cutoff frequency of the current frame includes: The first harmonic order and the second harmonic order are determined based on the cutoff frequency of the current frame and the fundamental frequency of the current frame. The speech signal is reconstructed using the harmonic model based on the first harmonic order and the second harmonic order. Wherein, the first harmonic order is the harmonic order corresponding to the voiced tone in the current frame reconstructed by the harmonic model, and the second harmonic order is the harmonic order corresponding to the unvoiced tone in the current frame reconstructed by the harmonic model.

3. The method according to claim 1 or 2, characterized in that, Decoding the audio stream to obtain the cutoff frequency of the current frame includes: Decode the audio bitstream to obtain the cutoff frequency of the target frame, where the target frame refers to the frame that is encoded with a cutoff frequency and is closest to the current frame; The cutoff frequency of the current frame is obtained by interpolation based on the cutoff frequency of the target frame.

4. The method according to any one of claims 1-3, characterized in that, When the speech signal is reconstructed using the harmonic model, a random phase is added to the unvoiced segment of the current frame.

5. The method according to any one of claims 1-4, characterized in that, The method further includes: Obtain the speech feature parameters of the current frame, and determine the amplitude coefficients of each harmonic in the current frame based on the speech feature parameters of the current frame; The step of reconstructing the speech signal using a harmonic model based on the cutoff frequency of the current frame includes: The speech signal is reconstructed using a harmonic model based on the amplitude coefficients of each harmonic in the current frame and the cutoff frequency of the current frame.

6. The method according to any one of claims 1-5, characterized in that, The speech signal is obtained by processing the signal reconstructed from the harmonic model using the gain of the current frame.

7. An audio encoding method, characterized in that, The method includes: Get the current frame of the audio signal; Send an audio stream, the audio stream including the cutoff frequency of the current frame, the cutoff frequency being used to distinguish the frequency bands of unvoiced and voiced sounds in the current frame.

8. The method according to claim 7, characterized in that, Before sending the audio stream, the method further includes: Determine the cutoff frequency of the current frame; Quantization is performed based on the cutoff frequency to obtain the audio bitstream.

9. The method according to claim 8, characterized in that, Determining the cutoff frequency of the current frame includes: Based on the spectrum of the current frame, determine the signal energy distribution of the current frame; Based on the signal energy distribution of the current frame, determine the autocorrelation function of the current frame; The cutoff frequency of the current frame is determined based on the autocorrelation function of the current frame.

10. The method according to claim 8 or 9, characterized in that, The method further includes: Determine the gain and speech feature parameters of the current frame; The quantization based on the cutoff frequency includes: The cutoff frequency, fundamental frequency, gain, and speech feature parameters of the current frame are quantized.

11. The method according to claim 10, characterized in that, The quantization of the cutoff frequency, fundamental frequency, gain, and speech feature parameters of the current frame includes: The fundamental frequency, gain, and speech feature parameters of the current frame are first quantized, and the cutoff frequency of the current frame is second quantized. The first quantization and the second quantization are different.

12. The method according to any one of claims 8-11, characterized in that, Determining the cutoff frequency of the current frame includes: Given that the current frame is M frames away from the frame where the cutoff frequency was previously determined, the cutoff frequency of the current frame is determined, where M is an integer greater than or equal to 1.

13. An audio encoding / decoding system, characterized in that, It includes a first device and a second device, the first device being used to perform the method of any one of claims 1-6, and the second device being used to perform the method of any one of claims 7-13.

14. The audio encoding and decoding system according to claim 13, characterized in that, When the first device is a terminal, the first device is also used to receive an audio stream sent by the third device.

15. The audio encoding and decoding system according to claim 13, characterized in that, In the case where the first device is a server, the first device is also used to send the voice signal obtained by decoding the audio bitstream to the third device.

16. An electronic device, characterized in that, The electronic device includes: Memory, used to store computer programs; A transceiver is used to receive or transmit radio signals; A processor for executing the computer program, causing the electronic device to perform the method as described in any one of claims 1-6 or the method as described in any one of claims 7-12.

17. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program code that, when executed by a processing circuit, implements the method as described in any one of claims 1-6 or 7-12.

18. A computer program product, characterized in that, The computer program product is used to run on a computer to implement the method as described in any one of claims 1-6 or 7-12.

Images