3D audio signal coding method and apparatus, and encoder

Abstract

This application discloses a three-dimensional audio signal coding method and apparatus, and an encoder, and relates to the multimedia field. The method includes: after obtaining a fourth quantity coefficient in a current frame of a three-dimensional audio signal and a frequency domain feature value of the fourth quantity coefficient, an encoder selects a third quantity representative coefficient from the fourth quantity coefficient based on the frequency domain feature value of the fourth quantity coefficient, and selects a second quantity representative virtual speaker in the current frame from a candidate virtual speaker set based on the third quantity representative coefficient, and then encodes the current frame based on the second quantity representative virtual speaker in the current frame to obtain a bitstream. The encoder selects the representative virtual speaker from the candidate virtual speaker set by using a small number of representative coefficients to represent all coefficients. This effectively reduces the computational complexity performed by the encoder to search for virtual speakers and the computational complexity of performing compression coding on the three-dimensional audio signal, and thus reduces the computational load of the encoder.

Description

[Technical field]

[0001] This application claims priority to Chinese Patent Application No. 202110535832.3, entitled “Three-dimensional audio signal coding method and apparatus, and encoder,” filed with the China National Intellectual Property Office on May 17, 2021, which is incorporated by reference in its entirety.

[0002] This application relates to the multimedia field, and in particular to a three-dimensional audio signal coding method and apparatus, and an encoder. [Background technology]

[0003] With the rapid development of high-performance computers and signal processing technology, listeners have increasingly higher requirements for voice and audio experience. Immersive audio can meet people's requirements in this aspect. For example, three-dimensional audio technology is widely used in wireless communication (e.g., 4G / 5G) voice, virtual reality / augmented reality, media audio, and other aspects. Three-dimensional audio technology is an audio technology for acquiring, processing, transmitting, rendering, and reproducing sound and three-dimensional sound field information in the real world to provide spatial, enveloping, and immersive sound. This provides listeners with an extraordinary "immersive" hearing experience.

[0004] Typically, an acquisition device (e.g., a microphone) acquires a large amount of data to record three-dimensional sound field information, and transmits a three-dimensional audio signal to a reproduction device (e.g., a speaker or a headset), so that the reproduction device reproduces the three-dimensional audio. Because the three-dimensional sound field information includes a large amount of data, a large amount of storage space is required to store the data, and a high bandwidth is required to transmit the three-dimensional audio signal. To solve the above-mentioned problem, the three-dimensional audio signal can be compressed, and the compressed data can be stored or transmitted. Currently, an encoder can compress a three-dimensional audio signal by using multiple pre-configured virtual speakers. However, the computational complexity of performing compression coding on the three-dimensional audio signal by the encoder is high. Therefore, how to reduce the computational complexity of performing compression coding on the three-dimensional audio signal is an urgent problem to be solved. Summary of the Invention [Means for solving the problem]

[0005] This application provides a three-dimensional audio signal coding method and a three-dimensional audio signal coding device, and an encoder, to reduce the computational complexity of performing compression coding on a three-dimensional audio signal.

[0006] According to a first aspect, the application provides a three-dimensional audio signal encoding method. The method may be performed by an encoder, and specifically includes the following steps: After obtaining a fourth quantity coefficient in a current frame of a three-dimensional audio signal and a frequency domain feature value of the fourth quantity coefficient, the encoder selects a third quantity representative coefficient from the fourth quantity coefficient based on the frequency domain feature value of the fourth quantity coefficient, selects a second quantity representative virtual speaker in the current frame from a candidate virtual speaker set based on the third quantity representative coefficient, and encodes the current frame based on the second quantity representative virtual speaker in the current frame to obtain a bitstream. The fourth quantity coefficient includes a third quantity representative coefficient. The third quantity is less than the fourth quantity. This indicates that the third quantity representative coefficient is a part of the fourth quantity coefficient.

[0007] The current frame of the three-dimensional audio signal is a higher order ambisonics (HOA) signal, and frequency domain feature values ​​of the coefficients are determined based on the coefficients of the HOA signal.

[0008] The encoder selects some coefficients as representative coefficients from all coefficients in the current frame, and selects a representative virtual speaker from the candidate virtual speaker set by using a small number of representative coefficients to represent all coefficients in the current frame, which effectively reduces the computational complexity performed by the encoder to search for virtual speakers, thus reducing the computational complexity of performing compression coding on the three-dimensional audio signal, and reducing the computational load of the encoder.

[0009] Further, the encoder encoding the current frame based on the second amount of representative virtual speakers in the current frame to obtain a bitstream includes the encoder generating virtual speaker signals based on the current frame and the second amount of representative virtual speakers in the current frame, and encoding the virtual speaker signals to obtain a bitstream.

[0010] Since the frequency domain feature values ​​of the coefficients in the current frame represent the sound field characteristics of the three-dimensional audio signal, the encoder selects a representative coefficient in the current frame having a representative sound field component based on the frequency domain feature values ​​of the coefficients in the current frame. By using the representative coefficient, the representative virtual speaker in the current frame selected from the candidate virtual speaker set can completely represent the sound field characteristics of the three-dimensional audio signal. This helps to further improve the accuracy of generating virtual speaker signals by the encoder by performing compression coding on the three-dimensional audio signal to be encoded by using the representative virtual speaker in the current frame, and also to increase the compression ratio for performing compression coding on the three-dimensional audio signal, and reduce the bandwidth occupied by the encoder for transmitting the bitstream.

[0011] In a possible implementation, the step of selecting a representative coefficient of the third quantity from the coefficients of the fourth quantity based on the frequency domain feature values ​​of the coefficients of the fourth quantity includes the encoder selecting a representative coefficient from at least one subband included in a spectral range indicated by the coefficients of the fourth quantity based on the frequency domain feature values ​​of the coefficients of the fourth quantity to obtain the representative coefficient of the third quantity.

[0012] For example, the step of selecting a representative coefficient from at least one subband included in a spectral range indicated by the coefficient of the fourth quantity based on the frequency domain feature value of the coefficient of the fourth quantity to obtain a representative coefficient of the third quantity includes the encoder selecting Z representative coefficients from each of the at least one subband based on the frequency domain feature value of the coefficient in each subband to obtain a representative coefficient of the third quantity, where Z is a positive integer. The encoder selects the representative coefficient based on the frequency domain feature value of the coefficient in the spectral range indicated by all the coefficients in the current frame. This allows a representative coefficient to be selected from each subband and improves equalization for the encoder to select a representative coefficient from the spectral range indicated by all the coefficients in the current frame.

[0013] In another example, when the at least one subband includes at least two subbands, the step of selecting a representative coefficient from at least one subband included in a spectral range indicated by the coefficient of the fourth amount based on the frequency domain feature value of the coefficient of the fourth amount to obtain a representative coefficient of the third amount includes the encoder determining a weight for each of the at least two subbands based on the frequency domain feature value of the first candidate coefficient in each subband, adjusting the frequency domain feature value of the second candidate coefficient in each subband based on the weight of each subband to obtain an adjusted frequency domain feature value of the second candidate coefficient in each subband, and the first candidate coefficient and the second candidate coefficient are some coefficients in the subband, and determining the representative coefficient of the third amount based on the adjusted frequency domain feature value of the second candidate coefficient in the at least two subbands and the frequency domain feature value of the coefficient other than the second candidate coefficient in the at least two subbands. In this way, the encoder adjusts the probability that a coefficient in a subband is selected based on the weight of the subband. This further improves the accuracy of representing the coefficients of all subbands by representative coefficients selected by the encoder with respect to the sound field distribution and audio characteristics.

[0014] The encoder may divide the spectral range by an unequal division to obtain at least two sub-bands, where the at least two sub-bands each include a different amount of coefficients. Alternatively, the encoder may divide the spectral range by an equal division to obtain at least two sub-bands, where the at least two sub-bands each include the same amount of coefficients.

[0015] In another possible implementation, the step of selecting the second amount of representative virtual speakers in the current frame from the candidate virtual speaker set based on the third amount of representative coefficients includes the encoder determining the first amount of virtual speakers and the first amount of voting values ​​based on the third amount of representative coefficients in the current frame, the candidate virtual speaker set, and the number of votes, and selecting the second amount of representative virtual speakers in the current frame from the first amount of virtual speakers based on the first amount of voting values. The second amount is less than the first amount. This indicates that the second amount of representative virtual speakers in the current frame are some virtual speakers in the candidate virtual speaker set. It can be understood that the virtual speakers correspond one-to-one with the voting values. For example, the first amount of virtual speakers includes the first virtual speaker, the first amount of voting values ​​includes the voting values ​​of the first virtual speaker, and the first virtual speaker corresponds to the voting values ​​of the first virtual speaker. The vote value of the first virtual speaker represents a priority of the first virtual speaker. The candidate virtual speaker set includes a fifth amount of virtual speakers. The fifth amount of virtual speakers includes a first amount of virtual speakers. The first amount is less than or equal to the fifth amount. The vote count is an integer equal to or greater than 1, and the vote count is less than or equal to the fifth amount. The second amount is preset, or the second amount is determined based on the current frame.

[0016] Currently, during the search for a virtual speaker, the encoder uses the result of correlation calculation between the three-dimensional audio signal to be encoded and the virtual speaker as a measurement index for selecting the virtual speaker. Also, if the encoder transmits one virtual speaker for each coefficient, the purpose of efficient data compression cannot be achieved, and a heavy computation load is imposed on the encoder. In the virtual speaker selection method provided in this embodiment of this application, the encoder votes for each virtual speaker in the candidate virtual speaker set by using a small number of representative coefficients to represent all coefficients in the current frame, and selects a representative virtual speaker in the current frame based on the voting value. Furthermore, the encoder compresses and encodes the three-dimensional audio signal to be encoded by using the representative virtual speaker in the current frame. This not only effectively increases the compression ratio for performing compression coding on the three-dimensional audio signal, but also reduces the computational complexity performed by the encoder to search for the virtual speaker, thus reducing the computational complexity of performing compression coding on the three-dimensional audio signal and reducing the computational load of the encoder.

[0017] The second amount represents an amount of representative virtual speakers in the current frame selected by the encoder. A larger second amount indicates a larger amount of representative virtual speakers in the current frame and a larger amount of sound field information of the three-dimensional audio signal. A smaller second amount indicates a smaller amount of representative virtual speakers in the current frame and a smaller amount of sound field information of the three-dimensional audio signal. Thus, the second amount may be set to control the amount of representative virtual speakers in the current frame selected by the encoder. For example, the second amount may be preset. In another example, the second amount may be determined based on the current frame. For example, the value of the second amount may be 1, 2, 4, or 8.

[0018] In another possible implementation, the step of selecting the second amount of representative virtual speakers in the current frame from the first amount of virtual speakers based on the first amount of voting values ​​includes the encoder obtaining a seventh amount of virtual speakers and a seventh amount of final voting values ​​in the current frame corresponding to the current frame based on the first amount of voting values ​​and the sixth amount of final voting values ​​in the previous frame, and selecting the second amount of representative virtual speakers in the current frame from the seventh amount of virtual speakers based on the seventh amount of final voting values ​​in the current frame. The second amount is less than the seventh amount. This indicates that the second amount of representative virtual speakers in the current frame are some of the seventh amount of virtual speakers. The seventh amount of virtual speakers includes the first amount of virtual speakers, and the seventh amount of virtual speakers includes the sixth amount of virtual speakers. The virtual speakers included in the sixth amount of virtual speakers are representative virtual speakers in the previous frame of the three-dimensional audio signal used to encode the previous frame. The sixth quantity of virtual speakers included in the representative virtual speaker set in the previous frame have a one-to-one correspondence with the sixth quantity of final voting values ​​in the previous frame.

[0019] During the search for virtual speakers, the positions of the real sound sources do not necessarily coincide with the positions of the virtual speakers, so that the virtual speakers and the real sound sources do not necessarily form a one-to-one correspondence. In addition, in a real complex scenario, a virtual speaker set that includes a limited amount of virtual speakers may not be able to represent all the sound sources in the sound field. In this case, the virtual speakers seen in different frames may change frequently, which greatly affects the auditory experience of the listener, and causes significant discontinuity and noise in the decoded and reconstructed three-dimensional audio signal. In the virtual speaker selection method provided in this embodiment of this application, the representative virtual speaker in the previous frame is inherited. Specifically, for the same number of virtual speakers, the initial voting value in the current frame is adjusted by using the final voting value in the previous frame, so that the encoder is more likely to select the representative virtual speaker in the previous frame. This alleviates frequent changes of virtual speakers in different frames, improves the continuity of the signal direction between frames, improves the stability of the sound image of the reconstructed three-dimensional audio signal, and ensures the sound quality of the reconstructed three-dimensional audio signal.

[0020] In another possible implementation, the method includes an encoder obtaining a first correlation between a representative virtual speaker set in a current frame and a previous frame, and obtaining a fourth amount of coefficients in the current frame of the three-dimensional audio signal and frequency domain feature values ​​of the fourth amount of coefficients if the first correlation does not satisfy the reuse condition. The representative virtual speaker set in the previous frame includes a sixth amount of virtual speakers. The virtual speakers included in the sixth amount of virtual speakers are representative virtual speakers in a previous frame of the three-dimensional audio signal used to encode the previous frame. The first correlation is used to determine whether to reuse the representative virtual speaker set in the previous frame when the current frame is encoded.

[0021] In this way, the encoder can first determine whether to reuse the representative virtual speaker set in the previous frame to encode the current frame. If the encoder reuses the representative virtual speaker set in the previous frame to encode the current frame, the encoder does not need to perform the virtual speaker search process again. This effectively reduces the computational complexity performed by the encoder to search for virtual speakers, and thus reduces the computational complexity of performing compression coding on the three-dimensional audio signal, and reduces the computational load of the encoder. This also makes it easier to mitigate the frequent changes of the virtual speakers in different frames, enhances the continuity of the directions between frames, and improves the stability of the sound image of the reconstructed three-dimensional audio signal, and ensures the sound quality of the reconstructed three-dimensional audio signal. If the encoder cannot reuse the representative virtual speaker set in the previous frame to encode the current frame, the encoder reselects the representative coefficient, votes for each virtual speaker in the candidate virtual speaker set by using the representative coefficient in the current frame, and selects a representative virtual speaker in the current frame based on the voting value, so as to reduce the computational complexity of performing compression coding on the three-dimensional audio signal, and reduce the computational load of the encoder.

[0022] Optionally, the method further comprises the encoder further obtaining the current frame of the three-dimensional audio signal for performing compression encoding on the current frame of the three-dimensional audio signal to obtain a bitstream, and sending the bitstream to the decoder side.

[0023] According to a second aspect, the application provides a three-dimensional audio signal encoding device. The device includes a module for performing a three-dimensional audio signal encoding method according to the first aspect or any one of the possible forms of the first aspect. For example, the three-dimensional audio signal encoding device includes a coefficient selection module, a virtual speaker selection module, and an encoding module. The coefficient selection module is configured to obtain a fourth quantity of coefficients in a current frame of the three-dimensional audio signal and a frequency domain feature value of the fourth quantity of coefficients. The coefficient selection module is further configured to select a third quantity of representative coefficients from the fourth quantity of coefficients based on the frequency domain feature value of the fourth quantity of coefficients, the third quantity being less than the fourth quantity. The virtual speaker selection module is configured to select a second quantity of representative virtual speaker in the current frame from the candidate virtual speaker set based on the third quantity of representative coefficients. The encoding module is configured to encode the current frame based on the second quantity of representative virtual speaker in the current frame to obtain a bitstream. These modules can perform corresponding functions in the method example of the first aspect. For details, please refer to the detailed explanation in the method example, and the details will not be repeated here.

[0024] According to a third aspect, the application provides an encoder, the encoder comprising at least one processor and a memory, the memory being configured to store a group of computer instructions, the execution of which by the processor results in the execution of the operation steps of the three-dimensional audio signal encoding method according to the first aspect or any one of the possible implementation aspects of the first aspect.

[0025] According to a fourth aspect, the application provides a system. The system includes an encoder according to the third aspect and a decoder. The encoder is configured to perform the operation steps of the three-dimensional audio signal encoding method according to the first aspect or any one of the possible implementation aspects of the first aspect. The decoder is configured to decode a bitstream generated by the encoder.

[0026] According to a fifth aspect, the application provides a computer readable storage medium comprising computer software instructions which, when executed on an encoder, cause the encoder to perform the operational steps of a method according to the first aspect or any one of the possible implementation aspects of the first aspect.

[0027] According to a sixth aspect, the application provides a computer program product which, when executed on an encoder, enables the encoder to perform the operation steps of a method according to the first aspect or any one of the possible implementation aspects of the first aspect.

[0028] In this application, based on the implementation aspects provided in the above aspects, the implementation aspects can be further combined to provide more implementation aspects. [Brief description of the drawings]

[0029] [Figure 1] FIG. 1 is a schematic diagram of the structure of an audio coding system according to an embodiment of this application; [Diagram 2] FIG. 1 is a schematic diagram of a scenario of an audio coding system according to an embodiment of this application; [Diagram 3] FIG. 2 is a schematic diagram of an encoder structure according to an embodiment of the present application; [Figure 4] 1 is a schematic flowchart of a three-dimensional audio encoding method according to an embodiment of the present application; [Figure 5A]1 is a schematic flowchart of a virtual speaker selection method according to an embodiment of the present application; [Figure 5B] 1 is a schematic flowchart of a virtual speaker selection method according to an embodiment of the present application; [Figure 6] 1 is a schematic flowchart of a three-dimensional audio signal encoding method according to an embodiment of this application; [Figure 7A] 1 is a schematic flow chart of a method for selecting representative coefficients in a three-dimensional audio signal according to an embodiment of the present application; [Figure 7B] 1 is a schematic flow chart of a method for selecting representative coefficients in a three-dimensional audio signal according to an embodiment of the present application; [Figure 8] 1 is a schematic flowchart of a virtual speaker selection method according to an embodiment of the present application; [Figure 9] 4 is a schematic flowchart of another virtual speaker selection method according to an embodiment of the present application; [Figure 10] 4 is a schematic flowchart of another virtual speaker selection method according to an embodiment of the present application; [Figure 11] FIG. 1 is a schematic diagram of the structure of a three-dimensional audio signal encoding device according to this application; [Figure 12] 1 is a schematic diagram of the structure of an encoder according to the present application; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0030] To make the following description of the embodiments clear and concise, a brief description of the related art is provided first.

[0031] Sound is a continuous wave produced by the vibration of an object. An object that vibrates and produces sound waves is called a sound source. When sound waves are transmitted through a medium (e.g. air, solid, or liquid), the hearing organs of humans and animals can detect the sound.

[0032] Sound waves are characterized by their pitch, intensity, and timbre. Pitch describes how high or low a sound is. Intensity describes the volume of the sound. Sound intensity is sometimes called loudness or volume. The unit of sound intensity is the decibel (dB). Timbre is also called sound quality.

[0033] The frequency of a sound wave determines its pitch value. The higher the frequency, the higher the pitch. The number of times an object vibrates per second is called frequency. The unit of frequency is hertz (Hz). The sound frequencies that the human ear can detect range from 20 Hz to 20,000 Hz.

[0034] The amplitude of a sound wave determines the sound intensity. The greater the amplitude, the greater the sound intensity. The shorter the distance from the sound source, the greater the sound intensity.

[0035] The waveform of the sound wave determines the tone, and includes square wave, sawtooth wave, sine wave, pulse wave, etc.

[0036] Sounds may be classified into regular and irregular sounds based on the characteristics of sound waves. Irregular sounds are sounds generated from a sound source by irregular vibration. Abnormal sounds are noises that affect, for example, people's work, study, rest, etc. Regular sounds are sounds generated from a sound source by regular vibration. Normal sounds include sounds and music. When sounds are represented by electricity, regular sounds are analog signals that change continuously in the time-frequency domain. The analog signals may be called audio signals. Audio signals are information carriers that carry voice, music, and sound effects.

[0037] The human auditory system has the ability to distinguish the spatial distribution of sound sources in space, and thus, when hearing sounds in space, listeners can detect the direction of the sound in addition to its pitch, intensity, and timbre.

[0038] As people pay more attention to the hearing experience and have higher and higher requirements for quality, three-dimensional audio technology emerges accordingly to enhance the sense of depth, immersion, and spatiality of sound. In this way, the listener not only feels the sound emitted by the sound source from the front, back, left, and right, but also feels that the space where the listener is located is surrounded by a spatial sound field (abbreviated as sound field) created by the sound source, and the sound spreads all around. This creates an "immersive" sound effect that makes the listener feel as if they are in a movie theater, concert hall, etc.

[0039] In three-dimensional audio technology, the space outside the human ear is a system, and the signal received by the eardrum is a three-dimensional audio signal output by the system outside the ear by filtering the sound generated by the sound source. For example, the system outside the human ear may be defined as a system impulse response h(n), an arbitrary sound source may be defined as x(n), and the signal received by the eardrum is the convolution result of x(n) and h(n). The three-dimensional audio signal in the embodiment of this application may be a higher order ambisonics (HOA) signal. Three-dimensional audio may also be called three-dimensional sound effect, spatial audio, three-dimensional sound field reconstruction, virtual 3D audio, binaural audio, etc.

[0040] When sound waves are transmitted through an ideal medium, it is well known that the wave speed is k=w / c and the angular frequency is w=2πf, where f is the frequency of the sound wave and c is the speed of sound. The sound pressure P satisfies equation (1), where ∇ 2 is the Laplace operator. ∇ 2 P+k 2 P=0 formula (1)

[0041] It is assumed that the spatial system outside the human ear is a sphere, the listener is at the center of the sphere, the sound transmitted from outside the sphere is projected onto the sphere, and the sound outside the sphere is filtered. It is assumed that the sound source is distributed on the sphere, and the sound field generated by the sound source on the sphere is used to fit the sound field generated by the original sound source. That is, the three-dimensional audio technology is a sound field fitting method. Specifically, the equation in Equation (1) is solved in a spherical coordinate system. In the passive spherical domain, the equation in Equation (1) is solved to the following Equation (2).

number

[0042] where r represents the radius of the sphere, θ represents the azimuth angle, φ represents the elevation angle, k represents the wave speed, s represents the amplitude of an ideal plane wave, and m represents the sequence number of the order of the three-dimensional audio signal (or the sequence number of the order of the HOA signal).

number

number

number

number

number

[0043] Equation (3) may be substituted into equation (2), transforming equation (2) into equation (4).

number

[0044]

number

[0045] A three-dimensional audio signal is an information carrier that carries the spatial location information of a sound source in a sound field and describes the sound field of a listener in space. Equation (4) shows that the sound field can be expanded on a sphere based on spherical harmonics, that is, the sound field can be decomposed into multiple overlapping plane waves. Therefore, the sound field described by the three-dimensional audio signal can be represented by multiple overlapping plane waves, and the sound field can be reconstructed by using the three-dimensional audio signal coefficients.

[0046] Compared with a 5.1-channel audio signal or a 7.1-channel audio signal, the Nth-order HOA signal is (N+1) 2Since the three-dimensional audio signal has a large number of channels, the HOA signal contains more data to describe the spatial information of the sound field. When an acquisition device (e.g., a microphone) transmits a three-dimensional audio signal to a playback device (e.g., a speaker), a high bandwidth needs to be consumed. Currently, an encoder can perform compression encoding on the three-dimensional audio signal via spatial squeezed surround audio coding (S3AC) or directional audio coding (DirAC) to obtain a bitstream, and transmit the bitstream to a playback device. The playback device decodes the bitstream, reconstructs the three-dimensional audio signal, and reproduces the reconstructed three-dimensional audio signal. This reduces the amount of data and bandwidth usage during the transmission of the three-dimensional audio signal to the playback device. However, the computational complexity performed by the encoder to perform compression encoding on the three-dimensional audio signal is high, and excessive computing resources of the encoder are occupied. Therefore, how to reduce the computational complexity of performing compression coding on the three-dimensional audio signal is an urgent problem to be solved.

[0047] The embodiments of this application provide an audio coding technique, in particular a three-dimensional audio coding technique directed to three-dimensional audio signals, and more specifically a coding technique for representing three-dimensional audio signals by using a small number of channels to improve conventional audio coding systems. Audio coding (or commonly referred to as coding) includes two parts: audio encoding and audio decoding. Audio encoding is performed at the source side and typically involves processing (e.g., compressing) the original audio to reduce the amount of data for representing the original audio and achieve more efficient storage and / or transmission. Audio decoding is performed at the destination side and typically involves performing an inverse process to the encoder to reconstruct the original audio. The encoding unit and the decoding unit are also collectively referred to as a codec. Hereinafter, implementation aspects of the embodiments of this application will be described in detail with reference to the accompanying drawings.

[0048] 1 is a schematic diagram of the structure of an audio coding system according to an embodiment of this application. The audio coding system 100 includes a source device 110 and a destination device 120. The source device 110 is configured to perform compression encoding on a three-dimensional audio signal to obtain a bitstream, and send the bitstream to the destination device 120. The destination device 120 decodes the bitstream, reconstructs the three-dimensional audio signal, and plays the reconstructed three-dimensional audio signal.

[0049] Specifically, the source device 110 includes an audio capture device 111 , a pre-processor 112 , an encoder 113 , and a communication interface 114 .

[0050] The audio capture device 111 is configured to capture original audio. The audio capture device 111 may be any type of audio capture device for capturing real-world sounds and / or any type of audio generation device. For example, the audio capture device 111 is a computer audio processor for generating computer audio. Alternatively, the audio capture device 111 may be any type of memory or internal memory for storing audio. The audio includes real-world sounds, sounds of a virtual scene (e.g., VR or augmented reality, AR), and / or any combination thereof.

[0051] The pre-processor 112 is configured to receive the original audio captured by the audio capturing device 111, and pre-process the original audio to obtain a three-dimensional audio signal. For example, the pre-processing performed by the pre-processor 112 includes channel switching, audio format conversion, noise removal, etc.

[0052] The encoder 113 is configured to receive the three-dimensional audio signal generated by the pre-processor 112 and perform compression encoding on the three-dimensional audio signal to obtain a bitstream. For example, the encoder 113 may include a spatial encoder 1131 and a core encoder 1132. The spatial encoder 1131 is configured to select (or search) virtual speakers from a set of candidate virtual speakers based on the three-dimensional audio signal, and generate virtual speaker signals based on the three-dimensional audio signal and the virtual speakers. The virtual speaker signals may also be referred to as playback signals. The core encoder 1132 is configured to encode the virtual speaker signals to obtain a bitstream.

[0053] The communication interface 114 is configured to receive the bitstream generated by the encoder 113 and transmit the bitstream to the destination device 120 via the communication channel 130, whereby the destination device 120 reconstructs the three-dimensional audio signal based on the bitstream.

[0054] The destination device 120 includes a player 121, a post-processor 122, a decoder 123, and a communication interface 124.

[0055] The communication interface 124 is configured to receive the bitstream transmitted by the communication interface 114 and to transmit the bitstream to the decoder 123 such that the decoder 123 reconstructs the three-dimensional audio signal based on the bitstream.

[0056] The communication interface 114 and the communication interface 124 may be configured to transmit and receive data related to the original audio via a direct communication link between the source device 110 and the destination device 120, e.g., a direct wired or wireless connection, or via any type of network, such as a wired network, a wireless network, or any combination thereof, or via any type of private or public network, or any combination thereof.

[0057] Each of communication interface 114 and communication interface 124 may be configured as a unidirectional communication interface, as indicated in FIG. 1 by an arrow pointing from source device 110 to destination device 120, corresponding to communication channel 130, or a bidirectional communication interface, and may be configured to send and receive messages, etc., to establish a connection, determine and exchange a communication link and / or any other information related to data transmission, such as the transmission of an encoded bitstream.

[0058] The decoder 123 is configured to decode the bitstream and reconstruct a three-dimensional audio signal. For example, the decoder 123 includes a core decoder 1231 and a spatial decoder 1232. The core decoder 1231 is configured to decode the bitstream to obtain virtual speaker signals. The spatial decoder 1232 is configured to reconstruct the three-dimensional audio signal based on the candidate virtual speaker sets and the virtual speaker signals to obtain a reconstructed three-dimensional audio signal.

[0059] The post-processor 122 is configured to receive the reconstructed three-dimensional audio signal generated by the decoder 123 and to post-process the reconstructed three-dimensional audio signal. For example, the post-processing performed by the post-processor 122 includes audio rendering, volume normalization, user interaction, audio format conversion, noise removal, etc.

[0060] The player 121 is configured to play back the reconstructed sound based on the reconstructed three-dimensional audio signal.

[0061] It should be noted that the audio capture device 111 and the encoder 113 may be integrated into a single physical device or may be located in different physical devices. This is not limited. For example, the source device 110 shown in FIG. 1 includes the audio capture device 111 and the encoder 113. This indicates that the audio capture device 111 and the encoder 113 are integrated into one physical device. In this case, the source device 110 may also be referred to as an acquisition device. For example, the source device 110 is a media gateway of a radio access network, a media gateway of a core network, a transcoding device, a media resource server, an AR device, a VR device, a microphone, or another audio capture device. If the source device 110 does not include the audio capture device 111, it indicates that the audio capture device 111 and the encoder 113 are two different physical devices, and the source device 110 can acquire the original audio from another device (e.g., an audio capture device or an audio storage device).

[0062] Moreover, the player 121 and the decoder 123 may be integrated into one physical device or located in different physical devices. This is not limited. For example, the destination device 120 shown in FIG. 1 includes the player 121 and the decoder 123. This indicates that the player 121 and the decoder 123 are integrated into one physical device. In this case, the destination device 120 is also called a playback device, and the destination device 120 has a decoding function and a function of playing the reconstructed audio. For example, the destination device 120 is a speaker, a headset, or another audio playback device. If the destination device 120 does not include the player 121, it indicates that the player 121 and the decoder 123 are two different physical devices. After decoding the bitstream and reconstructing the three-dimensional audio signal, the destination device 120 transmits the reconstructed three-dimensional audio signal to another playback device (e.g., a speaker or a headset), and the other playback device plays the reconstructed three-dimensional audio signal.

[0063] 1, the source device 110 and the destination device 120 may be integrated into one physical device or may be located in different physical devices, but this is not limited thereto.

[0064] For example, as shown in (a) of FIG. 2, the source device 110 may be a microphone in a recording studio, and the destination device 120 may be a speaker. The source device 110 may obtain original audio of various musical instruments and send the original audio to a codec device. The codec device performs codec processing on the original audio to obtain a reconstructed three-dimensional audio signal. The destination device 120 plays the reconstructed three-dimensional audio signal. As another example, the source device 110 may be a microphone in a terminal device, and the destination device 120 may be a headset. The source device 110 may obtain external sounds or audio synthesized by the terminal device.

[0065] In another example, as shown in FIG. 2(b), the source device 110 and the destination device 120 are integrated into a virtual reality (VR), an augmented reality (AR), a mixed reality (MR), or an extended reality (XR) device. In this case, the VR / AR / MR / XR device has the capability to capture the original audio, play the audio, and perform coding. The source device 110 may capture sounds made by the user and sounds made by virtual objects in the virtual environment in which the user is located.

[0066] In these embodiments, the source device 110 or its corresponding functions and the destination device 120 or its corresponding functions may be implemented by using the same hardware and / or software, separate hardware and / or software, or any combination thereof. Based on this description, the presence and division of different units or functions in the source device 110 and / or the destination device 120 shown in Fig. 1 may vary depending on the actual device and application, which is obvious to those skilled in the art.

[0067] The structure of the audio coding system is merely an example for explanation. In some possible implementations, the audio coding system may further include another device. For example, the audio coding system may further include a device-side device or a cloud-side device. After obtaining the original audio, the source device 110 pre-processes the original audio to obtain a three-dimensional audio signal, and transmits the three-dimensional audio to the device-side device or the cloud-side device, so that the device-side device or the cloud-side device realizes the function of encoding and decoding the three-dimensional audio signal.

[0068] The audio coding method provided in the embodiment of this application is mainly applied to the encoder side. The structure of the encoder is described in detail with reference to Fig. 3. As shown in Fig. 3, the encoder 300 includes a virtual speaker configuration unit 310, a virtual speaker set generation unit 320, an encoding analysis unit 330, a virtual speaker selection unit 340, a virtual speaker signal generation unit 350, and an encoding unit 360.

[0069] The virtual speaker configuration unit 310 is configured to generate virtual speaker configuration parameters based on the encoder configuration information to obtain multiple virtual speakers. The encoder configuration information includes, but is not limited to, the order of the three-dimensional audio signal (or usually called HOA order), the encoding bit rate, user-defined information, etc. The virtual speaker configuration parameters include, but are not limited to, the amount of virtual speakers, the order of the virtual speakers, the position coordinates of the virtual speakers, etc. For example, the amount of virtual speakers is 2048, 1669, 1343, 1024, 530, 512, 256, 128, or 64. The order of the virtual speakers may be any of 2nd to 6th orders. The position coordinates of the virtual speakers include azimuth and elevation.

[0070] The virtual speaker configuration parameters output by the virtual speaker configuration unit 310 are input for the virtual speaker set generation unit 320 .

[0071] The virtual speaker set generation unit 320 is configured to generate a candidate virtual speaker set based on the virtual speaker configuration parameters, where the candidate virtual speaker set includes multiple virtual speakers. Specifically, the virtual speaker set generation unit 320 determines multiple virtual speakers included in the candidate virtual speaker set based on the amount of virtual speakers, and determines coefficients in the virtual speakers based on the position information (e.g., coordinates) of the virtual speakers and the order of the virtual speakers. For example, the method for determining the coordinates of the virtual speakers includes, but is not limited to, generating multiple virtual speakers according to an equidistance rule, or generating multiple non-uniformly distributed virtual speakers according to the auditory perception principle, and then generating the coordinates of the virtual speakers based on the amount of virtual speakers.

[0072] The coefficients for the virtual speakers may also be generated according to the above-mentioned principle of generating a three-dimensional audio signal. s and φ s is set to the position coordinates of the virtual speaker,

number

[0073] The encoding analysis unit 330 is configured to perform encoding analysis on the three-dimensional audio signal, for example to analyze the sound field distribution features of the three-dimensional audio signal, in particular the amount of sound sources, the directivity of the sound sources, the dispersion of the sound sources, and other features of the three-dimensional audio signal.

[0074] The coefficients for the multiple virtual speakers included in the candidate virtual speaker sets output by the virtual speaker set generation unit 320 are input for the virtual speaker selection unit 340 .

[0075] The sound field distribution features of the three-dimensional audio signal output by the encoding analysis unit 330 are input for the virtual speaker selection unit 340 .

[0076] The virtual speaker selection unit 340 is configured to determine a representative virtual speaker that matches the three-dimensional audio signal based on the three-dimensional audio signal to be encoded, the sound field distribution characteristics of the three-dimensional audio signal, and the coefficients of the multiple virtual speakers.

[0077] Alternatively, the encoder 300 in this embodiment of the present application may not include the encoding analysis unit 330. Specifically, the encoder 300 may not analyze the input signal, and the virtual speaker selection unit 340 determines the representative virtual speaker by using a default configuration. For example, the virtual speaker selection unit 340 determines the representative virtual speaker that matches the three-dimensional audio signal based only on the coefficients of the three-dimensional audio signal and the multiple virtual speakers.

[0078] The encoder 300 can use a three-dimensional audio signal acquired from a acquisition device or a three-dimensional audio signal acquired by synthesis of an artificial audio object as input to the encoder 300. In addition, the three-dimensional audio signal input to the encoder 300 can be a three-dimensional audio signal in the time domain or a three-dimensional audio signal in the frequency domain, but this is not limited thereto.

[0079] The position information of the representative virtual speaker and the coefficients in the representative virtual speaker output by the virtual speaker selection unit 340 are input for the virtual speaker signal generation unit 350 and the encoding unit 360 .

[0080] The virtual speaker signal generation unit 350 is configured to generate a virtual speaker signal based on the three-dimensional audio signal and attribute information of the representative virtual speaker. The attribute information of the representative virtual speaker includes at least one of position information of the representative virtual speaker, a coefficient for the representative virtual speaker, and a coefficient for the three-dimensional audio signal. When the attribute information is the position information of the representative virtual speaker, the coefficient for the representative virtual speaker is determined based on the position information of the representative virtual speaker. When the attribute information includes a coefficient in the three-dimensional audio signal, the coefficient for the representative virtual speaker is obtained based on the coefficient in the three-dimensional audio signal. Specifically, the virtual speaker signal generation unit 350 calculates a virtual speaker signal based on the coefficient in the three-dimensional audio signal and the coefficient in the representative virtual speaker.

[0081] For example, assume that matrix A represents the coefficients in the virtual speaker and matrix X represents the HOA coefficients in the HOA signal. Matrix X is the inverse matrix of matrix A. The theoretical optimal solution w is obtained by using the least squares method, where w denotes the virtual speaker signal. The virtual speaker signal satisfies equation (5). w=A -1 X formula (5)

[0082] A -1 indicates the inverse matrix of matrix A, and the size of matrix A is (M×C), where C indicates the amount of representative virtual speakers and M indicates the amount of sound channels of the N-th order HOA signal. a indicates a coefficient in the representative virtual speaker. The size of matrix X is (M×L), where L indicates the amount of coefficients in the HOA signal. x indicates a coefficient in the HOA signal. The coefficient in the representative virtual speaker may be an HOA coefficient in the representative virtual speaker, or an ambisonics coefficient in the representative virtual speaker. For example,

number

number

[0083] The virtual speaker signals output by the virtual speaker signal generation unit 350 are input for the encoding unit 360 .

[0084] The encoding unit 360 is configured to perform core encoding on the virtual speaker signals to obtain a bitstream, including but not limited to transforming, quantizing, psychoacoustic modeling, noise shaping, bandwidth extension, downmixing, arithmetic encoding, bitstream generation, etc.

[0085] Note that the spatial encoder 1131 may include a virtual speaker configuration unit 310, a virtual speaker set generation unit 320, an encoding analysis unit 330, a virtual speaker selection unit 340, and a virtual speaker signal generation unit 350. That is, the virtual speaker configuration unit 310, the virtual speaker set generation unit 320, the encoding analysis unit 330, the virtual speaker selection unit 340, and the virtual speaker signal generation unit 350 realize the function of the spatial encoder 1131. The core encoder 1132 may include an encoding unit 360. That is, the encoding unit 360 realizes the function of the core encoder 1132.

[0086] The encoder shown in Figure 3 may generate one virtual speaker signal or multiple virtual speaker signals, which may be obtained by the encoder shown in Figure 3 through multiple runs or may be obtained by the encoder shown in Figure 3 through a single run.

[0087] The coding process of a three-dimensional audio signal will be described below with reference to the accompanying drawings. Figure 4 is a schematic flowchart of a three-dimensional audio encoding method according to an embodiment of this application. Here, an explanation is given by using an example in which the source device 110 and the destination device 120 of Figure 1 perform the three-dimensional audio signal coding process. As shown in Figure 4, the method includes the following steps:

[0088] S410: The source device 110 obtains the current frame of the three-dimensional audio signal.

[0089] As described in the previous embodiment, when the originating device 110 is accompanied by the audio capturing device 111, the originating device 110 can capture the original audio by using the audio capturing device 111. Optionally, the originating device 110 may alternatively receive the original audio captured by another device, or may capture the original audio from a memory in the originating device 110 or another memory. The original audio may include at least one of real-world sounds captured in real time, audio stored in the device, and audio captured by synthesis of multiple audios. In this embodiment, the method of capturing the original audio and the type of the original audio are not limited.

[0090] After obtaining the original audio, the source device 110 generates a three-dimensional audio signal based on the three-dimensional audio technology and the original audio, so as to provide an "immersive" sound effect to the listener during the playback of the original audio. For a specific method for generating a three-dimensional audio signal, please refer to the description of the pre-processor 112 in the above embodiment and the description of the related art.

[0091] Also, the audio signal is a continuous analog signal. During processing of the audio signal, the audio signal may be sampled first to generate a digital signal of a sequence of frames. A frame may include multiple sampling points. Alternatively, the frame may be sampling points obtained by sampling. Alternatively, the frame may include subframes obtained by dividing the frame. Alternatively, the frame may be subframes obtained by dividing the frame. For example, if the length of a frame is L sampling points and the frame is divided into N subframes, each subframe corresponds to L / N sampling points. Audio encoding and decoding usually means processing a sequence of audio frames including multiple sampling points.

[0092] The audio frame may include a current frame or a previous frame. The current frame or the previous frame described in the embodiment of this application may be a frame or a subframe. The current frame is a frame on which a coding process is currently performed. The previous frame is a frame on which a coding process was performed immediately before the current moment. The previous frame may be a frame one moment before the current moment or a frame several moments before the current moment. In this embodiment of this application, the current frame of the three-dimensional audio signal is a frame of the three-dimensional audio signal on which a coding process was performed at the current moment, and the previous frame is a frame of the three-dimensional audio signal on which a coding process was performed at a moment before the current moment. The current frame of the three-dimensional audio signal may be a current frame to be encoded of the three-dimensional audio signal. The current frame of the three-dimensional audio signal may be referred to as a current frame for short. The previous frame of the three-dimensional audio signal may be referred to as a previous frame for short.

[0093] S420: The originating device 110 determines a set of candidate virtual speakers.

[0094] In some cases, the candidate virtual speaker set is pre-configured in the memory of the source device 110. The source device 110 may retrieve the candidate virtual speaker set from the memory. The candidate virtual speaker set includes a plurality of virtual speakers. The virtual speakers represent virtual speakers in a spatial sound field. The virtual speakers are configured to calculate virtual speaker signals based on the three-dimensional audio signal such that the destination device 120 plays the reconstructed three-dimensional audio signal.

[0095] In another case, the virtual speaker configuration parameters are pre-configured in a memory of the originating device 110. The originating device 110 generates the candidate virtual speaker sets based on the virtual speaker configuration parameters. Optionally, the originating device 110 generates the candidate virtual speaker sets in real-time based on the computing resource (e.g., processor) capabilities of the originating device 110 and the characteristics (e.g., channels and data amount) of the current frame.

[0096] For a specific method of generating the candidate virtual speaker sets, please refer to the prior art and the description of the virtual speaker configuration unit 310 and the virtual speaker set generation unit 320 in the above embodiment.

[0097] S430: The source device 110 selects a representative virtual speaker in a current frame of the three-dimensional audio signal from a set of candidate virtual speakers based on the current frame.

[0098] The source device 110 votes for the virtual speakers based on the coefficients in the current frame and the coefficients in the virtual speakers, and selects a representative virtual speaker for the current frame from the candidate virtual speaker set based on the voting value of the virtual speaker. The candidate virtual speaker set is searched for a limited amount of representative virtual speakers in the current frame as the best matching virtual speaker for the current frame to be encoded, in order to compress data of the three-dimensional audio signal to be encoded.

[0099] Figures 5A and 5B are schematic flow charts of a virtual speaker selection method according to an embodiment of this application. The method process shown in Figures 5A and 5B is a description of a specific operation process included in S430 of Figure 4. Here, an explanation is given by using an example in which the encoder 113 of the source device 110 shown in Figure 1 performs the virtual speaker selection process. In particular, the function of the virtual speaker selection unit 340 is realized. As shown in Figures 5A and 5B, the method includes the following steps:

[0100] S510: The encoder 113 obtains the representative coefficients in the current frame.

[0101] The representative coefficient may be a frequency domain representative coefficient or a time domain representative coefficient. The frequency domain representative coefficient may also be called a frequency domain representative frequency or a spectrum representative coefficient. The time domain representative coefficient may also be called a time domain representative sampling point. For a specific method for obtaining the representative coefficient of the current frame, please refer to the following description of S610 and S620 in Figures 6, 7A, and 7B.

[0102] S520: The encoder 113 selects a representative virtual speaker for the current frame from the candidate virtual speaker set based on a voting value obtained by voting for virtual speakers in the candidate virtual speaker set based on the representative coefficient for the current frame, i.e., executes S440 to S460.

[0103] The encoder 113 votes for the virtual speakers in the candidate virtual speaker set based on the representative coefficients in the current frame and the coefficients in the virtual speakers, and selects (searches for) a representative virtual speaker for the current frame from the candidate virtual speaker set based on the final voting values ​​of the virtual speakers in the current frame. For a specific method for selecting a representative virtual speaker for the current frame, please refer to the following description of S630 in Figures 8 and 9.

[0104] It should be noted that the encoder first traverses the virtual speakers included in the candidate virtual speaker set and compresses the current frame by using a representative virtual speaker for the current frame selected from the candidate virtual speaker set. However, if the selection results of the virtual speakers in successive frames vary greatly, the sound image of the reconstructed three-dimensional audio signal becomes unstable, and the sound quality of the reconstructed three-dimensional audio signal deteriorates. In this embodiment of the application, the encoder 113 can update the initial voting values ​​of the virtual speakers for the current frame and included in the candidate virtual speaker set based on the final voting values ​​of the previous frame and the representative virtual speaker for the previous frame to obtain the final voting values ​​of the virtual speakers for the current frame, and then selects the representative virtual speaker for the current frame from the candidate virtual speaker set based on the final voting values ​​of the virtual speakers for the current frame. In this way, the representative virtual speaker for the current frame is selected based on the representative virtual speaker for the previous frame. Therefore, when selecting the representative virtual speaker for the current frame for the current frame, the encoder tends to select the same virtual speaker as the representative virtual speaker for the previous frame. This improves the directional continuity between successive frames, and solves the problem that the results of selecting virtual speakers for successive frames are significantly different. Optionally, this embodiment of the application may further include S530.

[0105] S530: The encoder 113 adjusts the initial voting values ​​of the virtual speakers in the candidate virtual speaker set in the current frame based on the final voting value of the representative virtual speaker in the previous frame in order to obtain the final voting value of the virtual speaker in the current frame.

[0106] After voting for the virtual speakers in the candidate virtual speaker set based on the representative coefficient of the current frame and the coefficient of the virtual speaker for obtaining the initial voting value of the virtual speaker for the current frame, the encoder 113 adjusts the initial voting value of the virtual speaker in the candidate virtual speaker set for the current frame based on the final voting value of the representative virtual speaker of the previous frame in the previous frame to obtain the final voting value of the virtual speaker for the current frame. The representative virtual speaker of the previous frame is the virtual speaker used when the encoder 113 encodes the previous frame. For a specific method for adjusting the initial voting value of the virtual speaker in the candidate virtual speaker set in the current frame, please refer to the following description of S6302a and S6302b in FIG. 9.

[0107] In some embodiments, if the current frame is the first frame of the original audio, the encoder 113 performs S510 and S520. If the current frame is any frame after the second frame of the original audio, the encoder 113 may first determine whether to reuse the representative virtual speaker of the previous frame to encode the current frame, or may determine whether to search for a virtual speaker to ensure directional continuity between successive frames and reduce encoding complexity. Optionally, this embodiment of the application may further include S540.

[0108] S540: The encoder 113 determines whether to search for a virtual speaker based on the representative virtual speakers of the current frame and the previous frame.

[0109] If the encoder 113 determines to search for a virtual speaker, it executes S510 to S530. Optionally, the encoder 113 can execute S510 first, that is, the encoder 113 obtains a representative coefficient of a current frame. The encoder 113 determines whether to search for a virtual speaker based on the representative coefficient of the current frame and the coefficient of a representative virtual speaker of a previous frame. If the encoder 113 determines to search for a virtual speaker, it executes S520 and S530.

[0110] If the encoder 113 determines not to search for a virtual speaker, it executes S550.

[0111] S550: The encoder 113 determines to reuse the representative virtual speaker of the previous frame for encoding the current frame.

[0112] The encoder 113 reuses the representative virtual speakers of the previous frame and the current frame to generate a virtual speaker signal, encodes the virtual speaker signal to obtain a bitstream, and sends the bitstream to the destination device 120, i.e., performs S450 and S460.

[0113] For a specific method for determining whether to search for a virtual speaker, see the description of S650 and S660 in FIG. 10 below.

[0114] S440: The source device 110 generates a virtual speaker signal based on a current frame of the three-dimensional audio signal and a representative virtual speaker for the current frame.

[0115] The source device 110 generates a virtual speaker signal based on the coefficient of the current frame and the coefficient of the representative virtual speaker of the current frame. For a specific method for generating a virtual speaker signal, please refer to the description of the virtual speaker signal generating unit 350 in the prior art and the above embodiment.

[0116] S450: The source device 110 encodes the virtual speaker signal to obtain a bitstream.

[0117] In order to compress the data of the three-dimensional audio signal to be encoded, the source device 110 may perform encoding operations such as conversion or quantization on the virtual speaker signals to generate a bitstream. For a specific method for generating a bitstream, please refer to the description of the encoding unit 360 in the prior art and the above embodiment.

[0118] S460: The source device 110 transmits a bitstream to the destination device 120.

[0119] After encoding all of the original audio, the source device 110 can send a bitstream of the original audio to the destination device 120. Alternatively, the source device 110 can encode the three-dimensional audio signal in real time on a frame-by-frame basis and send a bitstream of the encoded frames. For specific methods for sending the bitstream, please refer to the prior art and the descriptions of the communication interface 114 and the communication interface 124 in the above embodiments.

[0120] S470: The destination device 120 decodes the bitstream sent by the source device 110 and reconstructs the three-dimensional audio signal to obtain a reconstructed three-dimensional audio signal.

[0121] After receiving the bitstream, the destination device 120 decodes the bitstream to obtain virtual speaker signals, and reconstructs a three-dimensional audio signal based on the candidate virtual speaker set and the virtual speaker signal to obtain a reconstructed three-dimensional audio signal. The destination device 120 plays the reconstructed three-dimensional audio signal. Alternatively, the destination device 120 can send the reconstructed three-dimensional audio signal to another playback device, which then plays the reconstructed three-dimensional audio signal, so as to realize a more vivid "immersive" sound effect, as if the listener were in a movie theater, a concert hall, a virtual scene, etc.

[0122] Currently, in the process of searching for a virtual speaker, a correlation operation needs to be performed on each coefficient of the three-dimensional audio signal and the coefficient of each virtual speaker to measure the relationship between each virtual speaker in the candidate virtual speaker set and the three-dimensional audio signal. This increases the computational burden on the encoder. An embodiment of this application provides a method for selecting coefficients of a three-dimensional audio signal. The encoder performs a correlation operation on a representative coefficient of the three-dimensional audio signal and a coefficient for each virtual speaker to select a representative virtual speaker, thereby reducing the complexity of the computation performed by the encoder to search for a virtual speaker.

[0123] The method for selecting coefficients for a three-dimensional audio signal is described in detail below with reference to the accompanying drawings. FIG. 6 is a schematic flowchart of a three-dimensional audio signal encoding method according to an embodiment of this application. Here, an explanation is given by using an example in which the encoder 113 of the source device 110 of FIG. 1 performs a process of selecting coefficients in a three-dimensional audio signal. In particular, the function of the virtual speaker selection unit 340 is realized. The method process shown in FIG. 6 is a description of a specific operation process included in S510 of FIG. 5A. As shown in FIG. 6, the method includes the following steps:

[0124] S610: The encoder 113 obtains a fourth quantity coefficient in a current frame of the three-dimensional audio signal and a frequency domain feature value for the fourth quantity coefficient.

[0125] Assuming that the three-dimensional audio signal is an HOA signal, the encoder 113 samples the current frame of the HOA signal to obtain L (N+1) 2 number of sampling points, i.e., the coefficient of the fourth quantity can be obtained. N indicates the order of the HOA signal. For example, assuming that the duration of the current frame of the HOA signal is 20 milliseconds, the encoder 113 samples the current frame based on a frequency of 48 kHz to obtain 960 (N+1) sampling points in the time domain. 2 number of sampling points, which are sometimes called time-domain coefficients.

[0126] The frequency domain coefficients of the current frame of the three-dimensional audio signal can be obtained by a time-frequency transform based on the time domain coefficients of the current frame of the three-dimensional audio signal. The transform method from the time domain to the frequency domain is not limited. For example, the transform method from the time domain to the frequency domain is the Modified Discrete Cosine Transform (MDCT). In this case, the 960·(N+1) 2 frequency domain coefficients can be obtained. The frequency domain coefficients may also be called spectral coefficients or frequencies.

[0127] The frequency domain feature value of a sampling point satisfies the following equation: p(j)=norm(x(j)), where j=1, 2, ..., and L, L denotes the quantity of sampling moments, x denotes the frequency domain coefficients, e.g., MDCT coefficients, of the current frame of the three-dimensional audio signal, norm is the operation of calculating the 2-norm, and x(j) is the (N+1)-th sampling moment. 2 1 shows the frequency domain coefficients of sampling points.

[0128] Alternatively, the frequency domain feature value of the sampling point may be any channel coefficient in the HOA signal. Usually, the channel coefficient corresponding to the 0th order is selected. Therefore, the frequency domain feature value of the HOA signal satisfies the following equation: p(j)=x0(j), where x0(j) denotes the frequency domain coefficient at the jth 0th order frequency.

[0129] Alternatively, the frequency domain feature value of a sampling point may be the average value of multiple channel coefficients in the HOA signal. Therefore, the frequency domain feature value of the HOA signal satisfies the following formula: p(j)=mean(x(j)), where mean indicates the averaging operation.

[0130] S620: The encoder 113 selects a representative coefficient of the third quantity from the coefficients of the fourth quantity based on the frequency domain feature values ​​of the coefficients of the fourth quantity.

[0131] The encoder 113 divides the spectral range indicated by the coefficient of the fourth quantity into at least one subband. The encoder 113 divides the spectral range indicated by the coefficient of the fourth quantity into one subband. It can be understood that the spectral range of the subband is equal to the spectral range indicated by the coefficient of the fourth quantity. This is equivalent to the encoder 113 not dividing the spectral range indicated by the coefficient of the fourth quantity.

[0132] When the encoder 113 divides the spectral range indicated by the coefficients of the fourth quantity into at least two subbands, in some cases the encoder 113 divides the spectral range indicated by the coefficients of the fourth quantity equally into at least two subbands, each of which includes the same amount of coefficients.

[0133] In another case, the encoder 113 divides the spectral range represented by the fourth amount of coefficients unevenly, and the at least two sub-bands obtained by the division include different amounts of coefficients, or the at least two sub-bands obtained by the division include different amounts of coefficients. For example, the encoder 113 can divide the spectral range represented by the fourth amount of coefficients unevenly based on a low frequency range, a mid frequency range, and a high frequency range in the spectral range represented by the fourth amount of coefficients, so that each of the spectral ranges of the low frequency range, the mid frequency range, and the high frequency range includes at least one sub-band. At least one sub-band in the low frequency range includes the same amount of coefficients, respectively. At least one sub-band in the mid frequency range includes the same amount of coefficients, respectively. At least one sub-band in the high frequency range includes the same amount of coefficients, respectively. The sub-bands of the three spectral ranges of the low frequency range, the mid frequency range, and the high frequency range can include different amounts of coefficients.

[0134] For example, the encoder 113 divides the spectral range indicated by the fourth amount of coefficients into T subbands based on a psychoacoustic model. For example, T=44. The starting coefficient sequence number in the i-th subband is denoted as sfb[i], where i=1, 2, ..., and T, indicating that the value of i ranges from 1 to T. The amount of coefficients included in the i-th subband is denoted as b(i). Assuming that the low frequency range includes 10 subbands, b(1)=4 indicates that the 1st subband includes 4 coefficients, b(10)=4 indicates that the 10th subband includes 4 coefficients. The mid frequency range includes 20 subbands. b(11)=8 indicates that the 11th subband includes 8 coefficients, and b(30)=8 indicates that the 30th subband includes 8 coefficients. The high frequency range includes 14 subbands. b(31)=16 indicates that the 31st subband contains 16 coefficients, and b(44)=16 indicates that the 44th subband contains 16 coefficients.

[0135] Furthermore, the encoder 113 selects a representative coefficient from at least one subband included in a spectral range indicated by the coefficient of the fourth quantity based on the frequency domain feature value of the coefficient of the fourth quantity to obtain a representative coefficient of a third quantity, the third quantity being smaller than the fourth quantity, and the coefficient of the fourth quantity including the representative coefficient of the third quantity.

[0136] In a possible implementation, the method process shown in Figures 7A and 7B is a description of a specific operation process included in S620 of Figures 7A and 7B. As shown in Figures 7A and 7B, the method includes the following steps:

[0137] S6201: The encoder 113 selects Z representative coefficients from each of at least one subband based on frequency domain feature values ​​of the coefficients in each subband to obtain a third quantity of representative coefficients, where Z is a positive integer.

[0138] For example, the encoder 113 selects Z representative coefficients from each of the at least one subband according to a descending order of the frequency domain feature values ​​of the coefficients in each subband, and the selected Z representative coefficients from each subband constitute a third quantity of representative coefficients.

[0139] For example, the encoder 113 sorts the frequency domain feature values ​​of the b(i) coefficients in the i-th subband in descending order, and selects K(i) representative coefficients according to the descending order of the frequency domain feature values ​​of the b(i) coefficients in the i-th subband, starting from the coefficient with the largest frequency domain feature value in the i-th subband. The coefficient sequence numbers corresponding to the K(i) representative coefficients in the i-th subband are a i [j], where j=0, ..., and K(i)-1, indicating that the value of j ranges from 0 to K(i)-1. The value of K(i) may be set in advance or may be generated according to a predetermined rule. For example, the encoder 113 selects 50% of the coefficients with the maximum frequency domain feature value starting from the coefficient with the maximum frequency domain feature value in the i-th subband as representative coefficients.

[0140] In another possible implementation, when at least one subband includes at least two subbands, for each of the at least two subbands, the encoder 113 can first determine the weight of each of the at least two subbands, adjust the frequency domain feature value of the coefficient in each subband by using the weight of each subband, and then select a third amount of representative coefficients from the at least two subbands.As shown in Figure 7A and Figure 7B, S620 can further include the following steps.

[0141] S6202: The encoder 113 determines weights for each of the at least two subbands based on the frequency domain feature values ​​of the first candidate coefficient in each subband.

[0142] The first candidate coefficient may be some coefficients in a subband. The amount of the first candidate coefficient is not limited in this embodiment of the application, and there may be one first candidate coefficient or at least two first candidate coefficients. In some embodiments, the encoder 113 may select the first candidate coefficient according to the method described in S6201. It may be understood that the encoder 113 selects Z representative coefficients from each of the at least two subbands according to the descending order of the frequency domain feature values ​​of the coefficients in each subband, and uses the Z representative coefficients as the first candidate coefficients in each subband. For example, the at least two subbands include a first subband, and the Z representative coefficients selected from the first subband are used as the first candidate coefficients in the first subband.

[0143] The encoder 113 determines a weight for the subband based on the frequency domain feature value of the first candidate coefficient in the subband and the frequency domain feature values ​​of all the coefficients in the subband.

[0144] For example, the encoder 113 calculates the weight w(i) of the i-th subband based on the frequency domain feature values ​​of the candidate coefficients in the i-th subband and the frequency domain feature values ​​of all the coefficients in the i-th subband. The weight w(i) of the i-th subband satisfies Equation (6).

number

[0145] p denotes the frequency domain feature value of the coefficients of the current frame, K(i) denotes the amount of coefficients in the i-th subband, and a i Let [j] denote the coefficient sequence number of the jth coefficient in the ith subband, sfb[i] denote the starting coefficient sequence number in the ith subband, and b(i) denote the amount of coefficients included in the ith subband, j = 0, ..., K(i)-1, for i = 1, 2, ..., T.

[0146] S6203: The encoder 113 adjusts the frequency domain feature values ​​of the second candidate coefficients in each subband based on the weights of each subband to obtain adjusted frequency domain feature values ​​of the second candidate coefficients in each subband.

[0147] The second candidate coefficient may be some coefficients in a subband. The amount of the second candidate coefficient is not limited in this embodiment of the application, and there may be one second candidate coefficient or at least two second candidate coefficients. In some embodiments, the encoder 113 may select the second candidate coefficient according to the method described in S6201. It may be understood that the encoder 113 selects Z representative coefficients from each of the at least two subbands according to the descending order of the frequency domain feature values ​​of the coefficients in each subband, and uses the Z representative coefficients as the second candidate coefficients in each subband. In this case, the amount of the first candidate coefficients and the amount of the second candidate coefficients may be the same or different. For the first candidate coefficients and the second candidate coefficients in a subband, the first candidate coefficients and the second candidate coefficients may be the same coefficients or different coefficients. The encoder 113 may adjust the frequency domain feature values ​​of some coefficients in each subband.

[0148] Alternatively, the second candidate coefficients may be all the coefficients in the subband. In this case, the amount of the first candidate coefficients may be different from the amount of the second candidate coefficients. It can be understood that the encoder 113 adjusts the frequency domain feature values ​​of all the coefficients in each subband.

[0149] For example, the encoder 113 adjusts the frequency domain feature values ​​of the K(i) coefficients in the i-th subband based on the weight w(i) of the i-th subband, and the adjusted frequency domain feature values ​​of the K(i) coefficients in the i-th subband satisfy Equation (7). P'(a i [j])=P(a i [j])W(i) Equation (7)

[0150] j=1,2,...,K(i). P(a i [j]) denotes the frequency domain feature value corresponding to the jth coefficient in the ith subband, and P'(a i [j]) denotes the adjusted frequency domain feature value corresponding to the jth coefficient in the ith subband, K(i) denotes the amount of the coefficient in the ith subband, and a i [j] denotes the coefficient sequence number of the jth coefficient in the ith subband, and w(i) denotes the weight of the ith subband, j=0,...,K(i)-1,i=1,2,...,T.

[0151] S6204: The encoder 113 determines a representative coefficient of a third quantity based on the adjusted frequency domain feature value of the second candidate coefficient in the at least two subbands and the frequency domain feature value of a coefficient other than the second candidate coefficient in the at least two subbands.

[0152] The encoder 113 sorts the frequency domain feature values ​​of all coefficients in the at least two subbands in descending order, and selects a representative coefficient of the third quantity according to the descending order of the frequency domain feature values ​​of all coefficients in the at least two subbands, starting from the coefficient having the largest frequency domain feature value in the at least two subbands.

[0153] It may be understood that when the second candidate coefficients are some coefficients in a subband, the frequency domain feature values ​​of all the coefficients in the at least two subbands include the adjusted frequency domain feature values ​​of the second candidate coefficients in the at least two subbands and the frequency domain feature values ​​of the coefficients other than the second candidate coefficients. The encoder 113 determines the representative coefficient of the third quantity based on the adjusted frequency domain feature values ​​of the second candidate coefficients in the at least two subbands and the frequency domain feature values ​​of the coefficients other than the second candidate coefficients in the at least two subbands.

[0154] If the second candidate coefficients are all the coefficients in the subband, the frequency domain feature values ​​of all the coefficients in the at least two subbands are the adjusted frequency domain feature values ​​of the second candidate coefficients. The encoder 113 determines a representative coefficient of the third quantity based on the adjusted frequency domain feature values ​​of the second candidate coefficients in the at least two subbands.

[0155] The third amount may be preset or may be generated according to a preset rule, for example, the encoder 113 selects 20% of the coefficients having the maximum frequency domain feature value as the representative frequency from all the coefficients in the at least two subbands.

[0156] S630: The encoder 113 selects a second amount of representative virtual speaker for the current frame from the set of candidate virtual speakers based on the third amount of representative coefficient.

[0157] The encoder 113 performs a correlation operation on the representative coefficients of the third quantity in the current frame of the three-dimensional audio signal and the coefficients for each virtual speaker in the candidate virtual speaker set to select a representative virtual speaker for the second quantity for the current frame.

[0158] The encoder selects some coefficients as representative coefficients from all coefficients in the current frame, and selects a representative virtual speaker from the candidate virtual speaker set by using a small number of representative coefficients to represent all coefficients in the current frame. This effectively reduces the computational complexity performed by the encoder to search for a virtual speaker, and thus reduces the computational complexity of performing compression coding on the three-dimensional audio signal, and reduces the computational load of the encoder. For example, a frame of an N-th order HOA signal has 960·(N+1) 2 coefficients. In this embodiment, the first 10% of the coefficients can be selected to participate in the virtual speaker search. In this case, the encoding complexity is reduced by 90% compared to the encoding complexity when all the coefficients participate in the virtual speaker search.

[0159] S640: The encoder 113 encodes the current frame based on the second amount of representative virtual speakers for the current frame to obtain a bitstream.

[0160] The encoder 113 generates a virtual speaker signal based on the current frame and the second amount of representative virtual speakers of the current frame, and encodes the virtual speaker signal to obtain a bitstream. For a specific method for generating the bitstream, please refer to the description of the encoding units 360 and S450 in the prior art and the above embodiments.

[0161] After generating the bitstream, the encoder 113 transmits the bitstream to the destination device 120, so that the destination device 120 decodes the bitstream transmitted by the source device 110 and reconstructs the three-dimensional audio signal to obtain a reconstructed three-dimensional audio signal.

[0162] Since the frequency domain feature values ​​of the coefficients in the current frame represent the sound field characteristics of the three-dimensional audio signal, the encoder selects a representative coefficient in the current frame having a representative sound field component based on the frequency domain feature values ​​of the coefficients in the current frame. By using the representative coefficient, the representative virtual speaker in the current frame selected from the candidate virtual speaker set can completely represent the sound field characteristics of the three-dimensional audio signal. This helps to further improve the accuracy of generating virtual speaker signals by the encoder by performing compression coding on the three-dimensional audio signal to be encoded by using the representative virtual speaker in the current frame, and also to increase the compression ratio for performing compression coding on the three-dimensional audio signal, and reduce the bandwidth occupied by the encoder for transmitting the bitstream.

[0163] In this embodiment of the present application, the encoder 113 may select a representative virtual speaker of the second amount in the current frame based on a voting value obtained by voting for a virtual speaker in the candidate virtual speaker set based on a representative coefficient of the third amount in the current frame. The method process shown in Figure 8 is a description of a specific operation process included in S630 of Figure 7B. As shown in Figure 8, the method includes the following steps:

[0164] S6301: The encoder 113 determines a first quantity of virtual speakers and a first quantity of voting values ​​based on a representative coefficient of a third quantity of the current frame, a candidate virtual speaker set, and a voting count.

[0165] The vote count is used to limit the number of votes made for a virtual speaker. The vote count is an integer equal to or greater than 1, the vote count is equal to or less than the amount of virtual speakers included in the candidate virtual speaker set, and the vote count is equal to or less than the amount of virtual speaker signals transmitted by the encoder. For example, the candidate virtual speaker set includes a fifth amount of virtual speakers, the fifth amount of virtual speakers includes a first amount of virtual speakers, the first amount is equal to or less than the fifth amount, the vote count is an integer equal to or greater than 1, and the vote count is equal to or less than the fifth amount. The virtual speaker signal is also a transmission channel corresponding to the current frame for the representative virtual speaker of the current frame. Usually, the amount of virtual speaker signals is equal to or less than the amount of virtual speakers.

[0166] In a possible implementation, the number of votes may be pre-configured or may be determined based on the computational capabilities of the encoder, for example, the number of votes may be determined based on the encoding rate of the encoder and / or the encoding application scenario.

[0167] In another possible implementation, the number of votes is determined based on the amount of directional sound sources in the current frame. For example, if the amount of directional sound sources in the sound field is 2, the number of votes is set to 2.

[0168] This embodiment of the application provides three possible implementations for determining the first amount of virtual speakers and the first amount of voting values. In the following, the three schemes are described individually.

[0169] In a first possible implementation, the number of votes is equal to 1. After obtaining a plurality of representative coefficients by sampling, the encoder 113 obtains the vote values ​​obtained by voting for all virtual speakers in the candidate virtual speaker set based on each representative coefficient of the current frame, and accumulates the vote values ​​of the same number of virtual speakers to obtain a first amount of virtual speakers and a first amount of vote values. It can be understood that the candidate virtual speaker set includes a first amount of virtual speakers. The first amount is equal to the amount of virtual speakers included in the candidate virtual speaker set. Assuming that the candidate virtual speaker set includes a fifth amount of virtual speakers, the first amount is equal to the fifth amount. The first amount of vote values ​​includes the vote values ​​of all virtual speakers in the candidate virtual speaker set. The encoder 113 may use the first amount of vote values ​​as the final vote values ​​of the first amount of virtual speakers of the current frame, and execute S6302. That is, the encoder 113 selects a second amount of representative virtual speakers for the current frame from the first amount of virtual speakers based on the voting values ​​of the first amount.

[0170] The virtual speakers have a one-to-one correspondence with the voting values, i.e., one virtual speaker corresponds to one voting value. For example, the first amount of virtual speakers includes the first virtual speaker, the first amount of voting values ​​includes the voting values ​​of the first virtual speaker, and the first virtual speaker corresponds to the voting value of the first virtual speaker. The voting value of the first virtual speaker represents the priority of the first virtual speaker. Alternatively, the priority may be replaced with a preference. Specifically, the voting value of the first virtual speaker represents the preference to use the first virtual speaker to encode the current frame. It can be understood that a larger voting value of the first virtual speaker indicates a higher priority or preference of the first virtual speaker, and indicates a higher tendency for the encoder 113 to select the first virtual speaker to encode the current frame, compared to virtual speakers with a voting value lower than the voting value of the first virtual speaker in the candidate virtual speaker set.

[0171] In the second possible implementation, the difference from the first possible implementation is that after obtaining the voting values ​​obtained by voting all virtual speakers in the candidate virtual speaker set based on each representative coefficient in the current frame, the encoder 113 selects some voting values ​​from the voting values ​​obtained by voting all virtual speakers in the candidate virtual speaker set based on each representative coefficient, and accumulates the voting values ​​of the same number of virtual speakers among the virtual speakers corresponding to these voting values ​​to obtain a first amount of virtual speakers and a first amount of voting values. It can be understood that the candidate virtual speaker set includes a first amount of virtual speakers. The first amount is equal to or less than the amount of virtual speakers included in the candidate virtual speaker set. The first amount of voting values ​​includes the voting values ​​of some virtual speakers included in the candidate virtual speaker set, or the first amount of voting values ​​includes the voting values ​​of all virtual speakers included in the candidate virtual speaker set.

[0172] In the third possible implementation manner, the difference from the second possible implementation manner is that the number of votes is an integer equal to or greater than 2. For each representative coefficient of the current frame, the encoder 113 performs at least two votes for all virtual speakers in the candidate virtual speaker set, and selects the virtual speaker with the largest vote value each time. After performing at least two votes for all virtual speakers for each representative coefficient of the current frame, the encoder 113 accumulates the vote values ​​of the same number of virtual speakers to obtain a first amount of virtual speakers and a first amount of vote values.

[0173] S6302: The encoder 113 selects a second amount of representative virtual speakers for the current frame from the first amount of virtual speakers based on the voting values ​​of the first amount.

[0174] The encoder 113 selects a second amount of representative virtual speakers for the current frame from the first amount of virtual speakers based on the voting values ​​of the first amount, and the voting values ​​of the second amount of representative virtual speakers for the current frame are greater than a preset threshold.

[0175] Alternatively, the encoder 113 may select a representative virtual speaker for the second quantity of the current frame from the virtual speakers of the first quantity based on the voting values ​​of the first quantity. For example, the encoder 113 determines the voting values ​​of the second quantity from the voting values ​​of the first quantity according to a descending order of the voting values ​​of the first quantity, and uses a virtual speaker corresponding to the voting value of the second quantity among the virtual speakers of the first quantity as a representative virtual speaker for the second quantity of the current frame.

[0176] Optionally, if the voting values ​​of the different numbers of virtual speakers among the first amount of virtual speakers are the same and the voting values ​​of the different numbers of virtual speakers are greater than a preset threshold, the encoder 113 can use all the different numbers of virtual speakers as representative virtual speakers for the current frame.

[0177] It should be noted that the second amount is less than the first amount. The first amount of virtual speakers includes the second amount of representative virtual speakers of the current frame. The second amount may be preset, or the second amount may be determined based on the amount of sound sources in the sound field of the current frame. For example, the second amount may be directly equal to the amount of sound sources in the sound field of the current frame. Or, the amount of sound sources in the sound field of the current frame is processed based on a preset algorithm, and the amount obtained by the processing is used as the second amount. The preset algorithm can be designed according to requirements. For example, the preset algorithm may be as follows: the second amount = the amount of sound sources in the sound field of the current frame + 1; or the second amount = the amount of sound sources in the sound field of the current frame - 1.

[0178] The encoder votes for each virtual speaker in the candidate virtual speaker set by using a small number of representative coefficients to represent all coefficients of the current frame, and selects a representative virtual speaker for the current frame based on the voting value. Furthermore, the encoder compresses and encodes the three-dimensional audio signal to be encoded by using the representative virtual speaker in the current frame. This not only effectively increases the compression ratio for performing compression coding on the three-dimensional audio signal, but also reduces the computational complexity performed by the encoder to search for the virtual speakers, thus reducing the computational complexity of performing compression coding on the three-dimensional audio signal, and reducing the computational load of the encoder.

[0179] In order to improve the directional continuity between successive frames and solve the problem that the results of selecting virtual speakers of successive frames change greatly, the encoder 113 adjusts the initial voting values ​​of the virtual speakers in the candidate virtual speaker set in the current frame based on the final voting value of the representative virtual speaker of the previous frame in order to obtain the final voting value of the virtual speaker of the current frame. Figure 9 is a schematic flowchart of another virtual speaker selection method according to an embodiment of this application. The method process shown in Figure 9 is a description of a specific operation process included in S6302 of Figure 8.

[0180] S6302a: The encoder 113 obtains a seventh amount of virtual speakers and a seventh amount of final voting values ​​for the current frame corresponding to the current frame based on a first amount of initial voting values ​​for the current frame and a sixth amount of final voting values ​​for the previous frame.

[0181] The encoder 113 can determine a first amount of virtual speakers and a first amount of voting values ​​based on the current frame of the three-dimensional audio signal, the candidate virtual speaker set, and the number of votes, according to the method described in S6301, and then use the first amount of voting values ​​as initial voting values ​​for the first amount of virtual speakers for the current frame.

[0182] The virtual speakers have a one-to-one correspondence with the initial vote values ​​of the current frame, i.e., one virtual speaker corresponds to one initial vote value of the current frame. For example, the first amount of virtual speakers includes a first virtual speaker, the first amount of initial vote values ​​of the current frame includes the initial vote value of the first virtual speaker of the current frame, and the first virtual speaker corresponds to the initial vote value of the first virtual speaker of the current frame. The initial vote value of the first virtual speaker in the current frame represents a priority of using the first virtual speaker to encode the current frame.

[0183] The sixth quantity of virtual speakers included in the representative virtual speaker set in the previous frame have a one-to-one correspondence with the sixth quantity of final voting values ​​in the previous frame. The sixth quantity of virtual speakers may be representative virtual speakers of the previous frame of the three-dimensional audio signal used when the encoder 113 encodes the previous frame.

[0184] Specifically, the encoder 113 updates the first amount of initial vote values ​​of the current frame based on the sixth amount of final vote values ​​of the previous frame. Specifically, the encoder 113 calculates the sum of the initial vote values ​​of the current frame of the virtual speakers in the first amount of virtual speakers and the final vote values ​​of the previous frame of the same number of virtual speakers in the sixth amount of virtual speakers to obtain the seventh amount of final vote values ​​of the current frame corresponding to the seventh amount of virtual speakers and the current frame. The seventh amount of virtual speakers includes the first amount of virtual speakers, and the seventh amount of virtual speakers includes the sixth amount of virtual speakers.

[0185] S6302b: The encoder 113 selects a second quantity of representative virtual speakers for the current frame from the seventh quantity of virtual speakers based on the seventh quantity of final voting values ​​for the current frame.

[0186] The encoder 113 selects a second quantity of representative virtual speakers for the current frame from the seventh quantity of virtual speakers based on a final voting value of the seventh quantity of the current frame, and the final voting value of the current frame of the second quantity of representative virtual speakers for the current frame is greater than a preset threshold.

[0187] Alternatively, the encoder 113 may select a representative virtual speaker for the second quantity of the current frame from the seventh quantity of virtual speakers based on the final voting values ​​of the seventh quantity of the current frame. For example, the encoder 113 determines a final voting value for the second quantity of the current frame from the final voting values ​​of the seventh quantity of the current frame according to a descending order of the final voting values ​​of the seventh quantity of the current frame, and uses a virtual speaker that is in the seventh quantity of virtual speakers and associated with the final voting value of the second quantity of the current frame as a representative virtual speaker for the second quantity of the current frame.

[0188] Optionally, if the voting values ​​of the different numbers of virtual speakers among the seventh amount of virtual speakers are the same and the voting values ​​of the different numbers of virtual speakers are greater than a preset threshold, the encoder 113 can use all the different numbers of virtual speakers as representative virtual speakers for the current frame.

[0189] Note that the second amount is less than the seventh amount. The seventh amount of virtual speakers includes the second amount of representative virtual speakers for the current frame. The second amount may be preset, or the second amount may be determined based on the amount of sound sources in the sound field of the current frame.

[0190] In addition, before the encoder 113 encodes the next frame of the current frame, if the encoder 113 determines to reuse the representative virtual speaker of the previous frame to encode the next frame, the encoder 113 may use the second amount of the representative virtual speaker in the current frame as the second amount of the representative virtual speaker in the previous frame, and encode the next frame of the current frame by using the second amount of the representative virtual speaker in the previous frame.

[0191] During the search for virtual speakers, the positions of the real sound sources do not necessarily coincide with the positions of the virtual speakers, so that the virtual speakers and the real sound sources do not necessarily form a one-to-one correspondence. Moreover, in real complex scenarios, the virtual speakers may not be able to represent independent sound sources in the sound field. In this case, the virtual speakers seen in different frames may change frequently, and this frequent change will greatly affect the listener's auditory experience, causing significant discontinuity and noise in the decoding and reconstructed three-dimensional audio signal. In the virtual speaker selection method provided in this embodiment of this application, the representative virtual speaker in the previous frame is inherited. Specifically, for the same number of virtual speakers, the initial voting value in the current frame is adjusted by using the final voting value in the previous frame, so that the encoder is more likely to select the representative virtual speaker in the previous frame. This alleviates the frequent changes of the virtual speakers in different frames, improves the continuity of the directions between frames, improves the stability of the sound image of the reconstructed three-dimensional audio signal, and ensures the sound quality of the reconstructed three-dimensional audio signal. Furthermore, the parameters are adjusted to avoid long-term inheritance of the final vote value of the previous frame, which avoids cases where the algorithm cannot adapt to scenarios where the sound field is changing, e.g. when the sound source is moving.

[0192] In addition, an embodiment of this application further provides a virtual speaker selection method. The encoder can first determine whether to reuse the representative virtual speaker set of the previous frame to encode the current frame. If the encoder reuses the representative virtual speaker set in the previous frame to encode the current frame, the encoder does not need to perform the virtual speaker search process again. This effectively reduces the computational complexity performed by the encoder to search for virtual speakers, thus reducing the computational complexity of performing compression coding on the three-dimensional audio signal, and reducing the computational load of the encoder. If the encoder cannot reuse the representative virtual speaker set of the previous frame to encode the current frame, the encoder reselects the representative coefficient, votes for each virtual speaker in the candidate virtual speaker set by using the representative coefficient of the current frame, and selects a representative virtual speaker for the current frame based on the voting value, thereby reducing the computational complexity of performing compression coding on the three-dimensional audio signal, and reducing the computational load of the encoder. Figure 10 is a schematic flowchart of a virtual speaker selection method according to an embodiment of this application. As shown in FIG. 10, before the encoder 113 obtains the fourth quantity coefficients and the frequency domain feature values ​​of the fourth quantity coefficients of the current frame of the three-dimensional audio signal, i.e. before S610, the method includes the following steps:

[0193] S650: The encoder 113 obtains a first correlation between a representative virtual speaker set in a current frame and a previous frame of the three-dimensional audio signal.

[0194] The representative virtual speaker set in the previous frame includes a sixth amount of virtual speakers. The virtual speakers included in the sixth amount of virtual speakers are representative virtual speakers in the previous frame of the three-dimensional audio signal used to encode the previous frame. The first correlation represents a priority of reusing the representative virtual speaker set in the previous frame when the current frame is encoded. Alternatively, the priority may be replaced with a preference. Specifically, the first correlation is used to determine whether the representative virtual speaker set in the previous frame should be reused when the current frame is encoded. It can be understood that a higher first correlation with the representative virtual speaker set in the previous frame indicates a higher preference for the representative virtual speaker set in the previous frame, and indicates that the encoder 113 is more likely to select the representative virtual speaker for the previous frame to encode the current frame.

[0195] S660: The encoder 113 determines whether the first correlation satisfies a reuse condition.

[0196] If the first correlation does not satisfy the reuse condition, it indicates that the encoder 113 is more likely to search for a virtual speaker and encode the current frame based on the representative virtual speaker of the current frame, and S610 is executed, i.e., the encoder 113 obtains a fourth quantity coefficient of the current frame of the three-dimensional audio signal and a frequency domain feature value of the fourth quantity coefficient.

[0197] Optionally, after selecting the third quantity representative coefficient from the fourth quantity coefficients based on the frequency domain feature value of the fourth quantity coefficient, the encoder 113 can alternatively use the maximum representative coefficient of the third quantity representative coefficients as the coefficient of the current frame for obtaining the first correlation. In this case, the encoder 113 obtains the first correlation between the maximum representative coefficient of the third quantity representative coefficients of the current frame and the representative virtual speaker set of the previous frame. If the first correlation does not satisfy the reuse condition, S630 is executed. That is, the encoder 113 selects the second quantity representative virtual speaker of the current frame from the candidate virtual speaker set based on the third quantity representative coefficient.

[0198] If the first correlation satisfies the reuse condition, which indicates that the encoder 113 is more inclined to select the representative virtual speaker of the previous frame for encoding the current frame, the encoder 113 performs S670 and S680.

[0199] S670: The encoder 113 generates virtual speaker signals based on the representative virtual speaker sets in the current frame and the previous frame.

[0200] S680: The encoder 113 encodes the virtual speaker signal to obtain a bitstream.

[0201] In the virtual speaker selection method provided in this embodiment of the present application, whether to search for a virtual speaker is determined based on the correlation between the representative coefficient of the current frame and the representative virtual speaker of the previous frame, which effectively reduces the complexity on the encoder side while ensuring the accuracy of selecting the representative virtual speaker of the current frame based on the correlation.

[0202] It can be understood that to realize the functions in the above embodiments, the encoder includes corresponding hardware structures and / or software modules for performing the functions. Those skilled in the art should easily recognize that this application can be implemented by hardware or a combination of hardware and computer software in combination with the units and method steps in the examples described in the embodiments disclosed in this application. Whether the functions are performed by hardware or by hardware driven by computer software depends on the specific application scenario and the design constraints of the technical solution.

[0203] The 3D audio signal coding method provided in the embodiment has been described in detail above with reference to Figures 1 to 10. The 3D audio signal encoding device and encoder provided in the embodiment will now be described with reference to Figures 11 and 12.

[0204] Fig. 11 is a schematic diagram of the structure of a possible three-dimensional audio signal encoding device according to an embodiment. The three-dimensional audio signal encoding device may be configured to realize the function of encoding a three-dimensional audio signal in the method embodiment, and thus can achieve the beneficial effects of the method embodiment. In this embodiment, the three-dimensional audio signal encoding device may be the encoder 113 shown in Fig. 1, the encoder 300 shown in Fig. 3, or a module (e.g., a chip) applied to a terminal device or a server.

[0205] As shown in Fig. 11, the three-dimensional audio signal encoding device 1100 includes a communication module 1110, a coefficient selection module 1120, a virtual speaker selection module 1130, an encoding module 1140, and a storage module 1150. The three-dimensional audio signal encoding device 1100 is configured to implement the functions of the encoder 113 in the method embodiments shown in Figs. 6 to 10.

[0206] The communication module 1110 is configured to acquire a current frame of the three-dimensional audio signal. Optionally, the communication module 1110 may alternatively receive a current frame of the three-dimensional audio signal acquired by another device or acquire the current frame of the three-dimensional audio signal from the storage module 1150. The current frame of the three-dimensional audio signal is an HOA signal. The frequency domain feature values ​​of the coefficients are determined based on a two-dimensional vector. The two-dimensional vector includes the HOA coefficients of the HOA signal.

[0207] The coefficient selection module 1120 is configured to obtain a fourth quantity of coefficients of a current frame of the three-dimensional audio signal and frequency domain feature values ​​of the fourth quantity of coefficients.

[0208] The coefficient selection module 1120 is further configured to select a third quantity representative coefficient from the fourth quantity coefficients based on the frequency domain feature values ​​of the fourth quantity coefficients, the third quantity being less than the fourth quantity.

[0209] When the three-dimensional audio signal encoding device 1100 is configured to realize the functions of the encoder 113 in the method embodiments illustrated in Figures 6 to 10, the coefficient selection module 1120 is configured to realize the relevant functions in S610 and S620.

[0210] Specifically, the coefficient selection module 1120 is particularly configured to select representative coefficients from at least one subband included in a spectral range indicated by the fourth amount of coefficients according to the frequency domain feature values ​​of the fourth amount of coefficients to obtain a third amount of representative coefficients, where the at least two subbands include different amounts of coefficients, or the at least two subbands include the same amount of coefficients, respectively.

[0211] For example, the coefficient selection module 1120 is specifically configured to select Z representative coefficients from each subband based on the frequency domain feature values ​​of the coefficients in each subband to obtain a third quantity of representative coefficients, where Z is a positive integer.

[0212] In another example, when the at least one subband includes at least two subbands, the coefficient selection module 1120 is specifically configured to: determine weights for each of the at least two subbands based on the frequency domain feature values ​​of the first candidate coefficients in each subband; adjust the frequency domain feature values ​​of the second candidate coefficients in each subband based on the weights of each subband to obtain adjusted frequency domain feature values ​​of the second candidate coefficients in each subband; and determine a representative coefficient of the third quantity based on the adjusted frequency domain feature values ​​of the second candidate coefficients in the at least two subbands and the frequency domain feature values ​​of coefficients other than the second candidate coefficients in the at least two subbands, where the first candidate coefficients and the second candidate coefficients are some coefficients in the subband.

[0213] The virtual speaker selection module 1130 is configured to select a second quantity representative virtual speaker for the current frame from the set of candidate virtual speakers based on the third quantity representative factor.

[0214] When the three-dimensional audio signal encoding apparatus 1100 is configured to realize the functions of the encoder 113 in the method embodiments illustrated in Figures 6 to 10, the virtual speaker selection module 1130 is configured to realize the relevant functions in S630.

[0215] For example, the virtual speaker selection module 1130 determines a first amount of virtual speakers and a first amount of voting values ​​based on the representative coefficient of the third amount of the current frame, the candidate virtual speaker set, and the number of votes, where the virtual speakers correspond one-to-one with the voting values, the first amount of virtual speakers includes the first virtual speaker, the voting value of the first amount includes the voting value of the first virtual speaker, the first virtual speaker corresponds to the voting value of the first virtual speaker, and the voting value of the first virtual speaker is It is particularly configured to represent a preference for encoding the current frame using the first virtual speaker, the candidate virtual speaker set includes a fifth amount of virtual speakers, the fifth amount of virtual speakers includes the first amount of virtual speakers, the vote count is an integer equal to or greater than one, the vote count is less than or equal to the fifth amount, and to select a second amount of representative virtual speakers for the current frame from the first amount of virtual speakers based on the vote value of the first amount, the second amount being less than the first amount.

[0216] Optionally, the virtual speaker selection module 1130 is further configured to obtain a seventh amount of virtual speakers and a seventh amount of final voting values ​​of the current frame corresponding to the current frame based on the first amount of voting values ​​of the previous frame and the sixth amount of final voting values, wherein the seventh amount of virtual speakers includes the first amount of virtual speakers, the seventh amount of virtual speakers includes a sixth amount of virtual speakers, and the virtual speakers included in the sixth amount of virtual speakers are representative virtual speakers of a previous frame of the three-dimensional audio signal used to encode the previous frame, and to select a second amount of representative virtual speakers of the current frame from the seventh amount of virtual speakers based on the seventh amount of final voting values ​​of the current frame, wherein the second amount is less than the seventh amount.

[0217] Optionally, the virtual speaker selection module 1130 is further configured to obtain a first correlation between the representative virtual speaker sets of the current frame and the previous frame, the representative virtual speaker set of the previous frame including a sixth amount of virtual speakers, the virtual speakers included in the sixth amount of virtual speakers being representative virtual speakers of the previous frame of the three-dimensional audio signal used to encode the previous frame, the first correlation being used to determine whether to reuse the representative virtual speaker set of the previous frame when the current frame is encoded, and if the first correlation does not satisfy the reuse condition, obtain a fourth amount of coefficients of the current frame of the three-dimensional audio signal and frequency domain feature values ​​of the fourth amount of coefficients.

[0218] The encoding module 1140 is configured to encode the current frame based on the second amount of representative virtual speakers for the current frame to obtain a bitstream.

[0219] When the three-dimensional audio signal encoding apparatus 1100 is configured to realize the functions of the encoder 113 in the method embodiments shown in FIGS. 6 to 10, the encoding module 1140 is configured to realize the relevant functions in S640.

[0220] For example, the encoding module 1140 is specifically configured to generate a virtual speaker signal based on the current frame and a second amount of representative virtual speakers for the current frame, and encode the virtual speaker signal to obtain a bitstream.

[0221] The storage module 1150 is configured to store coefficients associated with the three-dimensional audio signal, the candidate virtual speaker sets, the representative virtual speaker set of the previous frame, the selected coefficients and virtual speakers, etc., so that the encoding module 1140 encodes the current frame to obtain a bitstream and transmits the bitstream to the decoder.

[0222] It should be understood that the three-dimensional audio signal encoding apparatus 1100 in this embodiment of the application can be implemented by using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD can be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof. When the three-dimensional audio signal encoding method shown in Figures 6 to 10 is implemented by software, the three-dimensional audio signal encoding apparatus 1100 and its modules can alternatively be software modules.

[0223] For more detailed descriptions of the communication module 1110, the coefficient selection module 1120, the virtual speaker selection module 1130, the encoding module 1140, and the storage module 1150, please directly refer to the relevant descriptions of the method embodiments shown in Figures 6 to 10. Details will not be repeated here.

[0224] 12 is a schematic diagram of the structure of an encoder 1200 according to one embodiment. As shown in FIG. 12, the encoder 1200 includes a processor 1210, a bus 1220, a memory 1230, and a communication interface 1240.

[0225] It should be understood that in this embodiment, the processor 1210 may be a central processing unit (CPU), or the processor 1210 may be another general purpose processor, a digital signal processing (DSP), an ASIC, an FPGA or other programmable logic device, discrete gate or transistor logic devices, discrete hardware components, etc. The general purpose processor may be a microprocessor or any conventional processor, etc.

[0226] Alternatively, the processor may be a graphics processing unit (GPU), a neural network processing unit (NPU), a microprocessor, or one or more integrated circuits for controlling program execution for the solutions of this application.

[0227] The communication interface 1240 is configured to facilitate communication between the encoder 1200 and external devices or components. In this embodiment, the communication interface 1240 is configured to receive three-dimensional audio signals.

[0228] The bus 1220 may include a channel for transmitting information between the aforementioned components (e.g., the processor 1210 and the memory 1230). In addition to a data bus, the bus 1220 may further include a power bus, a control bus, a status signal bus, etc. However, for clarity of explanation, the various buses are illustrated as the bus 1220 in the figures.

[0229] In one example, the encoder 1200 may include multiple processors. The processor may be a multi-core (multi-CPU) processor. A processor herein may be one or more devices, circuits, and / or computing units for processing data (e.g., computer program instructions). The processor 1210 may call up coefficients for the three-dimensional audio signal, candidate virtual speaker sets, representative virtual speaker sets of previous frames, selected coefficients and virtual speakers, etc. stored in the memory 1230.

[0230] 12 only uses an example in which the encoder 1200 has one processor 1210 and one memory 1230. Here, the processor 1210 and the memory 1230 indicate types of components or devices. In a particular embodiment, the amount of each type of component or device may be determined according to service requirements.

[0231] The memory 1230 may correspond to a storage medium configured to store information such as coefficients related to the three-dimensional audio signal in the method embodiments, candidate virtual speaker sets, representative virtual speaker sets of previous frames, and selected coefficients and virtual speakers, for example a magnetic disk such as a mechanical hard disk or a solid-state drive.

[0232] The encoder 1200 may be a general-purpose device or a dedicated device. For example, the encoder 1200 may be an X86-based or ARM-based server, or another dedicated server, such as a policy control and charging (PCC) server. The type of the encoder 1200 is not limited in this embodiment of the application.

[0233] It should be understood that the encoder 1200 according to this embodiment may correspond to the three-dimensional audio signal encoding apparatus 1100 in the embodiment, and may correspond to corresponding entities for performing any of the methods in Figures 6 to 10. In addition, the above and other operations and / or functions of the modules in the three-dimensional audio signal encoding apparatus 1100 are intended to implement corresponding processes of the methods in Figures 6 to 10, respectively. For the sake of brevity, the details will not be described again here.

[0234] The method steps in the embodiments may be implemented by hardware or by a processor executing software instructions. The software instructions may include corresponding software modules. The software modules may be stored in a random access memory (RAM), a flash memory, a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a register, a hard disk, a removable hard disk, a CD-ROM, or any other form of storage medium known in the art. For example, the storage medium is coupled to the processor such that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium may alternatively be components of the processor. The processor and the storage medium may be located in an ASIC. In addition, the ASIC may be located in a network device or a terminal device. Of course, the processor and the storage medium may exist as separate components in a network device or a terminal device.

[0235] All or part of the above-mentioned embodiments may be implemented by software, hardware, firmware, or any combination thereof. When software is used to implement the embodiments, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer programs or instructions are loaded and executed on a computer, all or part of the procedures or functions in the embodiments of this application are executed. The computer may be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user equipment, or other programmable device. The computer programs or instructions may be stored in a computer-readable storage medium or may be transmitted from one computer-readable storage medium to another. For example, the computer programs or instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center in a wired or wireless manner. The computer-readable storage medium may be any available medium accessible by a computer, or a data storage device such as a server or data center incorporating one or more available media. The usable medium may be a magnetic medium, such as a floppy disk, hard disk, or magnetic tape, an optical medium, such as a digital video disc (DVD), or a semiconductor medium, such as a solid state drive (SSD).

[0236] The above description is merely a specific implementation of this application, and is not intended to limit the scope of protection of this application. Any equivalent modifications or replacements that are easily conceived by those skilled in the art within the technical scope disclosed in this application shall be included in the scope of protection of this application. Therefore, the scope of protection of this application shall be subject to the scope of protection of the claims. [Explanation of symbols]

[0237] 100 Audio Coding System 110 Source Device 111 Audio Acquisition Device 112 Preprocessor 113 Encoder 114 Communication Interface 120 Destination Device 121 Player 122 Post Processor 123 Decoder 124 Communication Interface 130 Communication Channels 300 Encoder 310 Virtual speaker configuration unit 320 Virtual speaker set generation unit 330 Encoding Analysis Unit 340 Virtual Speaker Selection Unit 350 Virtual speaker signal generation unit 360 Encoding Units 1100 3D audio signal encoding device 1110 Communication Module 1120 Coefficient Selection Module 1130 Virtual Speaker Selection Module 1131 Spatial Encoder 1132 Core Encoder 1140 Encoding Module 1150 Storage Module 1200 Encoder 1210 Processor 1220 Bus 1230 Memory 1231 Core Decoder 1232 Spatial Decoder 1240 Communication Interface

Claims

1. A computer-implemented method for encoding a three-dimensional audio signal, comprising: obtaining a first correlation between a sixth amount of representative virtual speakers in a current frame and a previous frame of the three-dimensional audio signal, the sixth amount of representative virtual speakers being used to encode the previous frame, and the first correlation being used to decide whether to reuse the representative virtual speakers in the previous frame when the current frame is encoded; if the first correlation does not satisfy a reuse condition, obtaining a fourth quantity coefficient in a current frame of the three-dimensional audio signal and a frequency domain feature value of the fourth quantity coefficient; Selecting an integer number of representative coefficients from at least one subband included in a spectral range indicated by the coefficients of the fourth amount according to a descending order of the frequency domain characteristic values ​​of the coefficients in each subband, and selecting a representative coefficient of a third amount less than the fourth amount from the coefficients of the fourth amount; selecting a representative virtual speaker of a second amount in the current frame from a set of candidate virtual speakers based on a representative factor of the third amount; encoding the current frame based on the second amount of representative virtual speakers in the current frame to obtain a bitstream; The three-dimensional audio signal encoding method further comprises:

2. When the at least one subband includes at least two subbands, the step of selecting an integer number of representative coefficients from at least one subband included in a spectral range indicated by the coefficients of the fourth amount according to a descending order of frequency domain characteristic values ​​of coefficients in each subband, and selecting a representative coefficient of a third amount less than the fourth amount from the coefficients of the fourth amount includes: determining weights for each of the at least two subbands based on frequency domain feature values ​​of a first candidate coefficient in each subband; adjusting a frequency domain feature value of a second candidate coefficient in each subband based on the weight of each subband to obtain an adjusted frequency domain feature value of the second candidate coefficient in each subband, where the first candidate coefficient and the second candidate coefficient are some coefficients in the subband; determining a representative coefficient of the third quantity based on the adjusted frequency domain feature values ​​of a second candidate coefficient in the at least two subbands and frequency domain feature values ​​of coefficients other than the second candidate coefficient in the at least two subbands; 2. The method of claim 1, comprising:

3. If the first correlation satisfies a reuse condition, encoding the current frame based on the sixth amount of representative virtual speakers in the previous frame to obtain a bitstream; 3. The method of claim 1 or 2, further comprising:

4. The method according to claim 1 or 2, wherein the current frame of the three-dimensional audio signal is a Higher Order Ambisonics HOA signal, and the frequency domain feature values ​​of the coefficients are determined based on the coefficients of the HOA signal.

5. A virtual speaker selection module configured to obtain a first correlation between a sixth amount of representative virtual speaker sets in a current frame and a previous frame of a three-dimensional audio signal, the sixth amount of virtual speakers being used to encode the previous frame, and the first correlation being used to determine whether to reuse the representative virtual speaker set in the previous frame when the current frame is encoded; and if the first correlation does not satisfy a reuse condition, obtaining a fourth quantity coefficient in a current frame of the three-dimensional audio signal and a frequency domain feature value of the fourth quantity coefficient; a coefficient selection module configured to select an integer number of representative coefficients from at least one subband included in a spectral range indicated by the coefficients of the fourth amount according to a descending order of frequency domain feature values ​​of coefficients in each subband, and select a representative coefficient of a third amount less than the fourth amount from the coefficients of the fourth amount; The virtual speaker selection module is further configured to select a second amount of representative virtual speakers in the current frame from a candidate virtual speaker set based on a representative coefficient of the third amount when the first correlation does not satisfy a reuse condition; and an encoding module configured to encode the current frame based on the second amount of representative virtual speakers in the current frame to obtain a bitstream if the first correlation does not satisfy a reuse condition.

6. When the at least one subband includes at least two subbands, the coefficient selection module selects an integer number of representative coefficients from at least one subband included in a spectral range indicated by the coefficients of the fourth amount according to a descending order of the frequency domain feature values ​​of the coefficients in each subband, and selects a representative coefficient of a third amount less than the fourth amount from the coefficients of the fourth amount, determining weights for each of the at least two subbands based on frequency domain feature values ​​of a first candidate coefficient in each subband; Adjusting a frequency domain feature value of a second candidate coefficient in each subband according to the weight of each subband to obtain an adjusted frequency domain feature value of the second candidate coefficient in each subband, wherein the first candidate coefficient and the second candidate coefficient are some coefficients in the subband; determining a representative coefficient of the third quantity based on the adjusted frequency domain feature values ​​of a second candidate coefficient in the at least two subbands and frequency domain feature values ​​of coefficients other than the second candidate coefficient in the at least two subbands; The apparatus according to claim 5, specifically adapted for:

7. The encoding module includes: encoding the current frame based on the sixth amount of representative virtual speakers in the previous frame to obtain a bitstream if the first correlation satisfies a reuse condition; 7. The apparatus of claim 5 or 6, further configured to:

8. 7. The apparatus according to claim 5 or 6, wherein the current frame of the three-dimensional audio signal is a Higher Order Ambisonics HOA signal, and the frequency domain feature values ​​of the coefficients are determined based on the coefficients of the HOA signal.

9. 2. An encoder comprising at least one processor and a memory, the memory being configured to store a computer program such that, when executed by the at least one processor, the method for encoding a three-dimensional audio signal according to claim 1 is performed.

10. A system comprising an encoder according to claim 9 and a decoder, the encoder configured to perform the operational steps of the method according to claim 1, the decoder configured to decode a bitstream generated by the encoder.

11. A computer program, which, when executed, performs the method for encoding a three-dimensional audio signal according to claim 1.

12. 20. A computer-readable storage medium comprising a computer program, the computer program being configured, when executed in an encoder, to enable the encoder to perform the three-dimensional audio signal encoding method of claim 1.

13. A computer implemented method for storing a bitstream, comprising the steps of: obtaining a first correlation between a sixth amount of representative virtual speakers in a current frame and a previous frame of the three-dimensional audio signal, the sixth amount of representative virtual speakers being used to encode the previous frame, the first correlation being used to decide whether to reuse the representative virtual speakers in the previous frame when the current frame is encoded; if the first correlation does not satisfy a reuse condition, obtaining a fourth quantity coefficient in a current frame of the three-dimensional audio signal and a frequency domain feature value of the fourth quantity coefficient; selecting an integer number of representative coefficients from at least one subband included in a spectral range indicated by the coefficients of the fourth amount according to a descending order of the frequency domain characteristic values ​​of the coefficients in each subband based on the frequency domain characteristic values ​​of the coefficients of the fourth amount, and selecting a representative coefficient of a third amount less than the fourth amount from the coefficients of the fourth amount, wherein the third amount is less than the fourth amount; selecting a representative virtual speaker of a second amount in the current frame from a set of candidate virtual speakers based on a representative factor of the third amount; encoding the current frame based on the second amount of representative virtual speakers in the current frame to obtain a bitstream; If the first correlation satisfies a reuse condition, encoding the current frame based on the sixth amount of representative virtual speakers in the previous frame to obtain a bitstream; The method of storage further comprising the step of storing the bitstream in a computer readable storage medium.

Images