A method and device for audio encoding and decoding
By using the encoding parameters of the previous frame to determine the encoding parameters of the current frame, the problem of high encoding complexity in HOA technology is solved, and the encoding efficiency and simplicity of data processing are improved.
Patent Information
- Application Number
- CN202110530309.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-05-14
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2041-05-14
Smart Images

Figure CN115346537B_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present application relate to the field of coding and decoding technology, and in particular, to an audio coding and decoding method and device. Background Art
[0002] Three-dimensional audio technology is an audio technology that acquires, processes, transmits, renders and plays back sound events and three-dimensional sound field information in the real world. Three-dimensional audio technology gives sound a strong sense of space, envelopment and immersion, giving people an extraordinary auditory experience of "being there". Higher-order ambisonics (HOA) technology has the properties of being independent of speaker layout during recording, encoding and playback, and the rotatable playback characteristics of HOA format data. It has higher flexibility when playing back three-dimensional audio, and has therefore received more extensive attention and research.
[0003] In order to achieve better audio hearing effects, HOA technology requires a large amount of data to record more detailed sound scene information. Although this three-dimensional audio signal sampling and storage based on the scene is more conducive to the preservation and transmission of audio signal spatial information, as the HOA order increases, the amount of data will also increase. A large amount of data makes transmission and storage difficult, so the HOA signal needs to be encoded and decoded.
[0004] The HOA signal to be encoded generates a virtual speaker signal and a residual signal through encoding, and then further encodes the virtual speaker signal and the residual signal to obtain a bit stream. Generally, when encoding the virtual speaker signal and the residual signal, the virtual speaker signal and the residual signal of each frame are coded and decoded. However, only the correlation between the signals of the current frame is considered, and the virtual speaker signal and the residual signal of each frame are encoded, resulting in high computational complexity and low coding efficiency. Summary of the invention
[0005] The embodiments of the present application provide an audio encoding and decoding method and device to solve the problem of high computational complexity.
[0006] In the first aspect, an embodiment of the present application provides an audio encoding method, including: obtaining an audio channel signal of a current frame, wherein the audio channel signal of the current frame is obtained by spatially mapping an original high-order stereo reverberation HOA signal through a first target virtual speaker; when it is determined that the first target virtual speaker and the second target virtual speaker meet a set condition, determining the first encoding parameter of the audio channel signal of the current frame according to the second encoding parameter of the audio channel signal of the previous frame of the current frame, wherein the audio channel signal of the previous frame corresponds to the second target virtual speaker; encoding the audio channel signal of the current frame according to the first encoding parameter; and writing the encoding result of the audio channel signal of the current frame into a bitstream. Through the above method, when encoding the current frame, if the virtual speakers matching the previous frame are adjacent, the encoding parameters of the current frame can be determined according to the encoding parameters of the previous frame, so that there is no need to recalculate the encoding parameters of the current frame, which can improve the encoding efficiency.
[0007] In one possible design, the method further includes: writing the first encoding parameter into the bitstream. In the above design, the encoding parameter determined according to the encoding parameter of the previous frame is written into the bitstream as the encoding parameter of the current frame, so that the other end obtains the encoding parameter while improving the encoding efficiency.
[0008] In one possible design, the first encoding parameter includes one or more of an inter-channel pairing parameter, an inter-channel auditory space parameter, or an inter-channel bit allocation parameter.
[0009] In a possible design, the inter-channel auditory space parameter includes one or more of an inter-channel level difference ILD, an inter-channel time difference ITD, or an inter-channel phase difference IPD.
[0010] In a possible design, the setting condition includes that the first spatial position overlaps with the second spatial position; and the determining the first encoding parameter of the audio channel signal of the current frame according to the second encoding parameter of the audio channel signal of the previous frame includes: using the second encoding parameter of the audio channel signal of the previous frame as the first encoding parameter of the audio channel signal of the current frame. Through the above design, when the spatial position of the target virtual speaker of the previous frame overlaps with the spatial position of the target virtual speaker of the current frame, the encoding parameters of the previous frame are reused as the encoding parameters of the current frame. Considering the inter-frame spatial correlation between the audio channel signals, there is no need to calculate the encoding parameters of the current frame, which can improve the encoding efficiency.
[0011] In one possible design, the method further includes: writing a multiplexing identifier into a bitstream, wherein the value of the multiplexing identifier is a first value, and the first value indicates that the first encoding parameter of the audio channel signal of the current frame multiplexes the second encoding parameter. In the above design, the method of notifying the decoding side to determine the encoding parameters of the current frame by writing the multiplexing identifier into the bitstream is simple and effective.
[0012] In one possible design, the first spatial position includes the first coordinate of the first target virtual speaker, the second spatial position includes the second coordinate of the second target virtual speaker, and the overlap of the first spatial position and the second spatial position includes the first coordinate being the same as the second coordinate; or the first spatial position includes the first serial number of the first target virtual speaker, the second spatial position includes the second serial number of the second target virtual speaker, and the overlap of the first spatial position and the second spatial position includes the first serial number being the same as the second serial number; or the first spatial position includes the first HOA coefficient of the first target virtual speaker, the second spatial position includes the second HOA coefficient of the second target virtual speaker, and the overlap of the first spatial position and the second spatial position includes the first HOA coefficient being the same as the second HOA coefficient. In the above design, the spatial position is characterized by coordinates, serial numbers or HOA coefficients to determine whether the virtual speaker of the previous frame overlaps with the virtual speaker of the current frame, which is simple and effective.
[0013] In a possible design, the first target virtual speaker includes M virtual speakers, and the second target virtual speaker includes N virtual speakers; the setting condition includes that the first spatial position of the first target virtual speaker does not overlap with the second spatial position of the second target virtual speaker and the mth virtual speaker included in the first target virtual speaker is located within a set range centered on the nth virtual speaker included in the second target virtual speaker, wherein m traverses a positive integer less than or equal to M, and n traverses a positive integer less than or equal to N; the first encoding parameter of the audio channel signal of the current frame is determined according to the second encoding parameter of the audio channel signal of the previous frame, including: adjusting the second encoding parameter according to the set ratio to obtain the first encoding parameter. In the above design, when the spatial position of the target virtual speaker of the previous frame does not overlap with the spatial position of the target virtual speaker of the current frame but is adjacent, the encoding parameter of the current frame is adjusted by the encoding parameter of the previous frame. Considering the inter-frame spatial correlation between the audio channel signals, there is no need to calculate the encoding parameter of the current frame by a complex calculation method, which can improve the encoding efficiency.
[0014] In which, in the embodiment of the present invention, the first coding parameter can be one coding parameter or multiple coding parameters, and the adjustment can be reduction, or enlargement, or partial reduction and another part unchanged, or partial enlargement and another part unchanged, or partial reduction and another part enlargement, or partial reduction, partial unchanged and partial enlargement.
[0015] In a possible design, when the first spatial position includes the first coordinates of the first target virtual speaker and the second spatial position includes the second coordinates of the second target virtual speaker, whether the mth virtual speaker is located within a set range centered on the nth virtual speaker is determined by a correlation between the mth virtual speaker and the nth virtual speaker, wherein the correlation satisfies the following conditions:
[0016]
[0017] Among them, R represents the correlation, norm() represents the normalization operation, and M H A matrix consisting of the coordinates of the virtual speakers included in the first target virtual speaker of the current frame, is the transpose of the matrix composed of the coordinates of the virtual speakers included in the second target virtual speaker of the previous frame; when the correlation is greater than the set value, the mth virtual speaker is located within the set range centered on the nth virtual speaker. The above design provides a simple and effective way to determine the proximity relationship between the virtual speakers of the previous frame and the virtual speakers of the current frame.
[0018] In one possible design, the method further includes: writing a multiplexing identifier into a bit stream, the value of the multiplexing identifier being a second value, the second value indicating that a first encoding parameter of an audio channel signal of the current frame is obtained by adjusting the second encoding parameter according to a set ratio.
[0019] In one possible design, the method further includes: writing the set ratio into the bitstream. Through the above design, the set ratio is notified to the decoding side through the bitstream, so that the decoding side determines the encoding parameters of the current frame according to the set ratio, so that the decoding side obtains the encoding parameters while improving the encoding efficiency.
[0020] In a second aspect, an embodiment of the present application provides an audio decoding method, comprising: parsing a multiplexing identifier from a bitstream, wherein the multiplexing identifier indicates that a first encoding parameter of an audio channel signal of a current frame is determined by a second encoding parameter of an audio channel signal of a previous frame of the current frame; determining the first encoding parameter according to the second encoding parameter of the audio channel signal of the previous frame; and decoding the audio channel signal of the current frame from the bitstream according to the first encoding parameter. Through the above design, the decoding side does not need to parse the encoding parameters from the bitstream, which can improve the decoding efficiency.
[0021] In one possible design, the first encoding parameter is determined according to the second encoding parameter of the audio channel signal of the previous frame, including: when the value of the multiplexing identifier is a first value, the first value indicates that the first encoding parameter multiplexes the second encoding parameter, and the second encoding parameter is obtained as the first encoding parameter. Through the above design, there is no need to decode each encoding parameter from the bitstream, only the multiplexing identifier needs to be decoded, which can improve decoding efficiency.
[0022] In one possible design, the first encoding parameter is determined according to the second encoding parameter of the audio channel signal of the previous frame, including: when the value of the multiplexing identifier is a second value, the second value indicates that the first encoding parameter is obtained by adjusting the second encoding parameter according to a set ratio, and the first encoding parameter is obtained by adjusting the second encoding parameter according to the set ratio.
[0023] In one possible design, the method further includes: when the value of the multiplexing identifier is a second value, decoding from the code stream to obtain the set ratio.
[0024] In a possible design, the encoding parameters of the audio channel signals include one or more of inter-channel pairing parameters, inter-channel auditory space parameters, or inter-channel bit allocation parameters.
[0025] In the third aspect, the embodiment of the present application provides an audio encoding device, and the beneficial effects can be found in the relevant description of the first aspect, which will not be repeated here. The audio encoding device includes several functional units for implementing any one of the methods of the first aspect. For example, the audio encoding device may include a spatial encoding unit for obtaining an audio channel signal of a current frame, wherein the audio channel signal of the current frame is obtained by spatially mapping the original high-order stereo reverberation HOA signal by a first target virtual speaker; a core encoding unit for determining the first encoding parameter of the audio channel signal of the current frame according to the second encoding parameter of the audio channel signal of the previous frame of the current frame when it is determined that the first target virtual speaker and the second target virtual speaker meet the set conditions, and the audio channel signal of the previous frame corresponds to the second target virtual speaker; the audio channel signal of the current frame is encoded according to the first encoding parameter, and the encoding result of the audio channel signal of the current frame is written into the bitstream.
[0026] In a possible design, the core encoding unit is also used to write the first encoding parameter into the bit stream.
[0027] In one possible design, the first encoding parameter includes one or more of an inter-channel pairing parameter, an inter-channel auditory space parameter, or an inter-channel bit allocation parameter.
[0028] In one possible design, the set condition includes that the first spatial position of the first target virtual speaker overlaps with the second spatial position of the second target virtual speaker; and the core encoding unit is specifically used to use the second encoding parameter of the audio channel signal of the previous frame as the first encoding parameter of the audio channel signal of the current frame.
[0029] In one possible design, the core encoding unit is also used to write a multiplexing identifier into a bit stream, and the value of the multiplexing identifier is a first value, and the first value indicates that the first encoding parameter of the audio channel signal of the current frame multiplexes the second encoding parameter.
[0030] In one possible design, the first spatial position includes the first coordinate of the first target virtual speaker, the second spatial position includes the second coordinate of the second target virtual speaker, and the overlap of the first spatial position and the second spatial position includes the first coordinate being the same as the second coordinate; or the first spatial position includes the first serial number of the first target virtual speaker, the second spatial position includes the second serial number of the second target virtual speaker, and the overlap of the first spatial position and the second spatial position includes the first serial number being the same as the second serial number; or the first spatial position includes the first HOA coefficient of the first target virtual speaker, the second spatial position includes the second HOA coefficient of the second target virtual speaker, and the overlap of the first spatial position and the second spatial position includes the first HOA coefficient being the same as the second HOA coefficient.
[0031] In one possible design, the first target virtual speaker includes M virtual speakers, and the second target virtual speaker includes N virtual speakers; the set conditions include that the first spatial position of the first target virtual speaker does not overlap with the second spatial position of the second target virtual speaker and the mth virtual speaker included in the first target virtual speaker is located within a set range centered on the nth virtual speaker included in the second target virtual speaker, wherein m traverses a positive integer less than or equal to M, and n traverses a positive integer less than or equal to N; the core encoding unit is specifically used to adjust the second encoding parameter according to a set ratio to obtain the first encoding parameter.
[0032] In a possible design, when the first spatial position includes the first coordinates of the first target virtual speaker and the second spatial position includes the second coordinates of the second target virtual speaker, whether the mth virtual speaker is located within a set range centered on the nth virtual speaker is determined by a correlation between the mth virtual speaker and the nth virtual speaker, wherein the correlation satisfies the following conditions:
[0033]
[0034] Among them, R represents the correlation, norm() represents the normalization operation, and M H A matrix consisting of the coordinates of the virtual speakers included in the first target virtual speaker of the current frame, The transpose of the matrix consisting of the coordinates of the virtual speakers included in the second target virtual speaker of the previous frame;
[0035] When the correlation is greater than a set value, the mth virtual speaker is located within a set range centered on the nth virtual speaker.
[0036] In one possible design, the core encoding unit is also used to write a multiplexing identifier into a bit stream, and the value of the multiplexing identifier is a second value, and the second value indicates that the first encoding parameter of the audio channel signal of the current frame is obtained by adjusting the second encoding parameter according to a set ratio.
[0037] In a possible design, the core encoding unit is also used to write the set ratio into the bit stream.
[0038] In the fourth aspect, an embodiment of the present application provides an audio decoding device, and the beneficial effects can be found in the relevant description of the second aspect, which will not be repeated here. The audio decoding device includes several functional units for implementing any one of the methods of the third aspect. For example, the audio decoding device may include: a core decoding unit, used to parse a multiplexing identifier from a bitstream, wherein the multiplexing identifier indicates that a first encoding parameter of an audio channel signal of a current frame is determined by a second encoding parameter of an audio channel signal of a previous frame of the current frame; the first encoding parameter is determined according to the second encoding parameter of the audio channel signal of the previous frame; the audio channel signal of the current frame is decoded from the bitstream according to the first encoding parameter; a spatial decoding unit, used to spatially decode the audio channel signal to obtain a high-order stereo reverberation HOA signal.
[0039] In one possible design, the core decoding unit is specifically used to, when the value of the multiplexing identifier is a first value, indicate that the first encoding parameter multiplexes the second encoding parameter, and obtain the second encoding parameter as the first encoding parameter.
[0040] In one possible design, the core decoding unit is specifically used to, when the value of the multiplexing identifier is a second value, the second value indicates that the first encoding parameter is obtained by adjusting the second encoding parameter according to a set ratio, and the first encoding parameter is obtained by adjusting the second encoding parameter according to the set ratio.
[0041] In one possible design, the core decoding unit is specifically used to decode from the code stream to obtain the set ratio when the value of the multiplexing identifier is a second value.
[0042] In a possible design, the encoding parameters of the audio channel signals include one or more of inter-channel pairing parameters, inter-channel auditory space parameters, or inter-channel bit allocation parameters.
[0043] In a fifth aspect, an embodiment of the present application provides an audio encoder, wherein the video encoder is used to encode the HOA signal. Exemplarily, the audio encoder can implement the method described in the first aspect. The audio encoder can include the device described in any design of the third aspect.
[0044] In a sixth aspect, an embodiment of the present application provides an audio decoder, wherein the video decoder is used to decode the HOA signal from the bitstream. Exemplarily, the audio decoder can implement the method described in any one of the designs of the second aspect. The audio decoder includes the device described in any one of the designs of the fourth aspect.
[0045] In a seventh aspect, an embodiment of the present application provides an audio encoding device, comprising: a non-volatile memory and a processor coupled to each other, wherein the processor calls a program code stored in the memory to execute the method described in the first aspect or any design of the first aspect.
[0046] In an eighth aspect, an embodiment of the present application provides an audio decoding device, comprising: a non-volatile memory and a processor coupled to each other, wherein the processor calls a program code stored in the memory to execute the method described in the second aspect or any design of the second aspect.
[0047] In a ninth aspect, an embodiment of the present application provides a computer-readable storage medium storing a program code, wherein the program code includes instructions for executing part or all of the steps of any one of the methods of the first to second aspects.
[0048] In a tenth aspect, an embodiment of the present application provides a computer program product, which, when executed on a computer, enables the computer to execute part or all of the steps of any one of the methods from the first aspect to the second aspect.
[0049] In an eleventh aspect, an embodiment of the present application provides a computer-readable storage medium, comprising a code stream obtained by any method of the first aspect.
[0050] It should be understood that the beneficial effects of the third to tenth aspects of the present application can be found in the relevant descriptions of the first and second aspects and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS
[0051] Figure 1A is a schematic block diagram of an audio encoding and decoding system 100 in an embodiment of the present application;
[0052] Figure 1B This is a schematic block diagram of the audio encoding and decoding process in an embodiment of the present application;
[0053] Figure 1C This is a schematic block diagram of another audio encoding and decoding system in an embodiment of the present application;
[0054] Figure 1D This is a schematic block diagram of another audio encoding and decoding system in an embodiment of the present application;
[0055] Figure 2A This is a schematic diagram of the structure of the audio encoding component in an embodiment of the present application;
[0056] Figure 2B This is a schematic diagram of the structure of the audio decoding component in the embodiment of the present application;
[0057] Figure 3A This is a flowchart of an audio encoding method in an embodiment of the present application;
[0058] Figure 3B This is a flowchart of another audio encoding method in an embodiment of the present application;
[0059] Figure 4A This is a flowchart of an audio encoding and decoding method in an embodiment of the present application;
[0060] Figure 4B This is a flowchart of another audio encoding and decoding method in an embodiment of the present application;
[0061] Figure 5 This is a schematic block diagram of the audio encoding process in an embodiment of the present application;
[0062] Figure 6 This is a schematic diagram of an audio encoding device in an embodiment of the present application;
[0063] Figure 7 Schematic diagram of an audio decoding device in an embodiment of the present application. DETAILED DESCRIPTION
[0064] The following is a description of the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. In the following description, reference is made to the drawings that form a part of the present disclosure and show the specific aspects of the embodiments of the present application or the specific aspects of the embodiments of the present application in an illustrative manner. It should be understood that the embodiments of the present application can be used in other aspects and may include structural or logical changes not depicted in the drawings. Therefore, the following detailed description should not be understood in a restrictive sense, and the scope of the present application is defined by the appended claims. For example, it should be understood that the disclosure of the described method can be equally applicable to the corresponding device or system for performing the method, and vice versa. For example, if one or more specific method steps are described, the corresponding device may include one or more units such as functional units to perform the one or more method steps described (for example, one unit performs one or more steps, or multiple units, each of which performs one or more of the multiple steps), even if such one or more units are not explicitly described or illustrated in the drawings. On the other hand, for example, if a specific device is described in terms of one or more units such as functional units, the corresponding method may include a step to perform the functionality of the one or more units (e.g., a step to perform the functionality of the one or more units, or multiple steps, each of which performs the functionality of one or more of the multiple units), even if such one or more steps are not explicitly described or illustrated in the drawings. Further, it should be understood that, unless otherwise explicitly stated, the features of the various exemplary embodiments and / or aspects described herein may be combined with each other.
[0065] The words "first", "second" and similar terms mentioned herein do not indicate any order, quantity or importance, but are only used to distinguish different components. Similarly, the words "a" or "an" and similar terms do not indicate a quantity limitation, but indicate the existence of at least one. The words "connected" or "connected" and similar terms are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect.
[0066] The term "multiple" as used herein refers to two or more than two. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the related objects are in an "or" relationship.
[0067] The following describes the system architecture used in the embodiments of the present application. Figure 1A As shown, Figure 1A A schematic block diagram of an audio encoding and decoding system 100 used in an embodiment of the present application is given as an example. Figure 1AAs shown, the audio encoding and decoding system 100 may include an audio encoding component 110 and an audio decoding component 120. The audio encoding component 110 is used to perform audio encoding on the HOA signal (or 3D audio signal). Optionally, the audio encoding component 110 may be implemented by software, or may be implemented by hardware, or may be implemented by a combination of software and hardware, which is not specifically limited in the embodiments of the present application.
[0068] See also Figure 1B As shown, the audio encoding component 110 may encode the HOA signal (or 3D audio signal) by the following steps:
[0069] 1) Performing audio preprocessing on the obtained HOA signal. The preprocessing may include filtering out the low-frequency part of the HOA signal, for example, taking 20 Hz or 50 Hz as a cutoff point, and extracting the azimuth information in the HOA signal.
[0070] The HOA signal may be collected by the audio collection component and sent to the audio encoding component 110. Optionally, the audio collection component may be arranged in the same device as the audio encoding component 110; or, the audio collection component may be arranged in a different device from the audio encoding component 110.
[0071] 2) Encode (Audio encoding) and package (File / Segmentencapsulation) the audio preprocessed signal to obtain a bit stream.
[0072] 3) The audio encoding component 110 delivers the code stream to the audio decoding component 120 at the decoding end through a transmission channel.
[0073] The audio decoding component 120 is used to decode the code stream generated by the audio encoding component 110 to obtain the HOA signal.
[0074] Optionally, the audio encoding component 110 and the audio decoding component 120 may be connected by wire or wirelessly. The audio decoding component 120 obtains the bit stream generated by the audio encoding component 110 through the connection; or, the audio encoding component 110 stores the generated bit stream in the memory, and the audio decoding component 120 reads the bit stream in the memory. Optionally, the audio decoding component 120 may be implemented by software; or, it may also be implemented by hardware; or, it may also be implemented by a combination of software and hardware, which is not limited in the embodiments of the present application.
[0075] The audio decoding component 120 decodes the bit stream to obtain the HOA signal, which may include the following steps:
[0076] 1) Unpack the code stream (File / Segment decapsulation).
[0077] 2) Performing audio decoding on the unpacked signal to obtain a decoded signal.
[0078] 3) Perform audio rendering on the decoded signal.
[0079] 4) The rendered signal is mapped to the listener's headphones or speakers. The listener's headphones can be independent headphones or headphones on terminal devices such as glasses.
[0080] Optionally, the audio encoding component 110 and the audio decoding component 120 may be arranged in the same device; or, they may also be arranged in different devices. The device may be a mobile terminal with audio signal processing function, such as a mobile phone, a tablet computer, a laptop computer and a desktop computer, a Bluetooth speaker, a voice recorder, a wearable device, etc., or a network element with audio signal processing capability in a core network or a wireless network, such as a media gateway, a transcoding device, a media resource server, etc., or an audio codec used in a virtual reality (VR) streaming service, which is not limited in the embodiments of the present application.
[0081] Schematically, refer to Figure 1C In this embodiment, the audio encoding component 110 is set in the mobile terminal 130, and the audio decoding component 120 is set in the mobile terminal 140. The mobile terminal 130 and the mobile terminal 140 are independent electronic devices with audio signal processing capabilities, and the mobile terminal 130 and the mobile terminal 140 are connected via a wireless or wired network.
[0082] Optionally, the mobile terminal 130 includes an audio collection component 131 , an audio encoding component 110 and a channel encoding component 132 , wherein the audio collection component 131 is connected to the audio encoding component 110 , and the audio encoding component 110 is connected to the audio encoding component 132 .
[0083] Optionally, the mobile terminal 140 includes an audio playback component 141, an audio decoding component 120 and a channel decoding component 142, wherein the audio playback component 141 is connected to the audio decoding component 120, and the audio decoding component 120 is connected to the channel coding component 132. After the mobile terminal 130 collects the HOA signal through the audio collection component 131, the audio coding component 110 encodes the HOA signal to obtain a coded stream; then, the channel coding component 132 encodes the coded stream to obtain a transmission signal.
[0084] Mobile terminal 130 sends the transmission signal to mobile terminal 140 via a wireless or wired network, for example, the transmission signal may be sent to mobile terminal 140 via a communication device of a wireless or wired network. The communication devices of the wired or wireless network to which mobile terminal 130 and mobile terminal 140 belong may be the same or different.
[0085] After receiving the transmission signal, the mobile terminal 140 decodes the transmission signal through the channel decoding component 142 to obtain a coded code stream (which can be referred to as a code stream for short); decodes the coded code stream through the audio decoding component 120 to obtain an HOA signal; and plays the HOA signal through the audio playing component.
[0086] Schematically, refer to Figure 1D In the embodiment of the present application, an example is given in which the audio encoding component 110 and the audio decoding component 120 are arranged in a network element 150 having audio signal processing capability in the same core network or wireless network.
[0087] Optionally, the network element 150 includes a channel decoding component 151, an audio decoding component 120, an audio encoding component 110, and a channel encoding component 152. The channel decoding component 151 is connected to the audio decoding component 120, the audio decoding component 120 is connected to the audio encoding component 110, and the audio encoding component 110 is connected to the channel encoding component 152.
[0088] After receiving the transmission signal sent by other devices, the channel decoding component 151 decodes the transmission signal to obtain a first coded code stream; decodes the first coded code stream through the audio decoding component 120 to obtain an HOA signal; encodes the HOA signal through the audio encoding component 110 to obtain a second coded code stream; and encodes the second coded code stream through the channel encoding component 152 to obtain a transmission signal.
[0089] The other device may be a mobile terminal with audio signal processing capability; or may be other network elements with audio signal processing capability, which is not limited in this embodiment.
[0090] Optionally, the audio encoding component 110 and the audio decoding component 120 in the network element may transcode the encoded code stream sent by the mobile terminal.
[0091] Optionally, in this embodiment, the device equipped with the audio encoding component 110 is referred to as an audio encoding device. In actual implementation, the audio encoding device may also have an audio decoding function, which is not limited in this embodiment of the present application. The device equipped with the audio decoding component 120 may be referred to as an audio decoding device.
[0092] Schematically, see Figure 2AAs shown, the audio encoding component 110 may include a spatial encoder 210 and a core encoder 220. The HOA signal to be encoded is encoded by the spatial encoder 210 to obtain an audio channel signal, that is, the HOA to be encoded is passed through the spatial encoder 210 to generate a virtual speaker signal and a residual signal; the core encoder 220 encodes the audio channel signal to obtain a bit stream.
[0093] Schematically, see Figure 2B As shown, the audio decoding component 120 may include a core decoder 230 and a spatial decoder 240. After receiving the bitstream, the core decoder 230 decodes the bitstream to obtain an audio channel signal; then the spatial decoder 240 obtains a reconstructed HOA signal based on the decoded audio channel signal (virtual speaker signal and residual signal).
[0094] As an example, the spatial encoder 210 and the core encoder 220 may be two independent processing units. The spatial decoder 240 and the core decoder 230 may be two independent processing units. The core encoder 220 generally encodes the audio channel signal as a plurality of single channel signals or stereo channel signals or multi-channel signals.
[0095] The core encoder 220 encodes the audio channel signal of each frame. One possible method is to calculate the encoding parameters of the audio channel signal of each frame, and then encode the audio channel signal of the current frame according to the calculated encoding parameters and write them into the bitstream, and write the encoding parameters into the bitstream. However, this method only considers the correlation between audio channel signals and ignores the spatial correlation between frames of audio channel signals, resulting in low encoding efficiency.
[0096] Since the audio channel signal is obtained by mapping the target virtual speaker on the original HOA signal, there is a certain connection between the inter-frame correlation of the audio channel signal and the selection of the virtual speaker of the HOA signal. When the spatial positions of the virtual speakers are the same or adjacent, the audio channel signal has a strong correlation between the frames. Based on this, considering the inter-frame correlation of the audio channel signal, an embodiment of the present application provides a coding and decoding method, through the proximity relationship between the virtual speaker corresponding to the current frame and the virtual speaker corresponding to the previous frame, if they are adjacent or overlapped in position, the coding parameters of the current frame can be determined according to the coding parameters of the previous frame, so that the coding parameters of the current frame are no longer calculated by the calculation algorithm of each coding parameter, which can improve the coding efficiency.
[0097] Before describing the coding and decoding scheme provided by the embodiment of the present application in detail, some concepts that may be involved in the embodiment of the present application are briefly introduced below. The terms used in the implementation method of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.
[0098] (1) The HOA signal is a three-dimensional (3D) representation of the sound field. The HOA signal is usually represented by multiple spherical harmonic coefficients (SHC) or other hierarchical elements. According to the HOA theory, for an ideal signal with a specific direction (for example, a point sound source signal or a plane wave signal in the far field), the corresponding HOA signal only has amplitude differences between the channels, so it can be represented by a single-channel signal and a set of proportional coefficients corresponding to each channel. In HOA technology, the HOA signal is usually converted into an actual speaker signal for playback, or the HOA signal is converted into a virtual loudspeaker (VL) signal and then mapped to the speaker signal corresponding to both ears for playback. The selection of the (virtual) speaker is crucial to the quality of the reconstructed signal.
[0099] (2) The current frame refers to a certain length of sample points obtained by collecting audio signals, such as 960 points or 1024 points. The previous frame refers to the frame before the current frame. For example, if the current frame is the nth frame, the previous frame is the n-1th frame. The previous frame can also be called the previous frame.
[0100] (3) Audio channel signals may include multi-channel virtual speaker signals, or include multi-channel virtual speaker signals and residual signals. For example, the HOA signal to be encoded is mapped to multiple virtual speakers to obtain a multi-channel virtual speaker signal and a residual signal. The channel data of the virtual speaker and the number of channels of the residual signal may be pre-set. The audio channel signal may also be referred to as a transmission channel, and other names may also be used, which are not specifically limited in the present application. As an example, the virtual speaker signal may be obtained by selecting a target virtual speaker that matches the HOA signal of the current frame to be encoded from a virtual speaker set according to a matching projection algorithm, and obtaining the virtual speaker signal according to the HOA signal of the current frame and the selected target virtual speaker. The residual signal may be obtained based on the HOA signal to be encoded and the virtual speaker signal.
[0101] (4) Coding parameters: For example, the coding parameters may include one or more of an inter-channel pairing parameter, an inter-channel auditory space parameter, or an inter-channel bit allocation parameter.
[0102] The inter-channel pairing parameter is used to characterize the pairing relationship (or grouping relationship) between the channels to which the multiple audio signals included in the audio channel signal belong. Inter-channel pairing is a calculation method for pairing the transmission channels of the audio signal by correlation and other criteria to achieve efficient coding of the transmission channel.
[0103] As an example, the audio channel signal may include a virtual speaker signal and a residual signal. The following is an exemplary description of the determination of the inter-channel configuration parameters:
[0104] For example, the audio channel signal can be divided into two groups, the virtual speaker signal is a group, called the virtual speaker signal group, and the residual signal is a group, called the residual signal group. The virtual speaker signal group includes M virtual speaker signals composed of a single channel, M is a positive integer greater than 2, and the residual signal group includes N residual signals composed of a single channel, N is a positive integer greater than 2. For example, M=4, N=4. The channel pairing result can be a pairing of two channels, or a pairing of three or more channels, or an unpaired pairing of channels. Taking the pairing of two channels as an example, the channel pairing parameter refers to the selection result of a pair of different signals in each group. Taking the virtual speaker signal group as an example, for example, the virtual speaker signal group includes 4 channels, namely channel 1, channel 2, channel 3, and channel 4. For example, the channel pairing parameter can be a pairing of channel 1 and channel 2, a pairing of channel 3 and channel 4, or a pairing of channel 1 and channel 3, a pairing of channel 2 and channel 4, or a pairing of channel 1 and channel 2, and an unpairing of channel 3 and channel 4. The method for determining the channel pairing parameter is not specifically limited in this application. As an example, the channel pairing parameters may be determined by constructing a channel correlation matrix W, for example, see formula (1):
[0105]
[0106] Among them, m11-m44 represent the correlation between two channels. Further, the diagonal elements of the matrix are set to 0 to obtain W', see formula (2):
[0107]
[0108] The principle of pairing between channels may be the sequence number of the element in W′ when it reaches the maximum value, and in this case, the pairing parameter between channels may be the sequence number of the matrix element.
[0109] The inter-channel auditory space parameter is used to characterize the degree of human ear's perception of the acoustic image characteristics of the auditory space. Exemplarily, the inter-channel auditory space parameter may include one or more of the inter-channel level difference (ILD) (also referred to as the inter-channel level difference), the inter-channel time difference (ITD) (also referred to as the inter-channel time difference), or the inter-channel phase difference (IPD) (also referred to as the inter-channel phase difference).
[0110] Taking the ILD parameter as an example, the ILD parameter may be the ratio of the signal energy of each channel in the audio channel signal to the average energy of all channels. As an example, the ILD parameter may be composed of two parameters: the absolute value of the ratio of each channel and the adjustment direction value. The embodiment of the present application does not specifically limit the determination method of ILD, ITD or IPD.
[0111] Taking the ITD parameter as an example, for example, the audio channel signal includes two channel signals, channel 1 and channel 2, respectively, then the ITD parameter may be the ratio of the time difference between the two channels in the audio channel signal. Taking the IPD parameter as an example, for example, the audio channel signal includes two channel signals, channel 1 and channel 2, respectively, then the IPD parameter may be the ratio of the phase difference between the two channels in the audio channel signal.
[0112] The inter-channel bit allocation parameter is used to characterize the bit allocation relationship of the channels to which the multiple audio signals included in the audio channel signal respectively belong during encoding. Exemplarily, the inter-channel bit allocation can be implemented by the inter-channel bit allocation method based on energy. For example, the channels to be allocated bits include 4 channels, namely channel 1, channel 2, channel 3, and channel 4. The channels to be allocated bits can be the channels to which the multiple audio signals included in the audio channel signal belong, or they can be multiple channels obtained by downmixing the audio channel signals after channel pairing, or they can be multiple channels obtained after inter-channel ILD calculation and inter-channel pairing downmixing. The bit allocation ratio of channel 1, channel 2, channel 3 and channel 4 can be obtained through inter-channel bit allocation, and the ratio of the bit allocation can be used as the inter-channel bit allocation parameter, for example, channel 1 occupies 3 / 16, channel 2 occupies 5 / 16, channel 3 occupies 6 / 16 and channel 4 occupies 2 / 16. The method adopted for inter-channel bit allocation is not specifically limited in the embodiment of the present application.
[0113] See also Figure 3A and Figure 3B As shown, it is a flowchart of an encoding method provided by an exemplary embodiment of the present application. The encoding method can be implemented by an audio encoding device, or by an audio encoding component, or by a core encoder. In the subsequent description, the implementation by the audio encoding component is taken as an example.
[0114] 301 , obtaining an audio channel signal of a current frame, where the audio channel signal of the current frame is obtained by spatially mapping an original HOA signal through a first target virtual speaker.
[0115] In a possible example, the first target virtual speaker may include one or more virtual speakers, or may include one or more virtual speaker groups. Each speaker group may include one or more virtual speakers. The number of virtual speakers included in different virtual speaker groups may be the same or different. Each virtual speaker in the first target virtual speaker performs spatial mapping on the original HOA signal to obtain an audio channel signal. The audio channel signal may include audio signals of one or more channels. For example, a virtual speaker performs spatial mapping on the original HOA signal to obtain an audio channel signal of one channel.
[0116] For example, the first target virtual speaker includes M virtual speakers, where M is a positive integer. The audio channel signal of the current frame may include virtual speaker signals of M channels. The virtual speaker signals of M channels correspond one-to-one to the M virtual speakers.
[0117] The number of speakers included in the first target virtual speaker can be related to the encoding rate or transmission rate, or can be related to the complexity of the audio encoding component, or can be determined by configuration. For example, when the encoding rate is low, such as when it is equal to 128kbps, M=1, when the encoding rate is medium, such as when it is equal to 384kbps, M=4, and when the encoding rate is high, such as when it is equal to 768kbps, M=7. For another example, when the encoder complexity is low, M=1, when the encoder complexity is medium, M=2, and when the encoder complexity is high, M=6. For another example: when the encoding rate is 128kbps and the encoding complexity requirement is low, M=1.
[0118] 302. When it is determined that the second target virtual speaker corresponding to the audio channel signal of the previous frame of the first target virtual speaker and the current frame meets a set condition, determine the first encoding parameter of the audio channel signal of the current frame according to the second encoding parameter of the audio channel signal of the previous frame.
[0119] Exemplarily, the first encoding parameter may include one or more of an inter-channel pairing parameter, an inter-channel auditory space parameter, or an inter-channel bit allocation parameter.
[0120] For example, determining that the first target virtual speaker and the second target virtual speaker corresponding to the audio channel signal of the previous frame of the current frame meet the set condition can be understood as determining that the proximity relationship between the first target virtual speaker and the second target virtual speaker corresponding to the audio channel signal of the previous frame of the current frame meets the set condition, or understood as the proximity between the first target virtual speaker and the second target virtual speaker corresponding to the audio channel signal of the previous frame of the current frame. The proximity relationship can be understood as the spatial position relationship between the first target virtual speaker and the second target virtual speaker, or the proximity relationship can be characterized by the spatial correlation between the first target virtual speaker and the second target virtual speaker.
[0121] As an example, whether the set condition is met can be determined by the spatial position of the first target virtual speaker and the spatial position of the second target virtual speaker. For ease of distinction, the spatial position of the first target virtual speaker is referred to as the first spatial position, and the spatial position of the second target virtual speaker is referred to as the second spatial position. It can be understood that the first target virtual speaker may include M virtual speakers, then the first spatial position may include the spatial position of each virtual speaker in the M virtual speakers. The second target virtual speaker may include N virtual speakers, then the second spatial position may include the spatial position of each virtual speaker in the N virtual speakers. M and N are both positive integers greater than 1. M and N may be the same or different. Exemplarily, the spatial position of the target virtual speaker can be characterized by coordinates or serial numbers or HOA coefficients. Optionally, M=N.
[0122] In some possible embodiments, the first target virtual speaker and the second target virtual speaker corresponding to the audio channel signal of the previous frame of the current frame meet the set condition, which may include that the first spatial position overlaps with the second spatial position, or it may be understood that the proximity relationship meets the set condition. When the first spatial position overlaps with the second spatial position, the second encoding parameter may be reused as the first encoding parameter, that is, the encoding parameter of the audio channel signal of the previous frame is used as the encoding parameter of the audio channel signal of the current frame.
[0123] When the first target virtual speaker and the second target virtual speaker both include multiple virtual speakers, the number of virtual speakers included in the first target virtual speaker and the second target virtual speaker is the same, and the first spatial position overlaps with the second spatial position, which can be described as a one-to-one overlap of the spatial positions of the multiple virtual speakers included in the first target virtual speaker and the spatial positions of the multiple virtual speakers included in the second target virtual speaker.
[0124] For example, when the spatial position is represented by coordinates, for the sake of distinction, the coordinates of the first target virtual speaker are called the first coordinates, and the coordinates of the second target virtual speaker are called the second coordinates, that is, the first spatial position includes the first coordinates of the first target virtual speaker, and the second spatial position includes the second coordinates of the second target virtual speaker. Then, the first spatial position overlaps with the second spatial position, that is, the first coordinate is the same as the second coordinate. It should be understood that when the first target virtual speaker and the second target virtual speaker both include multiple virtual speakers, the coordinates of the multiple virtual speakers included in the first target virtual speaker are the same as the coordinates of the multiple virtual speakers included in the second target virtual speaker.
[0125] For another example, when the spatial position is represented by the serial number of the virtual speaker, in order to facilitate distinction, the serial number of the first target virtual speaker is called the first serial number, and the serial number of the second target virtual speaker is called the second serial number, that is, the first spatial position includes the first serial number of the first target virtual speaker, and the second spatial position includes the second serial number of the second target virtual speaker, then the first spatial position overlaps with the second spatial position, that is, the first serial number is the same as the second serial number. It should be understood that when the first target virtual speaker and the second target virtual speaker both include multiple virtual speakers, the serial numbers of the multiple virtual speakers included in the first target virtual speaker correspond to the serial numbers of the multiple virtual speakers included in the second target virtual speaker.
[0126] For another example, when the spatial position is characterized by the HOA coefficient of the virtual speaker, in order to facilitate distinction, the HOA coefficient of the first target virtual speaker is called the first HOA coefficient, and the HOA coefficient of the second target virtual speaker is called the second HOA coefficient, that is, the first spatial position includes the first HOA coefficient of the first target virtual speaker, and the second spatial position includes the second HOA coefficient of the second target virtual speaker, then the first spatial position overlaps with the second spatial position, that is, the first HOA coefficient is the same as the second HOA coefficient. It should be understood that when the first target virtual speaker and the second target virtual speaker both include multiple virtual speakers, the HOA coefficients of the multiple virtual speakers included in the first target virtual speaker are the same as the HOA coefficients of the multiple virtual speakers included in the second target virtual speaker.
[0127] In some other possible embodiments, the second target virtual speaker corresponding to the audio channel signal of the previous frame of the current frame satisfies the set condition, which may include that the first spatial position does not overlap with the second spatial position and the multiple virtual speakers included in the first target virtual speaker correspond one-to-one to the set range centered on the multiple virtual speakers included in the second target virtual speaker. It can also be understood that the proximity relationship satisfies the set condition. For example, it can be determined whether the mth virtual speaker included in the first target virtual speaker is located within the set range centered on the nth virtual speaker included in the second target virtual speaker, m traverses a positive integer less than or equal to M, and n traverses a positive integer less than or equal to N, so as to determine whether the second target virtual speaker corresponding to the audio channel signal of the previous frame of the first target virtual speaker and the current frame satisfies the set condition. For example, when the first spatial position does not overlap with the second spatial position, if the multiple virtual speakers included in the first target virtual speaker correspond one-to-one to the set range centered on the multiple virtual speakers included in the second target virtual speaker, the second encoding parameter of the audio channel signal of the previous frame can be adjusted according to the set ratio to obtain the second encoding parameter of the audio channel signal of the current frame. For another example, when the first spatial position does not overlap with the second spatial position, if the multiple virtual speakers included in the first target virtual speaker are located one-to-one within a set range centered on the multiple virtual speakers included in the second target virtual speaker, the audio channel signal of the current frame can partially reuse the second encoding parameters of the audio channel signal of the previous frame. For example, the encoding parameters of the virtual speaker signal in the audio channel signal of the current frame reuse the encoding parameters of the virtual speaker signal in the audio channel signal of the previous frame, and the encoding parameters of the residual signal in the audio channel signal of the current frame do not reuse the encoding parameters of the virtual speaker signal in the audio channel signal of the previous frame. For another example, the encoding parameters of the virtual speaker signal in the audio channel signal of the current frame reuse the encoding parameters of the virtual speaker signal in the audio channel signal of the previous frame, and the encoding parameters of the residual signal in the audio channel signal of the current frame are obtained by adjusting the encoding parameters of the virtual speaker signal in the audio channel signal of the previous frame according to the set ratio.
[0128] Taking the current frame audio channel signal including two virtual speaker signals, H1 and H2 as an example, the first target virtual speaker includes two virtual speakers, virtual speaker 1-1 and virtual speaker 1-2. Taking the previous frame audio channel signal including two virtual speaker signals, FH1 and FH2 as an example, the second target virtual speaker includes two virtual speakers, virtual speaker 2-1 and virtual speaker 2-2. Virtual speaker 1-1 is located within a set range centered on virtual speaker 2-1, and virtual speaker 1-2 is located within a set range centered on virtual speaker 2-2, then the proximity relationship between the first target virtual speaker and the second target virtual speaker meets the set condition.
[0129] For example, taking the first spatial position including the first coordinate and the second spatial position including the second coordinate as an example, the coordinates of the virtual speaker are represented by (horizontal angle azi, elevation angle ele). The coordinates of the virtual speaker 1-1 are (H1_pos_aiz, H1_pos_ele), and the coordinates of the virtual speaker 1-2 are (H2_pos_aiz, H2_pos_ele). The coordinates of the virtual speaker 2-1 are (FH1_pos_aiz, FH1_pos_ele), and the coordinates of the virtual speaker 2-2 are (FH2_pos_aiz, FH2_pos_ele). When H1_Pos_azi∈[HF1_Pos_azi±TH1] and H1_Pos_ele∈[HF1_Pos_ele±TH2] and H2_Pos_azi∈[HF2_Pos_azi±TH3] and H2_Pos_ele∈[HF1_Pos_ele±TH4], the proximity relationship between the first target virtual speaker and the second target virtual speaker satisfies the set condition, that is, the multiple virtual speakers included in the first target virtual speaker are located one-to-one within the set range centered on the multiple virtual speakers included in the second target virtual speaker. Among them, TH1, TH2, TH3 and TH4 are set thresholds for characterizing the set range. For example, TH1, TH2, TH3 and TH4 can be the same or different, or TH1=TH3, TH2=TH4.
[0130] For example, take the example that the first spatial position includes the first serial number and the second spatial position includes the second serial number. The serial number of virtual speaker 1-1 is H1_Ind, and the serial number of virtual speaker 1-2 is H2_Ind. The serial number of virtual speaker 2-1 is FH1_Ind, and the serial number of virtual speaker 2-2 is FH2_Ind. When H1_Ind∈[FH1_Ind±TH5] and H2_Ind∈[FH2_Ind±TH6], the first target virtual speaker and the second target virtual speaker meet the set conditions, that is, the multiple virtual speakers included in the first target virtual speaker are one-to-one correspondingly located within the set range centered on the multiple virtual speakers included in the second target virtual speaker. Among them, TH5 and TH6 are set thresholds used to characterize the set range. Optionally, TH5=TH6.
[0131] For example, take the example that the first spatial position includes the first HOA coefficient and the second spatial position includes the second HOA coefficient. The HOA coefficient of virtual speaker 1-1 is H1_Coef, and the HOA coefficient of virtual speaker 1-2 is H2_Coef. The HOA coefficient of virtual speaker 2-1 is FH1_Coef, and the HOA coefficient of virtual speaker 2-2 is FH2_Coef. When H1_Coef∈[FH1_Coef±TH7] and H2_Ind∈[HF2_Ind±TH8], the first target virtual speaker and the second target virtual speaker meet the set conditions, that is, the multiple virtual speakers included in the first target virtual speaker are located one-to-one within the set range centered on the multiple virtual speakers included in the second target virtual speaker. Among them, TH7 and TH8 are set thresholds used to characterize the set range. Optionally, TH7=TH8.
[0132] In some possible embodiments, the audio encoding component may also determine whether the first target virtual speaker and the second target virtual speaker satisfy a set condition by determining a correlation between the first target virtual speaker and the second target virtual speaker.
[0133] As an example, the audio encoding component may determine the correlation between the first target virtual speaker and the second target virtual speaker according to the first coordinate of the first target virtual speaker and the second coordinate of the second target virtual speaker.
[0134] For example, when the audio encoding component determines that the first coordinate of the first target virtual speaker is the same as the second coordinate of the second target virtual speaker, the correlation R = 1. In this case, the first encoding parameter can reuse the second encoding parameter.
[0135] For another example, when the audio encoding component determines that the first coordinate of the first target virtual speaker is not completely the same as the second coordinate of the second target virtual speaker, the correlation can be determined by the following formula (3).
[0136]
[0137] Among them, R represents the correlation, norm() represents the normalization operation, S() represents the operation of determining the distance, and H m represents the coordinates of the mth virtual speaker in the first target virtual speaker, FH n represents the coordinates of the nth virtual speaker in the second target virtual speaker. m ,FH n ) indicates determining the distance between the mth virtual speaker included in the first target virtual speaker and the nth virtual speaker included in the second target virtual speaker. m traverses a positive integer not greater than N, and n traverses a positive integer not greater than N. N is the virtual speaker included in the first target virtual speaker and the second target virtual speaker.
[0138] For another example, when the audio encoding component determines that the first coordinate of the first target virtual speaker is not completely the same as the second coordinate of the second target virtual speaker, the correlation can be determined by the following formula (4).
[0139] The first target virtual speakers of the current frame include N virtual speakers, namely: H1, H2, ... HN, and the second target virtual speakers of the previous frame include N virtual speakers, namely: FH1, FH2, ... FHN.
[0140]
[0141] Among them, M H A matrix consisting of the coordinates of the virtual speakers included in the first target virtual speaker of the current frame, The second target virtual speaker for the previous frame includes the transpose of the matrix consisting of the coordinates of the virtual speakers.
[0142] For example:
[0143]
[0144]
[0145] For another example, the correlation between the first target virtual speaker and the second target virtual speaker determined according to the first coordinate of the first target virtual speaker and the second coordinate of the second target virtual speaker satisfies the condition shown in the following formula (5):
[0146]
[0147] Among them, R represents the correlation, norm() represents the normalization operation, and max() represents the maximum value operation of the elements in the brackets. represents the horizontal angle of the i-th virtual speaker included in the first target virtual speaker, represents the horizontal angle of the i-th virtual speaker included in the second target virtual speaker, represents the pitch angle of the i-th virtual speaker included in the first target virtual speaker, represents the pitch angle of the i-th virtual speaker included in the first target virtual speaker.
[0148] When the correlation is not equal to 1 and is greater than a set value, the first coding parameter may partially reuse the second coding parameter, or the first coding parameter may be obtained by adjusting the second coding parameter according to a set ratio. For example, the set value is a number greater than 0.5 and less than 1.
[0149] 303, encode the audio channel signal of the current frame according to the first encoding parameter and write it into a bitstream. It can also be described as encoding the audio channel signal of the current frame according to the first encoding parameter to obtain an encoding result, and writing the encoding result into a bitstream.
[0150] In some possible embodiments, when the first spatial position of the first target virtual speaker overlaps with the second spatial position of the second target virtual speaker, the second encoding parameter is multiplexed as the first encoding parameter to encode the audio channel signal of the current frame and write it into the bitstream.
[0151] In some other possible embodiments, when the first spatial position does not overlap with the second spatial position, if the multiple virtual speakers included in the first target virtual speaker are located one-to-one within a set range centered on the multiple virtual speakers included in the second target virtual speaker, the second encoding parameter can be adjusted according to the set ratio to obtain the first encoding parameter.
[0152] For example, the setting ratio is represented by α, and the first encoding parameter of the audio channel signal of the current frame = α*the second encoding parameter of the audio channel signal of the previous frame, where α has a value range of (0,1). The first encoding parameter may include one or more of an inter-channel pairing parameter, an inter-channel auditory space parameter, or an inter-channel bit allocation parameter. In some examples, the value of α may be different for different encoding parameters. For example, the value of α corresponding to the inter-channel pairing parameter is α1, and the value of α corresponding to the inter-channel bit allocation parameter is α2.
[0153] Furthermore, the audio encoding component also needs to notify the audio decoding component of the first encoding parameter of the audio channel signal of the current frame through the code stream.
[0154] In some embodiments, the audio encoding component may write the first encoding parameter in the bitstream to notify the audio decoding component of the first encoding parameter of the audio channel signal of the current frame. Figure 3A As shown, the audio encoding component further executes 304a to write the first encoding parameter into the bitstream.
[0155] Combination Figure 3A The encoding method described, see Figure 4A As shown, the decoding side can decode by the following decoding method. The method on the decoding side can be executed by an audio decoding device, or by an audio decoding component, or by a core encoder. The following takes the method on the decoding side executed by the audio decoding component as an example.
[0156] 405a, the audio encoding component sends the code stream to the audio decoding component, so that the audio decoding component receives the code stream.
[0157] 406a, the audio decoding component decodes the bitstream to obtain a first encoding parameter.
[0158] 407a, the audio decoding component decodes the bitstream according to the first encoding parameter to obtain the audio channel signal of the current frame.
[0159] In some other embodiments, the audio encoding component may write a multiplexing identifier in the bitstream, and indicate how to obtain the first encoding parameter of the audio channel signal of the current frame through different values of the multiplexing identifier. Figure 3B As shown, the audio encoding component further executes 304b to encode the multiplexing flag into the bitstream. The multiplexing flag is used to indicate that the first encoding parameter of the audio channel signal of the current frame is determined by the second encoding parameter of the audio channel signal of the previous frame.
[0160] In one possible manner, when the first spatial position of the first target virtual speaker overlaps with the second spatial position of the second target virtual speaker, the multiplexing identifier is a first value to indicate that the first encoding parameter of the audio channel signal of the current frame multiplexes the second encoding parameter. Optionally, in this manner, the first encoding parameter may no longer be written into the code stream, reducing resource occupancy and improving transmission efficiency. Optionally, when the first spatial position of the first target virtual speaker does not overlap with the second spatial position of the second target virtual speaker, the multiplexing identifier is a third value to indicate that the first encoding parameter of the audio channel signal of the current frame does not multiplex the second encoding parameter, and the determined first encoding parameter may be written into the code stream. The first encoding parameter may be determined according to the second encoding parameter or may be obtained by calculation. For example, when the first spatial position does not overlap with the second spatial position, if the multiple virtual speakers included in the first target virtual speaker correspond one to one to a set range centered on the multiple virtual speakers included in the second target virtual speaker, the second encoding parameter may be adjusted according to the set ratio to obtain the first encoding parameter, and then the obtained first encoding parameter is written into the code stream and the multiplexing identifier having a value of the third value is written into the code stream. For another example, when the first target virtual speaker and the second target virtual speaker do not meet the set condition, the first encoding parameter of the audio channel signal of the current frame can be calculated, the first encoding parameter can be written into the bitstream, and the multiplexing identifier having a third value can be written into the bitstream. For example, the first value is 0 and the third value is 1, or the first value is 1 and the third value is 0. Of course, the first value and the third value can also be other values, which are not limited in the embodiments of the present application.
[0161] In another possible manner, when the first spatial position of the first target virtual speaker overlaps with the second spatial position of the second target virtual speaker, a multiplexing identifier is written into the code stream, and the multiplexing identifier is a first value to indicate that the first encoding parameter of the audio channel signal of the current frame multiplexes the second encoding parameter. The second encoding parameter is adjusted according to a set ratio to obtain the first encoding parameter, and the multiplexing identifier is written into the code stream, and the multiplexing identifier is a second value to indicate that the first encoding parameter of the audio channel signal of the current frame is obtained by adjusting the second encoding parameter according to a set ratio. Optionally, the audio encoding component can also write the set ratio into the code stream. In some examples, when the first target virtual speaker and the second target virtual speaker do not meet the set conditions, the first encoding parameter of the audio channel signal of the current frame can be calculated, the first encoding parameter is written into the code stream, and the multiplexing identifier with a value of a third value is written into the code stream. For example, the first value is 11, the second value is 01, and the third value is 00. Of course, the first value, the second value, and the third value can also be other values, which are not limited in the embodiments of the present application.
[0162] Combination Figure 3B For the corresponding encoding method, see Figure 4B As shown, the decoding side can decode by the following decoding method. The method on the decoding side can be executed by an audio decoding device, or by an audio decoding component, or by a core encoder. The following takes the method on the decoding side executed by the audio decoding component as an example.
[0163] 405b, the audio encoding component sends the code stream to the audio decoding component, so that the audio decoding component receives the code stream.
[0164] 406b, the audio decoding component decodes the bit stream to obtain a multiplexing identifier.
[0165] 407b, when the multiplexing identifier indicates that the first encoding parameter of the audio channel signal of the current frame is determined by the second encoding parameter of the audio channel signal of the previous frame, the audio decoding component determines the first encoding parameter according to the second encoding parameter.
[0166] 408b: Decode the bitstream according to the first encoding parameter to obtain the audio channel signal of the current frame.
[0167] In some scenarios, the multiplexing identifier may include two values. For example, the value of the multiplexing identifier is a first value, indicating that the first coding parameter of the audio channel signal of the current frame multiplexes the second coding parameter. The value of the multiplexing identifier is a third value, indicating that the first coding parameter of the audio channel of the current frame does not multiplex the second coding parameter. The audio decoding component decodes the multiplexing identifier from the bitstream to obtain the multiplexing identifier. When the value of the multiplexing identifier is the first value, the second coding parameter is multiplexed as the first coding parameter, and the audio channel signal of the current frame is decoded from the bitstream according to the multiplexed second coding parameter. When the value of the multiplexing identifier is the third value, the first coding parameter of the audio channel signal of the current frame is decoded from the bitstream, and then the audio channel signal of the current frame is decoded from the bitstream according to the first coding parameter obtained by decoding.
[0168] In some other scenarios, the multiplexing identifier may include more than two values, and the multiplexing identifier is a first value to indicate that the first encoding parameter of the audio channel signal of the current frame multiplexes the second encoding parameter. The multiplexing identifier takes a second value to indicate that the second encoding parameter is adjusted according to a set ratio to obtain the first encoding parameter. The multiplexing identifier takes a third value to indicate that the first encoding parameter is obtained by decoding from the code stream. The audio decoding component decodes the code stream to obtain the multiplexing identifier. When the value of the multiplexing identifier is the first value, the second encoding parameter is multiplexed as the first encoding parameter, and the audio channel signal of the current frame is decoded from the code stream according to the multiplexed second encoding parameter. When the value of the multiplexing identifier is the second value, the second encoding parameter is adjusted according to the set ratio to obtain the first encoding parameter, and then the audio channel signal of the current frame is decoded from the code stream according to the obtained first encoding parameter. Optionally, the set ratio can be pre-configured in the audio decoding component, and the audio decoding component can obtain the configured set ratio, so as to adjust the second encoding parameter according to the set ratio to obtain the first encoding parameter. The set ratio can be written into the code stream by the audio encoding component, and the audio decoding component can obtain the set ratio by decoding from the code stream. When the value of the multiplexing identifier is the third value, the first coding parameter of the audio channel signal of the current frame is obtained by decoding from the bitstream, and then the audio channel signal of the current frame is obtained by decoding from the bitstream according to the first coding parameter obtained by decoding.
[0169] In some possible embodiments, the first encoding parameter includes one or more of an inter-channel pairing parameter, an inter-channel auditory space parameter, or an inter-channel bit allocation parameter.
[0170] When the first encoding parameter includes multiple parameters, one multiplexing identifier may be used for different parameters, or different multiplexing identifiers may be used for multiple parameters.
[0171] For example, the same multiplexing identifier may be used for different parameters. When the multiplexing identifier is a first value, it indicates that the first encoding parameters include the second encoding parameters of the audio channel signal of the previous frame.
[0172] Different multiplexing identifiers may be used to describe different parameters.
[0173] As an example, the first encoding parameter includes an inter-channel pairing parameter. For example, the multiplexing identifier Flag_1 is used to indicate whether the inter-channel pairing parameters of the audio channel signal of the current frame reuse the inter-channel pairing parameters of the audio channel signal of the previous frame. For example, when Flag_1=1, it indicates that the inter-channel pairing parameters of the audio channel signal of the current frame reuse the inter-channel pairing parameters of the audio channel signal of the previous frame; when Flag_1=0, it indicates that the inter-channel pairing parameters of the audio channel signal of the current frame do not reuse the inter-channel pairing parameters of the audio channel signal of the previous frame. For another example, when Flag_1=11, it indicates that the inter-channel pairing parameters of the audio channel signal of the current frame reuse the inter-channel pairing parameters of the audio channel signal of the previous frame; when Flag_1=00, it indicates that the inter-channel pairing parameters of the audio channel signal of the current frame do not reuse the inter-channel pairing parameters of the audio channel signal of the previous frame; Flag_1=01 (or 10), it indicates that the inter-channel pairing parameters of the audio channel signal of the current frame are obtained by adjusting the inter-channel pairing parameters of the audio channel signal of the previous frame according to the set ratio, or it indicates that the inter-channel pairing parameters of the audio channel signal of the current frame partially reuse the inter-channel pairing parameters of the audio channel signal of the previous frame.
[0174] As another example, the first encoding parameter includes an inter-channel auditory space parameter, wherein the inter-channel auditory space parameter includes one or more of ILD, IPD or ITD.
[0175] In a possible manner, when the inter-channel auditory space parameter includes multiple parameters, a multiplexing flag may indicate whether the multiple parameters included in the inter-channel auditory space parameter of the audio channel signal of the current frame multiplex the inter-channel auditory space parameter of the audio channel signal of the previous frame.
[0176] For example, taking the inter-channel auditory space parameters including ILD, IPD and ITD as an example. The multiplexing identifier Flag_2 is used to indicate whether the inter-channel auditory space parameters (including ILD, IPD and ITD) of the audio channel signal of the current frame reuse the inter-channel auditory space parameters of the audio channel signal of the previous frame. For example, when Flag_2=1, it indicates that the inter-channel auditory space parameters of the audio channel signal of the current frame reuse the inter-channel auditory space parameters of the audio channel signal of the previous frame; when Flag_2=0, it indicates that the inter-channel auditory space parameters of the audio channel signal of the current frame do not reuse the inter-channel auditory space parameters of the audio channel signal of the previous frame. For another example, when Flag_2=11, it indicates that the inter-channel auditory space parameters of the audio channel signal of the current frame reuse the inter-channel auditory space parameters of the audio channel signal of the previous frame; when Flag_2=00, it indicates that the inter-channel auditory space parameters of the audio channel signal of the current frame do not reuse the inter-channel auditory space parameters of the audio channel signal of the previous frame; Flag_2=01 (or 10), it indicates that the inter-channel auditory space parameters of the audio channel signal of the current frame are obtained by adjusting the inter-channel auditory space parameters of the audio channel signal of the previous frame according to the set ratio, or it indicates that the inter-channel auditory space parameters of the audio channel signal of the current frame partially reuse the inter-channel auditory space parameters of the audio channel signal of the previous frame.
[0177] In another possible manner, when the inter-channel auditory space parameters include multiple parameters, different parameters use different multiplexing identifiers. Take the inter-channel auditory space parameters including ILD, IPD and ITD as an example. The multiplexing identifier Flag_2-1 is used to indicate whether the ILD of the audio channel signal of the current frame reuses the ILD of the audio channel signal of the previous frame. The multiplexing identifier Flag_2-2 is used to indicate whether the ITD of the audio channel signal of the current frame reuses the ITD of the audio channel signal of the previous frame. The multiplexing identifier Flag_2-3 is used to indicate whether the IPD of the audio channel signal of the current frame reuses the IPD of the audio channel signal of the previous frame.
[0178] As another example, the first encoding parameter includes an inter-channel bit allocation parameter. For example, the multiplexing identifier Flag_3 is used to indicate whether the inter-channel bit allocation parameter of the audio channel signal of the current frame reuses the inter-channel bit allocation parameter of the audio channel signal of the previous frame. For example, when Flag_3=1, it indicates that the inter-channel bit allocation parameter of the audio channel signal of the current frame reuses the inter-channel bit allocation parameter of the audio channel signal of the previous frame; when Flag_3=0, it indicates that the inter-channel bit allocation parameter of the audio channel signal of the current frame does not reuse the inter-channel bit allocation parameter of the audio channel signal of the previous frame. For another example, when Flag_3=11, it indicates that the inter-channel bit allocation parameters of the audio channel signals of the current frame reuse the inter-channel bit allocation parameters of the audio channel signals of the previous frame; when Flag_3=00, it indicates that the inter-channel bit allocation parameters of the audio channel signals of the current frame do not reuse the inter-channel bit allocation parameters of the audio channel signals of the previous frame; Flag_3=01 (or 10), indicating that the inter-channel bit allocation parameters of the audio channel signals of the current frame are obtained by adjusting the inter-channel bit allocation parameters of the audio channel signals of the previous frame according to the set ratio, or indicating that the inter-channel bit allocation parameters of the audio channel signals of the current frame partially reuse the inter-channel bit allocation parameters of the audio channel signals of the previous frame.
[0179] The following is an exemplary description of the generation process of the HOA coefficient of the virtual speaker involved in the embodiment of the present application. The HOA coefficient of the virtual speaker can also be generated in other ways, which are not specifically limited in the embodiment of the present application.
[0180] Taking the propagation of sound waves in an ideal medium as an example, the wave number is k = w / c, the angular frequency w = 2πf, f is the sound wave frequency, and c is the sound speed. Then the sound pressure p satisfies the following formula (6), where is the Laplace operator:
[0181]
[0182] Solve the equation p in formula (6) in spherical coordinates. In the passive spherical region, the solution p of the equation can be expressed as the following formula (7):
[0183]
[0184] In the above formula (7), r represents the radius of the sphere, θ represents the horizontal angle, represents the pitch angle, k represents the wave number, s represents the amplitude of the ideal plane wave, and m represents the order number of the HOA. is the spherical Bessel function, also known as the radial basis function, The first j in represents the imaginary unit. The part does not change with angle. That is θ, Spherical harmonics of the direction, is the spherical harmonic function of the sound source direction.
[0185] Its Ambisonics coefficient can be expressed as formula (8):
[0186]
[0187] According to formula (8), the expanded form corresponding to formula (7) is further obtained as shown in formula (9):
[0188]
[0189] Formula (9) shows that the sound field can be expanded on the sphere according to spherical harmonics, using the coefficient Alternatively, if the coefficients are known, Can be based on Reconstruct the sound field. Truncate the above formula to the Nth term with coefficient As an approximate description of the sound field, it is called the N-order HOA coefficient, which can also be called the Ambisonics coefficient. The P-order Ambisonics coefficient has a total of (P+1) 2 channels. Among them, Ambisonics signals above the first order are also called HOA signals. In one possible configuration, the HOA order can be 2 to 10. By superimposing the coefficients of the spherical harmonics corresponding to a sampling point of the HOA signal, the spatial sound field at the time corresponding to the sampling point can be reconstructed.
[0190] According to the above description, the HOA coefficient of the virtual speaker can be generated. s and Set to the coordinates of the virtual speaker, that is, the horizontal angle (θ s ) and pitch angle According to formula (8), the HOA coefficient of the speaker, also called the Ambisonics coefficient, can be obtained.
[0191] For the third-order HOA signal, let the amplitude of the ideal plane wave s = 1, and the corresponding 16-channel HOA coefficients can be obtained by spherical harmonics The calculation formula of the 16-channel HOA coefficients corresponding to the third-order HOA signal is shown in Table 1.
[0192] Table 1
[0193]
[0194]
[0195] In Table 1, θ represents the horizontal angle of the loudspeaker. represents the elevation angle of the loudspeaker. l represents the HOA order, l = 0, 1…P; m represents the directional parameter in each order, m = -l,…,l. According to the expression in polar coordinates in Table 1, the 16 channel coefficients corresponding to the 3rd order HOA signal can be obtained according to the loudspeaker position coordinates.
[0196] The following is an exemplary description of the method for determining the target virtual speaker of the current frame and the method for generating the audio channel signal. The target virtual speaker of the current frame and the method for generating the audio channel signal may also be determined in other ways, which are not specifically limited in the embodiments of the present application.
[0197] A1, the audio encoding component determines the number of virtual speakers included in the first target virtual speaker and the number of virtual speaker signals included in the audio channel signal.
[0198] The first target number M of virtual speakers cannot exceed the total number of virtual speakers. For example, the virtual speaker set includes 1024 virtual speakers, and the number K of virtual speaker signals (virtual speaker signals to be transmitted by the encoder) cannot exceed the first target number M of virtual speakers.
[0199] Among them, the number M of virtual speakers included in the first target virtual speaker can be related to the encoding rate, can also be related to the complexity of the encoder, or can be specified by the user. For example, when the rate is low, such as when it is equal to 128kbps, M=1, when the rate is medium, such as when it is equal to 384kbps, M=4, when the rate is high, such as when it is equal to 768kbps, M=7; when the encoder complexity is low, M=1, when the encoder complexity is medium, M=2, when the encoder complexity is high, M=6. For another example: when the encoding rate is 128kbps and the encoding complexity requirement is low, M=1.
[0200] Optionally, the number M of the first target virtual speakers can also be obtained through a scene signal type parameter. For example, the scene signal type parameter can be an eigenvalue after SVD decomposition of the HOA signal to be encoded in the current frame. The number d of sound sources in different directions in the sound field can be obtained through the scene signal type parameter, and the number M of the first target virtual speakers satisfies 1≤N≤d.
[0201] A2: Determine a virtual speaker in a first target virtual speaker according to the HOA signal to be encoded and the candidate virtual speaker set.
[0202] First, calculate the speaker voting value P of the i-th round of the j-th frequency point of the HOA signal to be encoded jil , determine the matching speaker number g of the i-th round at the j-th frequency point j,i and its corresponding voting value The representative point can be determined first according to the HOA signal to be encoded in the current frame, and then the speaker voting value can be calculated according to the representative point of the HOA signal to be encoded. The speaker voting value can also be directly calculated according to each point of the HOA signal to be encoded in the current frame. The representative point can be a representative sample point in the time domain or a representative frequency point in the frequency domain.
[0203] The speaker set in the i-th round can be a virtual speaker set, including Q virtual speakers; or a subset selected from the virtual speaker set according to a pre-set rule. The speaker sets used in different rounds can be the same or different.
[0204] This embodiment takes L' representative frequency points of the HOA signal to be encoded and uses a virtual speaker set as the speaker for calculating the voting value in each round as an example, and provides a method for calculating the speaker voting value: the speaker voting value is obtained by projecting the HOA coefficient of the signal to be encoded and the HOA coefficient of the speaker.
[0205] The specific steps include:
[0206] (1) Calculate the projection value of the HOA coefficient of the jth frequency point of the signal to be encoded and the HOA coefficient of the lth speaker to obtain the voting value P of the lth speaker in the i-th round jil , l=1,2…Q.
[0207] The following is an implementation method for obtaining the projection value:
[0208] P jil =log(E jil ) or P jil =E jil ;
[0209]
[0210] where θ is the azimuth and is the pitch angle, is the HOA coefficient of the jth frequency point of the signal to be encoded, is the HOA coefficient of the lth speaker, l=1,2…Q, Q is the total number of speakers.
[0211] (2) According to the voting value P jil , l = 1, 2…Q, get the matching speaker g for the i-th round of voting corresponding to the j-th frequency point j,i .
[0212] For example, the matching speaker g for the i-th round of voting corresponding to the j-th frequency point j,iThe selection criterion is to select the loudspeaker with the largest absolute value of the voting value from the voting values corresponding to the Q loudspeakers in the i-th round of voting corresponding to the j-th frequency point as the matching loudspeaker for the j-th frequency point in the i-th round of voting, and its serial number is g j,i When l = g j,i When
[0213] (3) If i is less than the number of voting rounds I, then the HOA coefficient of the loudspeaker selected in the i-th round of voting at the j-th frequency point is subtracted from the HOA signal of the j-th frequency point to be encoded, and the result is the HOA signal to be encoded required for calculating the loudspeaker voting value in the next round at the j-th frequency point:
[0214]
[0215] Where E jig is the voting value of the matching speaker in the i-th round of voting at the j-th frequency point. The right side of the formula is the HOA coefficient of the signal to be coded for the i-th round of voting corresponding to the j-th frequency point. is the HOA coefficient of the signal to be encoded for the i+1th round of voting corresponding to the jth frequency point, w is the weight, and the value that can be preset satisfies 0≤w≤1. In addition, an adaptive weight calculation method is given:
[0216] Where norm is the operation to obtain the two-norm. is the HOA coefficient of the matching speaker in the i-th round of voting at the j-th frequency point.
[0217] (4) Repeat (1) to (3) until the voting values of the matching speakers in each round of the jth sample point are calculated.
[0218] (5) Repeat (1) to (4) until the voting values of the matching speakers at all frequency points are calculated.
[0219] Secondly, according to the matching speaker number g of each representative frequency point in each round j,i and its corresponding voting value Calculate the total vote value VOTE of each matching speaker g : VOTE g =∑P jig or VOTE g =VOTE g +P jig .
[0220] The specific implementation is the voting value of all matching speakers with equal serial numbers. The total vote value corresponding to the matching speaker is accumulated to obtain. For example:
[0221]
[0222] The best matching speaker set is determined based on the total voting value of the matching speakers. Specifically, the total voting value VOTE of all matching speakers g Make a selection based on the total vote value VOTE g The size of C voted matching speakers is selected as the best matching speaker set, and then the position coordinates of the best matching speaker set are obtained.
[0223] A3, based on the position coordinates of the best matching speaker set, calculate the HOA coefficient matrix A[f g1 ,f g2 ,…,f gC ].
[0224] A4, calculate the virtual speaker signal H based on the HOA coefficient matrix and the best matching speaker set: H = A -1 X.
[0225] Among them, A -1 Represents the inverse matrix of matrix A. The size of matrix A is (M×C), C is the number of speakers that won the vote, and M is the number of channels of the HOA coefficient of order N. M=(N+1) 2 , a represents the HOA coefficient of the best matching speaker, for example,
[0226]
[0227] Where X represents the HOA coefficient of the signal to be encoded, the size of the matrix X is (M×L), M is the number of channels of the HOA coefficient of order N, L is the number of frequency points, and x represents the HOA coefficient of the signal to be encoded, for example,
[0228]
[0229] The following describes the encoding method flow provided by the embodiment of the present application in combination with a specific scenario. Take the example of an audio encoding component including a spatial encoder and a core encoder.
[0230] B1, the spatial encoder performs spatial encoding processing on the HOA signal to be encoded to obtain the audio channel signal of the current frame and the attribute information of the first target virtual speaker of the audio channel of the current frame, and transmits it to the core encoder. The attribute information of the first target virtual speaker includes one or more of the coordinates, sequence number or HOA coefficients of the first target virtual speaker.
[0231] B2, the core encoder performs core encoding processing on the audio channel signal to obtain a bit stream.
[0232] The core coding process may include but is not limited to transformation, psychoacoustic model processing, downmixing processing, bandwidth extension, quantization and entropy coding, etc. The core coding process may process audio channel signals in the frequency domain or in the time domain, which is not limited here.
[0233] The coding parameters used in the downmixing process may include one or more of inter-channel pairing parameters, inter-channel auditory space parameters or inter-channel bit allocation parameters. That is, when performing the downmixing process, it may include inter-channel pairing processing, channel signal adjustment processing, inter-channel bit allocation processing, etc.
[0234] For example, see Figure 5 The figure shows a possible coding process diagram.
[0235] The HOA signal to be encoded is processed by the spatial encoder to output the audio channel signal of the current frame and the attribute information of the first target virtual speaker of the audio channel of the current frame. Take the audio channel signal as a time domain signal as an example. The core encoder performs transient detection on the audio channel signal, and then performs windowing transformation on the signal after transient detection to obtain a frequency domain signal. Further, noise shaping processing is performed on the frequency domain signal to obtain the shaped audio channel signal. Then, downmixing processing is performed on the audio channel signal after noise shaping processing, which may include channel pairing operations, channel signal adjustment, and channel signal bit allocation operations. The embodiment of the present application does not specifically limit the processing order of channel pairing operations, channel signal adjustment, and channel signal bit allocation operations. See. Figure 5As shown, the inter-channel pairing process is first performed, and the inter-channel pairing process is performed specifically according to the inter-channel pairing parameters, and the inter-channel pairing parameters and / or the multiplexing identifier are encoded into the bitstream. The inter-channel pairing parameters can determine whether the inter-channel pairing parameters of the current frame reuse the inter-channel pairing parameters of the previous frame according to the attribute information of the first target virtual speaker of the current frame (the coordinates, serial number or HOA coefficient of the first target virtual speaker) and the attribute information of the second target virtual speaker of the previous frame (the coordinates, serial number or HOA coefficient of the second target virtual speaker). According to the determined inter-channel pairing parameters of the current frame, the audio channel signal after the noise shaping process of the current frame is subjected to the inter-channel pairing process to obtain the paired audio channel signal. Then, the channel signal adjustment is performed for the paired audio channel signal, for example, the channel signal adjustment is performed on the paired audio channel signal according to the inter-channel auditory space parameters to obtain the adjusted audio channel signal, and the inter-channel auditory space parameters and / or the multiplexing identifier are encoded into the bitstream. The inter-channel auditory space parameters can determine whether the inter-channel auditory space parameters of the current frame reuse the inter-channel auditory space parameters of the previous frame according to the attribute information of the first target virtual speaker of the current frame (the coordinates, sequence number or HOA coefficient of the first target virtual speaker) and the attribute information of the second target virtual speaker of the previous frame (the coordinates, sequence number or HOA coefficient of the second target virtual speaker). Further, the adjusted audio channel signal is subjected to inter-channel bit allocation processing according to the inter-channel bit allocation parameters, and the inter-channel bit allocation parameters and / or the multiplexing identifier are encoded into the bit stream. The inter-channel bit allocation parameters can determine whether the inter-channel bit allocation parameters of the current frame reuse the inter-channel bit allocation parameters of the previous frame according to the attribute information of the first target virtual speaker of the current frame (the coordinates, sequence number or HOA coefficient of the first target virtual speaker) and the attribute information of the second target virtual speaker of the previous frame (the coordinates, sequence number or HOA coefficient of the second target virtual speaker). After the inter-channel bit allocation processing, quantization, entropy coding and bandwidth adjustment can be further performed to obtain the bit stream.
[0236] According to the same inventive concept as the above method, the present application embodiment provides an audio encoding device. Figure 6 As shown, the audio encoding device may include a spatial encoding unit 601, which is used to obtain an audio channel signal of a current frame, where the audio channel signal of the current frame is obtained by spatially mapping an original high-order stereo reverberation HOA signal through a first target virtual speaker; a core encoding unit 602, which is used to determine a first encoding parameter of the audio channel signal of the current frame according to a second encoding parameter of the audio channel signal of the previous frame when it is determined that the first target virtual speaker and the second target virtual speaker corresponding to the audio channel signal of the previous frame of the current frame meet a set condition; encode the audio channel signal of the current frame according to the first encoding parameter and write it into a bitstream.
[0237] In a possible design, the core encoding unit 602 is also used to write the first encoding parameters into the bit stream.
[0238] In one possible design, the first encoding parameter includes one or more of an inter-channel pairing parameter, an inter-channel auditory space parameter, or an inter-channel bit allocation parameter.
[0239] In one possible design, the set condition includes that the first spatial position overlaps with the second spatial position; and the core encoding unit 602 is specifically used to use the second encoding parameter of the audio channel signal of the previous frame as the first encoding parameter of the audio channel signal of the current frame.
[0240] In one possible design, the core encoding unit 602 is also used to write a multiplexing identifier into a bit stream, and the value of the multiplexing identifier is a first value, and the first value indicates that the first encoding parameter of the audio channel signal of the current frame multiplexes the second encoding parameter.
[0241] In one possible design, the first spatial position includes the first coordinate of the first target virtual speaker, the second spatial position includes the second coordinate of the second target virtual speaker, and the overlap of the first spatial position and the second spatial position includes the first coordinate being the same as the second coordinate; or the first spatial position includes the first serial number of the first target virtual speaker, the second spatial position includes the second serial number of the second target virtual speaker, and the overlap of the first spatial position and the second spatial position includes the first serial number being the same as the second serial number; or the first spatial position includes the first HOA coefficient of the first target virtual speaker, the second spatial position includes the second HOA coefficient of the second target virtual speaker, and the overlap of the first spatial position and the second spatial position includes the first HOA coefficient being the same as the second HOA coefficient.
[0242] In one possible design, the first target virtual speaker includes M virtual speakers, and the second target virtual speaker includes N virtual speakers; the setting conditions include that the first spatial position does not overlap with the second spatial position and the mth virtual speaker included in the first target virtual speaker is located within a set range centered on the nth virtual speaker included in the second target virtual speaker, wherein m traverses a positive integer less than or equal to M, and n traverses a positive integer less than or equal to N; the core encoding unit 602 is specifically used to adjust the second encoding parameter according to a set ratio to obtain the first encoding parameter.
[0243] In a possible design, when the first spatial position includes the first coordinates of the first target virtual speaker and the second spatial position includes the second coordinates of the second target virtual speaker, whether the mth virtual speaker is located within a set range centered on the nth virtual speaker is determined by a correlation between the mth virtual speaker and the nth virtual speaker, wherein the correlation satisfies the following conditions:
[0244]
[0245] Among them, R represents the correlation, norm() represents the normalization operation, and M H A matrix consisting of the coordinates of the virtual speakers included in the first target virtual speaker of the current frame, The transpose of the matrix consisting of the coordinates of the virtual speakers included in the second target virtual speaker of the previous frame;
[0246] When the correlation is greater than a set value, the mth virtual speaker is located within a set range centered on the nth virtual speaker, where m traverses a positive integer less than or equal to M, and n traverses a positive integer less than or equal to N.
[0247] In one possible design, the core encoding unit 602 is also used to write a multiplexing identifier into a bit stream, and the value of the multiplexing identifier is a second value, and the second value indicates that the first encoding parameter of the audio channel signal of the current frame is obtained by adjusting the second encoding parameter according to a set ratio.
[0248] In a possible design, the core encoding unit is also used to write the set ratio into the bit stream.
[0249] According to the same inventive concept as the above method, the embodiment of the present application provides an audio decoding device. Figure 7 As shown, the audio decoding device may include a core decoding unit 701, which is used to parse a multiplexing identifier from a bitstream, wherein the multiplexing identifier indicates that a first encoding parameter of an audio channel signal of a current frame is determined by a second encoding parameter of an audio channel signal of a previous frame of the current frame; the first encoding parameter is determined according to the second encoding parameter of the audio channel signal of the previous frame; the audio channel signal of the current frame is decoded from the bitstream according to the first encoding parameter; and a spatial decoding unit 702, which is used to perform spatial decoding on the audio channel signal to obtain a high-order stereo reverberation HOA signal.
[0250] In one possible design, the core decoding unit 701 is specifically used to, when the value of the multiplexing identifier is a first value, indicate that the first encoding parameter multiplexes the second encoding parameter, and obtain the second encoding parameter as the first encoding parameter.
[0251] In one possible design, the core decoding unit 701 is specifically used to, when the value of the multiplexing identifier is a second value, the second value indicates that the first encoding parameter is obtained by adjusting the second encoding parameter according to a set ratio, and the first encoding parameter is obtained by adjusting the second encoding parameter according to the set ratio.
[0252] In a possible design, the core decoding unit 701 is specifically used to decode from the code stream to obtain the set ratio when the value of the multiplexing identifier is a second value.
[0253] In a possible design, the encoding parameters of the audio channel signals include one or more of inter-channel pairing parameters, inter-channel auditory space parameters, or inter-channel bit allocation parameters.
[0254] For example, at the decoding end, Figure 7 In the example, the position of the core decoding unit 701 corresponds to Figure 2B In other words, the specific implementation of the function of the core decoding unit 701 can be found in Figure 2B The location of the spatial decoding unit 702 corresponds to Figure 2B In other words, the specific implementation of the function of the spatial decoding unit 702 can be seen in Figure 2B Specific details of the spatial decoder 240 are described in detail.
[0255] For example, at the encoding end, Figure 6 In the example, the position of the spatial coding unit 601 corresponds to Figure 2A In other words, the specific implementation of the function of the spatial coding unit 601 can be found in Figure 2A The location of the core encoding unit 602 corresponds to Figure 2A In other words, the specific implementation of the function of the core encoding unit 602 can be found in Figure 2A Specific details of the core encoder 220 in.
[0256] It should also be noted that the core coding unit 602 and the specific implementation process of the core coding unit 602 can be referred to Figure 3A , Figure 3B or Figure 5 For the sake of brevity, the detailed description of the embodiments will not be repeated here.
[0257] Those skilled in the art will appreciate that the functions described in conjunction with the various illustrative logic blocks, modules, and algorithm steps disclosed herein can be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions described in the various illustrative logic blocks, modules, and steps can be stored or transmitted as one or more instructions or codes on a computer-readable medium and executed by a processing unit according to hardware. Computer-readable media may include computer-readable storage media, which corresponds to tangible media, such as data storage media, or includes any communication media that facilitates the transfer of computer programs from one place to another (e.g., according to a communication protocol). In this manner, computer-readable media may generally correspond to (1) non-temporary tangible computer-readable storage media, or (2) communication media, such as signals or carrier waves. Data storage media may be any available media that can be accessed by one or more computers or one or more processors to retrieve instructions, codes, and / or data structures for implementing the techniques described in this application. A computer program product may include computer-readable media.
[0258] As an example and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage, flash memory, or any other medium that can be used to store the desired program code in the form of instructions or data structures and can be accessed by a computer. Also, any connection is properly referred to as a computer-readable medium. For example, if instructions are transmitted from a website, server or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio and microwave are included in the definition of media. However, it should be understood that the computer-readable storage media and data storage media do not include connections, carrier waves, signals or other temporary media, but are actually directed to non-temporary tangible storage media. As used herein, disks and optical disks include compact disks (CDs), laser optical disks, optical optical disks, digital versatile disks (DVDs), and Blu-ray disks, where disks typically reproduce data magnetically, while optical disks reproduce data optically using lasers. Combinations of the above should also be included within the scope of computer-readable media.
[0259] Instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Thus, the term "processor" as used herein may refer to any of the aforementioned structures or any other structures suitable for implementing the techniques described herein. Additionally, in some aspects, the functions described by the various illustrative logic blocks, modules, and steps described herein may be provided within dedicated hardware and / or software modules configured for encoding and decoding, or incorporated in a combined codec. Moreover, the techniques may be fully implemented in one or more circuits or logic elements.
[0260] The techniques of the present application may be implemented in a variety of devices or apparatuses, including a wireless handset, an integrated circuit (IC), or a set of ICs (e.g., a chipset). Various components, modules, or units are described in this application to emphasize functional aspects of devices for performing the disclosed techniques, but do not necessarily need to be implemented by different hardware units. In fact, as described above, the various units may be combined in a codec hardware unit in conjunction with appropriate software and / or firmware, or provided by interoperating hardware units (including one or more processors as described above).
[0261] In the above embodiments, the description of each embodiment has different emphases. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0262] The above is only an exemplary embodiment of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
Claims
1. An audio encoding method, characterized in that: include: Obtaining an audio channel signal of a current frame, where the audio channel signal of the current frame is obtained by spatially mapping an original high-order ambisonic reverberation (HOA) signal through a first target virtual speaker; When it is determined that the first target virtual speaker and the second target virtual speaker meet the set condition, determining the first encoding parameter of the audio channel signal of the current frame according to the second encoding parameter of the audio channel signal of the previous frame of the current frame, the audio channel signal of the previous frame corresponding to the second target virtual speaker; Encoding the audio channel signal of the current frame according to the first encoding parameter; Writing the encoding result of the audio channel signal of the current frame into the bit stream; The first target virtual speaker includes M virtual speakers, the second target virtual speaker includes N virtual speakers, the setting condition includes: a first spatial position of the first target virtual speaker does not overlap with a second spatial position of the second target virtual speaker, and the mth virtual speaker included in the first target virtual speaker is located within a setting range centered on the nth virtual speaker included in the second target virtual speaker, and determining the first encoding parameter of the audio channel signal of the current frame according to the second encoding parameter of the audio channel signal of the previous frame includes: Adjust the second encoding parameter according to a set ratio to obtain the first encoding parameter; Among them, m traverses positive integers less than or equal to M, and n traverses positive integers less than or equal to N.
2. The method according to claim 1, characterized in that The method further comprises: The first encoding parameter is written into the bitstream.
3. The method according to claim 1, characterized in that The first encoding parameter includes one or more of an inter-channel pairing parameter, an inter-channel auditory space parameter, or an inter-channel bit allocation parameter.
4. The method according to any one of claims 1 to 3, characterized in that: When the setting condition includes that the first spatial position of the first target virtual speaker overlaps with the second spatial position of the second target virtual speaker, determining the first encoding parameter of the audio channel signal of the current frame according to the second encoding parameter of the audio channel signal of the previous frame includes: The second encoding parameter of the audio channel signal of the previous frame is used as the first encoding parameter of the audio channel signal of the current frame.
5. The method according to claim 4, characterized in that The method further comprises: A multiplexing flag is written into a bitstream, wherein a value of the multiplexing flag is a first value, and the first value indicates that a first encoding parameter of an audio channel signal of the current frame multiplexes the second encoding parameter.
6. The method according to claim 4, characterized in that The first spatial position includes a first coordinate of the first target virtual speaker, the second spatial position includes a second coordinate of the second target virtual speaker, and the overlap of the first spatial position and the second spatial position includes that the first coordinate is the same as the second coordinate; or The first spatial position includes a first serial number of the first target virtual speaker, the second spatial position includes a second serial number of the second target virtual speaker, and the overlap of the first spatial position and the second spatial position includes that the first serial number is the same as the second serial number; or The first spatial position includes a first HOA coefficient of the first target virtual speaker, the second spatial position includes a second HOA coefficient of the second target virtual speaker, and the overlap of the first spatial position and the second spatial position includes that the first HOA coefficient is the same as the second HOA coefficient.
7. The method according to claim 1 or 2, characterized in that: When the first spatial position includes the first coordinate of the first target virtual speaker and the second spatial position includes the second coordinate of the second target virtual speaker, whether the mth virtual speaker is located within a set range centered on the nth virtual speaker is determined by a correlation between the mth virtual speaker and the nth virtual speaker, wherein the correlation satisfies the following conditions: Among them, R represents the correlation, norm() represents the normalization operation, and M H A matrix consisting of the coordinates of the virtual speakers included in the first target virtual speaker of the current frame, The transpose of the matrix consisting of the coordinates of the virtual speakers included in the second target virtual speaker of the previous frame; When the correlation is greater than a set value, the mth virtual speaker is located within a set range centered on the nth virtual speaker.
8. The method according to claim 7, characterized in that The method further comprises: A multiplexing identifier is written into a bitstream, wherein a value of the multiplexing identifier is a second value, and the second value indicates that a first encoding parameter of an audio channel signal of the current frame is obtained by adjusting the second encoding parameter according to a set ratio.
9. The method according to claim 7, characterized in that The method further includes: writing the set ratio into the code stream.
10. An audio encoding device, characterized in that: include: A spatial encoding unit, configured to obtain an audio channel signal of a current frame, wherein the audio channel signal of the current frame is obtained by spatially mapping an original high-order ambisonic reverberation HOA signal through a first target virtual speaker; a core encoding unit, configured to, when it is determined that the first target virtual speaker and the second target virtual speaker meet a set condition, determine a first encoding parameter of an audio channel signal of a current frame according to a second encoding parameter of an audio channel signal of a previous frame of the current frame, the audio channel signal of the previous frame corresponding to the second target virtual speaker; encode the audio channel signal of the current frame according to the first encoding parameter, and write the encoding result of the audio channel signal of the current frame into a bitstream; The first target virtual speaker includes M virtual speakers, the second target virtual speaker includes N virtual speakers, the set conditions include that the first spatial position of the first target virtual speaker does not overlap with the second spatial position of the second target virtual speaker and the mth virtual speaker included in the first target virtual speaker is located within a set range centered on the nth virtual speaker included in the second target virtual speaker, and the core encoding unit is specifically used to adjust the second encoding parameter according to a set ratio to obtain the first encoding parameter, wherein m traverses a positive integer less than or equal to M, and n traverses a positive integer less than or equal to N.
11. The device according to claim 10, characterized in that The core encoding unit is further configured to write the first encoding parameter into a bit stream.
12. The device according to claim 10, characterized in that The first encoding parameter includes one or more of an inter-channel pairing parameter, an inter-channel auditory space parameter, or an inter-channel bit allocation parameter.
13. The device according to any one of claims 10 to 12, characterized in that: When the set condition includes that the first spatial position of the first target virtual speaker overlaps with the second spatial position of the second target virtual speaker, the core encoding unit is specifically used to use the second encoding parameter of the audio channel signal of the previous frame as the first encoding parameter of the audio channel signal of the current frame.
14. The device according to claim 13, characterized in that The core encoding unit is further used to write a multiplexing identifier into a bitstream, wherein the value of the multiplexing identifier is a first value, and the first value indicates that the first encoding parameter of the audio channel signal of the current frame multiplexes the second encoding parameter.
15. The device according to claim 13, characterized in that The first spatial position includes a first coordinate of the first target virtual speaker, the second spatial position includes a second coordinate of the second target virtual speaker, and the overlap of the first spatial position and the second spatial position includes that the first coordinate is the same as the second coordinate; or The first spatial position includes a first serial number of the first target virtual speaker, the second spatial position includes a second serial number of the second target virtual speaker, and the overlap of the first spatial position and the second spatial position includes that the first serial number is the same as the second serial number; or The first spatial position includes a first HOA coefficient of the first target virtual speaker, the second spatial position includes a second HOA coefficient of the second target virtual speaker, and the overlap of the first spatial position and the second spatial position includes that the first HOA coefficient is the same as the second HOA coefficient.
16. The device according to claim 10 or 11, characterized in that When the first spatial position includes the first coordinate of the first target virtual speaker and the second spatial position includes the second coordinate of the second target virtual speaker, whether the mth virtual speaker is located within a set range centered on the nth virtual speaker is determined by a correlation between the mth virtual speaker and the nth virtual speaker, wherein the correlation satisfies the following conditions: Among them, R represents the correlation, norm() represents the normalization operation, and M H A matrix consisting of the coordinates of the virtual speakers included in the first target virtual speaker of the current frame, The transpose of the matrix consisting of the coordinates of the virtual speakers included in the second target virtual speaker of the previous frame; When the correlation is greater than a set value, the mth virtual speaker is located within a set range centered on the nth virtual speaker.
17. The device according to claim 16, characterized in that The core encoding unit is further used to write a multiplexing identifier into a bitstream, wherein the value of the multiplexing identifier is a second value, and the second value indicates that the first encoding parameter of the audio channel signal of the current frame is obtained by adjusting the second encoding parameter according to a set ratio.
18. The device according to claim 16, characterized in that The core encoding unit is further configured to write the set ratio into the bit stream.
19. An audio encoding device, characterized in that: include: A non-volatile memory and a processor coupled to each other, wherein the processor calls a program code stored in the memory to execute the method according to any one of claims 1 to 9.
Citation Information
Patent Citations
Indicating frame parameter reusability for coding vectors
CN105917408A
