Decoding method and electronic device

MYPI2025006994A0Undetermined Publication Date: 2025-01-10HUAWEI TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
MYPI2025006994
Authority / Receiving Office
MY · MY
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-07-10
Filing Date
2024-06-28
Publication Date
2025-01-10

AI Technical Summary

Technical Problem

In extended reality (XR) applications, it is difficult for the existing technology to efficiently decode and render immersive audio, especially under multi-point advanced stereo reverberation (HOA) technology, the decoding end needs to obtain or locally generate scene audio signals from other devices or generate locally. metadata, resulting in an increase in rendering time and affecting the user experience.

Method used

Provide a decoding method, by analyzing the metadata of the scene audio signal from the code stream, and rendering it directly on the decoding end, avoiding the need to obtain or locally generate metadata from other devices, improving decoding efficiency and reducing the computational volume. This saves rendering time and improves user experience.

Benefits of technology

This method can significantly improve the metadata efficiency of the decoding side to acquire scene audio signals, reduce the calculation amount of metadata generated, save rendering time, improve user experience, and match user needs and device computing capabilities.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure PI2025006994_DRAWING_001
    Figure PI2025006994_DRAWING_001
  • Figure PI2025006994_DRAWING_002
    Figure PI2025006994_DRAWING_002
  • Figure PI2025006994_DRAWING_003
    Figure PI2025006994_DRAWING_003
Patent Text Reader

Abstract

A decoding method and an electronic device are provided. The method includes: first receiving a bitstream; then parsing the bitstream to obtain metadata of a scene audio signal; and obtaining a reconstructed audio signal of the scene audio signal based on the bitstream, where the scene audio signal describes a sound field of a sound source in a scene, the scene includes a plurality of microphones, the metadata includes first metadata, the first metadata is used in combination with a distance between two of the plurality of microphones to determine a virtual loudspeaker radius, the virtual loudspeaker radius and a virtual loudspeaker signal are used for rendering to obtain a rendered audio signal, and the virtual loudspeaker signal is generated based on the reconstructed audio signal. In this way, a decoder side does not need to obtain the metadata of the scene audio signal from another electronic device, and does not need to locally generate the metadata of the scene audio signal, so that a calculation amount of generating the metadata of the scene audio signal is reduced. Therefore, time for obtaining a binaural rendered signal through rendering can be reduced, thereby improving user experience.
Need to check novelty before this filing date? Find Prior Art

Description

Decoding method and electronic device

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on July 10, 2023, with application number 202310843541.X and application name “Decoding Method and Electronic Device”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The embodiments of the present application relate to the field of coding and decoding, and in particular to a decoding method and electronic device. Background Art

[0003] To meet the immersive audio needs of extended reality (XR) applications, the MPEG-I Audio standard provides a 6-degree-of-freedom (DoF) audio technology based on multi-point higher-order ambisonics (HOA). First, several sound sources and multi-point HOA microphones are deployed in the room. Next, the sound sources play audio signals, and the multi-point HOA microphones collect the sound in the room to obtain multi-point HOA signals. After that, the multi-point HOA signals pass through the audio encoder to obtain a bitstream. After decoding the bitstream, the audio decoder renders the decoded signal based on the HOA metadata, user position and posture information, and other information to obtain a binaural rendering signal.

[0004] Summary of the Invention

[0005] In view of this, the present application provides a decoding method and electronic device. In this decoding method, a decoding end can parse the metadata of the scene audio signal from the bitstream. In this way, the decoding end does not need to obtain the metadata of the scene audio signal from other electronic devices, nor does it need to generate the metadata of the scene audio signal locally, thereby saving the time of rendering the binaural rendering signal and improving the user experience.

[0006] In a first aspect, an embodiment of the present application provides an encoding method, comprising: first, obtaining a scene audio signal and metadata of the scene audio signal; wherein the scene audio signal is used to describe the sound field of a sound source in a scene, the scene including multiple microphones, and the metadata includes first metadata, the first metadata being used to determine a virtual speaker radius based on the distance between two of the multiple microphones, the virtual speaker radius and the virtual speaker signal being used for rendering to obtain a rendered audio signal, the virtual speaker signal being generated based on a reconstructed audio signal of the scene audio signal; and then, encoding the scene audio signal and the metadata to obtain a bitstream. In this way, after the bitstream is transmitted to a decoder, the decoder can parse the metadata of the scene audio signal from the bitstream and apply the metadata of the scene audio signal to a subsequent rendering process to obtain a rendered audio signal. Consequently, the decoder does not need to obtain the metadata of the scene audio signal from other electronic devices or generate the metadata of the scene audio signal locally, thereby improving the efficiency of the decoder in obtaining the metadata of the scene audio signal and reducing the computational complexity of generating the metadata of the scene audio signal. This can save time in rendering the rendered audio signal and improve the user experience.

[0007] Exemplarily, the rendered audio signal may refer to an audio signal output to a playback module (such as headphones) for playback, and may be a binaural rendering signal.

[0008] Exemplarily, the distances between any two microphones among the multiple microphones are equal.

[0009] Exemplarily, the scene audio signal may include: a multi-point HOA signal (wherein the HOA signal may include a three-dimensional HOA signal and a two-dimensional HOA signal (also referred to as a planar HOA signal)) and a three-dimensional audio signal; the three-dimensional audio signal may refer to other audio signals in the scene audio signal in addition to the HOA signal. This application uses the scene audio signal as a multi-point HOA signal as an example for description.

[0010] For example, the microphones can be HOA microphones. Several sound sources and multi-point HOA microphones can be deployed in a room (i.e., a real scene). Then, the sound sources play audio signals, and the multi-point HOA microphones collect the sounds in the room to generate multi-point HOA signals.

[0011] For example, a scene audio signal may refer to an audio signal to be encoded by an encoder, and a reconstructed audio signal of the scene audio signal may refer to an audio signal obtained by a decoder or a decoding module within the encoder based on a bitstream obtained by encoding the scene audio signal. The reconstructed audio signal may also be referred to as a reconstructed scene audio signal.

[0012] It should be noted that this application does not limit the order of encoding scene audio signals and encoding metadata.

[0013] According to the first aspect, the method further includes: writing a first identifier into the bitstream; wherein the first identifier is used to indicate rendering complexity, and the rendering complexity is used to determine the number of virtual speakers. In this way, the decoder can perform rendering based on the number of virtual speakers, the virtual speaker radius, and the virtual speaker signal to obtain a rendered audio signal.

[0014] For example, different rendering complexities correspond to different audio qualities of the binaural rendering signals obtained by rendering; wherein, the value of the first identifier can be customized by the user or determined based on the computing power of the decoding end, so that the decoding end can render a binaural rendering signal whose audio quality matches factors such as user needs and the computing power of the electronic device.

[0015] According to the first aspect, or any implementation of the first aspect above, a second identifier is written into the bitstream; wherein the second identifier is used to indicate whether the bitstream contains metadata of the scene audio signal.

[0016] The decoder may have multiple renderers (each using a different rendering method), including those that render based on metadata of the scene audio signal and those that render based on other information. Furthermore, the second identifier can also be used to indicate whether to render based on metadata of the scene audio signal. In this way, the decoder can determine which renderer or rendering algorithm to use based on the value of the second identifier parsed from the bitstream.

[0017] According to the first aspect, or any implementation of the first aspect above, the second identifier is further used to indicate whether the code stream also includes the first identifier.

[0018] The multiple renderers included in the decoding end may include: a renderer that performs rendering based on the metadata of the scene audio signal and the first identifier, and a renderer that performs rendering based on other information. Furthermore, the second identifier may also be used to indicate whether rendering is performed based on the metadata of the scene audio signal and the first identifier. In this way, the decoding end can determine which renderer or rendering algorithm to use for rendering based on the value of the second identifier parsed from the bitstream.

[0019] According to the first aspect, or any implementation of the first aspect above, the method further includes: writing a third identifier into the bitstream; wherein the third identifier is used to indicate whether the bitstream contains the first identifier. In this case, the second identifier may be used only to indicate whether the bitstream contains metadata.

[0020] According to the first aspect, or any implementation of the first aspect above, obtaining metadata for a scene audio signal includes: obtaining scene description information for the scene; and determining metadata based on the scene description information. In this way, a binaural rendering signal generated based on the metadata of the scene audio signal is closer to the actual audio signal heard by a user in a real environment.

[0021] Exemplarily, scene description information can be used to describe scene information, such as information including but not limited to: identification, location information, characteristics and direction of multiple HOA microphones in a real scene, coordinates and materials of objects contained in the real scene, identification, location information, characteristics and direction of sound sources in the real scene, critical distance of the room in the real scene, acoustic reverberation parameters (such as direct-to-Reverberation energy Ratio (DRR), reverberation time (Reverberation Time) (such as T20 (time for energy to decay by 20dB), T30, T60, etc.), clarity (Clarity) (such as C50 (the ratio of sound energy before and after 50ms), C80, etc.), etc.), etc., and this application does not impose any restrictions on this.

[0022] For example, the critical distance of a room can be defined as the distance from the point where the direct sound energy density and the reverberant sound energy density are equal in a certain direction to the center of the sound source when the sound source is stably radiating sound waves indoors (in a real environment). This distance is also called the diffuse field distance. When the sound source's radiation directivity is spherical, the critical distance is equal in all directions, and is also called the reverberation radius.

[0023] According to the first aspect, or any implementation of the first aspect above, metadata is determined based on the scene description information of the scene, including: when the scene description information includes the room critical distance of the scene, first metadata is determined based on the product value of the room critical distance and a preset value.

[0024] According to the first aspect, or any implementation of the first aspect above, metadata is determined based on scene description information of the scene, including: when the scene description information includes acoustic reverberation parameters of the scene, determining the reverberation type of the scene based on the acoustic reverberation parameters of the scene; and determining first metadata based on the reverberation type of the scene.

[0025] There are multiple reverb types, and different reverb types correspond to different first metadata. For example, the first reverb type (corresponding to a dry sound scene or a non-reverberant room) corresponds to first metadata of 0.5, while the second reverb type (corresponding to a wet sound scene or a reverberant room) corresponds to first metadata of 1.5.

[0026] According to the first aspect, or any implementation of the first aspect above, the metadata also includes second metadata, and the metadata is determined based on the scene description information of the scene, including: when the scene description information includes position information of multiple microphones, the position information of the multiple microphones is used as the second metadata; wherein the second metadata is used to determine the distance between two microphones among the multiple microphones.

[0027] According to the first aspect, or any implementation of the first aspect above, the metadata also includes second metadata, and the metadata is determined based on the scene description information of the scene, including: when the scene description information includes position information of multiple microphones, determining the distance between two microphones among the multiple microphones based on the position information of the multiple microphones; and using the distance between the two microphones among the multiple microphones as the second metadata.

[0028] According to the first aspect, or any implementation of the first aspect above, obtaining metadata for the scene audio signal includes obtaining metadata for the scene audio signal input by a user. In other words, the user can customize the metadata for the scene audio signal, thereby ensuring that the binaural rendering signal subsequently rendered by the decoding end matches the user's personalized needs.

[0029] According to the first aspect, or any implementation of the first aspect above, the code stream includes: syntax elements of metadata of the scene audio signal and corresponding description information.

[0030] For example, the syntax elements and corresponding description information of the metadata of the scene audio signal can be added to the original syntax structure of the bitstream (or called the high-level syntax element table); in this way, the encoding end can write the metadata of the scene audio signal into the bitstream according to the syntax elements and corresponding description information of the metadata of the scene audio signal in the syntax structure of the bitstream.

[0031] According to the first aspect, or any implementation of the first aspect above, the code stream includes: a syntax element of the first identifier and corresponding description information.

[0032] Exemplarily, the syntax element of the first identifier and the corresponding description information can be added to the original syntax structure of the code stream; in this way, the encoder can write the first identifier into the code stream according to the syntax element of the first identifier and the corresponding description information in the syntax structure of the code stream.

[0033] According to the first aspect, or any implementation of the first aspect above, the code stream includes: a syntax element of the second identifier and corresponding description information.

[0034] Exemplarily, the syntax elements of the second identifier and the corresponding description information can be added to the original syntax structure of the code stream; in this way, the encoder can write the second identifier into the code stream according to the syntax elements of the second identifier and the corresponding description information in the syntax structure of the code stream.

[0035] According to the first aspect, or any implementation of the first aspect above, the code stream includes: a syntax element of a third identifier and corresponding description information.

[0036] According to the first aspect, or any implementation of the first aspect above, the scene audio signal includes a multi-point high-order ambisonic reverberation (HOA) signal.

[0037] According to the first aspect, or any implementation of the first aspect above, the syntax element of the first metadata is distanceFactor.

[0038] According to the first aspect, or any implementation of the first aspect above, the syntax element of the first identifier is complexityLevel.

[0039] According to the first aspect, or any implementation of the first aspect above, the syntax element of the second identifier is hoaGroupHasLowProfileConfig.

[0040] On the second aspect, an embodiment of the present application provides a decoding method, which includes: receiving a bit stream; parsing metadata of a scene audio signal from the bit stream; obtaining a reconstructed audio signal of the scene audio signal based on the bit stream; the scene audio signal is used to describe the sound field of the sound source in the scene, the scene includes multiple microphones, the metadata includes first metadata, the first metadata is used to determine the virtual speaker radius with the distance between two microphones among the multiple microphones, the virtual speaker radius and the virtual speaker signal are used for rendering to obtain a rendered audio signal, and the virtual speaker signal is generated based on the reconstructed audio signal. In this way, the decoding end does not need to obtain metadata of the scene audio signal from other electronic devices, nor does it need to generate metadata of the scene audio signal locally, which improves the efficiency of obtaining metadata of the scene audio signal and reduces the computational complexity of generating metadata of the scene audio signal; thereby saving time in rendering to obtain binaural rendering signals and improving user experience.

[0041] According to a second aspect, the method further includes: parsing a first identifier from the bitstream; wherein the first identifier is used to indicate rendering complexity, and the rendering complexity is used to determine the number of virtual speakers.

[0042] According to the second aspect, or any implementation of the second aspect above, the method further includes: parsing a second identifier from the bitstream; wherein the value of the second identifier is a first preset identifier value, indicating that the bitstream includes metadata. In other words, when the value of the second identifier is the first preset identifier value, it can be determined that the decoder rendered according to the metadata of the scene audio signal and that the bitstream contains metadata of the scene audio signal; and further parsing the bitstream can be performed to obtain the metadata of the scene audio signal.

[0043] According to the second aspect, or any implementation of the second aspect above, when the value of the second identifier is the first preset identifier value, it indicates that the bitstream also includes the first identifier. That is, when the value of the second identifier is the first preset identifier value, the decoder can determine that rendering is performed based on the metadata of the scene audio signal and the first identifier, and can also determine that the bitstream also includes the first identifier; and can then continue to parse the bitstream to obtain the first identifier.

[0044] According to the second aspect, or any implementation of the second aspect above, the method further includes: parsing a third identifier from the code stream; wherein, when the value of the third identifier is the second preset identifier value, it indicates that the code stream includes the first identifier.

[0045] According to the second aspect, or any implementation of the second aspect, the metadata further includes second metadata, and the second metadata is used to determine the distance between two microphones among the multiple microphones.

[0046] According to the second aspect, or any implementation of the second aspect above, the scene audio signal includes a multi-point high-order ambisonics (HOA) signal, and the reconstructed audio signal includes a reconstructed audio signal of the multi-point HOA signal.

[0047] According to the second aspect, or any implementation of the second aspect above, the bitstream includes: syntax elements of metadata and corresponding description information; and parsing metadata of the scene audio signal from the bitstream includes: parsing the metadata from the bitstream based on the syntax elements of the metadata and the corresponding description information.

[0048] According to the second aspect, or any implementation of the second aspect above, the codestream includes: syntax elements of the first identifier and corresponding description information, and parsing the first identifier from the codestream includes: parsing the first identifier from the codestream according to the syntax elements of the first identifier and the corresponding description information.

[0049] According to the second aspect, or any implementation of the second aspect above, the codestream includes: syntax elements of the second identifier and corresponding description information, and parsing the second identifier from the codestream includes: parsing the second identifier from the codestream according to the syntax elements of the second identifier and the corresponding description information.

[0050] According to the second aspect, or any implementation of the second aspect above, the bitstream includes: a syntax element of a third identifier and corresponding description information, and parsing the third identifier from the bitstream includes: parsing the third identifier from the bitstream according to the syntax element of the third identifier and the corresponding description information.

[0051] According to the second aspect, or any implementation of the second aspect, the syntax element of the first metadata is distanceFactor.

[0052] According to the second aspect, or any implementation of the second aspect above, the syntax element of the first identifier is complexityLevel.

[0053] According to the second aspect, or any implementation of the second aspect, the syntax element of the second identifier is hoaGroupHasLowProfileConfig.

[0054] According to the second aspect, or any implementation of the second aspect above, the method also includes: generating a virtual speaker signal based on a reconstructed audio signal of the scene audio signal; determining a virtual speaker radius based on the distance between the first metadata and two microphones among the multiple microphones; rendering according to the virtual speaker radius and the virtual speaker signal to obtain a rendered audio signal.

[0055] The second aspect and any implementation of the second aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the second aspect and any implementation of the second aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here.

[0056] In a third aspect, an embodiment of the present application provides an encoding method, which includes: first, obtaining a scene audio signal; then, encoding the scene audio signal and a first identifier to obtain a bit stream; the first identifier is used to indicate the rendering complexity, the rendering complexity is used to determine the number of virtual speakers, the reconstructed audio signal of the scene audio signal is used to generate a virtual speaker signal, and the virtual speaker signal and the number of virtual speakers are used for rendering to obtain a rendered audio signal.

[0057] In this way, the decoding end can render according to the number of virtual speakers, the virtual speaker radius, and the virtual speaker signal to obtain a rendered audio signal. For example, different rendering complexities correspond to different audio qualities of the binaural rendering signal obtained by rendering. The value of the first identifier can be set by the user or determined based on the computing power of the decoding end. In this way, the decoding end can render a binaural rendering signal whose audio quality matches factors such as user requirements and the computing power of the electronic device.

[0058] According to a third aspect, the method further includes: writing a second identifier into the code stream; wherein the second identifier is used to indicate whether the code stream contains the first identifier.

[0059] According to the third aspect, or any implementation of the third aspect above, the code stream includes: a syntax element of the first identifier and corresponding description information.

[0060] According to the third aspect, or any implementation of the third aspect above, the code stream includes: a syntax element of the second identifier and corresponding description information.

[0061] According to the third aspect, or any implementation of the third aspect above, the scene audio signal includes a multi-point high-order ambisonic reverberation (HOA) signal.

[0062] According to the third aspect, or any implementation of the third aspect above, the syntax element of the first identifier is complexityLevel.

[0063] The third aspect and any implementation of the third aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the third aspect and any implementation of the third aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here.

[0064] In a fourth aspect, an embodiment of the present application also provides a decoding method, which includes: receiving a bit stream; parsing a first identifier from the bit stream; obtaining a reconstructed audio signal based on the bit stream, wherein the first identifier is used to indicate the rendering complexity, the rendering complexity is used to determine the number of virtual speakers, the reconstructed audio signal is used to generate a virtual speaker signal, and the virtual speaker signal and the number of virtual speakers are used for rendering to obtain a rendered audio signal.

[0065] According to a fourth aspect, the method further includes: parsing a second identifier from the code stream; and indicating that the code stream includes the first identifier when the value of the second identifier is a preset identifier value.

[0066] According to the fourth aspect, or any implementation of the fourth aspect above, the codestream includes: a syntax element of the first identifier and corresponding description information, and parsing the first identifier from the codestream includes: parsing the first identifier from the codestream according to the syntax element of the first identifier and the corresponding description information.

[0067] According to the fourth aspect, or any implementation of the fourth aspect above, the codestream includes: a syntax element of the second identifier and corresponding description information, and parsing the second identifier from the codestream includes: parsing the second identifier from the codestream according to the syntax element of the second identifier and the corresponding description information.

[0068] According to the fourth aspect, or any implementation of the fourth aspect above, the reconstructed audio signal includes a reconstructed audio signal of a multi-point high-order ambisonics (HOA) signal.

[0069] According to the fourth aspect, or any implementation of the fourth aspect above, the syntax element of the first identifier is complexityLevel.

[0070] According to the fourth aspect, or any implementation of the fourth aspect above, the method also includes: generating a virtual speaker signal based on the reconstructed audio signal of the scene audio signal; determining the number of virtual speakers based on the first identifier; rendering according to the number of virtual speakers and the virtual speaker signal to obtain a rendered audio signal.

[0071] The fourth aspect and any implementation of the fourth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the fourth aspect and any implementation of the fourth aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here.

[0072] In a fifth aspect, an embodiment of the present application provides an encoding device, the encoding device comprising:

[0073] A first data acquisition module is configured to acquire a scene audio signal and metadata of the scene audio signal; wherein the scene audio signal is used to describe a sound field of a sound source in a scene, the scene includes multiple microphones, the metadata includes first metadata, the first metadata is used to determine a virtual speaker radius based on a distance between two microphones among the multiple microphones, the virtual speaker radius and the virtual speaker signal are used for rendering to obtain a rendered audio signal, and the virtual speaker signal is generated based on a reconstructed audio signal of the scene audio signal;

[0074] The first encoding module is used to encode the scene audio signal and metadata to obtain a bit stream.

[0075] In addition, the encoding device may further include a communication module.

[0076] It should be understood that the encoding device can also execute the above-mentioned first aspect and any one of the implementation methods of the first aspect. The corresponding effects can also refer to the technical effects corresponding to the above-mentioned first aspect and any one of the implementation methods of the first aspect, which will not be repeated here.

[0077] In a sixth aspect, an embodiment of the present application provides a decoding device, the decoding device comprising:

[0078] A first receiving module, configured to receive a code stream;

[0079] A first parsing module is used to parse metadata of the scene audio signal from the bit stream;

[0080] The first signal acquisition module is used to obtain a reconstructed audio signal of a scene audio signal based on a bit stream; the scene audio signal is used to describe the sound field of a sound source in the scene, the scene includes multiple microphones, the metadata includes first metadata, the first metadata is used to determine a virtual speaker radius with a distance between two microphones among the multiple microphones, the virtual speaker radius and the virtual speaker signal are used for rendering to obtain a rendered audio signal, and the virtual speaker signal is generated based on the reconstructed audio signal.

[0081] In addition, the decoding device may further include a communication module.

[0082] It should be understood that the decoding device can also execute the above-mentioned second aspect and any one of the implementation methods of the second aspect. The corresponding effects can also refer to the technical effects corresponding to the above-mentioned second aspect and any one of the implementation methods of the second aspect, which will not be repeated here.

[0083] In a seventh aspect, an embodiment of the present application provides an encoding device, the encoding device comprising:

[0084] A second data acquisition module is used to acquire a scene audio signal;

[0085] The second encoding module is used to encode the scene audio signal and the first identifier to obtain a code stream; the first identifier is used to indicate the rendering complexity, the rendering complexity is used to determine the number of virtual speakers, the reconstructed audio signal of the scene audio signal is used to generate a virtual speaker signal, and the virtual speaker signal and the number of virtual speakers are used for rendering to obtain a rendered audio signal.

[0086] In addition, the encoding device may further include a communication module.

[0087] It should be understood that the encoding device can also execute the above-mentioned third aspect and any one of the implementation methods of the third aspect. The corresponding effects can also refer to the technical effects corresponding to the above-mentioned third aspect and any one of the implementation methods of the third aspect, which will not be repeated here.

[0088] In an eighth aspect, an embodiment of the present application provides a decoding device, the decoding device comprising:

[0089] A second receiving module, configured to receive a code stream;

[0090] A second parsing module, configured to parse the code stream to obtain a first identifier;

[0091] The second signal acquisition module is used to obtain a reconstructed audio signal of the scene audio signal according to the bit stream; the first identifier is used to indicate the rendering complexity, the rendering complexity is used to determine the number of virtual speakers, the reconstructed audio signal is used to generate a virtual speaker signal, and the virtual speaker signal and the number of virtual speakers are used for rendering to obtain a rendered audio signal.

[0092] In addition, the decoding device may further include a communication module.

[0093] It should be understood that the decoding device can also execute the above-mentioned fourth aspect and any one of the implementation methods of the fourth aspect. The corresponding effects can also refer to the technical effects corresponding to the above-mentioned fourth aspect and any one of the implementation methods of the fourth aspect, which will not be repeated here.

[0094] In the ninth aspect, an embodiment of the present application provides an electronic device, comprising: a memory and a processor, the memory being coupled to the processor; the memory storing program instructions, which, when executed by the processor, enables the electronic device to execute the encoding method in the first aspect or any possible implementation of the first aspect; or, enables the electronic device to execute the encoding method in the third aspect or any possible implementation of the third aspect.

[0095] The ninth aspect and any implementation of the ninth aspect respectively correspond to the first aspect and any implementation of the first aspect, or respectively correspond to the third aspect and any implementation of the third aspect. The technical effects corresponding to the ninth aspect and any implementation of the ninth aspect can refer to the technical effects corresponding to the first aspect and any implementation of the first aspect, or refer to the technical effects corresponding to the third aspect and any implementation of the third aspect, and no further details are given here.

[0096] In the tenth aspect, an embodiment of the present application provides an electronic device, comprising: a memory and a processor, the memory being coupled to the processor; the memory storing program instructions, which, when executed by the processor, enables the electronic device to execute the decoding method in the second aspect or any possible implementation of the second aspect; or, enables the electronic device to execute the decoding method in the fourth aspect or any possible implementation of the fourth aspect.

[0097] The tenth aspect and any implementation of the tenth aspect respectively correspond to the second aspect and any implementation of the second aspect, or respectively correspond to the fourth aspect and any implementation of the fourth aspect. The technical effects corresponding to the tenth aspect and any implementation of the tenth aspect can refer to the technical effects corresponding to the above-mentioned second aspect and any implementation of the second aspect, or refer to the technical effects corresponding to the above-mentioned fourth aspect and any implementation of the fourth aspect, and no further details are given here.

[0098] In the eleventh aspect, an embodiment of the present application provides a chip comprising one or more interface circuits and one or more processors; the one or more processors receive or send data through the one or more interface circuits, and when the one or more processors execute computer instructions, the method in the first aspect or any possible implementation of the first aspect is executed; or, the method in the third aspect or any possible implementation of the third aspect is executed.

[0099] The eleventh aspect and any implementation of the eleventh aspect respectively correspond to the first aspect and any implementation of the first aspect, or respectively correspond to the third aspect and any implementation of the third aspect. The technical effects corresponding to the eleventh aspect and any implementation of the eleventh aspect can refer to the technical effects corresponding to the first aspect and any implementation of the first aspect, or refer to the technical effects corresponding to the third aspect and any implementation of the third aspect, and are not further described here.

[0100] In the twelfth aspect, an embodiment of the present application provides a chip comprising one or more interface circuits and one or more processors; the one or more processors receive or send data through the one or more interface circuits, and when the one or more processors execute computer instructions, the method in the second aspect or any possible implementation of the second aspect is executed; or, the method in the fourth aspect or any possible implementation of the fourth aspect is executed.

[0101] The twelfth aspect and any implementation of the twelfth aspect respectively correspond to the second aspect and any implementation of the second aspect, or respectively correspond to the fourth aspect and any implementation of the fourth aspect. The technical effects corresponding to the twelfth aspect and any implementation of the twelfth aspect can refer to the technical effects corresponding to the above-mentioned second aspect and any implementation of the second aspect, or can refer to the technical effects corresponding to the above-mentioned fourth aspect and any implementation of the fourth aspect, and no further details are given here.

[0102] In the thirteenth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a computer or a processor, it enables the computer or the processor to execute the encoding method in the first aspect or any possible implementation of the first aspect; or, it enables the computer or the processor to execute the encoding method in the third aspect or any possible implementation of the third aspect.

[0103] The thirteenth aspect and any implementation of the thirteenth aspect respectively correspond to the first aspect and any implementation of the first aspect, or respectively correspond to the third aspect and any implementation of the third aspect. The technical effects corresponding to the thirteenth aspect and any implementation of the thirteenth aspect can refer to the technical effects corresponding to the first aspect and any implementation of the first aspect, or refer to the technical effects corresponding to the third aspect and any implementation of the third aspect, which are not repeated here.

[0104] In the fourteenth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a computer or a processor, it enables the computer or the processor to execute the decoding method in the second aspect or any possible implementation of the second aspect; or, it enables the computer or the processor to execute the decoding method in the fourth aspect or any possible implementation of the fourth aspect.

[0105] The fourteenth aspect and any implementation of the fourteenth aspect respectively correspond to the second aspect and any implementation of the second aspect, or respectively correspond to the fourth aspect and any implementation of the fourth aspect. The technical effects corresponding to the fourteenth aspect and any implementation of the fourteenth aspect may refer to the technical effects corresponding to the above-mentioned second aspect and any implementation of the second aspect, or may refer to the technical effects corresponding to the above-mentioned fourth aspect and any implementation of the fourth aspect, and no further details are given here.

[0106] In a fifteenth aspect, an embodiment of the present application provides a computer program product, which includes computer instructions. When the computer instructions are executed by a computer or a processor, the computer or the processor executes the encoding method in the first aspect or any possible implementation of the first aspect, or the computer or the processor executes the encoding method in the third aspect or any possible implementation of the third aspect.

[0107] The fifteenth aspect and any implementation of the fifteenth aspect respectively correspond to the first aspect and any implementation of the first aspect, or respectively correspond to the third aspect and any implementation of the third aspect. The technical effects corresponding to the fifteenth aspect and any implementation of the fifteenth aspect may refer to the technical effects corresponding to the first aspect and any implementation of the first aspect, or refer to the technical effects corresponding to the third aspect and any implementation of the third aspect, and no further details are given here.

[0108] In the sixteenth aspect, an embodiment of the present application provides a computer program product, which includes computer instructions. When the computer instructions are executed by a computer or a processor, the computer or the processor executes the encoding method in the second aspect or any possible implementation of the second aspect, or the computer or the processor executes the encoding method in the fourth aspect or any possible implementation of the fourth aspect.

[0109] The sixteenth aspect and any implementation of the sixteenth aspect respectively correspond to the second aspect and any implementation of the second aspect, or respectively correspond to the fourth aspect and any implementation of the fourth aspect. The technical effects corresponding to the sixteenth aspect and any implementation of the sixteenth aspect may refer to the technical effects corresponding to the above-mentioned second aspect and any implementation of the second aspect, or may refer to the technical effects corresponding to the above-mentioned fourth aspect and any implementation of the fourth aspect, and no further details are given here.

[0110] In a seventeenth aspect, an embodiment of the present application further provides an XR device, the XR device comprising: a processor and an audio playback module;

[0111] The processor is configured to execute the steps in the second aspect and any one of the implementations of the second aspect;

[0112] The audio playback module is used to play the rendered audio signal involved in the second aspect and any one of the implementations of the second aspect.

[0113] The seventeenth aspect and any implementation of the seventeenth aspect correspond to the second aspect and any implementation of the second aspect, respectively. The technical effects corresponding to the seventeenth aspect and any implementation of the seventeenth aspect can be referred to the technical effects corresponding to the above-mentioned second aspect and any implementation of the second aspect, and will not be repeated here.

[0114] In an eighteenth aspect, an embodiment of the present application further provides an XR device, the XR device comprising: a processor and an audio playback module;

[0115] The processor is configured to execute the steps in the fourth aspect and any one of the implementations of the fourth aspect;

[0116] The audio playback module is used to play the rendered audio signal involved in the fourth aspect and any one of the implementation methods of the fourth aspect.

[0117] The eighteenth aspect and any implementation of the eighteenth aspect correspond to the fourth aspect and any implementation of the fourth aspect, respectively. The technical effects corresponding to the eighteenth aspect and any implementation of the eighteenth aspect can be referred to the technical effects corresponding to the fourth aspect and any implementation of the fourth aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0118] FIG1A is a schematic diagram illustrating an exemplary application scenario;

[0119] FIG1B is a schematic diagram illustrating an exemplary application scenario;

[0120] FIG1C is a schematic diagram illustrating an exemplary application scenario;

[0121] FIG1D is a schematic diagram illustrating an exemplary application scenario;

[0122] FIG2A is a schematic diagram illustrating an exemplary compression framework of a scene audio signal;

[0123] FIG2B is a schematic diagram of an exemplary transcoding framework of a scene audio signal;

[0124] FIG2C is a schematic diagram illustrating an exemplary rendering process;

[0125] FIG2D is a schematic diagram of an exemplary scene;

[0126] FIG2E is a schematic diagram illustrating an exemplary scenario;

[0127] FIG3A is a schematic diagram illustrating an exemplary encoding process;

[0128] FIG3B is a schematic diagram illustrating an exemplary code stream structure;

[0129] FIG4A is a schematic diagram illustrating an exemplary decoding process;

[0130] FIG4B is a schematic diagram illustrating an exemplary process of generating a binaural rendering signal;

[0131] FIG5A is a schematic diagram illustrating an exemplary encoding process;

[0132] FIG5B is a schematic diagram illustrating an exemplary code stream structure;

[0133] FIG6A is a schematic diagram illustrating an exemplary decoding process;

[0134] FIG6B is a schematic diagram illustrating an exemplary process of generating a binaural rendering signal;

[0135] FIG7A is a schematic diagram illustrating an exemplary encoding process;

[0136] FIG7B is a schematic diagram illustrating an exemplary code stream structure;

[0137] FIG8 is a schematic diagram illustrating an exemplary decoding process;

[0138] FIG9A is a schematic diagram illustrating an exemplary encoding process;

[0139] FIG9B is a schematic diagram illustrating an exemplary code stream structure;

[0140] FIG10 is a schematic diagram illustrating an exemplary decoding process;

[0141] FIG11 is a schematic diagram illustrating an exemplary encoding process;

[0142] FIG12 is a schematic diagram illustrating an exemplary decoding process;

[0143] FIG13 is a schematic diagram showing an exemplary process of generating a binaural rendering signal;

[0144] FIG14 is a schematic diagram illustrating an exemplary encoding device;

[0145] FIG15 is a schematic diagram illustrating an exemplary decoding device;

[0146] FIG16 is a schematic diagram illustrating an exemplary encoding device;

[0147] FIG17 is a schematic diagram illustrating an exemplary decoding device;

[0148] FIG18 is a schematic structural diagram of an exemplary device. DETAILED DESCRIPTION

[0149] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0150] The term "and / or" in this article is merely a description of the association relationship between associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0151] In the description and claims of the embodiments of this application, the terms "first" and "second" are used to distinguish different objects, rather than to describe a specific order of objects. For example, the terms "first target object" and "second target object" are used to distinguish different objects, rather than to describe a specific order of objects.

[0152] In the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0153] In the description of the embodiments of this application, unless otherwise specified, "multiple" means two or more. For example, "multiple processing units" means two or more processing units; "multiple systems" means two or more systems.

[0154] In order to make the description of the following embodiments clear and concise, a brief introduction to the relevant technology is first given.

[0155] Sound is a continuous wave produced by the vibration of an object. The object that vibrates and emits sound waves is called the sound source. As sound waves propagate through a medium (such as air, solid, or liquid), they are detected by the human or animal auditory system.

[0156] The characteristics of sound waves include pitch, intensity, and timbre. Pitch indicates the highness of a sound. Intensity indicates how loud a sound is. Intensity can also be called loudness or volume. The unit of intensity is decibel (dB). Timbre is also called timbre quality.

[0157] The frequency of a sound wave determines its pitch. The higher the frequency, the higher the pitch. The frequency is the number of times an object vibrates in one second, measured in hertz (Hz). The human ear can detect sounds with frequencies between 20Hz and 20,000Hz.

[0158] The amplitude of a sound wave determines its intensity. The greater the amplitude, the greater the intensity. The closer to the sound source, the greater the intensity.

[0159] The waveform of the sound wave determines the timbre, which includes square wave, sawtooth wave, sine wave and pulse wave.

[0160] Based on the characteristics of sound waves, sounds can be divided into regular sounds and irregular sounds. Irregular sounds are sounds produced by the irregular vibration of a sound source. For example, irregular sounds can disrupt people's work, study, and rest. Regular sounds are sounds produced by the regular vibration of a sound source. Regular sounds include speech and music. When represented electrically, regular sounds are analog signals that continuously vary in the time-frequency domain. These analog signals are called audio signals. Audio signals are information carriers that carry speech, music, and sound effects.

[0161] Since human hearing has the ability to distinguish the positional distribution of sound sources in space, when listeners hear sounds in space, in addition to being able to feel the pitch, intensity and timbre of the sounds, they can also feel the direction of the sounds.

[0162] As people's attention to and demand for higher quality in their auditory systems grows, three-dimensional audio technology has emerged to enhance the sense of depth, presence, and spatiality. This allows listeners to perceive not only sounds from sources in front, behind, left, and right, but also a sense of being enveloped by the spatial sound field (or "sound field") generated by these sources, with the sound spreading outward in all directions. This creates an immersive sound experience that transports listeners to places like a movie theater or concert hall.

[0163] As we all know, when sound waves propagate in an ideal medium, the wave number is k = w / c and the angular frequency is w = 2πf, where f is the sound wave frequency and c is the sound speed. The sound pressure p satisfies formula (1), is the Laplace operator.

[0164] Assume that the spatial system outside the human ear is a sphere, with the listener at the center of the sphere. Sounds from outside the sphere have a projection on the sphere, filtering out sounds outside the sphere. Assuming that the sound sources are distributed on this sphere, the sound field generated by the sound sources on the sphere is used to fit the sound field generated by the original sound source. In other words, three-dimensional audio technology is a method for fitting sound fields. Specifically, equation (1) is solved in a spherical coordinate system. Within the passive spherical region, equation (1) is solved as follows (2).

[0165] Among them, r represents the radius of the sphere, θ represents the horizontal angle information (or azimuth information), represents the pitch angle information (or called elevation angle information), k represents the wave number, s represents the amplitude of the ideal plane wave, and m represents the order number of the HOA signal (or called the order number of the HOA signal). represents the spherical Bessel function, which is also called the radial basis function, where the first j represents the imaginary unit, Does not change with angle. express Spherical harmonics of the direction, The spherical harmonic function representing the direction of the sound source. The HOA signal satisfies formula (3).

[0166] Substituting formula (3) into formula (2), formula (2) can be transformed into formula (4).

[0167] Among them, m is truncated to the Nth item, that is, m=N, with As an approximate description of the sound field; at this time, It can be called the HOA coefficient (which can be used to represent the Nth order HOA signal). The sound field refers to the area in the medium where sound waves exist. N is an integer greater than or equal to 1, and the Nth order HOA signal includes (N+1) 2 channels of audio signals.

[0168] Formula (4) shows that the sound field can be expanded on a sphere using spherical harmonics, meaning that the sound field can be decomposed into the superposition of multiple plane waves. Therefore, the sound field described by the HOA signal can be expressed as the superposition of multiple plane waves, and the sound field can be reconstructed using the HOA coefficients.

[0169] The scene audio signal involved in the embodiment of the present application is an information carrier that carries the spatial position information of the sound source in the sound field, which describes the sound field of the listener in the space, or can refer to a signal used to describe the sound field of the sound source. Among them, the scene audio signal may include: a multi-point HOA signal (wherein the HOA signal may include a three-dimensional HOA signal and a two-dimensional HOA signal (also referred to as a planar HOA signal)) and a three-dimensional audio signal; the three-dimensional audio signal may refer to other audio signals in the scene audio signal except the HOA signal. This application is explained by taking the scene audio signal as a multi-point HOA signal as an example.

[0170] Figure 1A is a schematic diagram of an exemplary application scenario. Figure 1A illustrates an application scenario in which a user uses an augmented reality (AR) device to experience an AR concert. It should be understood that users can also use AR devices to experience other AR projects such as AR science lectures, AR cinemas, and so on, and this application does not limit this.

[0171] Among them, the real scene where the user is in Figure 1A and the virtual scene corresponding to the AR project experienced by the user are the same scene, that is, the real scene where the user is in is a concert hall, and the virtual scene corresponding to the AR project experienced by the user is a virtual concert hall.

[0172] 1A , exemplarily, when a user wants to experience an AR concert in an empty concert hall, he or she can wear an AR device; then, the user can select the AR concert hall as the AR project that the user wants to experience in the AR device, and select the music file that the user wants to experience and perform a playback operation. Subsequently, the AR device can process the music file selected by the user, such as rendering, to obtain a rendered audio signal (for example, it can be a spatial audio signal (i.e., a binaural rendering signal), and this application takes the rendered audio signal as a binaural rendering signal as an example) and play the binaural rendering signal through headphones; and the AR device can also display the performance picture through the display screen, so that the user can see the performer playing at the corresponding position on the concert hall stage through the display screen. In this way, through the combination of vision and hearing, the user can experience the concert immersively.

[0173] Figure 1B is a schematic diagram illustrating an exemplary application scenario. Figure 1B illustrates an application scenario in which a user uses a virtual reality (VR) device to experience a VR cinema. It should be understood that users can also use VR devices to experience other VR projects, such as a VR concert hall, a VR science lecture hall, and so on, and this application does not limit this.

[0174] Among them, the real scene where the user is in Figure 1B is not the same as the virtual scene corresponding to the VR project that the user experiences, that is, the real scene where the user is in is the living room, and the virtual scene corresponding to the VR project that the user experiences is a virtual cinema.

[0175] Referring to Figure 1B , for example, when a user wishes to experience a VR cinema in their living room, they can wear a VR device. Next, the user can select VR Cinema as the VR project they wish to experience, select a movie file, and play it. The VR device can then process, such as rendering, the audio file contained in the movie file to generate a binaural rendering signal, which is then played through headphones. The VR device can also display the movie footage on a display screen. This allows the user to immerse themselves in the movie through a combination of visual and auditory senses.

[0176] Figure 1C is a schematic diagram of an exemplary application scenario. In Figure 1C, an application scenario is shown in which a user uses (mobile phone + headphones) to experience a concert.

[0177] It should be understood that users can also use (mobile phone + headphones) to experience other listening projects such as crosstalk, movies, etc., and this application does not limit this. In addition, the mobile phone is only an example of this application. This application can also use terminal devices such as tablets, laptops, personal computers, smart watches, etc. + headphones to experience listening.

[0178] Among them, the real scene where the user is in Figure 1C and the virtual scene corresponding to the listening project experienced by the user are the same scene, that is, the real scene where the user is in is a concert hall, and the virtual scene corresponding to the listening project experienced by the user is a virtual concert hall.

[0179] Referring to Figure 1C , for example, when a user wishes to experience a concert in an unoccupied concert hall, they can put on headphones and connect them to their phone. The user can then select the concert as the desired listening experience on their phone, select the desired music file, and play it. The phone can then process the selected music file, such as rendering it, to generate a binaural rendering signal. This signal can then be sent to the headphones for playback. Furthermore, the phone can also play a video of the concert performance. This allows the user to immerse themselves in the concert hall using their phone.

[0180] Figure 1D is a schematic diagram of an exemplary application scenario. In Figure 1D, it shows an application scenario in which a user uses (mobile phone + earphones) to experience crosstalk.

[0181] It should be understood that users can also use (mobile phone + headphones) to experience other listening projects such as concerts, movies, etc., and this application does not limit this. In addition, the mobile phone is only an example of this application. This application can also use terminal devices such as tablets, laptops, personal computers, smart watches, etc. + headphones to listen to music.

[0182] Among them, the real scene where the user is in Figure 1D is not the same as the virtual scene corresponding to the listening project experienced by the user, that is, the real scene where the user is in is the bedroom, and the virtual scene corresponding to the listening project experienced by the user is the virtual studio.

[0183] Referring to Figure 1D , for example, when a user wishes to experience crosstalk in their bedroom, they can wear headphones and connect them to their phone. Next, the user can select crosstalk as the desired listening item on their phone, select the desired crosstalk file, and play it. The phone can then process the selected crosstalk file, such as rendering it, to generate a binaural rendering signal. This signal can then be sent to the headphones for playback. Furthermore, the phone can also play the crosstalk video. This allows users to immersively experience crosstalk in their bedroom using their phone.

[0184] For ease of explanation, the terminal device in FIG. 1A to FIG. 1D is referred to as a first electronic device.

[0185] In Figures 1A to 1D, the music files, audio files contained in movie files, and crosstalk files stored in the first electronic device (also called the decoder) are all obtained from the second electronic device (also called the encoder) and are encoded scene audio signals, i.e., bitstreams. Furthermore, the first electronic device needs to decode the bitstream to obtain a reconstructed audio signal, and then render the reconstructed audio signal to obtain a binaural rendering signal.

[0186] For example, several sound sources and multi-point HOA microphones can be deployed in a room (i.e., a real-world scenario). Next, the sound sources play audio signals, and the multi-point HOA microphones collect the sounds in the room, generating multi-point HOA signals. The multi-point HOA signals can then be used to describe the sound field of the sound source in the real-world scenario. The distance between any two HOA microphones in the multi-point HOA microphones is equal. In this way, the second electronic device can obtain the multi-point HOA signals from the multi-point HOA microphones. The multi-point HOA signals are then encoded to generate a bitstream and sent to the first electronic device.

[0187] Furthermore, for the scenes in Figure 1A and Figure 1B, the second electronic device can also model the real scene to obtain a virtual scene. The second electronic device can then encode the virtual scene to obtain a code stream and send it to the first electronic device. It should be understood that a third electronic device can also model the scene and send the resulting code stream to the first or second electronic device.

[0188] Among them, real scenes refer to objectively existing scenes, such as cinemas, concert halls, lecture halls, etc.; virtual scenes are scenes constructed using virtual technology and are not objectively existing scenes, such as virtual cinemas, virtual concert halls, virtual lecture halls, etc.

[0189] FIG2A is a schematic diagram illustrating an exemplary compression framework of a scene audio signal.

[0190] 2A , exemplarily, the first electronic device may include a first audio acquisition module, a first scene audio encoding module, a first channel encoding module, a first channel decoding module, a first scene audio decoding module, a first rendering module, and a first audio playback module. It should be understood that the first electronic device may include more or fewer modules than those shown in FIG2A , and this application does not limit this. The first rendering module may also be referred to as a first renderer, the first scene audio encoding module may also be referred to as a first audio encoder, and the first scene audio decoding module may also be referred to as a first audio decoder.

[0191] 2A , exemplarily, the second electronic device may include a second audio acquisition module, a second scene audio encoding module, a second channel encoding module, a second channel decoding module, a second scene audio decoding module, a second rendering module, and a second audio playback module. It should be understood that the second electronic device may include more or fewer modules than those shown in FIG2A , and this application does not limit this. The second rendering module may also be referred to as a second renderer, the second scene audio encoding module may also be referred to as a second audio encoder, and the second scene audio decoding module may also be referred to as a second audio decoder.

[0192] Exemplarily, the process of encoding and transmitting a scene audio signal by a first electronic device to a second electronic device, and decoding and audio playback by the second electronic device can be as follows: a first audio acquisition module (such as an HOA microphone connected to the first electronic device) can perform audio acquisition and output the scene audio signal to the first scene audio encoding module. Next, the first scene audio encoding module can encode the scene audio signal and output a code stream to the first channel encoding module. Thereafter, the first channel encoding module can channel encode the code stream and transmit the channel-encoded code stream to the second electronic device via a wireless or wired network communication device. Then, a second channel decoding module of the second electronic device can channel decode the received data to obtain a code stream and output the code stream to a second scene audio decoding module. Next, the second scene audio decoding module can decode the code stream to obtain a reconstructed audio signal; the reconstructed audio signal is then output to a second rendering module, which renders the reconstructed scene signal to obtain a binaural rendering signal; the binaural rendering signal is then sent to the second audio playback module, which performs audio playback.

[0193] It should be understood that the process of the second electronic device encoding and transmitting the scene audio signal to the first electronic device, and the first electronic device decoding and audio playback is similar to the above-mentioned process of the first electronic device transmitting the scene audio signal to the second electronic device, and the second electronic device playing back the audio, and will not be repeated here.

[0194] Exemplarily, the first electronic device and the second electronic device may include, but are not limited to: personal computers, computer workstations, smart phones, tablet computers, servers, smart cameras, smart cars or other types of cellular phones, media consumption devices, wearable devices, set-top boxes, game consoles, etc.

[0195] For example, the present application can be specifically applied to VR / AR scenarios. In one possible embodiment, the first electronic device is a server and the second electronic device is a VR / AR device. In one possible embodiment, the second electronic device is a server and the first electronic device is a VR / AR device.

[0196] It should be noted that, when the first electronic device and the second electronic device are VR / AR devices, the first audio playback module and the second audio playback module can be built-in headphones of the VR / AR device; when the first electronic device and the second electronic device are terminal devices other than VR / AR devices, the first audio playback module and the second audio playback module can be external headphones of the other terminal devices.

[0197] For example, when a first electronic device encodes a scene audio signal and a second electronic device decodes it to obtain a reconstructed audio signal, the first electronic device may be referred to as an encoding end and the second electronic device may be referred to as a decoding end. When a second electronic device encodes a scene audio signal and a first electronic device decodes it to obtain a reconstructed audio signal, the second electronic device may be referred to as an encoding end and the first electronic device may be referred to as a decoding end.

[0198] FIG2B is a schematic diagram illustrating an exemplary transcoding framework of a scene audio signal.

[0199] 2B (1), illustratively, a wireless or core network device may include: a channel decoding module, other audio decoding modules, a scene audio encoding module, and a channel encoding module. The wireless or core network device may be used for audio transcoding.

[0200] Exemplarily, the specific application scenario of Figure 2B (1) may be: when the first electronic device is not provided with a scene audio encoding module but only with other audio encoding modules; and the second electronic device is only provided with a scene audio decoding module but not with other audio decoding modules, in order to enable the second electronic device to decode and play back the scene audio signal encoded by the first electronic device using other audio encoding modules, wireless or core network equipment can be used for transcoding.

[0201] Specifically, the first electronic device uses other audio encoding modules to encode the scene audio signal to obtain a first code stream; and then performs channel encoding on the first code stream and sends it to the wireless or core network device. Then, the channel decoding module of the wireless or core network device can perform channel decoding and output the first code stream obtained by channel decoding to the other audio decoding modules. After that, the other audio decoding modules decode the first code stream to obtain the scene audio signal and output the scene audio signal to the scene audio encoding module. Then, the scene audio encoding module can encode the scene audio signal to obtain a second code stream and output the second code stream to the channel encoding module. The channel encoding module performs channel encoding on the second code stream and sends it to the second electronic device. In this way, the second electronic device can call the scene audio decoding module to decode the second code stream obtained by channel decoding to obtain a reconstructed audio signal; the reconstructed audio signal can then be played back.

[0202] 2B (2), illustratively, a wireless or core network device may include: a channel decoding module, a scene audio decoding module, other audio encoding modules, and a channel encoding module. The wireless or core network device may be used for audio transcoding.

[0203] Exemplarily, the specific application scenario of Figure 2B (2) may be: when the first electronic device is only provided with a scene audio encoding module and no other audio encoding modules; and the second electronic device is not provided with a scene audio decoding module and only provided with other audio decoding modules, in order to enable the second electronic device to decode and play back the scene audio signal encoded by the scene audio encoding module of the first electronic device, wireless or core network equipment may be used for transcoding.

[0204] Specifically, the first electronic device uses a scene audio encoding module to encode the scene audio signal to obtain a first code stream; and then performs channel encoding on the first code stream and sends it to a wireless or core network device. Then, the channel decoding module of the wireless or core network device can perform channel decoding and output the first code stream obtained by channel decoding to the scene audio decoding module. After that, the scene audio decoding module decodes the first code stream to obtain a scene audio signal and outputs the scene audio signal to other audio encoding modules. Then, other audio encoding modules can encode the scene audio signal to obtain a second code stream and output the second code stream to the channel encoding module. The channel encoding module performs channel encoding on the second code stream and sends it to the second electronic device. In this way, the second electronic device can call other audio decoding modules to decode the second code stream obtained by channel decoding to obtain a reconstructed audio signal; the reconstructed audio signal can then be played back.

[0205] Based on FIG. 2A and FIG. 2B , the process of rendering to obtain binaural rendering signals is described.

[0206] Figure 2C is a schematic diagram of an exemplary rendering process. Figure 2C shows a schematic diagram of the rendering process of a renderer (which can be the first renderer or the second renderer).

[0207] Referring to Figure 2C , after the decoder obtains the reconstructed audio signal from the bitstream, it can input the reconstructed audio signal, metadata of the scene audio signal, and the user's position and posture into the renderer. The renderer then renders the reconstructed audio signal based on the metadata of the scene audio signal, the user's position and posture, to obtain a binaural rendering signal. The user posture can refer to the user's head posture.

[0208] For example, when the decoding end is an AR / VR device, the sensors in the AR / VR device can obtain the user's position and posture. When the decoding end is another terminal device, the positioning module in the decoding end can obtain the user's position, and the headphones connected to the decoding end can obtain the user's posture.

[0209] Exemplarily, the renderer may use a virtual loudspeaker object (VLO) algorithm for rendering to obtain a binaural rendering signal. Specifically, the reconstructed audio signal may be first converted into a virtual loudspeaker signal; specifically, the following formula (5) may be referred to:

[0210] S=DX (5)

[0211] X=[x1,......,x L ] T , L=(N+1) 2 (6)

[0212] S=[s1,......,s L ] T , L=(N+1) 2 (7)

[0213] In formula (5), S is the virtual speaker signal, expressed as shown in formula (7), and N is greater than 0; that is, the reconstructed audio signal is converted into virtual speaker signals for L virtual speakers. In formula (5), X is the reconstructed audio signal (i.e., the reconstructed audio signal of the HOA signal), expressed as shown in formula (6). In formula (5), D is the decoding matrix, with a size of L*L.

[0214] Next, the virtual speaker signal can be rendered based on the metadata of the scene audio signal, the user position, and the user posture to obtain a binaural rendering signal; wherein the metadata of the scene audio signal may include first metadata and second metadata. The process of generating the binaural rendering signal can refer to the following S11 to S16:

[0215] S11 , determining Y virtual speaker signals Sn . . . Sm closest to the user position from the virtual speaker signals of the L virtual speakers.

[0216] Where Y = mn.

[0217] S12: Determine the distance d1 between the two HOA microphones according to the second metadata.

[0218] In one possible approach, the second metadata is position information of multiple HOA microphones in the scene. In this case, the distance d1 between any two HOA microphones can be calculated based on the position information of the multiple HOA microphones in the scene.

[0219] In a possible manner, the second metadata is the distance d1 between any two HOA microphones in the multi-point HOA microphones.

[0220] FIG2D is a schematic diagram illustrating an exemplary scenario.

[0221] For example, Figure 2D shows two HOA microphones in a scene, with Y = 4. That is, a single HOA signal (i.e., the audio signal of a sound source (guitar) received by a single HOA microphone) can be represented by virtual speaker signals from four virtual speakers. The distance d1 between the two HOA microphones is shown in Figure 2D .

[0222] S13 : Determine a virtual speaker radius r1 according to the first metadata and the distance d1 between the two HOA microphones.

[0223] For example, the virtual speaker radius r1 may be as shown in FIG6B , and the calculation formula of the virtual speaker radius r1 may be as shown in the following formula (8):

[0224] r=kd (8)

[0225] In formula (8), k represents the first metadata. That is, when k remains unchanged, the larger d is, the larger r is.

[0226] FIG2E is a schematic diagram of an exemplary scenario. FIG2E can refer to the description of FIG2D and will not be repeated here. The difference between FIG2E and FIG2D is that the distance d1 between the two HOA microphones is different; the distance d1 between the two HOA microphones in FIG2E is greater than the distance d1 between the two HOA microphones in FIG2D. Correspondingly, the virtual speaker radius r1 in FIG2E is also larger.

[0227] S14 , selecting head-related impulse responses HRIRn…HRIRm corresponding to the virtual speaker signals Sn…Sm from a head-related impulse response (HRIR) library according to the user posture and the virtual speaker radius r1 .

[0228] Among them, Sn corresponds to HRIRn, Sn+1 corresponds to HRIRn+1, ..., and so on, Sm corresponds to HRIRm.

[0229] S15 , performing calculations based on Sn . . . Sm and corresponding HRIRn . . . HRIRm to obtain a binaural rendering signal for each of the Y virtual speakers.

[0230] S16 , adding the binaural rendering signal of each virtual loudspeaker in the Y virtual loudspeakers to obtain a binaural rendering signal.

[0231] 2D or 2E , by adding the binaural rendering signals of the two virtual speakers, a binaural rendering signal at the user's location can be obtained.

[0232] In an embodiment of the present application, the encoder can obtain metadata for the scene audio signal and write it into the bitstream. This allows the decoder to parse the metadata from the bitstream and apply it to the subsequent rendering process to obtain a binaural rendering signal. The process for obtaining metadata for the scene audio signal and the encoding and decoding process can be referenced in the following embodiment.

[0233] FIG3A is a schematic diagram illustrating an exemplary encoding process.

[0234] S301: Acquire a scene audio signal and metadata of the scene audio signal.

[0235] Exemplarily, the encoding end may obtain multi-point HOA signals from the multi-point HOA microphones as scene audio signals.

[0236] Exemplarily, in the present application, the metadata of the scene audio signal includes relevant information of the scene audio signal, and the metadata of the scene audio signal is used in the process of rendering to obtain a binaural rendering signal.

[0237] Exemplarily, in a possible manner of S301 , the encoder may obtain scene description information of the scene, and then determine metadata of the scene audio signal according to the scene description information of the scene; the specific process will be described later.

[0238] Exemplarily, scene description information can be used to describe scene information, such as but not limited to: identification, location information, characteristics and direction of multiple HOA microphones in a real scene, coordinates and materials of objects contained in the real scene, identification, location information, characteristics and direction of sound sources in the real scene, room critical distance of the real scene (room critical distance refers to the distance from the point where the direct sound energy density is equal to the reverberation sound energy density in a certain direction to the center of the sound source when the sound source stably radiates sound waves indoors (in a real scene), also known as the diffusion field distance. When the radiation directivity of the sound source is spherical, the critical distances in each direction are equal, which is also called the reverberation radius), acoustic reverberation parameters (such as direct-to-Reverberation energy Ratio (DRR), reverberation time (such as T20 (time for energy to decay by 20dB), T30, T60, etc.), clarity (such as C50 (the ratio of sound energy before and after 50ms), C80, etc.), etc.), etc., and this application does not impose any restrictions on this.

[0239] Exemplarily, in a possible manner of S301 , metadata of the scene audio signal input by the user may be acquired; that is, the metadata of the scene audio signal may be customized by the user.

[0240] In the embodiment of FIG3A , the metadata of the acquired scene audio signal only includes the first metadata, wherein the first metadata can be used to determine a virtual speaker radius based on a distance between two microphones among a plurality of microphones, and the microphones can be HOA microphones.

[0241] S302: Encode the scene audio signal and metadata of the scene audio signal to obtain a bitstream.

[0242] The encoding process of the scene audio signal can be as follows:

[0243] In a first possible encoding method, the scene audio signal may be directly encoded to obtain encoded data of the scene audio signal and write the encoded data into a bit stream.

[0244] In a second possible encoding method, a target virtual speaker can be first selected from a preset virtual speaker set based on the first scene audio signal; then, a first virtual speaker signal is generated based on the first scene audio signal and attribute information of the target virtual speaker; then, a second scene audio signal is determined using the attribute information of the target virtual speaker and the first virtual speaker signal; then, a residual signal is generated based on the first scene audio signal and the second scene audio signal; then, the first virtual speaker signal and the residual signal are encoded to obtain encoded data of the scene audio signal and written into the bitstream. The first scene audio signal refers to the original scene audio signal to be encoded.

[0245] In a third possible encoding method, the attribute information of the target virtual speaker can be determined based on the scene audio signal, where the scene audio signal includes audio signals of L channels, where L is a positive integer; the attribute information of the first audio signal and the target virtual speaker in the scene audio signal is encoded to obtain the encoded data of the scene audio signal and write it into the bitstream; wherein the first audio signal is the audio signal of G channels in the scene audio signal, where G is a positive integer less than or equal to L.

[0246] For example, in the embodiment of FIG3A , the present application can pre-add the scene audio signal metadata syntax elements and corresponding description information to the original syntax structure of the bitstream (or called the high-level syntax element table). In this way, the encoder can write the scene audio signal metadata into the bitstream based on the syntax elements and corresponding description information of the scene audio signal metadata in the bitstream syntax structure.

[0247] For example, the syntax structure of the code stream can refer to the following Table 1:

[0248] Table 1

[0249] Referring to Table 1, in Table 1, Syntax represents a syntax element, No.of bits represents the number of bits of data identified by the syntax element, and Mnemonic represents a comment or explanation of the syntax element.

[0250] Referring to Table 1, exemplarily, the syntax structure of the code stream includes multiple syntax elements: syntax element 1, syntax element 2, syntax element 3, ..., syntax element n, ..., syntax element n+h, and the syntax element "distanceFactor" newly added in this application; wherein, "distanceFactor" is a syntax element of the metadata of the scene audio signal, and n and h are positive integers.

[0251] Continuing to refer to Table 1, the "1" in the fourth row of Table 1 indicates that the data identified by the syntax element 2 in the code stream is represented by 1 bit, and the "bslbf" in the fourth row of Table 1 indicates that the data identified by the syntax element 2 in the code stream is a bit string with the left bit first.

[0252] The "3" in the fifth row of Table 1 indicates that the data identified by the syntax element 3 in the code stream is represented by 3 bits, and the "bslbf" in the fifth row of Table 1 indicates that the data identified by the syntax element 3 in the code stream is a bit string with the left bit first.

[0253] The "4" in the seventh row of Table 1 indicates that the data identified by the syntax element n in the code stream is represented by 4 bits, and the "uimsbf" in the seventh row of Table 1 indicates that the data identified by the syntax element n in the code stream is an unsigned integer with the high bit first.

[0254] The "8" in the last row of Table 1 indicates that the metadata for the scene audio signal in the bitstream can be represented using 8 bits. The "uimsbf" in the last row of Table 1 indicates that the metadata for the scene audio signal in the bitstream is an unsigned integer with the most significant bit first. It should be understood that Table 1 is merely an example for this application; the metadata for the scene audio signal can also be represented using other numbers of bits, such as 16 bits, and this application does not impose any restrictions on this. Furthermore, in addition to being represented using "distanceFactor," the syntax elements for the metadata for the scene audio signal can also be represented using other character strings, and this application does not impose any restrictions on this either.

[0255] For example, if the syntax structure of the bitstream is as shown in Table 1, then during the execution of S302, the metadata of the scene audio signal can be converted into an 8-bit binary number; then, the 8-bit binary number is written after the data identified by the syntax element n+h in the bitstream.

[0256] Exemplarily, S301 to S302 may be executed by an encoder.

[0257] FIG3B is a schematic diagram of an exemplary code stream structure, wherein the code stream structure of the code stream in the embodiment of FIG3A may be as shown in FIG3B , that is, the metadata of the scene audio signal is located after the encoded data of the scene audio signal.

[0258] It should be understood that in the bitstream of the present application, the metadata of the scene audio signal may also be located before the encoded data of the scene audio signal, and the present application does not impose any restrictions on this; and the bitstream of the present application may also include other data, and the present application does not impose any restrictions on this.

[0259] It should be noted that, in the code stream of the present application, the metadata of the scene audio signal and the encoded data of the scene audio signal may be adjacent or not adjacent, and the present application does not impose any restriction on this.

[0260] Fig. 4A is a schematic diagram illustrating an exemplary decoding process, wherein the decoding process of Fig. 4A corresponds to the encoding process of Fig. 3A.

[0261] S401, receiving a code stream.

[0262] S402: parse the metadata of the scene audio signal from the bitstream.

[0263] S403: Obtain a reconstructed audio signal of the scene audio signal according to the code stream.

[0264] S401 to S402 are executed by the decoder. This application does not limit the execution order of S402 and S403.

[0265] Exemplarily, the process of obtaining the reconstructed audio signal of the scene audio signal according to the bit stream may be as follows:

[0266] In the first decoding method (corresponding to the first encoding method described above), the reconstructed audio signal of the scene audio signal can be directly decoded from the bit stream. In this case, S403 can be executed by the decoder.

[0267] In the second decoding method (corresponding to the second encoding method described above), a virtual speaker signal and a residual signal can be decoded from the bitstream. Then, based on the target virtual speaker attribute information, the residual signal, and the virtual speaker signal, a reconstructed audio signal of the scene audio signal is obtained. In this case, the decoding operation can be performed by the decoder, and the remaining steps can be performed by other modules.

[0268] In the third decoding method (corresponding to the third encoding method described above), the bitstream can be decoded to obtain a first reconstructed signal and attribute information of the target virtual speaker. Next, a virtual speaker signal corresponding to the target virtual speaker is generated based on the attribute information and the first reconstructed signal. Finally, reconstruction is performed based on the attribute information and the virtual speaker signal to obtain a reconstructed audio signal of the scene audio signal. In this case, the decoding operation can be performed by the decoder, and the remaining steps can be performed by other modules.

[0269] Exemplarily, the decoding end can parse the metadata of the scene audio signal from the bitstream based on the syntax elements and corresponding description information of the metadata of the field audio signal in the syntax structure of the bitstream; wherein, in the embodiment of Figure 4A, the metadata of the scene audio signal can only include the first metadata. Afterwards, the decoding end can render the first metadata parsed from the bitstream and the reconstructed audio signal obtained according to the bitstream to obtain a binaural rendering signal. In this case, the decoding end can obtain the second metadata from other devices or generate the second metadata locally, and this application does not impose any restrictions on this; and the number Y of virtual speakers in the rendering process can be a user-defined value or a default value, and this application does not impose any restrictions on this.

[0270] Fig. 4B is a schematic diagram of an exemplary binaural rendering signal generation process, which is based on Fig. 3A and Fig. 4A.

[0271] Referring to Figure 4B , illustratively, after encoding the scene audio signal and metadata of the scene audio signal to obtain a bitstream, the encoder transmits the bitstream to the decoder. Upon receiving the bitstream, the decoder can obtain a reconstructed audio signal of the scene audio signal based on the bitstream and parse the metadata of the scene audio signal from the bitstream. The decoder then renders the reconstructed audio signal of the scene audio signal based on the metadata of the scene audio signal, the user's position, and the user's posture to obtain a binaural rendering signal.

[0272] For example, the present application may also include a first identifier indicating rendering complexity in the bitstream; the rendering complexity is used to determine the number Y of virtual speakers during the rendering process. In other words, the decoder can render the virtual speaker signal based on the scene audio signal metadata, the rendering complexity indicated by the first identifier, the user's position, and the user's posture to obtain a binaural rendering signal. In this way, a binaural rendering signal can be rendered whose audio quality matches user needs, the computing power of the electronic device, and other factors.

[0273] Fig. 5A is a schematic diagram illustrating an exemplary encoding process. In the embodiment of Fig. 5A , metadata of the scene audio signal is determined according to scene description information, and the metadata of the scene audio signal may include first metadata and second metadata.

[0274] S501: Acquire a scene audio signal and scene description information of the scene.

[0275] Exemplarily, the encoding end may obtain the scene description information of the scene locally, or may obtain the scene description information of the scene from a third electronic device, and this application does not impose any restrictions on this.

[0276] S502: Determine metadata of the scene audio signal according to scene description information of the scene.

[0277] Exemplarily, the metadata of the scene audio signal may include first metadata and second metadata.

[0278] In one possible approach, the first metadata may be determined based on the product of the room critical distance and a preset value; specifically, the following formula (9) may be referred to:

[0279] Wherein, k in formula (9) is the first metadata, k∈[0.5,1.5]; 1.5 in formula (9) is a preset value.

[0280] In one possible approach, the reverberation type of a scene can be determined based on the acoustic reverberation parameters in the scene; then, first metadata can be determined based on the reverberation type of the scene. Specifically, corresponding first metadata can be pre-set for different reverberation types; for example, for the first reverberation type (corresponding to a dry sound scene or an undisturbed room), the corresponding first metadata is set to 0.5, and for the second reverberation type (corresponding to a wet sound scene or a reverberant room), the corresponding first metadata is set to 1.5. In this way, when the reverberation type of the scene is determined to be the first reverberation type, 0.5 can be used as the first metadata; when the reverberation type of the scene is determined to be the second reverberation type, 1.5 can be used as the first metadata.

[0281] In one possible approach, the location information of multiple HOA microphones in the scene included in the scene description information may be used as the second metadata.

[0282] In one possible approach, the distance between two HOA microphones in the multi-point HOA microphones may be determined based on the position information of the multi-point HOA microphones in the scene included in the scene description information; and then, the distance between the two HOA microphones in the multi-point HOA microphones may be used as the second metadata.

[0283] Exemplarily, when the first metadata is input by a user, the value range of the first metadata may be limited to k∈[0.25,4].

[0284] It should be understood that the first metadata may also include other generation methods, and the metadata of the scene audio signal may also include other metadata, which is not limited in this application.

[0285] S503: Encode the scene audio signal and metadata of the scene audio signal to obtain a bitstream.

[0286] For example, S503 may refer to the description of S302 above, which will not be repeated here.

[0287] S504: Write the first identifier into the code stream.

[0288] For example, multiple rendering complexity levels may be pre-set, as shown in Table 2:

[0289] Table 2

[0290] The number of virtual speakers in Table 2 is Y.

[0291] Referring to Table 2, illustratively, the rendering complexity can be divided into 5 levels, wherein, in levels 1 to 4, the more virtual speakers there are, the more convolutions are involved in the rendering process, the higher the rendering complexity, and the better the audio quality.

[0292] Referring to Table 2, for example, for level 0, N is the order of the HOA signal, and the number of virtual speakers is (N+1) 2 , the number of convolutions is 2(N+1) 2 ; Rendering complexity and audio quality both depend on the HOA signal.

[0293] It should be understood that Table 2 is only an example of the present application, and the rendering complexity level can be set as required, and the present application does not impose any restrictions on this.

[0294] In one possible approach, a target rendering complexity level may be selected from the multiple rendering complexity levels in Table 2 according to the computing capability of the decoding end; and the value of the first identifier may be set as the target rendering complexity level.

[0295] In one possible manner, the user may input a target rendering complexity level according to the multiple rendering complexity levels in Table 2, and set the value of the first identifier to the target rendering complexity level input by the user.

[0296] For example, in the embodiment of FIG5A , the present application can pre-add the syntax elements and corresponding description information of the scene audio signal's metadata, as well as the syntax elements and corresponding description information of the first identifier, to the original syntax structure of the bitstream. In this way, the encoder can write the first identifier into the bitstream based on the syntax elements and corresponding description information of the first identifier in the syntax structure of the bitstream; and write the metadata of the scene audio signal into the bitstream based on the syntax elements and corresponding description information of the metadata of the scene audio signal in the syntax structure of the bitstream.

[0297] It should be noted that this application does not limit the execution order of S503 and S504.

[0298] For example, the syntax structure of the code stream can refer to the following Table 3:

[0299] Table 3

[0300] Referring to Table 3, in the embodiment of FIG5A, the present application adds the syntax elements "distanceFactor" and "complexityLevel"; wherein, "distanceFactor" is the syntax element of the metadata of the scene audio signal, and "complexityLevel" is the syntax element of the first identifier.

[0301] The "3" in the last row of Table 3 indicates that the first identifier in the codestream can be represented using 3 bits. The "uimsbf" in the last row of Table 3 indicates that the first identifier in the codestream is an unsigned integer with the most significant bit first. It should be understood that Table 3 is merely an example of this application; the first identifier can also be represented using other numbers of bits, such as 4 bits, and this application does not impose any restrictions on this. Furthermore, in addition to being represented by "complexityLevel," the syntax element of the first identifier can also be represented by other character strings, and this application does not impose any restrictions on this either.

[0302] The explanations of other syntax elements in Table 3 can refer to the descriptions in Table 1 and will not be repeated here.

[0303] For example, if the syntax structure of the bitstream is as shown in Table 3, then during S503, the metadata of the scene audio signal can be converted into an 8-bit binary number; then, this 8-bit binary number is written into the bitstream after the data identified by syntax element n+1. Furthermore, during S504, the first identifier can be converted into a 3-bit binary number; then, this 3-bit binary number is written into the bitstream after the metadata of the scene audio signal.

[0304] Figure 5B is a schematic diagram of an exemplary bitstream structure. The bitstream structure in the embodiment of Figure 5A may be as shown in Figure 5B, where the metadata of the scene audio signal is located after the encoded data of the scene audio signal, and the first identifier is located after the metadata of the scene audio signal.

[0305] It should be understood that the present application does not limit the location of the metadata of the scene audio signal, the encoded data of the scene audio signal, and the first identifier in the bitstream.

[0306] It should be understood that any two data among the metadata of the scene audio signal, the encoded data of the scene audio signal and the first identifier in the code stream of the present application may be adjacent or non-adjacent, and the present application does not impose any limitation on this.

[0307] Fig. 6A is a schematic diagram illustrating an exemplary decoding process, wherein the decoding process of Fig. 6A corresponds to the encoding process of Fig. 5A.

[0308] S601, receiving a code stream.

[0309] S602: Obtain a reconstructed audio signal of the scene audio signal according to the code stream.

[0310] S603: Parse metadata of the scene audio signal from the bitstream.

[0311] S604: parse the code stream to obtain a first identifier.

[0312] For example, the present application does not limit the execution order of S602 , S603 and S604 .

[0313] Exemplarily, after receiving the code stream, the decoding end may obtain a reconstructed audio signal of the scene audio signal according to the code stream; for details, please refer to the description in the embodiment of FIG4A , which will not be repeated here.

[0314] Exemplarily, the decoding end may further parse the metadata of the scene audio signal from the bitstream based on the syntax elements and corresponding description information of the metadata of the scene audio signal in the syntax structure of the bitstream. Furthermore, the decoding end may further parse the first identifier from the bitstream based on the syntax elements and corresponding description information of the first identifier in the syntax structure of the bitstream.

[0315] Subsequently, the decoder can perform rendering based on the reconstructed signal of the scene audio signal, the first metadata, the second metadata, and the first identifier to obtain a binaural rendering signal. In this case, the decoder can parse the second metadata and the first metadata from the bitstream; and during the rendering process, the number Y of virtual speakers selected can be determined based on the first identifier.

[0316] Fig. 6B is a schematic diagram of an exemplary binaural rendering signal generation process, which is based on Fig. 5A and Fig. 6A.

[0317] Referring to Figure 6B , illustratively, after the encoder encodes the scene audio signal, metadata of the scene audio signal, and a first identifier into a bitstream, the encoder transmits the bitstream to the decoder. Upon receiving the bitstream, the decoder can obtain a reconstructed audio signal of the scene audio signal based on the bitstream, parse the metadata of the scene audio signal and the first identifier from the bitstream, and then render the reconstructed audio signal of the scene audio signal based on the metadata of the scene audio signal, the rendering complexity level indicated by the first identifier, and the user's position and posture to obtain a binaural rendering signal.

[0318] It should be noted that in Figure 6B, the input to the renderer may be the first identifier, or the rendering complexity level indicated by the first identifier, or the number Y of virtual speakers corresponding to the rendering complexity level indicated by the first identifier; this application does not impose any restrictions on this.

[0319] Exemplarily, there may be multiple renderers at the decoding end (each renderer adopts a different rendering method), wherein these multiple renderers include: a renderer that renders according to the metadata of the scene audio signal (for example, a renderer that renders using the method in S11 to S16 above) and a renderer that renders according to other information. Furthermore, a corresponding identification value can be set for each renderer. For example, for renderer A, the corresponding identification value is set to "0"; for renderer B, the corresponding identification value is set to "1"; for renderer C, the corresponding identification value is set to "2", and so on. When it is determined that the decoding end needs to render according to the metadata of the scene audio signal, in one possible way, the identification value of the renderer that renders according to the metadata of the scene audio signal (hereinafter referred to as the first identification value) can be used as the identification value of the second identification; in another possible way, the second identification value (such as "0" or "1", which can be set as required) can be used as the identification value of the second identification; and then the second identification is added to the bitstream. The second identification value is used to instruct the decoder to perform rendering based on the metadata of the scene audio signal, or in other words, the second identification value is used to indicate that the bitstream includes metadata of the scene audio signal. This facilitates the decoder to determine which renderer or rendering algorithm to use for rendering.

[0320] Fig. 7A is a schematic diagram illustrating an exemplary encoding process. It should be noted that Fig. 7A is shown based on Fig. 3A. In the embodiment of Fig. 7A, the metadata of the scene audio signal is determined according to the scene description information.

[0321] S701: Acquire a scene audio signal.

[0322] S702: Encode the scene audio signal to obtain a bit stream.

[0323] For example, S701 to S702 may refer to the above description and will not be repeated here.

[0324] S703: Write the second identifier into the code stream.

[0325] The first identification value and the second identification value described above can both be referred to as first preset identification values. That is, when it is determined that the decoding end needs to use the metadata of the scene audio signal for rendering, the value of the second identification value can be set to the first preset identification value, and then the first preset identification value can be written into the bitstream. Otherwise, the value of the second identification value can be set to another identification value (the other identification value is different from the first preset identification value), and then the first preset identification value can be written into the bitstream.

[0326] S704: Obtain scene description information of the scene.

[0327] S705: Determine metadata of the scene audio signal according to the scene description information of the scene.

[0328] S706: Write the metadata of the scene audio signal into the bitstream.

[0329] For example, S704 to S706 may refer to the description of S501 to S503 above, which will not be repeated here.

[0330] For example, in the embodiment of FIG. 7A , the present application can pre-add the syntax elements and corresponding description information of the metadata of the scene audio signal, as well as the syntax elements and corresponding description information of the second identifier, to the original syntax structure of the bitstream. In this way, the encoder can write the second identifier into the bitstream based on the syntax elements and corresponding description information of the second identifier in the syntax structure of the bitstream.

[0331] For example, the syntax structure of the code stream can refer to the following Table 4:

[0332] Table 4

[0333] Referring to Table 4, in the embodiment of Figure 7A, the present application adds new syntax elements "distanceFactor" and "hoaGroupHasLowProfileConfig"; among them, "distanceFactor" is the syntax element of the metadata of the scene audio signal, and "hoaGroupHasLowProfileConfig" is the syntax element of the second identifier.

[0334] The "1" in the fourth-to-last row of Table 4 indicates that the second identifier in the codestream can be represented by a single bit. The "bslbf" in the fourth-to-last row of Table 4 indicates that the second identifier in the codestream is a bit string with the left bit first. It should be understood that Table 4 is merely an example of this application; the second identifier can also be represented by other numbers of bits, such as two bits, without limitation. Furthermore, the syntax element of the second identifier can also be represented by a string, without limitation.

[0335] The explanation of other syntax elements in Table 4 can refer to the description in Table 1 and will not be repeated here.

[0336] For example, if the syntax structure of the bitstream is as shown in Table 4, then during S703, the second identifier can be converted into a 1-bit binary number; then, this 1-bit binary number is written after the data identified by syntax element n+1 in the bitstream. Furthermore, during S706, the metadata of the scene audio signal can be converted into an 8-bit binary number; then, this 8-bit binary number is written after the second identifier in the bitstream. When the syntax structure of the bitstream is as shown in Table 4, the value of the second identifier is "1."

[0337] In one possible approach, the first identifier value of the renderer that renders based on the metadata of the scene audio signal can also be used as the value of the second identifier. Assuming that the first identifier value of the renderer that renders based on the metadata of the scene audio signal is "3", the syntax structure of the code stream can be as shown in Table 5:

[0338] For example, the syntax structure of the code stream can refer to the following Table 5:

[0339] Table 5

[0340] FIG7B is a schematic diagram of an exemplary bitstream structure. The bitstream structure in the embodiment of FIG7A may be as shown in FIG7B , that is, the second identifier is located after the encoded data of the scene audio signal, and the metadata of the scene audio signal is located after the second identifier.

[0341] It should be understood that the present application does not limit the positions of the metadata of the scene audio signal, the encoded data of the scene audio signal, and the second identifier in the bitstream.

[0342] It should be understood that, in the code stream of the present application, any two of the metadata of the scene audio signal, the encoded data of the scene audio signal, and the second identifier may be adjacent or non-adjacent, and the present application does not impose any restrictions on this.

[0343] Fig. 8 is a schematic diagram illustrating an exemplary decoding process, which corresponds to the encoding process of Fig. 7A.

[0344] S801, receiving a code stream.

[0345] S802: Obtain a reconstructed audio signal of the scene audio signal according to the code stream.

[0346] For example, S801 to S802 may refer to the above description and will not be repeated here.

[0347] S803: Parse the second identifier from the code stream.

[0348] S804: Determine whether the value of the second identifier is the first preset identifier value.

[0349] Exemplarily, the second identifier may be first parsed from the bitstream; then, it is determined whether the second identifier is a first preset identifier value to determine whether metadata of the scene audio signal can be parsed from the bitstream.

[0350] For example, when the value of the second identifier is the first preset identifier value, it indicates that the bitstream contains metadata for the scene audio signal, and in this case, S805 can be executed. When the value of the second identifier is another identifier value, it indicates that the bitstream does not contain metadata for the scene audio signal, and in this case, the field after the second identifier in the bitstream is decoded.

[0351] Exemplarily, when the code stream structure is as shown in Table 4, it can be determined whether the value of the second identifier is 1. When the value of the second identifier is 1, it indicates that the code stream contains metadata of the scene audio signal; when the value of the second identifier is not 1, it indicates that the code stream does not contain metadata of the scene audio signal.

[0352] Exemplarily, when the codestream structure of the codestream is as shown in Table 5, it can be determined whether the value of the second identifier is 3. When the value of the second identifier is 3, it indicates that the codestream contains metadata of the scene audio signal; when the value of the second identifier is not 3, it indicates that the codestream does not contain metadata of the scene audio signal.

[0353] S805: parse the metadata of the scene audio signal from the bitstream.

[0354] Exemplarily, there may be multiple renderers at the decoding end (each renderer adopts a different rendering method), wherein these multiple renderers include: a renderer that renders according to the metadata of the scene audio signal and the first identifier (for example, a renderer that renders according to the method in S11 to S16 above) and a renderer that renders according to other information. Furthermore, a corresponding identification value can be set for each renderer. For example, for renderer A, the corresponding identification value is set to "0"; for renderer B, the corresponding identification value is set to "1"; for renderer C, the corresponding identification value is set to "2", and so on. When it is determined that the decoding end needs to render according to the metadata of the scene audio signal and the first identifier, in one possible way, the identification value of the renderer that renders according to the metadata of the scene audio signal and the first identifier (hereinafter referred to as the first identification value) can be used as the identification value of the second identifier; in another possible way, the second identification value (such as "0" or "1", which can be set as required) can be used as the identification value of the second identifier; and then the second identifier is added to the bitstream. The second identifier value is used to instruct the decoder to perform rendering based on the metadata of the scene audio signal and the first identifier. In other words, the second identifier value is used to indicate that the bitstream includes the metadata of the scene audio signal and the first identifier. This facilitates the decoder to determine which renderer or rendering algorithm to use for rendering.

[0355] Fig. 9A is a schematic diagram illustrating an exemplary encoding process. It should be noted that Fig. 9A is shown based on Fig. 5A. In the embodiment of Fig. 9A, the metadata of the scene audio signal is determined according to the scene description information.

[0356] S901: Acquire a scene audio signal.

[0357] S902: Encode the scene audio signal to obtain a bit stream.

[0358] For example, S901 to S902 may refer to the above description and will not be repeated here.

[0359] S903: Write the second identifier into the code stream.

[0360] S904: Obtain scene description information of the scene.

[0361] S905 : Determine metadata of the scene audio signal according to the scene description information of the scene.

[0362] S906: Write the metadata of the scene audio signal into the bitstream.

[0363] S907: Write the first identifier into the code stream.

[0364] For example, this application does not limit the execution order of S906 and S907.

[0365] Exemplarily, in the embodiment of FIG9A , the present application may pre-add the syntax elements and corresponding description information of the metadata of the scene audio signal, add the syntax elements and corresponding description information of the second identifier, and add the syntax elements and corresponding description information of the first identifier in the original syntax structure of the bitstream.

[0366] For example, the syntax structure of the code stream can refer to the following Table 6:

[0367] Table 6

[0368] Figure 9B is a schematic diagram of an exemplary code stream structure. Figure 9B is a schematic diagram of an exemplary code stream structure. The code stream structure in the embodiment of Figure 9A can be as shown in Figure 9B, that is, the second identifier is located after the encoded data of the scene audio signal, the metadata of the scene audio signal is located after the second identifier, and the first identifier is located after the metadata of the scene audio signal.

[0369] It should be understood that the present application does not limit the positions of the metadata of the scene audio signal, the encoded data of the scene audio signal, the first identifier, and the second identifier in the bitstream.

[0370] It should be understood that in the code stream of this application, any two of the metadata of the scene audio signal, the encoded data of the scene audio signal, the first identifier and the second identifier may be adjacent or non-adjacent, and this application does not impose any restrictions on this.

[0371] Figure 10 is a schematic diagram illustrating an exemplary decoding process, which corresponds to the encoding process of Figure 9A.

[0372] S1001, receiving a code stream.

[0373] S1002: Obtain a reconstructed audio signal of the scene audio signal according to the code stream.

[0374] S1003: Parse the second identifier from the code stream.

[0375] S1004: Determine whether the value of the second identifier is the first preset identifier value.

[0376] S1005: parse the bitstream to obtain metadata of the scene audio signal.

[0377] S1006: Parse the first identifier from the code stream.

[0378] For example, S1001 to S1006 may refer to the description of the above embodiment and will not be repeated here.

[0379] It should be noted that in the embodiments of Figures 5A to 10 , a third identifier can also be written into the bitstream during the encoding process. This third identifier can be used to indicate whether the bitstream includes the first identifier. In this case, the second identifier can be used only to indicate whether the bitstream includes metadata for the scene audio signal. For example, when the first identifier is written into the bitstream, the value of the third identifier can be set to the second preset identifier value, and then the second preset identifier value can be written into the bitstream.

[0380] Correspondingly, the present application can pre-add the syntax elements and corresponding description information of the metadata of the scene audio signal to the original syntax structure of the code stream, add the syntax elements and corresponding description information of the second identifier, add the syntax elements and corresponding description information of the first identifier, and add the syntax elements and corresponding description information of the third identifier.

[0381] For example, the syntax structure of the code stream can refer to the following Table 7:

[0382] Table 7

[0383] In Table 7, "hoaGroupHascomplexityConfig" is a syntax element of the third flag; and the second preset flag value is "1".

[0384] In this way, the decoding end can decode the third identifier from the code stream; when determining that the value of the third identifier is the second preset identifier value, it can be determined that the code stream includes the first identifier; and then decode the first identifier from the code stream.

[0385] Exemplarily, the present application may also only write the first identifier into the bitstream. In this case, the decoding end may obtain metadata of the scene audio signal from other electronic devices or generate metadata of the scene audio signal locally.

[0386] FIG11 is a schematic diagram illustrating an exemplary encoding process.

[0387] S1101, obtaining a scene audio signal;

[0388] S1102, encode the scene audio signal and the first identifier to obtain a bitstream; wherein the first identifier is used to indicate rendering complexity, the rendering complexity is used to determine the number of virtual speakers, the reconstructed audio signal of the scene audio signal is used to generate a virtual speaker signal, and the virtual speaker signal and the number of virtual speakers are used for rendering to obtain a rendered audio signal.

[0389] S1101 to S1102 can refer to the above description and will not be repeated here.

[0390] The syntax element of the first identifier and the corresponding description information may be added to the syntax structure of the code stream.

[0391] For example, the syntax structure of the code stream may be as shown in Table 8 below:

[0392] Table 8

[0393] FIG12 is a schematic diagram illustrating an exemplary decoding process. The decoding process in FIG12 corresponds to the encoding process in FIG11 .

[0394] S1201, receiving code stream;

[0395] S1202: Obtain a reconstructed audio signal of the scene audio signal according to the code stream.

[0396] S1203: Parse a first identifier from the bitstream, where the first identifier is used to indicate rendering complexity, the rendering complexity is used to determine the number of virtual speakers, the reconstructed audio signal is used to generate a virtual speaker signal, and the virtual speaker signal and the number of virtual speakers are used for rendering to obtain a rendered audio signal.

[0397] For example, S1201 to S1203 can refer to the above description and will not be repeated here.

[0398] In addition, based on Figures 11 and 12, the encoding end can also write a second identifier into the code stream, where the second identifier is used to indicate whether the code stream contains the first identifier; illustratively, when the first identifier is written into the code stream, the value of the second identifier can be set to a preset identifier value, and then the preset identifier value is written into the code stream.

[0399] For example, the syntax structure of the code stream can refer to the following Table 9:

[0400] Table 9

[0401] In Table 9, "hoaGroupHascomplexityConfig" is a syntax element of the second flag; and the preset flag value is "1".

[0402] In this way, the decoding end can decode the second identifier from the code stream; when it is determined that the value of the second identifier is the preset identifier value, it can be determined that the code stream includes the first identifier; and then the first identifier can be decoded from the code stream.

[0403] In addition, the syntax element of the second identifier and the corresponding description information may be added to the syntax structure of the code stream. Detailed descriptions may be referred to above and will not be repeated here.

[0404] Fig. 13 is a schematic diagram of an exemplary binaural rendering signal generation process. The embodiment of Fig. 13 is executed by a terminal device, which may include a decoder and a renderer. Fig. 13 is shown based on Fig. 4A.

[0405] S1301: Receive a code stream.

[0406] S1302: parse the bitstream to obtain metadata of the scene audio signal.

[0407] S1303: Obtain a reconstructed audio signal of the scene audio signal according to the code stream.

[0408] For example, S1301 to S1303 may refer to the description of S401 to S403 above, which will not be repeated here.

[0409] S1304: Generate a virtual speaker signal according to the reconstructed audio signal of the scene audio signal.

[0410] S1305 : Determine a virtual speaker radius according to the distance between the first metadata and two microphones among the multiple microphones.

[0411] S1306 , performing rendering according to the virtual speaker radius and the virtual speaker signal to obtain a rendered audio signal.

[0412] Among them, S1304 to S1306 can refer to the description of the above S11 to S16, and will not be repeated here.

[0413] It should be noted that S1305 to S1306 can be executed by the renderer; in one possible manner, S1301 to S1303 can be executed by the decoder; in one possible manner, S1301 to S1302 can be executed by the decoder, and S1303 can be implemented collaboratively by the decoder and other modules.

[0414] Figure 14 is a schematic diagram of an exemplary encoding device, wherein the encoding device can be used to execute the method of the aforementioned embodiment, and therefore, the beneficial effects that can be achieved can refer to the beneficial effects of the corresponding method provided above, which will not be repeated here.

[0415] A first data acquisition module 1401 is configured to acquire a scene audio signal and metadata of the scene audio signal, wherein the scene audio signal is used to describe a sound field of a sound source in a scene, the scene includes multiple microphones, the metadata includes first metadata, the first metadata is used to determine a virtual speaker radius based on a distance between two microphones among the multiple microphones, the virtual speaker radius and the virtual speaker signal are used for rendering to obtain a rendered audio signal, and the virtual speaker signal is generated based on a reconstructed audio signal of the scene audio signal;

[0416] The first encoding module 1402 is configured to encode the scene audio signal and metadata to obtain a bitstream.

[0417] Figure 15 is a schematic diagram of an exemplary decoding device, wherein the decoding device can be used to execute the method of the aforementioned embodiment, and therefore, the beneficial effects that can be achieved can refer to the beneficial effects of the corresponding method provided above, which will not be repeated here.

[0418] The first receiving module 1501 is used to receive a code stream;

[0419] A first parsing module 1502 is configured to parse the bitstream to obtain metadata of the scene audio signal;

[0420] The first signal acquisition module 1503 is used to obtain a reconstructed audio signal of the scene audio signal based on the bit stream; the scene audio signal is used to describe the sound field of the sound source in the scene, the scene includes multiple microphones, the metadata includes first metadata, the first metadata is used to determine the virtual speaker radius with the distance between two microphones among the multiple microphones, the virtual speaker radius and the virtual speaker signal are used for rendering to obtain a rendered audio signal, and the virtual speaker signal is generated based on the reconstructed audio signal.

[0421] Figure 16 is a schematic diagram of an exemplary encoding device, wherein the encoding device can be used to execute the method of the aforementioned embodiment, and therefore, the beneficial effects that can be achieved can refer to the beneficial effects of the corresponding method provided above, which will not be repeated here.

[0422] The second data acquisition module 1601 is used to acquire a scene audio signal;

[0423] The second encoding module 1602 is used to encode the scene audio signal and the first identifier to obtain a code stream; the first identifier is used to indicate the rendering complexity, the rendering complexity is used to determine the number of virtual speakers, the reconstructed audio signal of the scene audio signal is used to generate a virtual speaker signal, and the virtual speaker signal and the number of virtual speakers are used for rendering to obtain a rendered audio signal.

[0424] Figure 17 is a schematic diagram of an exemplary decoding device, wherein the decoding device can be used to execute the method of the aforementioned embodiment, and therefore, the beneficial effects that can be achieved can refer to the beneficial effects of the corresponding method provided above, which will not be repeated here.

[0425] The second receiving module 1701 is used to receive a code stream;

[0426] A second parsing module 1702 is configured to parse the code stream to obtain a first identifier;

[0427] The second signal acquisition module 1703 is used to obtain a reconstructed audio signal of the scene audio signal according to the bit stream; the first identifier is used to indicate the rendering complexity, the rendering complexity is used to determine the number of virtual speakers, the reconstructed audio signal is used to generate a virtual speaker signal, and the virtual speaker signal and the number of virtual speakers are used for rendering to obtain a rendered audio signal.

[0428] In an example, FIG18 shows a schematic block diagram of a device 1800 according to an embodiment of the present application. The device 1800 may include: a processor 1801 and a transceiver / transceiver pin 1802 , and optionally, a memory 1803 .

[0429] The various components of the device 1800 are coupled together via a bus 1804, wherein the bus 1804 includes, in addition to a data bus, a power bus, a control bus, and a status signal bus. However, for the sake of clarity, all buses are referred to as bus 1804 in the figure.

[0430] Optionally, the memory 1803 may be used to store instructions in the aforementioned method embodiment. The processor 1801 may be used to execute the instructions in the memory 1803 and control the receiving pin to receive a signal and control the transmitting pin to send a signal.

[0431] The apparatus 1800 may be the electronic device or a chip of the electronic device in the above method embodiment.

[0432] Exemplarily, the electronic device may be a terminal device or a server.

[0433] Among them, all relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module and will not be repeated here.

[0434] The present application also provides a chip comprising one or more interface circuits and one or more processors; the one or more processors receive or send data via the one or more interface circuits, and when the one or more processors execute computer instructions, the steps of the above-mentioned related methods are implemented. The interface circuit is transceiver / transceiver pin 1802.

[0435] This embodiment further provides a computer-readable storage medium, in which computer instructions are stored. When the computer instructions are executed on an electronic device, the electronic device executes the above-mentioned related method steps to implement the method in the above-mentioned embodiment.

[0436] This embodiment further provides a computer program product, which includes computer instructions. When the computer instructions are executed by a computer or a processor, the computer executes the above-mentioned related steps to implement the method in the above-mentioned embodiment.

[0437] In addition, an embodiment of the present application also provides a device, which can specifically be a chip, component or module, and the device may include a connected processor and memory; wherein the memory is used to store computer-executable instructions, and when the device is running, the processor can execute the computer-executable instructions stored in the memory to enable the chip to execute the methods in the above-mentioned method embodiments.

[0438] Among them, the electronic device, computer-readable storage medium, computer program product or chip provided in this embodiment are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0439] Through the description of the above implementation methods, technical personnel in the relevant field can understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0440] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of modules or units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0441] Units described as separate components may or may not be physically separate, and components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0442] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0443] Any content of each embodiment of this application, as well as any content of the same embodiment, can be freely combined. Any combination of the above content is within the scope of this application.

[0444] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0445] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms without departing from the purpose of this application and the scope of protection of the claims, all of which are within the protection of this application.

[0446] The steps of the method or algorithm described in conjunction with the disclosure of the embodiments of the present application can be implemented in a hardware manner, or can be implemented by a processor executing a software instruction. The software instruction can be composed of corresponding software modules, and the software module can be stored in a random access memory (Random Access Memory, RAM), a flash memory, a read-only memory (Read Only Memory, ROM), an erasable programmable read-only memory (Erasable Programmable ROM, EPROM), an electrically erasable programmable read-only memory (Electrically EPROM, EEPROM), a register, a hard disk, a mobile hard disk, a read-only compact disc (CD-ROM) or any other form of storage medium well known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and can write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC.

[0447] Those skilled in the art will appreciate that in one or more of the above examples, the functions described in the embodiments of the present application can be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer-readable storage media and communication media, wherein communication media include any media that facilitates the transmission of computer programs from one place to another. The storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0448] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms without departing from the purpose of this application and the scope of protection of the claims, all of which are within the protection of this application.

Claims

1. A decoding method, characterized in that: The method comprises: Receive code stream; Parsing metadata of the scene audio signal from the bitstream; According to the bit stream, a reconstructed audio signal of the scene audio signal is obtained; the scene audio signal is used to describe the sound field of the sound source in the scene, the scene includes multiple microphones, the metadata includes first metadata, the first metadata is used to determine a virtual speaker radius with a distance between two microphones among the multiple microphones, the virtual speaker radius and the virtual speaker signal are used for rendering to obtain a rendered audio signal, and the virtual speaker signal is generated according to the reconstructed audio signal.

2. The method according to claim 1, characterized in that: The method further comprises: Parsing a first identifier from the bit stream; The first identifier is used to indicate rendering complexity, and the rendering complexity is used to determine the number of the virtual speakers.

3. The method according to claim 1 or 2, characterized in that: The method further comprises: Parsing a second identifier from the code stream; Wherein, when the value of the second identifier is the first preset identifier value, it indicates that the code stream includes the metadata.

4. The method according to claim 3, characterized in that The method further comprises: When the value of the second identifier is the first preset identifier value, it indicates that the code stream also includes the first identifier.

5. The method according to claim 2, characterized in that: The method further comprises: Parsing a third identifier from the bit stream; When the value of the third identifier is a second preset identifier value, it indicates that the code stream includes the first identifier.

6. The method according to any one of claims 1 to 5, characterized in that: The metadata also includes second metadata for determining a distance between two microphones of the plurality of microphones.

7. The method according to any one of claims 1 to 6, characterized in that: The scene audio signal includes a multi-point high-order surround audio reverberation (HOA) signal, and the reconstructed audio signal includes a reconstructed audio signal of the multi-point HOA signal.

8. The method according to any one of claims 1 to 7, characterized in that: The code stream includes: syntax elements of the metadata and corresponding description information; The step of parsing metadata of the scene audio signal from the bitstream includes: The metadata is parsed from the bitstream according to the syntax elements of the metadata and the corresponding description information.

9. The method according to any one of claims 2 to 8, characterized in that: The code stream includes: a syntax element of a first identifier and corresponding description information, and the parsing of the first identifier from the code stream includes: The first identifier is parsed from the bitstream according to the syntax element of the first identifier and the corresponding description information.

10. The method according to any one of claims 3 to 9, characterized in that: The code stream includes: a syntax element of a second identifier and corresponding description information, and the parsing of the second identifier from the code stream includes: The second identifier is parsed from the bitstream according to the syntax element of the second identifier and the corresponding description information.

11. The method according to any one of claims 5 to 9, characterized in that: The code stream includes: a syntax element of a third identifier and corresponding description information, and the parsing of the third identifier from the code stream includes: The third identifier is parsed from the bitstream according to the syntax element of the third identifier and the corresponding description information.

12. The method according to claim 8, characterized in that The syntax element of the first metadata is distanceFactor.

13. The method according to claim 9, characterized in that The syntax element of the first identifier is complexityLevel.

14. The method according to claim 10, characterized in that The syntax element of the second identifier is hoaGroupHasLowProfileConfig.

15. The method according to any one of claims 1 to 14, characterized in that The method further comprises: generating a virtual loudspeaker signal according to the reconstructed audio signal of the scene audio signal; determining a virtual speaker radius according to a distance between the first metadata and two microphones among the plurality of microphones; Rendering is performed according to the virtual speaker radius and the virtual speaker signal to obtain the rendered audio signal.

16. A decoding method, characterized in that: include: Receive code stream; Parsing a first identifier from the bit stream; According to the bit stream, a reconstructed audio signal is obtained, wherein the first identifier is used to indicate rendering complexity, the rendering complexity is used to determine the number of virtual speakers, the reconstructed audio signal is used to generate a virtual speaker signal, and the virtual speaker signal and the number of virtual speakers are used for rendering to obtain a rendered audio signal.

17. The method according to claim 16, characterized in that The method further comprises: Parsing a second identifier from the code stream; When the value of the second identifier is a preset identifier value, it indicates that the code stream includes the first identifier.

18. The method according to claim 16 or 17, characterized in that The code stream includes: the syntax element of the first identifier and the corresponding description information, and the parsing of the first identifier from the code stream includes: The first identifier is parsed from the bitstream according to the syntax element of the first identifier and the corresponding description information.

19. The method according to claim 17 or 18, characterized in that The code stream includes: a syntax element of a second identifier and corresponding description information, and the parsing of the second identifier from the code stream includes: The second identifier is parsed from the bitstream according to the syntax element of the second identifier and the corresponding description information.

20. The method according to any one of claims 16 to 19, characterized in that The reconstructed audio signal includes a reconstructed audio signal of a multi-point high-order ambisonics (HOA) signal.

21. The method according to claim 18, characterized in that The syntax element of the first identifier is complexityLevel.

22. The method according to any one of claims 16 to 21, characterized in that The method further comprises: generating a virtual loudspeaker signal according to the reconstructed audio signal of the scene audio signal; Determining the number of the virtual speakers according to the first identifier; Rendering is performed according to the number of the virtual speakers and the virtual speaker signal to obtain the rendered audio signal.

23. A decoding device, characterized in that: The device comprises: A first receiving module, used for receiving a code stream; A first parsing module, used for parsing metadata of a scene audio signal from the bit stream; The first signal acquisition module is used to acquire a reconstructed audio signal of the scene audio signal according to the bit stream; the scene audio signal is used to describe the sound field of the sound source in the scene, the scene includes multiple microphones, the metadata includes first metadata, and the first metadata is used to describe the sound field of the sound source in the scene. A virtual speaker radius is determined based on a distance from two microphones among the plurality of microphones, the virtual speaker radius and a virtual speaker signal are used for rendering to obtain a rendered audio signal, and the virtual speaker signal is generated according to the reconstructed audio signal.

24. A decoding device, characterized in that: The device comprises: A second receiving module, used for receiving a code stream; A second parsing module, used for parsing a first identifier from the bit stream; A second signal acquisition module is used to obtain a reconstructed audio signal of the scene audio signal according to the bit stream; the first identifier is used to indicate rendering complexity, the rendering complexity is used to determine the number of virtual speakers, the reconstructed audio signal is used to generate a virtual speaker signal, and the virtual speaker signal and the number of virtual speakers are used for rendering to obtain a rendered audio signal.

25. An extended reality (XR) device, characterized in that: The XR device includes: a processor and an audio playback module; The processor is used to execute the decoding method according to any one of claims 1 to 22; The audio playback module is used to play the rendered audio signal as described in any one of claims 1 to 22 above.

26. An electronic device, characterized in that: include: a memory and a processor, the memory being coupled to the processor; The memory stores program instructions, and when the program instructions are executed by the processor, the electronic device executes the decoding method according to any one of claims 1 to 22.

27. A chip, characterized in that: It comprises one or more interface circuits and one or more processors; the one or more processors receive or send data through the one or more interface circuits, and when the one or more processors execute computer instructions, the steps of the method described in any one of claims 1 to claim 22 are executed.

28. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program runs on a computer or a processor, the computer or the processor executes the decoding method according to any one of claims 1 to 22.

29. A computer program product, characterized in that The computer program product comprises computer instructions, and when the computer instructions are executed by a computer or a processor, the steps of the method according to any one of claims 1 to 22 are performed.