Decoding method and electronic equipment

Abstract

The embodiment of the invention provides a decoding method and electronic equipment. The method comprises the following steps: firstly, receiving a code stream; analyzing metadata of the scene audio signal from the code stream; and obtaining a reconstructed audio signal of the scene audio signal according to the code stream, the scene audio signal is used for describing a sound field of a sound source in a scene, the scene comprises a plurality of microphones, the metadata comprises first metadata, and the first metadata is used for determining a virtual loudspeaker radius according to the distance between two microphones in the plurality of microphones; the virtual speaker radius and the virtual speaker signal are used for rendering to obtain a rendered audio signal, the virtual speaker signal being generated from the reconstructed audio signal. Therefore, the decoding end does not need to acquire the metadata of the scene audio signal from other electronic equipment, and does not need to generate the metadata of the scene audio signal locally, so that the calculation amount of generating the metadata of the scene audio signal is reduced; therefore, the time for obtaining the binaural rendering signal through rendering can be saved, and the user experience is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of coding and decoding, and in particular to a decoding method and an electronic device. Background Art

[0002] Aiming at the immersive audio requirements of Extended Reality (XR) applications, the MPEG-I audio standard provides a 6-degree-of-freedom (DoF) audio technology based on multi-point higher-order ambisonics (HOA): first, several sound sources and multi-point HOA microphones are deployed in the room; then, the sound sources play audio signals, and the multi-point HOA microphones collect the sounds in the room to obtain multi-point HOA signals; after that, the multi-point HOA signals pass through the audio encoder to obtain a bit stream, and after the audio decoder decodes the bit stream, it renders the decoded signal, HOA metadata, user position and posture information, and obtains a binaural rendering signal. Summary of the invention

[0003] In view of this, the present application provides a decoding method and an electronic device. In the decoding method, a decoding end can parse metadata of a scene audio signal from a bit stream, so that the decoding end does not need to obtain metadata of the scene audio signal from other electronic devices, nor does it need to generate metadata of the scene audio signal locally, thereby saving time for rendering to obtain binaural rendering signals and improving user experience.

[0004] In the first aspect, an embodiment of the present application provides an encoding method, which includes: first, obtaining a scene audio signal and metadata of the scene audio signal; wherein the scene audio signal is used to describe the sound field of the sound source in the scene, the scene includes multiple microphones, the metadata includes first metadata, the first metadata is used to determine the virtual speaker radius with the distance between two microphones of the multiple microphones, the virtual speaker radius and the virtual speaker signal are used for rendering to obtain a rendered audio signal, and the virtual speaker signal is generated according to the reconstructed audio signal of the scene audio signal; then, encoding the scene audio signal and the metadata to obtain a bitstream. In this way, after the bitstream is transmitted to the decoding end, the decoding end can parse the metadata of the scene audio signal from the bitstream, and apply the metadata of the scene audio signal to the subsequent rendering process to obtain the rendered audio signal; and then the decoding end does not need to obtain the metadata of the scene audio signal from other electronic devices, nor does it need to generate the metadata of the scene audio signal locally, which improves the efficiency of the decoding end in obtaining the metadata of the scene audio signal and reduces the computational complexity of generating the metadata of the scene audio signal; thereby, the time for rendering to obtain the rendered audio signal can be saved, and the user experience can be improved.

[0005] Exemplarily, the rendered audio signal may refer to an audio signal output to a playback module (such as headphones) for playback, and may be a binaural rendering signal.

[0006] Exemplarily, the distance between any two microphones among the multiple microphones is equal.

[0007] Exemplarily, the scene audio signal may include: a multi-point HOA signal (wherein the HOA signal may include a three-dimensional HOA signal and a two-dimensional HOA signal (also referred to as a planar HOA signal)) and a three-dimensional audio signal; the three-dimensional audio signal may refer to other audio signals in the scene audio signal except the HOA signal. This application takes the scene audio signal as a multi-point HOA signal as an example for explanation.

[0008] Exemplarily, the microphone may be an HOA microphone, wherein a plurality of sound sources and multi-point HOA microphones may be deployed in a room (that is, a real scene); then, the sound source plays an audio signal, and the multi-point HOA microphones collect the sound in the room to obtain a multi-point HOA signal.

[0009] Exemplarily, the scene audio signal may refer to an audio signal to be encoded by the encoding end, and the reconstructed audio signal of the scene audio signal refers to an audio signal obtained by the decoding end or a decoding module in the encoding end according to a bit stream obtained by encoding the scene audio signal. The reconstructed audio signal may also be referred to as a reconstructed scene audio signal.

[0010] It should be noted that the present application does not limit the order of encoding the scene audio signal and encoding the metadata.

[0011] According to the first aspect, the method further includes: writing a first identifier into a bitstream; wherein the first identifier is used to indicate rendering complexity, and the rendering complexity is used to determine the number of virtual speakers. In this way, the decoding end can render according to the number of virtual speakers, the radius of the virtual speakers, and the virtual speaker signal to obtain a rendered audio signal.

[0012] Exemplarily, different rendering complexities correspond to different audio qualities of binaural rendering signals obtained by rendering; wherein, the value of the first identifier can be customized by the user or determined based on the computing power of the decoding end, so that the decoding end can render a binaural rendering signal whose audio quality matches factors such as user requirements and the computing power of the electronic device.

[0013] According to the first aspect, or any implementation of the first aspect above, a second identifier is written into the bitstream; wherein the second identifier is used to indicate whether the bitstream contains metadata of the scene audio signal.

[0014] There may be multiple renderers at the decoding end (each renderer adopts a different rendering method), and these multiple renderers include: a renderer that renders according to the metadata of the scene audio signal and a renderer that renders according to other information; further, the second identifier can also be used to indicate whether to render according to the metadata of the scene audio signal. In this way, the decoding end can determine which renderer or rendering algorithm to use for rendering according to the value of the second identifier parsed from the bitstream.

[0015] According to the first aspect, or any implementation of the first aspect above, the second identifier is further used to indicate whether the bitstream also includes the first identifier.

[0016] The multiple renderers included in the decoding end may include: a renderer that renders according to the metadata of the scene audio signal and the first identifier and a renderer that renders according to other information; and the second identifier may also be used to indicate whether to render according to the metadata of the scene audio signal and the first identifier. In this way, the decoding end can determine which renderer or rendering algorithm to use for rendering according to the value of the second identifier parsed from the bitstream.

[0017] According to the first aspect, or any implementation of the first aspect above, the method further includes: writing a third identifier into the bitstream; wherein the third identifier is used to indicate whether the bitstream contains the first identifier. In this case, the second identifier can only be used to indicate whether the bitstream contains metadata.

[0018] According to the first aspect, or any implementation of the first aspect above, obtaining metadata of the scene audio signal includes: obtaining scene description information of the scene; and determining metadata according to the scene description information of the scene. In this way, the binaural rendering signal obtained by rendering according to the metadata of the scene audio signal is closer to the real audio signal heard by the user in the real environment.

[0019] Exemplarily, the scene description information can be used to describe the scene information, for example, it may include but is not limited to: the identification, location information, characteristics and direction of multiple HOA microphones in the real scene, the coordinates and materials of the objects contained in the real scene, the identification, location information, characteristics and direction of the sound source in the real scene, the critical distance of the room in the real scene, acoustic reverberation parameters (such as direct-to-Reverberation energy Ratio (DRR), reverberation time (Reverberation Time) (such as T20 (time for energy decay of 20dB), T30, T60, etc.), clarity (Clarity) (such as C50 (the ratio of sound energy before and after 50ms), C80, etc.), etc.), etc., and the present application does not impose any restrictions on this.

[0020] For example, the critical distance of a room can refer to the distance from the point where the sound energy density is equal to the reverberation sound energy density in a certain direction to the center of the sound source when the sound source radiates sound waves stably in the room (real environment), also known as the diffusion field distance. When the radiation directivity of the sound source is spherical, the critical distances in all directions are equal, which is also called the reverberation radius.

[0021] According to the first aspect, or any implementation of the first aspect above, metadata is determined based on scene description information of the scene, including: when the scene description information includes a room critical distance of the scene, first metadata is determined based on the product value of the room critical distance and a preset value.

[0022] According to the first aspect, or any implementation of the first aspect above, metadata is determined based on scene description information of the scene, including: when the scene description information includes acoustic reverberation parameters of the scene, determining the reverberation type of the scene based on the acoustic reverberation parameters of the scene; and determining first metadata based on the reverberation type of the scene.

[0023] The reverberation type may include multiple types, and different reverberation types correspond to different first metadata. For example, the first reverberation type (corresponding to a dry sound scene or a room without reverberation) corresponds to 0.5, and the second reverberation type (corresponding to a wet sound scene or a reverberation room) corresponds to 1.5.

[0024] According to the first aspect, or any implementation of the first aspect above, the metadata also includes second metadata, and the metadata is determined according to the scene description information of the scene, including: when the scene description information includes position information of multiple microphones, the position information of the multiple microphones is used as the second metadata; wherein the second metadata is used to determine the distance between two microphones among the multiple microphones

[0025] According to the first aspect, or any implementation of the first aspect above, the metadata also includes second metadata, and the metadata is determined according to scene description information of the scene, including: when the scene description information includes position information of multiple microphones, determining the distance between two microphones among the multiple microphones according to the position information of the multiple microphones; and using the distance between two microphones among the multiple microphones as the second metadata.

[0026] According to the first aspect, or any implementation of the first aspect above, obtaining metadata of the scene audio signal includes: obtaining metadata of the scene audio signal input by the user. That is, the user can customize the metadata of the scene audio signal, so that the binaural rendering signal obtained by subsequent rendering at the decoding end can match the user's personalized needs.

[0027] According to the first aspect, or any implementation manner of the first aspect above, the bitstream includes: syntax elements of metadata of the scene audio signal and corresponding description information.

[0028] Exemplarily, the syntax elements and corresponding description information of the metadata of the scene audio signal can be added to the original syntax structure of the bitstream (or called the high-level syntax element table); in this way, the encoder can write the metadata of the scene audio signal into the bitstream according to the syntax elements and corresponding description information of the metadata of the scene audio signal in the syntax structure of the bitstream.

[0029] According to the first aspect, or any implementation of the first aspect above, the code stream includes: a syntax element of a first identifier and corresponding description information.

[0030] Exemplarily, the syntax elements of the first identifier and the corresponding description information may be added to the original syntax structure of the bitstream; thus, the encoder may write the first identifier into the bitstream according to the syntax elements of the first identifier and the corresponding description information in the syntax structure of the bitstream.

[0031] According to the first aspect, or any implementation of the first aspect above, the code stream includes: a syntax element of the second identifier and corresponding description information.

[0032] Exemplarily, the syntax elements of the second identifier and the corresponding description information may be added to the original syntax structure of the bitstream; thus, the encoder may write the second identifier into the bitstream according to the syntax elements of the second identifier and the corresponding description information in the syntax structure of the bitstream.

[0033] According to the first aspect, or any implementation of the first aspect above, the code stream includes: a syntax element of a third identifier and corresponding description information.

[0034] According to the first aspect, or any implementation of the first aspect above, the scene audio signal includes a multi-point high-order stereo reverberation HOA signal.

[0035] According to the first aspect, or any implementation manner of the first aspect, the syntax element of the first metadata is distanceFactor.

[0036] According to the first aspect, or any implementation of the first aspect above, the syntax element of the first identifier is complexityLevel.

[0037] According to the first aspect, or any implementation of the first aspect, the syntax element of the second identifier is hoaGroupHasLowProfileConfig.

[0038] In the second aspect, an embodiment of the present application provides a decoding method, the method comprising: receiving a bit stream; parsing metadata of a scene audio signal from the bit stream; obtaining a reconstructed audio signal of the scene audio signal according to the bit stream; the scene audio signal is used to describe the sound field of a sound source in the scene, the scene includes multiple microphones, the metadata includes first metadata, the first metadata is used to determine a virtual speaker radius based on the distance between two microphones among the multiple microphones, the virtual speaker radius and the virtual speaker signal are used for rendering to obtain a rendered audio signal, and the virtual speaker signal is generated based on the reconstructed audio signal. In this way, the decoding end does not need to obtain metadata of the scene audio signal from other electronic devices, nor does it need to generate metadata of the scene audio signal locally, which improves the efficiency of obtaining metadata of the scene audio signal and reduces the amount of calculation for generating metadata of the scene audio signal; thereby, the time for rendering to obtain binaural rendering signals can be saved, and the user experience can be improved.

[0039] According to the second aspect, the method further includes: parsing a first identifier from the bitstream; wherein the first identifier is used to indicate rendering complexity, and the rendering complexity is used to determine the number of virtual speakers.

[0040] According to the second aspect, or any implementation of the second aspect above, the method further includes: parsing a second identifier from the bitstream; wherein, when the value of the second identifier is the first preset identifier value, it indicates that the bitstream includes metadata. That is, when the value of the second identifier is the first preset identifier value, it can be determined that the decoding end is rendered according to the metadata of the scene audio signal, and it can be determined that the bitstream contains the metadata of the scene audio signal; then, the bitstream can be further parsed to obtain the metadata of the scene audio signal.

[0041] According to the second aspect, or any implementation of the second aspect above, when the value of the second identifier is the first preset identifier value, it indicates that the bitstream also includes the first identifier. That is, when the value of the second identifier is the first preset identifier value, the decoding end can determine that the rendering is performed according to the metadata of the scene audio signal and the first identifier, and can also determine that the bitstream also includes the first identifier; then, the bitstream can continue to be parsed to obtain the first identifier.

[0042] According to the second aspect, or any implementation of the second aspect above, the method further includes: parsing a third identifier from the code stream; wherein, when the value of the third identifier is the second preset identifier value, it indicates that the code stream includes the first identifier.

[0043] According to the second aspect, or any implementation of the second aspect, the metadata further includes second metadata, and the second metadata is used to determine the distance between two microphones among the multiple microphones.

[0044] According to the second aspect, or any implementation of the second aspect, the scene audio signal includes a multi-point high-order stereo reverberation HOA signal, and the reconstructed audio signal includes a reconstructed audio signal of the multi-point HOA signal.

[0045] According to the second aspect, or any implementation of the second aspect above, the bitstream includes: syntax elements of metadata and corresponding description information; parsing metadata of the scene audio signal from the bitstream includes: parsing the metadata from the bitstream according to the syntax elements of the metadata and the corresponding description information.

[0046] According to the second aspect, or any implementation of the second aspect above, the code stream includes: a syntax element of a first identifier and corresponding description information, and parsing the first identifier from the code stream includes: parsing the first identifier from the code stream according to the syntax element of the first identifier and the corresponding description information.

[0047] According to the second aspect, or any implementation of the second aspect above, the code stream includes: a syntax element of the second identifier and corresponding description information, and parsing the second identifier from the code stream includes: parsing the second identifier from the code stream according to the syntax element of the second identifier and the corresponding description information.

[0048] According to the second aspect, or any implementation method of the above second aspect, the code stream includes: a syntax element of a third identifier and corresponding description information, and parsing the third identifier from the code stream includes: parsing the third identifier from the code stream according to the syntax element of the third identifier and the corresponding description information.

[0049] According to the second aspect, or any implementation of the second aspect, the syntax element of the first metadata is distanceFactor.

[0050] According to the second aspect, or any implementation of the second aspect, the syntax element of the first identifier is complexityLevel.

[0051] According to the second aspect, or any implementation of the second aspect, the syntax element of the second identifier is hoaGroupHasLowProfileConfig.

[0052] According to the second aspect, or any implementation of the second aspect above, the method also includes: generating a virtual speaker signal based on a reconstructed audio signal of the scene audio signal; determining a virtual speaker radius based on the distance between the first metadata and two microphones among the multiple microphones; rendering according to the virtual speaker radius and the virtual speaker signal to obtain a rendered audio signal.

[0053] The second aspect and any implementation of the second aspect correspond to the first aspect and any implementation of the first aspect respectively. The technical effects corresponding to the second aspect and any implementation of the second aspect can refer to the technical effects corresponding to the above-mentioned first aspect and any implementation of the first aspect, which will not be repeated here.

[0054] In a third aspect, an embodiment of the present application provides an encoding method, which includes: first, obtaining a scene audio signal; then, encoding the scene audio signal and a first identifier to obtain a bit stream; the first identifier is used to indicate the rendering complexity, and the rendering complexity is used to determine the number of virtual speakers, and the reconstructed audio signal of the scene audio signal is used to generate a virtual speaker signal, and the virtual speaker signal and the number of virtual speakers are used for rendering to obtain a rendered audio signal.

[0055] In this way, the decoding end can render according to the number of virtual speakers, the radius of the virtual speakers and the virtual speaker signal to obtain a rendered audio signal. Exemplarily, different rendering complexities correspond to different audio qualities of binaural rendering signals obtained by rendering; wherein the value of the first identifier can be set by a user or determined according to the computing power of the decoding end, so that the decoding end can render a binaural rendering signal whose audio quality matches factors such as user requirements and the computing power of the electronic device.

[0056] According to a third aspect, the method further includes: writing a second identifier into the code stream; wherein the second identifier is used to indicate whether the code stream contains the first identifier.

[0057] According to the third aspect, or any implementation of the third aspect above, the code stream includes: a syntax element of the first identifier and corresponding description information.

[0058] According to the third aspect, or any implementation of the third aspect above, the code stream includes: a syntax element of the second identifier and corresponding description information.

[0059] According to the third aspect, or any implementation of the third aspect, the scene audio signal includes a multi-point high-order stereo reverberation HOA signal.

[0060] According to the third aspect, or any implementation of the third aspect above, the syntax element of the first identifier is complexityLevel.

[0061] The third aspect and any implementation of the third aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the third aspect and any implementation of the third aspect can refer to the technical effects corresponding to the first aspect and any implementation of the first aspect, which will not be repeated here.

[0062] In a fourth aspect, an embodiment of the present application also provides a decoding method, which includes: receiving a bit stream; parsing a first identifier from the bit stream; obtaining a reconstructed audio signal based on the bit stream, wherein the first identifier is used to indicate a rendering complexity, the rendering complexity is used to determine the number of virtual speakers, the reconstructed audio signal is used to generate a virtual speaker signal, and the virtual speaker signal and the number of virtual speakers are used for rendering to obtain a rendered audio signal.

[0063] According to a fourth aspect, the method further includes: parsing a second identifier from the code stream; and indicating that the code stream includes the first identifier when the value of the second identifier is a preset identifier value.

[0064] According to the fourth aspect, or any implementation method of the fourth aspect above, the code stream includes: a syntax element of a first identifier and corresponding description information, and parsing the first identifier from the code stream includes: parsing the first identifier from the code stream according to the syntax element of the first identifier and the corresponding description information.

[0065] According to the fourth aspect, or any implementation method of the fourth aspect above, the code stream includes: a syntax element of the second identifier and corresponding description information, and parsing the second identifier from the code stream includes: parsing the second identifier from the code stream according to the syntax element of the second identifier and the corresponding description information.

[0066] According to the fourth aspect, or any implementation of the fourth aspect, the reconstructed audio signal includes a reconstructed audio signal of a multi-point high-order stereo reverberation HOA signal.

[0067] According to the fourth aspect, or any implementation of the fourth aspect above, the syntax element of the first identifier is complexityLevel.

[0068] According to the fourth aspect, or any implementation of the fourth aspect above, the method also includes: generating a virtual speaker signal based on a reconstructed audio signal of the scene audio signal; determining the number of virtual speakers based on the first identifier; rendering according to the number of virtual speakers and the virtual speaker signal to obtain a rendered audio signal.

[0069] The fourth aspect and any implementation of the fourth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the fourth aspect and any implementation of the fourth aspect can refer to the technical effects corresponding to the above-mentioned first aspect and any implementation of the first aspect, which will not be repeated here.

[0070] In a fifth aspect, an embodiment of the present application provides a coding device, the coding device comprising:

[0071] A first data acquisition module is used to acquire a scene audio signal and metadata of the scene audio signal; wherein the scene audio signal is used to describe the sound field of a sound source in the scene, the scene includes multiple microphones, the metadata includes first metadata, the first metadata is used to determine a virtual speaker radius with a distance between two microphones among the multiple microphones, the virtual speaker radius and the virtual speaker signal are used for rendering to obtain a rendered audio signal, and the virtual speaker signal is generated according to a reconstructed audio signal of the scene audio signal;

[0072] The first encoding module is used to encode the scene audio signal and metadata to obtain a bit stream.

[0073] In addition, the encoding device may further include a communication module.

[0074] It should be understood that the encoding device can also execute the above-mentioned first aspect and any one of the implementation methods of the first aspect, and the corresponding effects can also refer to the technical effects corresponding to the above-mentioned first aspect and any one of the implementation methods of the first aspect, which will not be repeated here.

[0075] In a sixth aspect, an embodiment of the present application provides a decoding device, the decoding device comprising:

[0076] A first receiving module, used for receiving a code stream;

[0077] A first parsing module, used for parsing metadata of a scene audio signal from a bit stream;

[0078] The first signal acquisition module is used to obtain a reconstructed audio signal of a scene audio signal according to a bit stream; the scene audio signal is used to describe the sound field of a sound source in the scene, the scene includes multiple microphones, the metadata includes first metadata, the first metadata is used to determine a virtual speaker radius from a distance between two microphones among the multiple microphones, the virtual speaker radius and the virtual speaker signal are used for rendering to obtain a rendered audio signal, and the virtual speaker signal is generated based on the reconstructed audio signal.

[0079] In addition, the decoding device may further include a communication module.

[0080] It should be understood that the decoding device can also execute the above-mentioned second aspect and any one of the implementation methods of the second aspect, and the corresponding effects can also refer to the technical effects corresponding to the above-mentioned second aspect and any one of the implementation methods of the second aspect, which will not be repeated here.

[0081] In a seventh aspect, an embodiment of the present application provides a coding device, the coding device comprising:

[0082] A second data acquisition module, used to acquire a scene audio signal;

[0083] The second encoding module is used to encode the scene audio signal and the first identifier to obtain a bit stream; the first identifier is used to indicate the rendering complexity, the rendering complexity is used to determine the number of virtual speakers, the reconstructed audio signal of the scene audio signal is used to generate a virtual speaker signal, the virtual speaker signal and the number of virtual speakers are used for rendering to obtain a rendered audio signal.

[0084] In addition, the encoding device may further include a communication module.

[0085] It should be understood that the encoding device can also execute the above-mentioned third aspect and any one of the implementation methods of the third aspect, and the corresponding effects can also refer to the technical effects corresponding to the above-mentioned third aspect and any one of the implementation methods of the third aspect, which will not be repeated here.

[0086] In an eighth aspect, an embodiment of the present application provides a decoding device, the decoding device comprising:

[0087] A second receiving module, used for receiving a code stream;

[0088] A second parsing module, used for parsing a first identifier from a bit stream;

[0089] The second signal acquisition module is used to obtain a reconstructed audio signal of the scene audio signal according to the bit stream; the first identifier is used to indicate the rendering complexity, the rendering complexity is used to determine the number of virtual speakers, the reconstructed audio signal is used to generate a virtual speaker signal, the virtual speaker signal and the number of virtual speakers are used for rendering to obtain a rendered audio signal.

[0090] In addition, the decoding device may further include a communication module.

[0091] It should be understood that the decoding device can also execute the above-mentioned fourth aspect and any one of the implementation methods of the fourth aspect, and the corresponding effects can also refer to the technical effects corresponding to the above-mentioned fourth aspect and any one of the implementation methods of the fourth aspect, which will not be repeated here.

[0092] In the ninth aspect, an embodiment of the present application provides an electronic device, comprising: a memory and a processor, the memory being coupled to the processor; the memory storing program instructions, which, when executed by the processor, enables the electronic device to execute the encoding method in the first aspect or any possible implementation of the first aspect; or, enables the electronic device to execute the encoding method in the third aspect or any possible implementation of the third aspect.

[0093] The ninth aspect and any implementation of the ninth aspect correspond to the first aspect and any implementation of the first aspect, respectively, or correspond to the third aspect and any implementation of the third aspect, respectively. The technical effects corresponding to the ninth aspect and any implementation of the ninth aspect can refer to the technical effects corresponding to the above-mentioned first aspect and any implementation of the first aspect, or can refer to the technical effects corresponding to the above-mentioned third aspect and any implementation of the third aspect, which will not be repeated here.

[0094] In the tenth aspect, an embodiment of the present application provides an electronic device, comprising: a memory and a processor, the memory being coupled to the processor; the memory storing program instructions, which, when executed by the processor, enables the electronic device to execute the decoding method in the second aspect or any possible implementation of the second aspect; or, enables the electronic device to execute the decoding method in the fourth aspect or any possible implementation of the fourth aspect.

[0095] The tenth aspect and any implementation of the tenth aspect correspond to the second aspect and any implementation of the second aspect, respectively, or correspond to the fourth aspect and any implementation of the fourth aspect, respectively. The technical effects corresponding to the tenth aspect and any implementation of the tenth aspect can refer to the technical effects corresponding to the above-mentioned second aspect and any implementation of the second aspect, or can refer to the above-mentioned fourth aspect and the technical effects corresponding to any implementation of the fourth aspect, which will not be repeated here.

[0096] In the eleventh aspect, an embodiment of the present application provides a chip, comprising one or more interface circuits and one or more processors; the one or more processors receive or send data through the one or more interface circuits, and when the one or more processors execute computer instructions, the method in the first aspect or any possible implementation of the first aspect is executed; or, the method in the third aspect or any possible implementation of the third aspect is executed.

[0097] The eleventh aspect and any implementation of the eleventh aspect correspond to the first aspect and any implementation of the first aspect, or correspond to the third aspect and any implementation of the third aspect, respectively. The technical effects corresponding to the eleventh aspect and any implementation of the eleventh aspect can refer to the technical effects corresponding to the above-mentioned first aspect and any implementation of the first aspect, or refer to the technical effects corresponding to the above-mentioned third aspect and any implementation of the third aspect, which will not be repeated here.

[0098] In the twelfth aspect, an embodiment of the present application provides a chip, comprising one or more interface circuits and one or more processors; the one or more processors receive or send data through the one or more interface circuits, and when the one or more processors execute computer instructions, the method in the second aspect or any possible implementation of the second aspect is executed; or, the method in the fourth aspect or any possible implementation of the fourth aspect is executed.

[0099] The twelfth aspect and any implementation of the twelfth aspect correspond to the second aspect and any implementation of the second aspect, respectively, or correspond to the fourth aspect and any implementation of the fourth aspect, respectively. The technical effects corresponding to the twelfth aspect and any implementation of the twelfth aspect can refer to the technical effects corresponding to the above-mentioned second aspect and any implementation of the second aspect, or can refer to the above-mentioned fourth aspect and the technical effects corresponding to any implementation of the fourth aspect, which will not be repeated here.

[0100] In the thirteenth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a computer or a processor, it enables the computer or the processor to execute the encoding method in the first aspect or any possible implementation of the first aspect; or, it enables the computer or the processor to execute the encoding method in the third aspect or any possible implementation of the third aspect.

[0101] The thirteenth aspect and any implementation of the thirteenth aspect correspond to the first aspect and any implementation of the first aspect, or correspond to the third aspect and any implementation of the third aspect, respectively. The technical effects corresponding to the thirteenth aspect and any implementation of the thirteenth aspect can refer to the technical effects corresponding to the above-mentioned first aspect and any implementation of the first aspect, or refer to the technical effects corresponding to the above-mentioned third aspect and any implementation of the third aspect, which will not be repeated here.

[0102] In the fourteenth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a computer or a processor, it enables the computer or the processor to execute the decoding method in the second aspect or any possible implementation of the second aspect; or, it enables the computer or the processor to execute the decoding method in the fourth aspect or any possible implementation of the fourth aspect.

[0103] The fourteenth aspect and any implementation of the fourteenth aspect correspond to the second aspect and any implementation of the second aspect, respectively, or correspond to the fourth aspect and any implementation of the fourth aspect, respectively. The technical effects corresponding to the fourteenth aspect and any implementation of the fourteenth aspect can refer to the technical effects corresponding to the above-mentioned second aspect and any implementation of the second aspect, or can refer to the above-mentioned fourth aspect and the technical effects corresponding to any implementation of the fourth aspect, which will not be repeated here.

[0104] In a fifteenth aspect, an embodiment of the present application provides a computer program product, which includes computer instructions. When the computer instructions are executed by a computer or a processor, the computer or the processor executes the encoding method in the first aspect or any possible implementation of the first aspect, or the computer or the processor executes the encoding method in the third aspect or any possible implementation of the third aspect.

[0105] The fifteenth aspect and any implementation of the fifteenth aspect correspond to the first aspect and any implementation of the first aspect, respectively, or correspond to the third aspect and any implementation of the third aspect, respectively. The technical effects corresponding to the fifteenth aspect and any implementation of the fifteenth aspect can refer to the technical effects corresponding to the above-mentioned first aspect and any implementation of the first aspect, or can refer to the above-mentioned third aspect and the technical effects corresponding to any implementation of the third aspect, which will not be repeated here.

[0106] In the sixteenth aspect, an embodiment of the present application provides a computer program product, which includes computer instructions. When the computer instructions are executed by a computer or a processor, the computer or the processor executes the encoding method in the second aspect or any possible implementation of the second aspect, or the computer or the processor executes the encoding method in the fourth aspect or any possible implementation of the fourth aspect.

[0107] The sixteenth aspect and any implementation of the sixteenth aspect correspond to the second aspect and any implementation of the second aspect, or correspond to the fourth aspect and any implementation of the fourth aspect, respectively. The technical effects corresponding to the sixteenth aspect and any implementation of the sixteenth aspect can refer to the technical effects corresponding to the above-mentioned second aspect and any implementation of the second aspect, or can refer to the above-mentioned fourth aspect and the technical effects corresponding to any implementation of the fourth aspect, which will not be repeated here.

[0108] In a seventeenth aspect, an embodiment of the present application further provides an XR device, the XR device comprising: a processor and an audio playback module;

[0109] The processor is used to execute the steps in the second aspect and any one of the implementations of the second aspect;

[0110] The audio playback module is used to play the rendered audio signal involved in the second aspect and any one of the implementations of the second aspect.

[0111] The seventeenth aspect and any implementation of the seventeenth aspect correspond to the second aspect and any implementation of the second aspect, respectively. The technical effects corresponding to the seventeenth aspect and any implementation of the seventeenth aspect can refer to the technical effects corresponding to the above-mentioned second aspect and any implementation of the second aspect, which will not be repeated here.

[0112] In an eighteenth aspect, an embodiment of the present application further provides an XR device, the XR device comprising: a processor and an audio playback module;

[0113] The processor is used to execute the steps in the fourth aspect and any one of the implementations of the fourth aspect;

[0114] The audio playback module is used to play the rendered audio signal involved in the fourth aspect and any one of the implementation methods of the fourth aspect.

[0115] The eighteenth aspect and any implementation of the eighteenth aspect correspond to the fourth aspect and any implementation of the fourth aspect, respectively. The technical effects corresponding to the eighteenth aspect and any implementation of the eighteenth aspect can refer to the technical effects corresponding to the fourth aspect and any implementation of the fourth aspect, which will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0116] Figure 1A is a schematic diagram of an application scenario shown as an example;

[0117] Figure 1B is a schematic diagram of an application scenario shown as an example;

[0118] Figure 1C is a schematic diagram of an application scenario shown as an example;

[0119] Figure 1D is a schematic diagram of an application scenario shown as an example;

[0120] Figure 2A is a schematic diagram of a compression framework of a scene audio signal shown as an example;

[0121] Figure 2B is a schematic diagram of a transcoding framework of a scene audio signal shown as an example;

[0122] Figure 2C is a schematic diagram of an exemplary rendering process;

[0123] Figure 2D A schematic diagram of a scene is shown as an example;

[0124] Figure 2E A schematic diagram of a scene is shown as an example;

[0125] Figure 3A is a schematic diagram of an exemplary encoding process;

[0126] Figure 3B is a schematic diagram of an exemplary code stream structure;

[0127] Figure 4A is a schematic diagram of an exemplary decoding process;

[0128] Figure 4B is a schematic diagram of a binaural rendering signal generation process shown as an example;

[0129] Figure 5A is a schematic diagram of an exemplary encoding process;

[0130] Figure 5B is a schematic diagram of an exemplary code stream structure;

[0131] Fig. 6A is a schematic diagram of an exemplary decoding process;

[0132] Figure 6B is a schematic diagram of a binaural rendering signal generation process shown as an example;

[0133] Fig. 7A is a schematic diagram of an exemplary encoding process;

[0134] Figure 7B is a schematic diagram of an exemplary code stream structure;

[0135] Figure 8 is a schematic diagram of an exemplary decoding process;

[0136] Fig.9A is a schematic diagram of an exemplary encoding process;

[0137] Fig. 9B is a schematic diagram of an exemplary code stream structure;

[0138] Fig.10 is a schematic diagram of an exemplary decoding process;

[0139] Fig.11 is a schematic diagram of an exemplary encoding process;

[0140] Fig.12 is a schematic diagram of an exemplary decoding process;

[0141] Fig.13 A schematic diagram of the generation process of binaural rendering signals is shown as an example;

[0142] Fig.14 is a schematic diagram of an exemplary encoding device;

[0143] Fig.15 is a schematic diagram of an exemplary decoding device;

[0144] Fig.16 is a schematic diagram of an exemplary encoding device;

[0145] Fig.17 is a schematic diagram of an exemplary decoding device;

[0146] Fig.18 Schematic diagram of the structure of the device shown as an example. DETAILED DESCRIPTION

[0147] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0148] The term "and / or" in this article is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0149] The terms "first" and "second" in the description and claims of the embodiments of the present application are used to distinguish different objects rather than to describe a specific order of objects. For example, a first target object and a second target object are used to distinguish different target objects rather than to describe a specific order of target objects.

[0150] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0151] In the description of the embodiments of the present application, unless otherwise specified, the meaning of "multiple" refers to two or more than two. For example, multiple processing units refer to two or more processing units; multiple systems refer to two or more systems.

[0152] In order to make the description of the following embodiments clear and concise, a brief introduction to the related technology is first given.

[0153] Sound is a continuous wave generated by the vibration of an object. The object that vibrates and emits sound waves is called a sound source. When sound waves propagate through a medium (such as air, solid or liquid), the auditory organs of humans or animals can perceive the sound.

[0154] The characteristics of sound waves include pitch, intensity and timbre. Pitch refers to the high or low pitch of a sound. Intensity refers to the size of a sound. Intensity can also be called loudness or volume. The unit of intensity is decibel (dB). Timbre is also called timbre quality.

[0155] The frequency of sound waves determines the pitch of the sound. The higher the frequency, the higher the pitch. The number of times an object vibrates in one second is called frequency, and the unit of frequency is hertz (Hz). The frequency of sound that the human ear can recognize is between 20Hz and 20,000Hz.

[0156] The amplitude of the sound wave determines the intensity of the sound. The greater the amplitude, the greater the sound intensity. The closer to the sound source, the greater the sound intensity.

[0157] The waveform of the sound wave determines the timbre, and the waveforms of the sound wave include square wave, sawtooth wave, sine wave and pulse wave.

[0158] According to the characteristics of sound waves, sounds can be divided into regular sounds and irregular sounds. Irregular sounds refer to sounds produced by irregular vibrations of the sound source. Irregular sounds are, for example, noises that affect people's work, study, and rest. Regular sounds refer to sounds produced by regular vibrations of the sound source. Regular sounds include speech and music. When sound is represented by electricity, regular sounds are analog signals that continuously change in the time-frequency domain. This analog signal can be called an audio signal. An audio signal is an information carrier that carries speech, music, and sound effects.

[0159] Since human hearing has the ability to distinguish the positional distribution of sound sources in space, when listeners hear sounds in space, in addition to being able to feel the pitch, volume and timbre of the sounds, they can also feel the direction of the sounds.

[0160] As people pay more and more attention to the experience of auditory systems and demand more quality, three-dimensional audio technology has emerged to enhance the sense of depth, presence and space of sound. Therefore, listeners can not only feel the sound from the front, back, left and right sound sources, but also feel the feeling of being surrounded by the spatial sound field (referred to as "sound field") generated by these sound sources, and the feeling of sound spreading all around, creating an "immersive" sound effect that makes listeners feel like they are in a theater or concert hall.

[0161] As we all know, when sound waves propagate in an ideal medium, the wave number is k = w / c and the angular frequency is w = 2πf, where f is the sound wave frequency and c is the sound speed. The sound pressure p satisfies formula (1), is the Laplace operator.

[0162]

[0163] Assume that the space system outside the human ear is a sphere, the listener is at the center of the sphere, the sound coming from outside the sphere has a projection on the sphere, and the sound outside the sphere is filtered out. Assume that the sound source is distributed on this sphere, and the sound field generated by the sound source on the sphere is used to fit the sound field generated by the original sound source. That is, three-dimensional audio technology is a method of fitting the sound field. Specifically, solve the equation of formula (1) in the spherical coordinate system. In the passive spherical region, the solution of the equation of formula (1) is the following formula (2).

[0164]

[0165] Among them, r represents the radius of the sphere, θ represents the horizontal angle information (or azimuth information), represents the pitch angle information (or called the elevation angle information), k represents the wave number, s represents the amplitude of the ideal plane wave, and m represents the order number of the HOA signal (or called the order number of the HOA signal). represents the spherical Bessel function, which is also called the radial basis function, where the first j represents the imaginary unit, Does not vary with angle. represents θ, Spherical harmonics of the direction, Spherical harmonics representing the direction of the sound source. The HOA signal satisfies formula (3).

[0166]

[0167] Substituting formula (3) into formula (2), formula (2) can be transformed into formula (4).

[0168]

[0169] Here, m is truncated to the Nth term, that is, m = N, with As an approximate description of the sound field; at this time, It can be called HOA coefficient (can be used to represent N-order HOA signal). The sound field refers to the area in the medium where sound waves exist. N is an integer greater than or equal to 1. The N-order HOA signal includes (N+1) 2 channels of audio signal.

[0170] Formula (4) shows that the sound field can be expanded on the sphere according to spherical harmonics, that is, the sound field can be decomposed into the superposition of multiple plane waves. Therefore, the sound field described by the HOA signal can be expressed by the superposition of multiple plane waves, and the sound field can be reconstructed by the HOA coefficients.

[0171] The scene audio signal involved in the embodiment of the present application is an information carrier that carries the spatial position information of the sound source in the sound field, which describes the sound field of the listener in the space, or can refer to a signal used to describe the sound field of the sound source. Among them, the scene audio signal may include: a multi-point HOA signal (wherein the HOA signal may include a three-dimensional HOA signal and a two-dimensional HOA signal (also referred to as a planar HOA signal)) and a three-dimensional audio signal; the three-dimensional audio signal may refer to other audio signals in the scene audio signal except the HOA signal. The present application takes the scene audio signal as a multi-point HOA signal as an example for explanation.

[0172] Figure 1A FIG. 1 is a schematic diagram of an exemplary application scenario. Figure 1A In the example, the user uses an augmented reality (AR) device to experience an AR concert. It should be understood that the user can also use the AR device to experience other AR projects such as AR science lectures, AR cinemas, etc., and this application does not limit this.

[0173] in, Figure 1A The real scene where the user is located and the virtual scene corresponding to the AR project that the user experiences are the same scene, that is, the real scene where the user is located is the concert hall, and the virtual scene corresponding to the AR project that the user experiences is the virtual concert hall.

[0174] Reference Figure 1A , exemplarily, when a user wants to experience an AR concert in an empty concert hall, he can wear an AR device; then, the user can select the AR concert hall as the AR project he wants to experience in the AR device, and select the music file he wants to experience and perform a playback operation. Subsequently, the AR device can process the music file selected by the user, such as rendering, to obtain a rendered audio signal (for example, it can be a spatial audio signal (i.e., a binaural rendering signal), and this application takes the rendered audio signal as a binaural rendering signal as an example for explanation) and play the binaural rendering signal through headphones; and the AR device can also display the performance picture through the display screen, so that the user can see the performer playing at the corresponding position on the concert hall stage through the display screen. In this way, through the combination of vision and hearing, users can experience the concert immersively.

[0175] Figure 1B FIG. 1 is a schematic diagram of an exemplary application scenario. Figure 1BIn the example, a user uses a virtual reality (VR) device to experience an application scenario of a VR cinema. It should be understood that the user can also use the VR device to experience other VR projects such as a VR concert hall, a VR science lecture hall, etc., and this application does not limit this.

[0176] in, Figure 1B The real scene where the user is located and the virtual scene corresponding to the VR project that the user experiences are not the same scene, that is, the real scene where the user is located is the living room, and the virtual scene corresponding to the VR project that the user experiences is the virtual cinema.

[0177] Reference Figure 1B For example, when a user wants to experience a VR cinema in the living room, he or she can wear a VR device; then, the user can select a VR cinema as the VR project to be experienced in the VR device, and select the movie file to be experienced and perform a playback operation. Subsequently, the VR device can process the audio file contained in the movie file, such as rendering, to obtain a binaural rendering signal and play the binaural rendering signal through headphones; and the VR device can also display the movie screen on the display screen; in this way, through the combination of vision and hearing, the user can experience the movie immersively.

[0178] Figure 1C FIG. 1 is a schematic diagram of an exemplary application scenario. Figure 1C In the figure, an application scenario is shown in which a user uses (mobile phone + headphones) to experience a concert.

[0179] It should be understood that users can also use (mobile phone + headphones) to experience other listening projects such as crosstalk, movies, etc., and this application does not limit this. In addition, the mobile phone is only an example of this application, and this application can also use terminal devices such as tablets, laptops, personal computers, smart watches, etc. + headphones to experience listening.

[0180] in, Figure 1C The real scene where the user is located and the virtual scene corresponding to the listening project experienced by the user are the same scene, that is, the real scene where the user is located is the concert hall, and the virtual scene corresponding to the listening project experienced by the user is the virtual concert hall.

[0181] Reference Figure 1CFor example, when a user wants to experience a concert in an empty concert hall, he or she can wear headphones and connect the headphones to a mobile phone. Then, the user can select the concert as the listening project he or she wants to experience in the mobile phone, and select the music file he or she wants to experience and perform a play operation. Subsequently, the mobile phone can process the music file selected by the user, such as rendering, to obtain a binaural rendering signal. Then, the binaural rendering signal can be sent to the headphones for playback. In addition, the mobile phone can also play a video of the concert performance. In this way, the user can use the mobile phone to immersively experience the concert in the concert hall.

[0182] Figure 1D FIG. 1 is a schematic diagram of an exemplary application scenario. Figure 1D In the figure, an application scenario in which a user uses (mobile phone + headphones) to experience crosstalk is shown.

[0183] It should be understood that users can also use (mobile phone + headphones) to experience other listening projects such as concerts, movies, etc., and this application does not limit this. In addition, the mobile phone is only an example of this application, and this application can also use terminal devices such as tablets, laptops, personal computers, smart watches, etc. + headphones to listen to music.

[0184] in, Figure 1D The real scene where the user is located and the virtual scene corresponding to the listening project experienced by the user are not the same scene, that is, the real scene where the user is located is the bedroom, and the virtual scene corresponding to the listening project experienced by the user is the virtual studio.

[0185] Reference Figure 1D For example, when a user wants to experience crosstalk in the bedroom, he can wear headphones and connect the headphones to the mobile phone; then, the user can select crosstalk as the listening item he wants to experience in the mobile phone, and select the crosstalk file he wants to experience and perform the play operation. Subsequently, the mobile phone can process the crosstalk file selected by the user, such as rendering, to obtain a binaural rendering signal; then, the binaural rendering signal can be sent to the headphones for playback; and the mobile phone can also play the crosstalk video. In this way, the user can use the mobile phone in the bedroom to immersively experience crosstalk.

[0186] For ease of explanation, Figure 1A to Figure 1D The terminal device in the embodiment is called the first electronic device.

[0187] in, Figure 1A to Figure 1D The music files, audio files contained in the movie files, and crosstalk files in the first electronic device (also called the decoding end) are all obtained from the second electronic device (also called the encoding end), and are encoded scene audio signals, that is, bit streams. Then, the first electronic device needs to decode the bit stream, obtain the reconstructed audio signal, and then render the reconstructed audio signal to obtain a binaural rendering signal.

[0188] For example, several sound sources and multi-point HOA microphones can be deployed in a room (that is, a real scene); then, the sound source plays an audio signal, and the multi-point HOA microphone collects the sound in the room to obtain a multi-point HOA signal; at this time, the multi-point HOA signal can be used to describe the sound field of the sound source in the real scene; wherein, the distance between any two HOA microphones in the multi-point HOA microphones is equal. In this way, the second electronic device can obtain the multi-point HOA signal from the multi-point HOA microphone; then, the multi-point HOA signal is encoded to obtain a code stream and sent to the first electronic device.

[0189] In addition, for Figure 1A The scenes and Figure 1B The second electronic device can also model the scene based on the real scene to obtain a virtual scene. Afterwards, the second electronic device can encode the virtual scene to obtain a code stream and send it to the first electronic device. It should be understood that the third electronic device can also model and send the code stream obtained by encoding the virtual scene to the first electronic device or the second electronic device.

[0190] Among them, the real scene refers to an objectively existing scene, such as a cinema, concert hall, lecture hall, etc.; the virtual scene is a scene constructed using virtual technology and is not an objectively existing scene, such as a virtual cinema, virtual concert hall, virtual lecture hall, etc.

[0191] Figure 2A FIG. 4 is a schematic diagram of an exemplary compression framework of a scene audio signal.

[0192] Reference Figure 2A , exemplarily, the first electronic device may include a first audio acquisition module, a first scene audio encoding module, a first channel encoding module, a first channel decoding module, a first scene audio decoding module, a first rendering module and a first audio playback module. It should be understood that the first electronic device may include more than Figure 2A The present application does not limit the number of modules shown. The first rendering module may also be referred to as a first renderer, the first scene audio encoding module may also be referred to as a first audio encoder, and the first scene audio decoding module may also be referred to as a first audio decoder.

[0193] Reference Figure 2A , exemplarily, the second electronic device may include a second audio acquisition module, a second scene audio encoding module, a second channel encoding module, a second channel decoding module, a second scene audio decoding module, a second rendering module, and a second audio playback module. It should be understood that the second electronic device may include more than Figure 2AThe present application does not limit the number of modules shown. The second rendering module may also be referred to as a second renderer, the second scene audio encoding module may also be referred to as a second audio encoder, and the second scene audio decoding module may also be referred to as a second audio decoder.

[0194] Exemplarily, the first electronic device encodes and transmits the scene audio signal to the second electronic device, and the process of decoding and audio playback by the second electronic device can be as follows: the first audio acquisition module (such as the HOA microphone connected to the first electronic device) can perform audio acquisition and output the scene audio signal to the first scene audio encoding module. Then, the first scene audio encoding module can encode the scene audio signal and output the code stream to the first channel encoding module. After that, the first channel encoding module can channel encode the code stream and transmit the channel-encoded code stream to the second electronic device through a wireless or wired network communication device. Then, the second channel decoding module of the second electronic device can channel decode the received data to obtain the code stream and output the code stream to the second scene audio decoding module. Then, the second scene audio decoding module can decode the code stream to obtain a reconstructed audio signal; then the reconstructed audio signal is output to the second rendering module, and the second rendering module renders the reconstructed scene signal to obtain a binaural rendering signal; then the binaural rendering signal is sent to the second audio playback module, and the second audio playback module performs audio playback.

[0195] It should be understood that the process of the second electronic device encoding and transmitting the scene audio signal to the first electronic device, decoding and audio playback by the first electronic device is similar to the above-mentioned process of the first electronic device transmitting the scene audio signal to the second electronic device, and audio playback by the second electronic device, and will not be repeated here.

[0196] Exemplarily, the first electronic device and the second electronic device may include, but are not limited to: personal computers, computer workstations, smart phones, tablet computers, servers, smart cameras, smart cars or other types of cellular phones, media consumption devices, wearable devices, set-top boxes, game consoles, etc.

[0197] Exemplarily, the present application can be specifically applied to VR / AR scenarios. In one possible manner, the first electronic device is a server, and the second electronic device is a VR / AR device. In one possible manner, the second electronic device is a server, and the first electronic device is a VR / AR device.

[0198] It should be noted that, when the first electronic device and the second electronic device are VR / AR devices, the first audio playback module and the second audio playback module may be built-in headphones of the VR / AR devices; when the first electronic device and the second electronic device are terminal devices other than VR / AR devices, the first audio playback module and the second audio playback module may be external headphones of the other terminal devices.

[0199] Exemplarily, when a scene audio signal is encoded by a first electronic device and a reconstructed audio signal is obtained by decoding by a second electronic device, the first electronic device may be referred to as an encoding end and the second electronic device may be referred to as a decoding end. When a scene audio signal is encoded by a second electronic device and a reconstructed audio signal is obtained by decoding by a first electronic device, the second electronic device may be referred to as an encoding end and the first electronic device may be referred to as a decoding end.

[0200] Figure 2B The figure is a schematic diagram of a transcoding framework of a scene audio signal shown as an example.

[0201] Reference Figure 2B (1) Exemplarily, the wireless or core network device may include: a channel decoding module, other audio decoding modules, a scene audio encoding module and a channel encoding module. The wireless or core network device may be used for audio transcoding.

[0202] For example, Figure 2B The specific application scenario of (1) may be: when the first electronic device is not provided with a scene audio encoding module but only with other audio encoding modules; and the second electronic device is only provided with a scene audio decoding module but not with other audio decoding modules, in order to enable the second electronic device to decode and play back the scene audio signal encoded by the first electronic device using other audio encoding modules, wireless or core network equipment may be used for transcoding.

[0203] Specifically, the first electronic device uses other audio encoding modules to encode the scene audio signal to obtain a first code stream; and the first code stream is channel-encoded and sent to the wireless or core network device. Then, the channel decoding module of the wireless or core network device can perform channel decoding and output the first code stream obtained by channel decoding to other audio decoding modules. After that, other audio decoding modules decode the first code stream to obtain the scene audio signal and output the scene audio signal to the scene audio encoding module. Then, the scene audio encoding module can encode the scene audio signal to obtain a second code stream and output the second code stream to the channel encoding module, and the channel encoding module performs channel encoding on the second code stream and sends it to the second electronic device. In this way, the second electronic device can call the scene audio decoding module, decode the second code stream obtained by channel decoding, and obtain a reconstructed audio signal; the reconstructed audio signal can be subsequently played back.

[0204] Reference Figure 2B (2) Exemplarily, the wireless or core network device may include: a channel decoding module, a scene audio decoding module, other audio encoding modules and a channel encoding module. The wireless or core network device may be used for audio transcoding.

[0205] For example, Figure 2B The specific application scenario of (2) may be: when the first electronic device is only provided with a scene audio encoding module and no other audio encoding modules are provided; and the second electronic device is not provided with a scene audio decoding module and only provided with other audio decoding modules, in order to enable the second electronic device to decode and play back the scene audio signal encoded by the first electronic device using the scene audio encoding module, wireless or core network equipment may be used for transcoding.

[0206] Specifically, the first electronic device uses a scene audio encoding module to encode the scene audio signal to obtain a first code stream; and the first code stream is channel-encoded and sent to a wireless or core network device. Then, the channel decoding module of the wireless or core network device can perform channel decoding and output the first code stream obtained by channel decoding to the scene audio decoding module. After that, the scene audio decoding module decodes the first code stream to obtain a scene audio signal and outputs the scene audio signal to other audio encoding modules. Then, other audio encoding modules can encode the scene audio signal to obtain a second code stream and output the second code stream to the channel encoding module, and the channel encoding module performs channel encoding on the second code stream and sends it to the second electronic device. In this way, the second electronic device can call other audio decoding modules to decode the second code stream obtained by channel decoding to obtain a reconstructed audio signal; the reconstructed audio signal can be subsequently played back.

[0207] based on Figure 2A and Figure 2B , the process of rendering to obtain binaural rendering signals is described.

[0208] Figure 2C FIG. 1 is a schematic diagram of an exemplary rendering process. Figure 2C What is shown in FIG. 1 is a schematic diagram of the rendering process of a renderer (which may be a first renderer or a second renderer).

[0209] Reference Figure 2C After the decoder obtains the reconstructed audio signal according to the bit stream, the reconstructed audio signal, the metadata of the scene audio signal, the user position and the user posture can be input into the renderer, and the renderer renders the reconstructed audio signal according to the metadata of the scene audio signal, the user position and the user posture to obtain a binaural rendering signal. The user posture can refer to the user's head posture.

[0210] For example, when the decoding end is an AR / VR device, the sensor in the AR / VR device can obtain the user's position and user's posture. When the decoding end is other terminal devices, the user's position can be obtained by the positioning module in the decoding end, and the user's posture can be obtained by the headphones connected to the decoding end.

[0211] Exemplarily, the renderer may use a virtual loudspeaker object (VLO) algorithm for rendering to obtain a binaural rendering signal. Specifically, the reconstructed audio signal may be first converted into a virtual loudspeaker signal; specifically, the following formula (5) may be referred to:

[0212] S=DX (5)

[0213] X=[x1,......,x L ] T , L=(N+1) 2 (6)

[0214] S=[s1,......,s L ] T , L=(N+1) 2 (7)

[0215] In formula (5), S is a virtual speaker signal, and its expression is shown in formula (7), and N is greater than 0; that is, the reconstructed audio signal is converted into a virtual speaker signal of L virtual speakers. In formula (5), X is a reconstructed audio signal (i.e., a reconstructed audio signal of a HOA signal), and its expression is shown in formula (6). In formula (5), D is a decoding matrix, and its size is L*L.

[0216] Next, the virtual speaker signal may be rendered according to the metadata of the scene audio signal, the user position, and the user posture to obtain a binaural rendering signal; wherein the metadata of the scene audio signal may include first metadata and second metadata. The process of generating the binaural rendering signal may refer to the following S11 to S16:

[0217] S11, determining Y virtual speaker signals Sn...Sm which are closest to the user position from the virtual speaker signals of the L virtual speakers.

[0218] Among them, Y=mn.

[0219] S12: Determine a distance d1 between two HOA microphones according to the second metadata.

[0220] In a possible manner, the second metadata is position information of multiple HOA microphones in the scene. In this case, the distance d1 between any two HOA microphones can be calculated based on the position information of the multiple HOA microphones in the scene.

[0221] In a possible manner, the second metadata is a distance d1 between any two HOA microphones in the multi-point HOA microphones.

[0222] Figure 2D A schematic diagram of a scene is shown for example.

[0223] For example, in Figure 2D , where two HOA microphones are shown in the scene, and Y=4; that is, for a single HOA signal (i.e., the audio signal of the sound source (guitar) received by a single HOA microphone), a virtual speaker signal of four virtual speakers can be used to represent it. The distance d1 between the two HOA microphones is as follows: Figure 2D shown.

[0224] S13: Determine a virtual speaker radius r1 according to the first metadata and the distance d1 between the two HOA microphones.

[0225] For example, the virtual speaker radius r1 may be Figure 6B As shown, the calculation formula of the virtual speaker radius r1 can be shown as follows (8):

[0226] r=kd (8)

[0227] In formula (8), k represents the first metadata. That is, when k remains unchanged, the larger d is, the larger r is.

[0228] Figure 2E A schematic diagram of a scene is shown for example. Figure 2E You can refer to Figure 2D The description of is not repeated here. Figure 2E and Figure 2D The difference is that the distance d1 between the two HOA microphones is different; Figure 2E The distance d1 between the two HOA microphones is greater than Figure 2D The distance d1 between the two HOA microphones corresponds to Figure 2E The radius r1 of the virtual speaker is also larger.

[0229] S14, selecting head related impulse responses HRIRn ... HRIRm corresponding to the virtual speaker signals Sn ... Sm from a head related impulse response (HRIR) library according to the user posture and the virtual speaker radius r1.

[0230] Among them, Sn corresponds to HRIRn, Sn+1 corresponds to HRIRn+1, ..., and so on, Sm corresponds to HRIRm.

[0231] S15, performing calculations based on Sn ... Sm and corresponding HRIRn ... HRIRm, to obtain binaural rendering signals of each of the Y virtual speakers.

[0232] S16, adding the binaural rendering signal of each virtual speaker in the Y virtual speakers to obtain a binaural rendering signal.

[0233] Reference Figure 2D or Figure 2E , by adding the binaural rendering signals of the two virtual speakers, the binaural rendering signal at the user's location can be obtained.

[0234] In the embodiment of the present application, the metadata of the scene audio signal can be obtained by the encoder and written into the bitstream, so that the decoder can parse the metadata of the scene audio signal from the bitstream and use the metadata of the scene audio signal in the subsequent rendering process to obtain the binaural rendering signal. The process of obtaining the metadata of the scene audio signal and the encoding and decoding process can refer to the following embodiment.

[0235] Figure 3A FIG. 1 is a schematic diagram of an exemplary encoding process.

[0236] S301: Acquire a scene audio signal and metadata of the scene audio signal.

[0237] Exemplarily, the encoding end may obtain multi-point HOA signals from multi-point HOA microphones as scene audio signals.

[0238] Exemplarily, in the present application, the metadata of the scene audio signal includes relevant information of the scene audio signal, and the metadata of the scene audio signal is used in the process of rendering to obtain a binaural rendering signal.

[0239] Exemplarily, in a possible manner of S301, the encoder may obtain scene description information of the scene, and then determine metadata of the scene audio signal according to the scene description information of the scene; the specific process will be described later.

[0240] Exemplarily, the scene description information can be used to describe the information of the scene, for example, it may include but is not limited to: the identification, location information, characteristics and direction of multiple HOA microphones in the real scene, the coordinates and materials of the objects contained in the real scene, the identification, location information, characteristics and direction of the sound source in the real scene, the room critical distance of the real scene (the room critical distance refers to the distance from the point where the direct sound energy density is equal to the reverberation sound energy density in a certain direction to the center of the sound source when the sound source stably radiates sound waves indoors (in the real scene), also known as the diffusion field distance. When the radiation directivity of the sound source is spherical, the critical distances in each direction are equal, which is also called the reverberation radius), acoustic reverberation parameters (such as direct-to-Reverberation energy Ratio (DRR), reverberation time (Reverberation Time) (such as T20 (time for energy decay of 20dB), T30, T60, etc.), clarity (Clarity) (such as C50 (the ratio of sound energy before and after 50ms), C80, etc.), etc.), etc., and the present application does not impose any restrictions on this.

[0241] Exemplarily, in a possible manner of S301, metadata of the scene audio signal input by the user may be acquired; that is, the metadata of the scene audio signal may be customized by the user.

[0242] exist Figure 3A In an embodiment, the metadata of the acquired scene audio signal includes only the first metadata, wherein the first metadata can be used to determine a virtual speaker radius based on a distance between two microphones among a plurality of microphones, which may be HOA microphones.

[0243] S302: Encode the scene audio signal and metadata of the scene audio signal to obtain a bitstream.

[0244] The encoding process of the scene audio signal can be as follows:

[0245] In a first possible encoding method, the scene audio signal may be directly encoded to obtain encoded data of the scene audio signal and write the encoded data into a bit stream.

[0246] In the second possible encoding method, a target virtual speaker can be first selected from a preset virtual speaker set according to the first scene audio signal; then, a first virtual speaker signal is generated according to the first scene audio signal and the attribute information of the target virtual speaker; then, the second scene audio signal is determined using the attribute information of the target virtual speaker and the first virtual speaker signal; then, a residual signal is generated according to the first scene audio signal and the second scene audio signal; then, the first virtual speaker signal and the residual signal are encoded to obtain the encoded data of the scene audio signal and write it into the bitstream. Among them, the first scene audio signal refers to the original scene audio signal to be encoded.

[0247] In a third possible encoding method, the attribute information of the target virtual speaker can be determined based on the scene audio signal, the scene audio signal includes audio signals of L channels, and L is a positive integer; the attribute information of the first audio signal and the target virtual speaker in the scene audio signal is encoded to obtain the encoded data of the scene audio signal and write it into the bit stream; wherein the first audio signal is the audio signal of G channels in the scene audio signal, and G is a positive integer less than or equal to L.

[0248] For example, in Figure 3A In the embodiment of the present application, the present application can pre-add the syntax elements and corresponding description information of the metadata of the scene audio signal to the original syntax structure (or high-level syntax element table) of the bitstream. In this way, the encoder can write the metadata of the scene audio signal into the bitstream according to the syntax elements and corresponding description information of the metadata of the scene audio signal in the syntax structure of the bitstream.

[0249] For example, the syntax structure of the code stream can refer to the following table 1:

[0250] Table 1

[0251]

[0252] Referring to Table 1, in Table 1, Syntax represents a syntax element, No.ofbits represents the number of bits of data identified by the syntax element, and Mnemonic represents a comment or explanation of the syntax element.

[0253] Referring to Table 1, exemplarily, the syntax structure of the code stream includes multiple syntax elements: syntax element 1, syntax element 2, syntax element 3, ..., syntax element n, ..., syntax element n+h, and the syntax element "distanceFactor" newly added in the present application; wherein, "distanceFactor" is a syntax element of the metadata of the scene audio signal, and n and h are positive integers.

[0254] Continuing to refer to Table 1, the "1" in the fourth row of Table 1 indicates that the data identified by the syntax element 2 in the code stream is represented by 1 bit, and the "bslbf" in the fourth row of Table 1 indicates that the data identified by the syntax element 2 in the code stream is a bit string with the left bit first.

[0255] The "3" in the fifth row of Table 1 indicates that the data identified by the syntax element 3 in the code stream is represented by 3 bits, and the "bslbf" in the fifth row of Table 1 indicates that the data identified by the syntax element 3 in the code stream is a bit string with the left bit first.

[0256] The "4" in the seventh row of Table 1 indicates that the data identified by the syntax element n in the code stream is represented by 4 bits, and the "uimsbf" in the seventh row of Table 1 indicates that the data identified by the syntax element n in the code stream is an unsigned integer with the high bit first.

[0257] The "8" in the last row of Table 1 indicates that the metadata of the scene audio signal in the code stream can be represented by 8 bits; the "uimsbf" in the last row of Table 1 indicates that the metadata of the scene audio signal in the code stream is an unsigned integer, and the high bit comes first. It should be understood that Table 1 is only an example of the present application, and the metadata of the scene audio signal can also be represented by other numbers of bits, such as 16 bits, and the present application does not limit this; in addition, the syntax elements of the metadata of the scene audio signal can be represented by other strings in addition to "distanceFactor", and the present application does not limit this either.

[0258] Exemplarily, if the syntax structure of the bitstream is as shown in Table 1, in the process of executing S302, the metadata of the scene audio signal can be converted into an 8-bit binary number; then, the 8-bit binary number is written after the data identified by the syntax element n+h in the bitstream.

[0259] Exemplarily, S301 to S302 may be executed by an encoder.

[0260] Figure 3B is a schematic diagram of an exemplary code stream structure. Figure 3A In the embodiment, the code stream structure of the code stream can be as follows: Figure 3B As shown, the metadata of the scene audio signal is located after the encoded data of the scene audio signal.

[0261] It should be understood that, in the bitstream of the present application, the metadata of the scene audio signal may also be located before the encoded data of the scene audio signal, and the present application does not impose any restrictions on this; and the bitstream of the present application may also include other data, and the present application does not impose any restrictions on this.

[0262] It should be noted that, in the bitstream of the present application, the metadata of the scene audio signal and the encoded data of the scene audio signal may be adjacent or not adjacent, and the present application does not impose any limitation on this.

[0263] Figure 4A FIG. 4 is a schematic diagram showing an exemplary decoding process. Figure 4A The decoding process and Figure 3A The encoding process corresponds to .

[0264] S401, receiving a code stream.

[0265] S402: parse the metadata of the scene audio signal from the bitstream.

[0266] S403: Obtain a reconstructed audio signal of the scene audio signal according to the bit stream.

[0267] S401 to S402 are executed by the decoder. The present application does not limit the execution order of S402 and S403.

[0268] Exemplarily, according to the bit stream, the process of obtaining the reconstructed audio signal of the scene audio signal may be as follows:

[0269] In the first decoding method (corresponding to the first encoding method mentioned above), the reconstructed audio signal of the scene audio signal can be directly decoded from the bit stream. In this case, S403 can be executed by the decoder.

[0270] In the second decoding method (corresponding to the second encoding method described above), a virtual speaker signal and a residual signal can be decoded from the bitstream; then, a reconstructed audio signal of the scene audio signal is obtained according to the attribute information of the target virtual speaker, the residual signal and the virtual speaker signal. At this time, the decoding operation can be performed by the decoder, and other steps can be performed by other modules.

[0271] In the third decoding method (corresponding to the third encoding method mentioned above), the bit stream can be decoded to obtain the first reconstructed signal and the attribute information of the target virtual speaker; then, based on the attribute information and the first reconstructed signal, a virtual speaker signal corresponding to the target virtual speaker is generated; then, reconstruction is performed based on the attribute information and the virtual speaker signal to obtain a reconstructed audio signal of the scene audio signal. At this time, the decoding operation can be performed by the decoder, and the other steps can be performed by other modules.

[0272] Exemplarily, the decoding end may parse the metadata of the scene audio signal from the bitstream according to the syntax elements of the metadata of the field audio signal and the corresponding description information in the syntax structure of the bitstream; wherein, Figure 4AIn an embodiment of the present invention, the metadata of the scene audio signal may include only the first metadata. Afterwards, the decoding end may render the first metadata parsed from the bitstream and the reconstructed audio signal obtained from the bitstream to obtain a binaural rendering signal. In this case, the decoding end may obtain the second metadata from other devices or generate the second metadata locally, which is not limited in this application; and the number Y of virtual speakers during the rendering process may be a user-defined value or a default value, which is not limited in this application.

[0273] Figure 4B The figure is a schematic diagram of an exemplary binaural rendering signal generation process. Figure 4B So Figure 3A and Figure 4A As shown on the basis.

[0274] Reference Figure 4B Exemplarily, after the encoder encodes the scene audio signal and the metadata of the scene audio signal to obtain a bitstream, the encoder transmits the bitstream to the decoder. After receiving the bitstream, the decoder can obtain a reconstructed audio signal of the scene audio signal according to the bitstream, and parse the metadata of the scene audio signal from the bitstream; then, according to the metadata of the scene audio signal, the user position and the user posture, the reconstructed audio signal of the scene audio signal is rendered to obtain a binaural rendering signal.

[0275] Exemplarily, the present application may also write a first identifier for indicating rendering complexity into the bitstream; wherein the rendering complexity is used to determine the number Y of virtual speakers during the rendering process; that is, the decoding end may render the virtual speaker signal according to the metadata of the scene audio signal, the rendering complexity indicated by the first identifier, the user position, and the user posture to obtain a binaural rendering signal. In this way, a binaural rendering signal may be rendered whose audio quality matches factors such as user requirements and the computing power of the electronic device.

[0276] Figure 5A FIG. 1 is a schematic diagram of an exemplary encoding process. Figure 5A In an embodiment of the present invention, the metadata of the scene audio signal is determined according to the scene description information, and the metadata of the scene audio signal may include first metadata and second metadata.

[0277] S501, obtaining a scene audio signal and scene description information of the scene.

[0278] Exemplarily, the encoding end may obtain the scene description information of the scene locally, or may obtain the scene description information of the scene from a third electronic device, and this application does not impose any limitation on this.

[0279] S502: Determine metadata of a scene audio signal according to scene description information of the scene.

[0280] Exemplarily, the metadata of the scene audio signal may include first metadata and second metadata.

[0281] In a possible manner, the first metadata may be determined according to the product value of the room critical distance and a preset value; specifically, the following formula (9) may be referred to:

[0282]

[0283] Wherein, k in formula (9) is the first metadata, k∈[0.5,1.5]; 1.5 in formula (9) is a preset value.

[0284] In one possible approach, the reverberation type of the scene can be determined based on the acoustic reverberation parameters in the scene; then, the first metadata can be determined based on the reverberation type of the scene. Specifically, the corresponding first metadata can be set in advance for different reverberation types; for example, for the first reverberation type (corresponding to a dry sound scene or a reverberation-free room), the corresponding first metadata is set to 0.5, and for the second reverberation type (corresponding to a wet sound scene or a reverberation room), the corresponding first metadata is set to 1.5. In this way, when the reverberation type of the scene is determined to be the first reverberation type, 0.5 can be used as the first metadata; when the reverberation type of the scene is determined to be the second reverberation type, 1.5 can be used as the first metadata.

[0285] In a possible manner, the location information of multiple HOA microphones in the scene included in the scene description information may be used as the second metadata.

[0286] In one possible manner, the distance between two HOA microphones in the multi-point HOA microphones may be determined according to the position information of the multi-point HOA microphones in the scene included in the scene description information; then, the distance between the two HOA microphones in the multi-point HOA microphones may be used as the second metadata.

[0287] Exemplarily, when the first metadata is input by a user, the value range of the first metadata may be limited to k∈[0.25,4].

[0288] It should be understood that the first metadata may also include other generation methods, and the metadata of the scene audio signal may also include other metadata, which is not limited in the present application.

[0289] S503: Encode the scene audio signal and metadata of the scene audio signal to obtain a bitstream.

[0290] Exemplarily, S503 may refer to the description of S302 above, which will not be repeated here.

[0291] S504: Write the first identifier into the code stream.

[0292] Exemplarily, multiple rendering complexity levels may be pre-set, as shown in Table 2:

[0293] Table 2

[0294]

[0295] The number of virtual speakers in Table 2 is Y.

[0296] Referring to Table 2, illustratively, the rendering complexity can be divided into 5 levels, wherein, in level 1 to level 4, the more virtual speakers there are, the more convolutions are involved in the rendering process, the higher the rendering complexity, and the better the audio quality.

[0297] Referring to Table 2, for example, for level 0, N is the order of the HOA signal, and the number of virtual speakers is (N+1) 2 , the number of convolutions is 2(N+1) 2 ; Both rendering complexity and audio quality depend on the HOA signal.

[0298] It should be understood that Table 2 is only an example of the present application, and the rendering complexity level can be set as required, and the present application does not impose any limitation on this.

[0299] In a possible manner, a target rendering complexity level may be selected from a plurality of rendering complexity levels in Table 2 according to the computing capability of the decoding end; and the value of the first identifier may be set as the target rendering complexity level.

[0300] In a possible manner, the user may input a target rendering complexity level according to the multiple rendering complexity levels in Table 2, and set the value of the first identifier to the target rendering complexity level input by the user.

[0301] For example, in Figure 5A In the embodiment of the present application, the present application can pre-add the syntax elements and corresponding description information of the metadata of the scene audio signal, and add the syntax elements and corresponding description information of the first identifier in the original syntax structure of the bitstream. In this way, the encoder can write the first identifier into the bitstream according to the syntax elements and corresponding description information of the first identifier in the syntax structure of the bitstream; and write the metadata of the scene audio signal into the bitstream according to the syntax elements and corresponding description information of the metadata of the scene audio signal in the syntax structure of the bitstream.

[0302] It should be noted that the present application does not limit the execution order of S503 and S504.

[0303] For example, the syntax structure of the code stream can refer to the following table 3:

[0304] Table 3

[0305]

[0306]

[0307] Refer to Table 3. Figure 5A In the embodiment of the present application, the new syntax elements "distanceFactor" and "complexityLevel" are added; wherein, "distanceFactor" is a syntax element of the metadata of the scene audio signal, and "complexityLevel" is a syntax element of the first identifier.

[0308] The "3" in the last row of Table 3 indicates that the first identifier in the code stream can be represented by 3 bits; the "uimsbf" in the last row of Table 3 indicates that the first identifier in the code stream is an unsigned integer, and the high bit comes first. It should be understood that Table 3 is only an example of the present application, and the first identifier can also be represented by other numbers of bits, such as 4 bits, and the present application does not limit this; in addition, the syntax element of the first identifier can be represented by other strings in addition to "complexityLevel", and the present application does not limit this either.

[0309] Among them, the explanation of other syntax elements in Table 3 can refer to the description in Table 1, which will not be repeated here.

[0310] For example, if the syntax structure of the code stream is as shown in Table 3, in the process of executing S503, the metadata of the scene audio signal can be converted into an 8-bit binary number; then, the 8-bit binary number is written after the data identified by the syntax element n+1 in the code stream. And in the process of executing S504, the first identifier can be converted into a 3-bit binary number; then, the 3-bit binary number is written after the metadata of the scene audio signal in the code stream.

[0311] Figure 5B is a schematic diagram of an exemplary code stream structure. Figure 5A The code stream structure in the embodiment can be as follows Figure 5B As shown, the metadata of the scene audio signal is located after the encoded data of the scene audio signal, and the first identifier is located after the metadata of the scene audio signal.

[0312] It should be understood that the present application does not limit the location of the metadata of the scene audio signal, the encoded data of the scene audio signal, and the first identifier in the bitstream.

[0313] It should be understood that any two of the metadata of the scene audio signal, the encoded data of the scene audio signal and the first identifier in the bitstream of the present application may be adjacent or non-adjacent, and the present application does not impose any limitation on this.

[0314] Fig. 6A FIG. 4 is a schematic diagram showing an exemplary decoding process. Fig. 6A The decoding process and Figure 5A The encoding process corresponds to .

[0315] S601, receiving a code stream.

[0316] S602: Obtain a reconstructed audio signal of the scene audio signal according to the bit stream.

[0317] S603: parse metadata of the scene audio signal from the bitstream.

[0318] S604: parse the code stream to obtain a first identifier.

[0319] Illustratively, the present application does not limit the execution order of S602 , S603 , and S604 .

[0320] Exemplarily, after receiving the bit stream, the decoder can obtain the reconstructed audio signal of the scene audio signal according to the bit stream; Figure 4A The description in the embodiments will not be repeated here.

[0321] Exemplarily, the decoding end may also parse the metadata of the scene audio signal from the bitstream according to the syntax elements of the metadata of the scene audio signal in the syntax structure of the bitstream and the corresponding description information. And, the decoding end may also parse the first identifier from the bitstream according to the syntax elements of the first identifier in the syntax structure of the bitstream and the corresponding description information.

[0322] Subsequently, the decoding end may render according to the reconstructed signal of the scene audio signal, the first metadata, the second metadata and the first identifier to obtain a binaural rendering signal. In this case, the decoding end may parse the second metadata and the first metadata from the bitstream; and during the rendering process, the number Y of virtual speakers selected may be determined according to the first identifier.

[0323] Figure 6B The figure is a schematic diagram of an exemplary binaural rendering signal generation process. Figure 6B So Figure 5A and Fig. 6A As shown on the basis.

[0324] Reference Figure 6BExemplarily, after the encoder encodes the scene audio signal, the metadata of the scene audio signal, and the first identifier into a bitstream, the encoder transmits the bitstream to the decoder. After receiving the bitstream, the decoder can obtain a reconstructed audio signal of the scene audio signal according to the bitstream; and parse the metadata of the scene audio signal and the first identifier from the bitstream; then, according to the metadata of the scene audio signal, the rendering complexity level indicated by the first identifier, the user position, and the user posture, the reconstructed audio signal of the scene audio signal is rendered to obtain a binaural rendering signal.

[0325] It should be noted that in Figure 6B In the example, what is input to the renderer may be the first identifier, or the rendering complexity level indicated by the first identifier, or the number Y of virtual speakers corresponding to the rendering complexity level indicated by the first identifier; this application does not impose any limitation on this.

[0326] Exemplarily, there may be multiple renderers at the decoding end (each renderer adopts a different rendering method), wherein the multiple renderers include: a renderer that renders according to the metadata of the scene audio signal (for example, a renderer that renders according to the method in S11 to S16 above) and a renderer that renders according to other information. Furthermore, a corresponding identification value can be set for each renderer, for example, for renderer A, the corresponding identification value is set to "0"; for renderer B, the corresponding identification value is set to "1"; for renderer C, the corresponding identification value is set to "2", and so on. When it is determined that the decoding end needs to render according to the metadata of the scene audio signal, in one possible way, the identification value of the renderer that renders according to the metadata of the scene audio signal (hereinafter referred to as the first identification value) can be used as the identification value of the second identification; in one possible way, the second identification value (such as "0" or "1", which can be set as required) can be used as the identification value of the second identification; and then the second identification is added to the bitstream. The second identification value is used to instruct the decoder to render according to the metadata of the scene audio signal, or in other words, the second identification value is used to indicate that the bitstream includes the metadata of the scene audio signal. In this way, it is convenient for the decoder to determine which renderer or rendering algorithm to use for rendering.

[0327] Fig. 7A FIG. 1 is a schematic diagram of an exemplary encoding process. It should be noted that: Fig. 7A is Figure 3A Based on the above, Fig. 7A In an embodiment of the present invention, metadata of the scene audio signal is determined according to scene description information.

[0328] S701, obtaining a scene audio signal.

[0329] S702: Encode the scene audio signal to obtain a bit stream.

[0330] Illustratively, S701 to S702 may refer to the above description and will not be described in detail herein.

[0331] S703, write the second identifier into the code stream.

[0332] The first identification value and the second identification value mentioned above can both be referred to as the first preset identification value. That is, when it is determined that the decoding end needs to use the metadata of the scene audio signal for rendering, the value of the second identification can be set to the first preset identification value, and then the first preset identification value can be written into the bitstream. Otherwise, the value of the second identification can be set to other identification values ​​(other identification values ​​are different from the first preset identification value), and then the first preset identification value can be written into the bitstream.

[0333] S704: Obtain scene description information of the scene.

[0334] S705: Determine metadata of the scene audio signal according to the scene description information of the scene.

[0335] S706, writing metadata of the scene audio signal into the bitstream.

[0336] Illustratively, S704 to S706 may refer to the description of S501 to S503 above, which will not be repeated here.

[0337] For example, in Fig. 7A In the embodiment of the present invention, the present application can pre-add the syntax elements and corresponding description information of the metadata of the scene audio signal, and add the syntax elements and corresponding description information of the second identifier in the original syntax structure of the bitstream. In this way, the encoder can write the second identifier into the bitstream according to the syntax elements and corresponding description information of the second identifier in the syntax structure of the bitstream.

[0338] For example, the syntax structure of the code stream may refer to the following table 4:

[0339] Table 4

[0340]

[0341]

[0342] Referring to Table 4, Fig. 7A In the embodiment of the present application, the new syntax elements "distanceFactor" and "hoaGroupHasLowProfileConfig" are added; wherein, "distanceFactor" is a syntax element of the metadata of the scene audio signal, and "hoaGroupHasLowProfileConfig" is a syntax element of the second identifier.

[0343] The "1" in the fourth-to-last row in Table 4 indicates that the second identifier in the code stream can be represented by 1 bit; the "bslbf" in the fourth-to-last row in Table 4 indicates that the second identifier in the code stream is a bit string, and the left bit comes first. It should be understood that Table 4 is only an example of the present application, and the second identifier can also be represented by other numbers of bits, such as 2 bits, and the present application does not limit this; in addition, the syntax element of the second identifier can also be represented by its string, and the present application does not limit this.

[0344] Among them, the explanation of other syntax elements in Table 4 can refer to the description in Table 1, which will not be repeated here.

[0345] Exemplarily, if the syntax structure of the code stream is as shown in Table 4, in the process of executing S703, the second identifier can be converted into a 1-bit binary number; then, the 1-bit binary number is written after the data identified by the syntax element n+1 in the code stream. And in the process of executing S706, the metadata of the scene audio signal can be converted into an 8-bit binary number; then, the 8-bit binary number is written after the second identifier in the code stream. When the syntax structure of the code stream is as shown in Table 4, the value of the second identifier is "1".

[0346] In a possible manner, the first identification value of the renderer that renders according to the metadata of the scene audio signal can also be used as the value of the second identification. Assuming that the first identification value of the renderer that renders according to the metadata of the scene audio signal is "3", the syntax structure of the code stream can be as shown in Table 5:

[0347] For example, the syntax structure of the code stream may refer to the following table 5:

[0348] Table 5

[0349]

[0350]

[0351] Figure 7B is a schematic diagram of an exemplary code stream structure. Fig. 7A The code stream structure in the embodiment can be as follows Figure 7B As shown, the second identifier is located after the encoded data of the scene audio signal, and the metadata of the scene audio signal is located after the second identifier.

[0352] It should be understood that the present application does not limit the locations of the metadata of the scene audio signal, the encoded data of the scene audio signal, and the second identifier in the bitstream.

[0353] It should be understood that, in the bitstream of the present application, any two of the metadata of the scene audio signal, the encoded data of the scene audio signal and the second identifier may be adjacent or non-adjacent, and the present application does not impose any limitation on this.

[0354] Figure 8 FIG. 4 is a schematic diagram showing an exemplary decoding process. Figure 8 The decoding process and Fig. 7A The encoding process corresponds to .

[0355] S801, receiving a code stream.

[0356] S802: Obtain a reconstructed audio signal of the scene audio signal according to the bit stream.

[0357] Exemplarily, S801 to S802 may refer to the above description and will not be described in detail here.

[0358] S803: parse the second identifier from the bitstream.

[0359] S804: Determine whether the value of the second identifier is the first preset identifier value.

[0360] Exemplarily, the second identifier may be parsed from the bitstream first; then, it is determined whether the second identifier is a first preset identifier value to determine whether the metadata of the scene audio signal may be parsed from the bitstream.

[0361] Exemplarily, when the value of the second identifier is the first preset identifier value, it indicates that the code stream contains metadata of the scene audio signal, and S805 can be executed. When the value of the second identifier is another identifier value, it indicates that the code stream does not contain metadata of the scene audio signal, and at this time, the field after the second identifier in the code stream is decoded.

[0362] Exemplarily, when the code stream structure of the code stream is as shown in Table 4, it can be determined whether the value of the second identifier is 1. When the value of the second identifier is 1, it indicates that the code stream contains metadata of the scene audio signal; when the value of the second identifier is not 1, it indicates that the code stream does not contain metadata of the scene audio signal.

[0363] Exemplarily, when the code stream structure of the code stream is as shown in Table 5, it can be determined whether the value of the second identifier is 3. When the value of the second identifier is 3, it means that the code stream contains metadata of the scene audio signal; when the value of the second identifier is not 3, it means that the code stream does not contain metadata of the scene audio signal.

[0364] S805: parse the metadata of the scene audio signal from the bitstream.

[0365] Exemplarily, there may be multiple renderers at the decoding end (each renderer adopts a different rendering method), wherein the multiple renderers include: a renderer that renders according to the metadata of the scene audio signal and the first identifier (for example, a renderer that renders according to the method in S11 to S16 above) and a renderer that renders according to other information. Furthermore, a corresponding identification value can be set for each renderer, for example, for renderer A, the corresponding identification value is set to "0"; for renderer B, the corresponding identification value is set to "1"; for renderer C, the corresponding identification value is set to "2", and so on. When it is determined that the decoding end needs to render according to the metadata of the scene audio signal and the first identifier, in one possible way, the identification value of the renderer that renders according to the metadata of the scene audio signal and the first identifier (hereinafter referred to as the first identification value) can be used as the identification value of the second identifier; in one possible way, the second identification value (such as "0" or "1", which can be set as required) can be used as the identification value of the second identifier; and then the second identifier is added to the bitstream. The second identification value is used to instruct the decoder to render according to the metadata of the scene audio signal and the first identification, or in other words, the second identification value is used to indicate that the bitstream includes the metadata of the scene audio signal and the first identification. In this way, it is convenient for the decoder to determine which renderer or rendering algorithm to use for rendering.

[0366] Fig.9A FIG. 1 is a schematic diagram of an exemplary encoding process. It should be noted that: Fig.9A is Figure 5A Based on the above, Fig.9A In an embodiment of the present invention, metadata of the scene audio signal is determined according to scene description information.

[0367] S901, obtaining a scene audio signal.

[0368] S902: Encode the scene audio signal to obtain a bit stream.

[0369] Exemplarily, S901 to S902 may refer to the above description and will not be described in detail here.

[0370] S903, write the second identifier into the code stream.

[0371] S904: Obtain scene description information of the scene.

[0372] S905: Determine metadata of the scene audio signal according to the scene description information of the scene.

[0373] S906, writing metadata of the scene audio signal into the bitstream.

[0374] S907, write the first identifier into the bit stream.

[0375] Illustratively, the present application does not limit the execution order of S906 and S907.

[0376] For example, in Fig.9A In an embodiment of the present application, the present application may pre-add the syntax elements and corresponding description information of the metadata of the scene audio signal, add the syntax elements and corresponding description information of the second identifier, and add the syntax elements and corresponding description information of the first identifier in the original syntax structure of the bitstream.

[0377] For example, the syntax structure of the code stream may refer to the following table 6:

[0378] Table 6

[0379]

[0380] Fig. 9B FIG. 1 is a schematic diagram of an exemplary code stream structure. Fig. 9B is a schematic diagram of an exemplary code stream structure. Fig.9A The code stream structure in the embodiment can be as follows Fig. 9B As shown, the second identifier is located after the encoded data of the scene audio signal, the metadata of the scene audio signal is located after the second identifier, and the first identifier is located after the metadata of the scene audio signal.

[0381] It should be understood that the present application does not limit the locations of the metadata of the scene audio signal, the encoded data of the scene audio signal, the first identifier, and the second identifier in the bitstream.

[0382] It should be understood that in the bitstream of the present application, any two of the metadata of the scene audio signal, the encoded data of the scene audio signal, the first identifier and the second identifier may be adjacent or non-adjacent, and the present application does not impose any restrictions on this.

[0383] Fig.10 FIG. 4 is a schematic diagram showing an exemplary decoding process. Fig.10 The decoding process and Fig.9A The encoding process corresponds to .

[0384] S1001, receiving a code stream.

[0385] S1002: Obtain a reconstructed audio signal of a scene audio signal according to a bit stream.

[0386] S1003: parse the second identifier from the code stream.

[0387] S1004: Determine whether the value of the second identifier is the first preset identifier value.

[0388] S1005: parse the metadata of the scene audio signal from the bitstream.

[0389] S1006: parse the first identifier from the bit stream.

[0390] Illustratively, S1001 to S1006 may refer to the description of the above embodiment and will not be described in detail here.

[0391] It should be noted that Figure 5A to Figure 10 In the embodiment, during the encoding process, a third identifier may be written into the bitstream, and the third identifier may be used to indicate whether the bitstream includes the first identifier; in this case, the second identifier may be used only to indicate whether the bitstream includes metadata of the scene audio signal. Exemplarily, when the first identifier is written into the bitstream, the value of the third identifier may be set to the second preset identifier value, and then the second preset identifier value may be written into the bitstream.

[0392] Correspondingly, the present application can pre-add the syntax elements and corresponding description information of the metadata of the scene audio signal in the original syntax structure of the bitstream, add the syntax elements and corresponding description information of the second identifier, add the syntax elements and corresponding description information of the first identifier, and add the syntax elements and corresponding description information of the third identifier.

[0393] For example, the syntax structure of the code stream may refer to the following table 7:

[0394] Table 7

[0395]

[0396]

[0397] In Table 7, "hoaGroupHascomplexityConfig" is a syntax element of the third identifier; and the second preset identifier value is "1".

[0398] In this way, the decoding end can decode the third identifier from the code stream; when it is determined that the value of the third identifier is the second preset identifier value, it can be determined that the code stream includes the first identifier; and then the first identifier is decoded from the code stream.

[0399] Exemplarily, the present application may also write only the first identifier into the bitstream. In this case, the decoding end may obtain metadata of the scene audio signal from other electronic devices or generate metadata of the scene audio signal locally.

[0400] Fig.11 FIG. 1 is a schematic diagram of an exemplary encoding process.

[0401] S1101, obtaining a scene audio signal;

[0402] S1102, encode the scene audio signal and the first identifier to obtain a bitstream; wherein the first identifier is used to indicate the rendering complexity, the rendering complexity is used to determine the number of virtual speakers, the reconstructed audio signal of the scene audio signal is used to generate a virtual speaker signal, and the virtual speaker signal and the number of virtual speakers are used for rendering to obtain a rendered audio signal.

[0403] S1101 to S1102 may refer to the above description and will not be elaborated here.

[0404] The syntax element of the first identifier and the corresponding description information may be added to the syntax structure of the code stream.

[0405] For example, the syntax structure of the code stream may be as shown in Table 8 below:

[0406] Table 8

[0407]

[0408] Fig.12 FIG. 4 is a schematic diagram showing an exemplary decoding process. Fig.12 The decoding process and Fig.11 The encoding process corresponds to .

[0409] S1201, receiving code stream;

[0410] S1202: Obtain a reconstructed audio signal of the scene audio signal according to the bit stream.

[0411] S1203, parse a first identifier from the bitstream, wherein the first identifier is used to indicate rendering complexity, the rendering complexity is used to determine the number of virtual speakers, the reconstructed audio signal is used to generate a virtual speaker signal, and the virtual speaker signal and the number of virtual speakers are used for rendering to obtain a rendered audio signal.

[0412] Exemplarily, S1201 to S1203 can refer to the above description and will not be repeated here.

[0413] In addition, Fig.11 and Fig.12 On the basis of, the encoding end can also write a second identifier into the bitstream, wherein the second identifier is used to indicate whether the bitstream contains the first identifier; exemplarily, when the first identifier is written into the bitstream, the value of the second identifier can be set to a preset identifier value, and then the preset identifier value is written into the bitstream.

[0414] For example, the syntax structure of the code stream may refer to the following table 9:

[0415] Table 9

[0416]

[0417] In Table 9, "hoaGroupHascomplexityConfig" is a syntax element of the second flag; and the preset flag value is "1".

[0418] In this way, the decoding end can decode the second identifier from the code stream; when it is determined that the value of the second identifier is the preset identifier value, it can be determined that the code stream includes the first identifier; and then the first identifier is decoded from the code stream.

[0419] In addition, the syntax element of the second identifier and the corresponding description information may also be added to the syntax structure of the code stream. For details, please refer to the above description, which will not be repeated here.

[0420] Fig.13 The figure is a schematic diagram of an exemplary binaural rendering signal generation process. Fig.13 The embodiment is performed by a terminal device, which may include a decoder and a renderer; Fig.13 is Figure 4A shown on the basis of.

[0421] S1301, receiving a code stream.

[0422] S1302: parse the metadata of the scene audio signal from the bitstream.

[0423] S1303: Obtain a reconstructed audio signal of the scene audio signal according to the bit stream.

[0424] Exemplarily, S1301 to S1303 may refer to the description of S401 to S403 above, which will not be repeated here.

[0425] S1304: Generate a virtual speaker signal according to the reconstructed audio signal of the scene audio signal.

[0426] S1305: Determine a virtual speaker radius according to the first metadata and the distance between two microphones among the multiple microphones.

[0427] S1306: Rendering is performed according to the virtual speaker radius and the virtual speaker signal to obtain a rendered audio signal.

[0428] Among them, S1304~S1306 can refer to the description of the above S11~S16, which will not be repeated here.

[0429] It should be noted that S1305~S1306 can be executed by the renderer; in one possible way, S1301~S1303 can be executed by the decoder; in one possible way, S1301~S1302 can be executed by the decoder, and S1303 can be implemented by the decoder and other modules in collaboration.

[0430] Fig.14 The schematic diagram of the coding device is shown as an example. The coding device can be used to execute the method of the above embodiment, so the beneficial effects that can be achieved can refer to the beneficial effects of the corresponding method provided above, which will not be repeated here.

[0431] A first data acquisition module 1401 is used to acquire a scene audio signal and metadata of the scene audio signal; wherein the scene audio signal is used to describe the sound field of a sound source in the scene, the scene includes multiple microphones, the metadata includes first metadata, the first metadata is used to determine a virtual speaker radius with a distance between two microphones among the multiple microphones, the virtual speaker radius and the virtual speaker signal are used for rendering to obtain a rendered audio signal, and the virtual speaker signal is generated according to a reconstructed audio signal of the scene audio signal;

[0432] The first encoding module 1402 is used to encode the scene audio signal and metadata to obtain a bit stream.

[0433] Fig.15 The schematic diagram of the decoding device is shown as an example. The decoding device can be used to execute the method of the above embodiment, so the beneficial effects that can be achieved can refer to the beneficial effects of the corresponding method provided above, which will not be repeated here.

[0434] The first receiving module 1501 is used to receive a code stream;

[0435] A first parsing module 1502, configured to parse the bitstream to obtain metadata of the scene audio signal;

[0436] The first signal acquisition module 1503 is used to obtain a reconstructed audio signal of a scene audio signal according to a bit stream; the scene audio signal is used to describe the sound field of a sound source in the scene, the scene includes multiple microphones, the metadata includes first metadata, the first metadata is used to determine a virtual speaker radius from a distance between two microphones among the multiple microphones, the virtual speaker radius and the virtual speaker signal are used for rendering to obtain a rendered audio signal, and the virtual speaker signal is generated based on the reconstructed audio signal.

[0437] Fig.16 The schematic diagram of the coding device is shown as an example. The coding device can be used to execute the method of the above embodiment, so the beneficial effects that can be achieved can refer to the beneficial effects of the corresponding method provided above, which will not be repeated here.

[0438] The second data acquisition module 1601 is used to acquire a scene audio signal;

[0439] The second encoding module 1602 is used to encode the scene audio signal and the first identifier to obtain a bit stream; the first identifier is used to indicate the rendering complexity, the rendering complexity is used to determine the number of virtual speakers, the reconstructed audio signal of the scene audio signal is used to generate a virtual speaker signal, the virtual speaker signal and the number of virtual speakers are used for rendering to obtain a rendered audio signal.

[0440] Fig.17 The schematic diagram of the decoding device is shown as an example. The decoding device can be used to execute the method of the above embodiment, so the beneficial effects that can be achieved can refer to the beneficial effects of the corresponding method provided above, which will not be repeated here.

[0441] The second receiving module 1701 is used to receive a code stream;

[0442] The second parsing module 1702 is used to parse the bit stream to obtain a first identifier;

[0443] The second signal acquisition module 1703 is used to obtain a reconstructed audio signal of the scene audio signal according to the bit stream; the first identifier is used to indicate the rendering complexity, the rendering complexity is used to determine the number of virtual speakers, the reconstructed audio signal is used to generate a virtual speaker signal, the virtual speaker signal and the number of virtual speakers are used for rendering to obtain a rendered audio signal.

[0444] In one example, Fig.18 A schematic block diagram of a device 1800 according to an embodiment of the present application is shown. The device 1800 may include: a processor 1801 and a transceiver / transceiver pin 1802 , and optionally, a memory 1803 .

[0445] The various components of the device 1800 are coupled together via a bus 1804, wherein the bus 1804 includes a power bus, a control bus, and a status signal bus in addition to a data bus. However, for the sake of clarity, various buses are referred to as bus 1804 in the figure.

[0446] Optionally, the memory 1803 may be used to store instructions in the aforementioned method embodiment. The processor 1801 may be used to execute the instructions in the memory 1803, and control the receiving pin to receive a signal, and control the sending pin to send a signal.

[0447] The device 1800 may be the electronic device or a chip of the electronic device in the above method embodiment.

[0448] Exemplarily, the electronic device may be a terminal device or a server.

[0449] Among them, all relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module, and will not be repeated here.

[0450] The embodiment of the present application also provides a chip, including one or more interface circuits and one or more processors; the one or more processors receive or send data through the one or more interface circuits, and when the one or more processors execute computer instructions, the above-mentioned related method steps implement the steps of the method in the above-mentioned embodiment. The interface circuit is a transceiver / transceiver pin 1802.

[0451] This embodiment further provides a computer-readable storage medium, in which computer instructions are stored. When the computer instructions are executed on an electronic device, the electronic device executes the above-mentioned related method steps to implement the method in the above-mentioned embodiment.

[0452] This embodiment further provides a computer program product, which includes computer instructions. When the computer instructions are executed by a computer or a processor, the computer executes the above-mentioned related steps to implement the method in the above-mentioned embodiment.

[0453] In addition, an embodiment of the present application also provides a device, which may specifically be a chip, component or module, and the device may include a connected processor and memory; wherein the memory is used to store computer-executable instructions, and when the device is running, the processor may execute the computer-executable instructions stored in the memory so that the chip executes the methods in the above-mentioned method embodiments.

[0454] Among them, the electronic device, computer-readable storage medium, computer program product or chip provided in this embodiment are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above and will not be repeated here.

[0455] Through the description of the above implementation methods, technical personnel in the relevant field can understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0456] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0457] The units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0458] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0459] Any content of each embodiment of the present application, as well as any content of the same embodiment, can be freely combined. Any combination of the above content is within the scope of the present application.

[0460] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0461] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

[0462] The steps of the method or algorithm described in conjunction with the disclosure of the embodiments of the present application can be implemented in a hardware manner, or can be implemented by a processor executing a software instruction. The software instruction can be composed of corresponding software modules, and the software module can be stored in a random access memory (Random Access Memory, RAM), a flash memory, a read-only memory (ReadOnly Memory, ROM), an erasable programmable read-only memory (Erasable Programmable ROM, EPROM), an electrically erasable programmable read-only memory (Electrically EPROM, EEPROM), a register, a hard disk, a mobile hard disk, a read-only compact disk (CD-ROM) or any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and can write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC.

[0463] Those skilled in the art should be aware that in one or more of the above examples, the functions described in the embodiments of the present application can be implemented with hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer-readable storage media and communication media, wherein the communication media include any media that facilitates the transmission of a computer program from one place to another. The storage medium can be any available medium that a general or special-purpose computer can access.

[0464] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A decoding method, characterized in that: The method comprises: Receive code stream; Parsing metadata of the scene audio signal from the bitstream; According to the bit stream, a reconstructed audio signal of the scene audio signal is obtained; the scene audio signal is used to describe the sound field of the sound source in the scene, the scene includes multiple microphones, the metadata includes first metadata, the first metadata is used to determine a virtual speaker radius with a distance between two microphones among the multiple microphones, the virtual speaker radius and the virtual speaker signal are used for rendering to obtain a rendered audio signal, and the virtual speaker signal is generated according to the reconstructed audio signal.

2. The method according to claim 1, characterized in that The method further comprises: Parsing a first identifier from the bit stream; The first identifier is used to indicate rendering complexity, and the rendering complexity is used to determine the number of the virtual speakers.

3. The method according to claim 1 or 2, characterized in that: The method further comprises: Parsing a second identifier from the code stream; Wherein, when the value of the second identifier is the first preset identifier value, it indicates that the code stream includes the metadata.

4. The method according to claim 3, characterized in that The method further comprises: When the value of the second identifier is the first preset identifier value, it indicates that the code stream also includes the first identifier.

5. The method according to claim 2, characterized in that: The method further comprises: Parsing a third identifier from the bit stream; When the value of the third identifier is a second preset identifier value, it indicates that the code stream includes the first identifier.

6. The method according to any one of claims 1 to 5, characterized in that: The metadata also includes second metadata for determining a distance between two microphones of the plurality of microphones.

7. The method according to any one of claims 1 to 6, characterized in that: The scene audio signal includes a multi-point high-order surround audio reverberation (HOA) signal, and the reconstructed audio signal includes a reconstructed audio signal of the multi-point HOA signal.

8. The method according to any one of claims 1 to 7, characterized in that: The code stream includes: syntax elements of the metadata and corresponding description information; The step of parsing metadata of the scene audio signal from the bitstream includes: The metadata is parsed from the bitstream according to the syntax elements of the metadata and the corresponding description information.

9. The method according to any one of claims 2 to 8, characterized in that: The code stream includes: a syntax element of a first identifier and corresponding description information, and the parsing of the first identifier from the code stream includes: The first identifier is parsed from the bitstream according to the syntax element of the first identifier and the corresponding description information.

10. The method according to any one of claims 3 to 9, characterized in that: The code stream includes: a syntax element of a second identifier and corresponding description information, and the parsing of the second identifier from the code stream includes: The second identifier is parsed from the bitstream according to the syntax element of the second identifier and the corresponding description information.

11. The method according to any one of claims 5 to 9, characterized in that: The code stream includes: a syntax element of a third identifier and corresponding description information, and the parsing of the third identifier from the code stream includes: The third identifier is parsed from the bitstream according to the syntax element of the third identifier and the corresponding description information.

12. The method according to claim 8, characterized in that The syntax element of the first metadata is distanceFactor.

13. The method according to claim 9, characterized in that The syntax element of the first identifier is complexityLevel.

14. The method according to claim 10, characterized in that The syntax element of the second identifier is hoaGroupHasLowProfileConfig.

15. The method according to any one of claims 1 to 14, characterized in that The method further comprises: generating a virtual loudspeaker signal according to the reconstructed audio signal of the scene audio signal; determining a virtual speaker radius according to a distance between the first metadata and two microphones among the plurality of microphones; Rendering is performed according to the virtual speaker radius and the virtual speaker signal to obtain the rendered audio signal.

16. A decoding method, characterized in that: include: Receive code stream; Parsing a first identifier from the bit stream; According to the bit stream, a reconstructed audio signal is obtained, wherein the first identifier is used to indicate rendering complexity, the rendering complexity is used to determine the number of virtual speakers, the reconstructed audio signal is used to generate a virtual speaker signal, and the virtual speaker signal and the number of virtual speakers are used for rendering to obtain a rendered audio signal.

17. The method according to claim 16, characterized in that The method further comprises: Parsing a second identifier from the code stream; When the value of the second identifier is a preset identifier value, it indicates that the code stream includes the first identifier.

18. The method according to claim 16 or 17, characterized in that The code stream includes: the syntax element of the first identifier and the corresponding description information, and the parsing of the first identifier from the code stream includes: The first identifier is parsed from the bitstream according to the syntax element of the first identifier and the corresponding description information.

19. The method according to claim 17 or 18, characterized in that The code stream includes: a syntax element of a second identifier and corresponding description information, and the parsing of the second identifier from the code stream includes: The second identifier is parsed from the bitstream according to the syntax element of the second identifier and the corresponding description information.

20. The method according to any one of claims 16 to 19, characterized in that The reconstructed audio signal includes a reconstructed audio signal of a multi-point high-order ambisonics (HOA) signal.

21. The method according to claim 18, characterized in that The syntax element of the first identifier is complexityLevel.

22. The method according to any one of claims 16 to 21, characterized in that The method further comprises: generating a virtual loudspeaker signal according to the reconstructed audio signal of the scene audio signal; Determining the number of the virtual speakers according to the first identifier; Rendering is performed according to the number of the virtual speakers and the virtual speaker signal to obtain the rendered audio signal.

23. A decoding device, characterized in that: The device comprises: A first receiving module, used for receiving a code stream; A first parsing module, used for parsing metadata of a scene audio signal from the bit stream; A first signal acquisition module is used to acquire a reconstructed audio signal of the scene audio signal according to the bit stream; the scene audio signal is used to describe the sound field of the sound source in the scene, the scene includes multiple microphones, the metadata includes first metadata, the first metadata is used to determine a virtual speaker radius from a distance between two microphones among the multiple microphones, the virtual speaker radius and the virtual speaker signal are used for rendering to obtain a rendered audio signal, and the virtual speaker signal is generated according to the reconstructed audio signal.

24. A decoding device, characterized in that: The device comprises: A second receiving module, used for receiving a code stream; A second parsing module, used for parsing a first identifier from the bit stream; A second signal acquisition module is used to obtain a reconstructed audio signal of the scene audio signal according to the bit stream; the first identifier is used to indicate rendering complexity, the rendering complexity is used to determine the number of virtual speakers, the reconstructed audio signal is used to generate a virtual speaker signal, and the virtual speaker signal and the number of virtual speakers are used for rendering to obtain a rendered audio signal.

25. An extended reality (XR) device, characterized in that: The XR device includes: a processor and an audio playback module; The processor is used to execute the decoding method according to any one of claims 1 to 22; The audio playback module is used to play the rendered audio signal as described in any one of claims 1 to 22 above.

26. An electronic device, characterized in that: include: a memory and a processor, the memory being coupled to the processor; The memory stores program instructions, and when the program instructions are executed by the processor, the electronic device executes the decoding method according to any one of claims 1 to 22.

27. A chip, characterized in that: It comprises one or more interface circuits and one or more processors; the one or more processors receive or send data through the one or more interface circuits, and when the one or more processors execute computer instructions, the steps of the method described in any one of claims 1 to claim 22 are executed.

28. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program runs on a computer or a processor, the computer or the processor executes the decoding method according to any one of claims 1 to 22.

29. A computer program product, characterized in that The computer program product comprises computer instructions, and when the computer instructions are executed by a computer or a processor, the steps of the method according to any one of claims 1 to 22 are performed.

Images