Audio processing methods and terminal equipment

Abstract

The invention relates to a method for processing audio and a terminal device. This method comprises decoding an audio bitstream to obtain audio optimization metadata, base audio metadata, and M decoded audio data segments; the audio optimization metadata includes the first metadata of the first optimized listening region and the first decoded audio mixing parameter corresponding to the first optimized listening region; rendering M decoded audio data segments based on the user's current location and base audio metadata, to obtain M rendered audio data segments; when the current location is within the first optimized listening region, performing the first audio mixing on the M rendered audio data segments based on the first decoded audio mixing parameter, to obtain M first mixed audio data segments; and blend M segments of the first audio blending data to obtain blended audio data that corresponds to the first optimized listening region.

Description

Audio processing method and terminal

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on January 28, 2022, with application number 202210109139.4 and invention name “A Audio Processing Method and Terminal”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of audio technology, and in particular to an audio processing method and terminal. Background Art

[0003] Mixing is an essential step in music production, and its quality can determine the success or failure of a musical work. The mixed audio output allows listeners to experience detailed and layered musical effects that are unavailable during live recordings, making the music more expressive.

[0004] Virtual reality (VR) technology is gradually being applied to the music field, giving rise to VR music scenes. Currently, when creating VR music scenes, creators typically assume that the user is in the sweet spot, a fixed position, when mixing the music signal. This allows for head movement (i.e., three degrees of freedom (3DoF)). However, users only experience the music best when they are in the sweet spot. If the user moves, the listening experience deteriorates, impacting their musical experience.

[0005] Summary of the Invention

[0006] Embodiments of the present application provide an audio processing method and a terminal for improving the listening experience of a user when he or she is moving freely.

[0007] To solve the above technical problems, the embodiments of the present application provide the following technical solutions:

[0008] In a first aspect, an embodiment of the present application provides an audio processing method, comprising:

[0009] Decoding the audio bitstream to obtain audio optimization metadata, basic audio metadata, and M decoded audio data, wherein the audio optimization metadata includes first metadata of a first optimized listening area and first decoded mixing parameters corresponding to the first optimized listening area, and M is a positive integer;

[0010] Rendering the M first decoded audio data according to the current location of the user and the basic audio metadata to obtain M rendered audio data;

[0011] When the current position is in the first optimized listening area, performing a first mixing process on the M rendered audio data according to the first decoded mixing parameter to obtain M first mixed audio data;

[0012] The M first mixed audio data are mixed to obtain mixed audio data corresponding to the first optimized listening area.

[0013] In the above scheme, since metadata of the first optimized listening zone and first decoded mixing parameters corresponding to the first optimized listening zone can be obtained in the embodiment of the present application, M first decoded audio data are rendered based on the user's current location and the basic audio metadata to obtain M rendered audio data. Next, it is determined that the user's current location is within the first optimized listening zone, and first mixing is performed on the M rendered audio data based on the first decoded mixing parameters to obtain M first mixed audio data. Finally, the M first mixed audio data are mixed to obtain mixed audio data corresponding to the first optimized listening zone. Therefore, in the embodiment of the present application, when the user's current location is within the first optimized listening zone, both mixing and data mixing are performed using the audio data corresponding to the first optimized listening zone. This can provide audio optimized metadata suitable for when the user freely moves to the first optimized listening zone, thereby improving the user's listening experience during free movement.

[0014] In a possible implementation, the audio optimization metadata further includes: second decoded mixing parameters corresponding to the first optimized listening area;

[0015] The method further includes performing a second mixing process on the mixed audio data according to the second decoded mixing parameters to obtain second mixed audio data corresponding to the first optimized listening area.

[0016] In the above solution, after obtaining the second decoded mixing parameters, the decoding terminal can further perform a second mixing process on the mixed audio data corresponding to the first optimized listening area based on the second decoded mixing parameters corresponding to the first optimized listening area to obtain second mixed audio data corresponding to the first optimized listening area. The second mixing process can generate second mixed audio data, which, when played, can improve the user's listening experience.

[0017] In a possible implementation, the second decoded mixing parameter includes at least one of the following: an identifier of the second mixed audio data, an equalizer parameter, a compressor parameter, and a reverberator parameter.

[0018] In the above scheme, the second decoded mixing parameters may include an identifier of the second mixed audio data. The second decoded mixing parameters may also include equalization parameters. For example, the equalization parameters may include an equalization parameter identifier, a gain value and a Q value for each frequency band, wherein the Q value is a parameter of the equalization filter, which represents the quality factor of the equalization filter and can be used to describe the bandwidth of the equalization filter. The second decoded mixing parameters may also include compressor parameters. For example, the compressor parameters may include a compressor identifier, a threshold value, a compression ratio, an attack time, a release time, and a gain compensation value. The second decoded mixing parameters may also include reverberator parameters. For example, the reverberator parameters may include a reverberation type, a reverberation time, a delay time, and a direct-to-reverberation ratio.

[0019] In a possible implementation, the audio optimization metadata further includes: N-1 difference parameters of N-1 second decoded mixing parameters corresponding to N-1 optimized listening zones excluding the first optimized listening zone among the N optimized listening zones, relative to the second decoded mixing parameters corresponding to the first optimized listening zone, where N is a positive integer.

[0020] In the above solution, the difference parameter is a parameter representing the difference between the N-1 second decoded mix parameters of the N-1 optimized listening zones excluding the first optimized listening zone, relative to the second decoded mix parameters corresponding to the first optimized listening zone. The difference parameter is not the N-1 second decoded mix parameters of the N-1 optimized listening zones. By including the difference parameter in the audio optimization metadata, the data size of the audio optimization metadata can be reduced, thereby improving data transmission efficiency and decoding efficiency.

[0021] In a possible implementation, the first decoded mixing parameter includes at least one of the following: an identifier of the rendered audio data, an equalizer parameter, a compressor parameter, and a reverberator parameter.

[0022] In the above solution, the first decoded mixing parameters may include an identifier of the rendered audio data, for example, including identifiers of M rendered audio data. The first decoded mixing parameters may also include equalization parameters, for example, the equalization parameters may include an equalization parameter identifier, a gain value for each frequency band, and a Q value. The first decoded mixing parameters may also include compressor parameters, for example, the compressor parameters may include a compressor identifier, a threshold value, a compression ratio, an attack time, a release time, and a gain compensation value. The first decoded mixing parameters may also include reverberator parameters, for example, the reverberator parameters may include a reverberation type, a reverberation time, a delay time, and a direct-to-reverberation ratio.

[0023] In a possible implementation, the method further includes:

[0024] Decoding the video image code stream to obtain decoded video image data and video image metadata, wherein the video image metadata includes: video metadata and image metadata;

[0025] performing rendering processing on the decoded video image data according to the video image metadata to obtain rendered video image data;

[0026] establishing a virtual scene according to the rendered video image data;

[0027] The first optimized listening zone is identified in the virtual scene according to the rendered video image data and the audio optimization metadata.

[0028] In the above scheme, the decoding terminal renders the decoded video image data according to the video image metadata to obtain rendered video image data. The decoding terminal can use the rendered video image data to establish a virtual scene. Finally, the decoding terminal identifies the first optimized listening area in the virtual scene according to the rendered video image data and the audio optimization metadata, so that the first optimized listening area is displayed in the virtual scene on the decoding terminal side, guiding the user to experience music in the optimized listening area and improving the user's listening effect.

[0029] In a possible implementation manner, the first metadata includes at least one of the following: a reference coordinate system of the first optimized listening zone, center position coordinates of the first optimized listening zone, and a shape of the first optimized listening zone.

[0030] In the above solution, the metadata of the first optimized listening zone may include a reference coordinate system, or the metadata of the first optimized listening zone may not include a reference coordinate system, for example, the first optimized listening zone uses a default coordinate system. The metadata of the first optimized listening zone may include descriptive information describing the first optimized listening zone, such as the center coordinates of the first optimized listening zone, the shape of the first optimized listening zone, etc. In the embodiment of the present application, the first optimized listening zone can have various shapes, for example, a sphere, a cube, a cylinder, or any other arbitrary shape.

[0031] In a possible implementation, the audio optimization metadata includes: N-1 difference parameters of N-1 first decoded mixing parameters corresponding to N-1 optimized listening zones excluding the first optimized listening zone among N optimized listening zones, relative to the first decoded mixing parameters corresponding to the first optimized listening zone, where N is a positive integer.

[0032] In the above solution, the difference parameter is a parameter representing the difference between the N-1 first decoded mix parameters of the N-1 optimized listening zones excluding the first optimized listening zone and the first decoded mix parameters corresponding to the first optimized listening zone. The difference parameter is not the N-1 first decoded mix parameters of the N-1 optimized listening zones. By including the difference parameter in the audio optimization metadata, the data size of the audio optimization metadata can be reduced, thereby improving data transmission efficiency and decoding efficiency.

[0033] In a second aspect, an embodiment of the present application further provides an audio processing method, comprising:

[0034] receiving audio optimization metadata, audio basic metadata, and M first audio data, wherein the audio optimization metadata includes: first metadata of a first optimized listening area and first mixing parameters corresponding to the first optimized listening area, and M is a positive integer;

[0035] compressing and encoding the audio optimization metadata, the audio basic metadata, and the M first audio data to obtain an audio code stream;

[0036] Send the audio code stream.

[0037] In the above scheme, audio optimization metadata, audio basic metadata and M first audio data are first received. The audio optimization metadata includes metadata of the first optimized listening area and first mixing parameters of the first optimized listening area. Therefore, it is possible to provide audio optimization metadata suitable for when the user freely moves to the first optimized listening area, thereby improving the user's listening effect when freely moving.

[0038] In a possible implementation, the audio optimization metadata further includes: a second mixing parameter change identifier, wherein:

[0039] The second mixing parameter change identifier is used to indicate whether the second mixing parameter corresponding to the first audio data of the current frame has changed relative to the second mixing parameter corresponding to the first audio data of the previous frame.

[0040] In the above-described solution, the transmitting terminal may set a second mixing parameter change identifier in the audio optimization metadata. The second mixing parameter change identifier is used to indicate whether the second mixing parameter corresponding to the first optimized listening zone has changed, thereby enabling the decoding terminal to determine whether the second mixing parameter corresponding to the first optimized listening zone has changed based on the second mixing parameter change identifier. For example, when the second mixing parameter of the first audio data of the current frame has changed relative to the second mixing parameter corresponding to the first audio data of the previous frame, the second mixing parameter change identifier is true. The transmitting terminal may also send change information about the second mixing parameter corresponding to the first audio data. The decoding terminal receives the change information about the second mixing parameter corresponding to the first audio data and, based on the change information, obtains the changed second mixing parameter corresponding to the first audio data of the current frame.

[0041] In a possible implementation, the audio optimization metadata further includes: second mixing parameters corresponding to the first optimized listening area.

[0042] In the above solution, when the production terminal performs two mixing processes, the audio optimization metadata obtained by the production terminal may include first metadata for the first optimized listening area, first mixing parameters corresponding to the first optimized listening area, and second mixing parameters corresponding to the first optimized listening area. After the decoding terminal obtains this audio optimization metadata, the decoding terminal also needs to perform two mixing processes. This two-step mixing process can improve the user's listening experience.

[0043] In a possible implementation, the audio optimization metadata further includes: second mixing parameters corresponding to a first optimized listening zone among N optimized listening zones, and N-1 difference parameters of N-1 second mixing parameters corresponding to N-1 optimized listening zones among the N optimized listening zones excluding the first optimized listening zone, relative to the second mixing parameters corresponding to the first optimized listening zone, where N is a positive integer.

[0044] In the above solution, the difference parameter is a parameter representing the difference between the N-1 second mixing parameters of the N-1 optimized listening zones excluding the first optimized listening zone and the second mixing parameters corresponding to the first optimized listening zone. The difference parameter is not the N-1 second mixing parameter of the N-1 optimized listening zones. By including the difference parameter in the audio optimization metadata, the data size of the audio optimization metadata can be reduced, thereby improving data transmission and decoding efficiency.

[0045] In a possible implementation, the second mixing parameter includes at least one of the following: an identifier of the first audio data, an equalizer parameter, a compressor parameter, and a reverberator parameter.

[0046] In the above solution, the second mixing parameters may include an identifier of the first audio data, for example, including the identifier of the first audio data. The second mixing parameters may also include equalization parameters, for example, the equalization parameters may include an equalization parameter identifier, a gain value for each frequency band, and a Q value. The second mixing parameters may also include compressor parameters, for example, the compressor parameters may include a compressor identifier, a threshold value, a compression ratio, an attack time, a release time, and a gain compensation value. The second mixing parameters may also include reverberator parameters, for example, the reverberator parameters may include a reverberation type, a reverberation time, a delay time, and a direct-to-reverberation ratio.

[0047] In a possible implementation, the audio optimization metadata further includes: N-1 difference parameters of N-1 first mixing parameters corresponding to N-1 optimized listening zones other than the first optimized listening zone among the N optimized listening zones, relative to the first mixing parameters corresponding to the first optimized listening zone.

[0048] In the above solution, the difference parameter is a parameter representing the difference between the N-1 first mixing parameters of the N-1 optimized listening zones excluding the first optimized listening zone and the first mixing parameters corresponding to the first optimized listening zone. The difference parameter is not the N-1 first mixing parameter of the N-1 optimized listening zones. By including the difference parameter in the audio optimization metadata, the data size of the audio optimization metadata can be reduced, thereby improving data transmission efficiency and decoding efficiency.

[0049] In a possible implementation, the first mixing parameter includes at least one of the following: an identifier of the first audio data, an equalizer parameter, a compressor parameter, and a reverberator parameter.

[0050] In the above solution, the first mixing parameters may include an identifier of the first audio data, for example, including identifiers of M first audio data. The first mixing parameters may also include equalization parameters, for example, the equalization parameters may include an equalization parameter identifier, a gain value for each frequency band, and a Q value. The first mixing parameters may also include compressor parameters, for example, the compressor parameters may include a compressor identifier, a threshold value, a compression ratio, an attack time, a release time, and a gain compensation value. The first mixing parameters may also include reverberator parameters, for example, the reverberator parameters may include a reverberation type, a reverberation time, a delay time, and a direct-to-reverberation ratio.

[0051] In a possible implementation, the first metadata of the first optimized listening zone includes at least one of the following: a reference coordinate system of the first optimized listening zone, center position coordinates of the first optimized listening zone, and a shape of the first optimized listening zone.

[0052] In the above solution, the metadata of the first optimized listening zone may include a reference coordinate system, or the metadata of the first optimized listening zone may not include a reference coordinate system, for example, the first optimized listening zone uses a default coordinate system. The metadata of the first optimized listening zone may include descriptive information describing the first optimized listening zone, such as the center coordinates of the first optimized listening zone, the shape of the first optimized listening zone, etc. In the embodiment of the present application, the first optimized listening zone can have various shapes, for example, a sphere, a cube, a cylinder, or any other arbitrary shape.

[0053] In a possible implementation, the audio optimization metadata further includes: center position coordinates of a first optimized listening zone among N optimized listening zones, and position offsets of center position coordinates of N-1 optimized listening zones among the N optimized listening zones except the first optimized listening zone relative to the center position coordinates of the first optimized listening zone, where N is a positive integer.

[0054] In the above solution, the position offset refers to the offset between the center coordinates of the N-1 optimized listening zones (excluding the first optimized listening zone) and the center coordinates of the first optimized listening zone, rather than the center coordinates of the N-1 optimized listening zones. By including the position offset in the audio optimization metadata, the data size of the audio optimization metadata can be reduced, improving data transmission and decoding efficiency.

[0055] In a possible implementation, the audio optimization metadata further includes: an optimized listening area change identifier, and / or a first mixing parameter change identifier, wherein:

[0056] The optimized listening area change identifier is used to indicate whether the first optimized listening area has changed;

[0057] The first mixing parameter change identifier is used to indicate whether the first mixing parameter corresponding to the first audio data of the current frame has changed relative to the first mixing parameter corresponding to the first audio data of the previous frame.

[0058] In the above solution, the transmitting terminal may set a first mixing parameter change identifier in the audio optimization metadata. The first mixing parameter change identifier is used to indicate whether the first mixing parameter corresponding to the first audio data of the current frame has changed relative to the first mixing parameter corresponding to the first audio data of the previous frame, thereby enabling the decoding terminal to determine whether the first mixing parameter corresponding to the first optimized listening area has changed based on the first mixing parameter change identifier. In addition, the transmitting terminal may set an optimized listening area change identifier in the audio optimization metadata. The optimized listening area change identifier is used to indicate whether the optimized listening area determined by the production terminal has changed, thereby enabling the decoding terminal to determine whether the optimized listening area has changed based on the optimized listening area change identifier.

[0059] In a third aspect, an embodiment of the present application further provides an audio processing method, comprising:

[0060] Acquire basic audio metadata and metadata of N optimized listening areas, where N is a positive integer and the N optimized listening areas include a first optimized listening area;

[0061] Rendering the M audio data to be processed according to the first optimized listening area and the basic audio metadata to obtain M rendered audio data corresponding to the first optimized listening area, where M is a positive integer;

[0062] Performing a first mixing process on the M rendered audio data to obtain M first mixed audio data and first mixing parameters corresponding to the first optimized listening area;

[0063] Audio optimization metadata is generated according to the first metadata of the first optimized listening area and the first mixing parameters, where the audio optimization metadata includes: the first metadata and the first mixing parameters.

[0064] In the above scheme, since the audio optimization metadata in the embodiment of the present application includes the first metadata and the first mixing parameters of the first optimized listening zone, it is possible to provide audio optimization metadata suitable for when the user freely moves to the first optimized listening zone, thereby improving the user's listening experience when freely moving.

[0065] In one possible implementation, the method further includes: performing a mixing process on the M first mixed audio data to obtain mixed audio data corresponding to the first optimized listening zone; performing a second mixing process on the mixed audio data to obtain second mixed audio data corresponding to the first optimized listening zone and second mixing parameters corresponding to the first optimized listening zone;

[0066] Generating audio optimization metadata according to the first metadata of the first optimized listening area and the first mixing parameter includes:

[0067] The audio optimization metadata is generated according to the first metadata of the first optimized listening area, the first mixing parameters, and the second mixing parameters.

[0068] In the above solution, when the production terminal performs two mixing processes, the audio optimization metadata obtained by the production terminal may include first metadata for the first optimized listening area, first mixing parameters corresponding to the first optimized listening area, and second mixing parameters corresponding to the first optimized listening area. After the decoding terminal obtains this audio optimization metadata, the decoding terminal also needs to perform two mixing processes. This two-step mixing process can further improve the user's listening experience.

[0069] In a possible implementation, the second mixing parameters include: an identifier of the second mixed audio data, an equalizer parameter, a compressor parameter, and a reverberator parameter.

[0070] In the above solution, the second mixing parameters may include an identifier of the second mixed audio data. The second mixing parameters may also include equalization parameters, for example, the equalization parameters may include an equalization parameter identifier, a gain value for each frequency band, and a Q value. The second mixing parameters may also include compressor parameters, for example, the compressor parameters may include a compressor identifier, a threshold value, a compression ratio, an attack time, a release time, and a gain compensation value. The second mixing parameters may also include reverberator parameters, for example, the reverberator parameters may include a reverberation type, a reverberation time, a delay time, and a direct-to-reverberation ratio.

[0071] In a possible implementation, the first mixing parameters include: an identifier of the first mixed audio data, an equalizer parameter, a compressor parameter, and a reverberator parameter.

[0072] In the above solution, the first mixing parameters may include an identifier of the first mixed audio data, for example, including identifiers of M first mixed audio data. The first mixing parameters may also include equalization parameters, for example, the equalization parameters may include an equalization parameter identifier, a gain value for each frequency band, and a Q value. The first mixing parameters may also include compressor parameters, for example, the compressor parameters may include a compressor identifier, a threshold value, a compression ratio, an attack time, a release time, and a gain compensation value. The first mixing parameters may also include reverberator parameters, for example, the reverberator parameters may include a reverberation type, a reverberation time, a delay time, and a direct-to-reverberation ratio.

[0073] In a possible implementation, obtaining metadata of N optimized listening areas includes:

[0074] Acquire video image metadata and video image data, wherein the video image metadata includes: video metadata and image metadata, and the video image data includes: video data and image data;

[0075] Rendering the video image data according to the video image metadata to obtain video scene information;

[0076] The metadata of the N optimized listening areas are obtained according to the video scene information.

[0077] In the above solution, the production terminal configures N optimized listening areas based on the generated video scene information, thereby generating metadata for the N optimized listening areas. Since the metadata for the N optimized listening areas is generated using the video scene information, an optimized listening area that better matches the video scene can be selected.

[0078] In a possible implementation, the rendering processing of the M to-be-processed audio data according to the first optimized listening area and the basic audio metadata includes:

[0079] adjusting the audio basic metadata according to the mixed audio data corresponding to the first optimized listening area to obtain adjusted audio basic metadata, wherein the mixed audio data is obtained by mixing the M first mixed audio data;

[0080] The rendering process is performed on the M to-be-processed audio data according to the first optimized listening area and the adjusted basic audio metadata.

[0081] In the above solution, the production terminal adjusts the audio basic metadata according to the mixed audio data corresponding to the first optimized listening area to obtain adjusted audio basic metadata. For example, the frequency response of one or more audio signals in the audio data or the position, gain, and other parameters of the audio signal in the audio basic metadata are adjusted, thereby adjusting the position, gain, and other parameters of the audio data. By adjusting the audio basic metadata, the user's listening experience can be further improved.

[0082] In a possible implementation, the first metadata of the first optimized listening zone includes at least one of the following: a reference coordinate system of the first optimized listening zone, center position coordinates of the first optimized listening zone, and a shape of the first optimized listening zone.

[0083] In the above solution, the metadata of the first optimized listening zone may include a reference coordinate system, or the metadata of the first optimized listening zone may not include a reference coordinate system, for example, the first optimized listening zone uses a default coordinate system. The metadata of the first optimized listening zone may include descriptive information describing the first optimized listening zone, such as the center coordinates of the first optimized listening zone, the shape of the first optimized listening zone, etc. In the embodiment of the present application, the first optimized listening zone can have various shapes, for example, a sphere, a cube, a cylinder, or any other arbitrary shape.

[0084] In a fourth aspect, an embodiment of the present application further provides a decoding terminal, including:

[0085] a decoding module, configured to decode the audio bitstream to obtain audio optimization metadata, basic audio metadata, and M decoded audio data, wherein the audio optimization metadata includes first metadata of a first optimized listening area and first decoded mixing parameters corresponding to the first optimized listening area, and M is a positive integer;

[0086] a rendering module, configured to render the M first decoded audio data according to the current location of the user and the basic audio metadata to obtain M rendered audio data;

[0087] a mixing module, configured to perform a first mixing process on the M rendered audio data according to the first decoded mixing parameter to obtain the M first mixed audio data when the current position is within the first optimized listening zone;

[0088] A mixing module is configured to perform mixing processing on the M first mixed audio data to obtain mixed audio data corresponding to the first optimized listening area.

[0089] In the fourth aspect of the present application, the constituent modules of the decoding terminal may further execute the steps described in the aforementioned first aspect and various possible implementations. For details, please refer to the aforementioned description of the first aspect and various possible implementations.

[0090] In a fifth aspect, an embodiment of the present application further provides a transmission terminal, including:

[0091] a receiving module, configured to receive audio optimization metadata, basic audio metadata, and the M first audio data, wherein the audio optimization metadata includes: first metadata for a first optimized listening area and first mixing parameters corresponding to the first optimized listening area, and M is a positive integer;

[0092] an encoding module, configured to compress and encode the audio optimization metadata, the audio basic metadata, and the M first audio data to obtain an audio code stream;

[0093] The sending module is used to send the audio code stream.

[0094] In the fifth aspect of the present application, the constituent modules of the transmission terminal may also execute the steps described in the aforementioned second aspect and various possible implementations. For details, please refer to the aforementioned description of the second aspect and various possible implementations.

[0095] In a sixth aspect, an embodiment of the present application further provides a production terminal, including:

[0096] an acquisition module, configured to acquire basic audio metadata and metadata of N optimized listening areas, where N is a positive integer and the N optimized listening areas include a first optimized listening area;

[0097] a rendering module, configured to perform rendering processing on M pieces of to-be-processed audio data according to the first optimized listening area and the basic audio metadata, so as to obtain M pieces of rendered audio data corresponding to the first optimized listening area, where M is a positive integer;

[0098] a mixing module, configured to perform a first mixing process on the M rendered audio data to obtain M first mixed audio data and first mixing parameters corresponding to the first optimized listening area;

[0099] A generating module is configured to generate audio optimization metadata according to first metadata of the first optimized listening area and the first mixing parameters, wherein the audio optimization metadata includes: the first metadata and the first mixing parameters.

[0100] In the sixth aspect of the present application, the component modules of the manufacturing terminal may also execute the steps described in the aforementioned third aspect and various possible implementations. For details, please refer to the aforementioned description of the third aspect and various possible implementations.

[0101] In a seventh aspect, an embodiment of the present application provides a computer-readable storage medium, wherein instructions are stored in the computer-readable storage medium, which, when executed on a computer, enables the computer to execute the methods described in the first to third aspects above.

[0102] In an eighth aspect, an embodiment of the present application provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the methods described in the first to third aspects above.

[0103] In the ninth aspect, an embodiment of the present application provides a communication device, which may include entities such as a terminal device or a chip, and the communication device includes: a processor, a memory; the memory is used to store instructions; the processor is used to execute the instructions in the memory, so that the communication device performs the method as described in any one of the first to third aspects above.

[0104] In a tenth aspect, the present application provides a chip system comprising a processor for supporting a decoding terminal, a transmission terminal, and a production terminal in implementing the functions described in the above aspects, such as sending or processing the data and / or information described in the above methods. In one possible design, the chip system further comprises a memory for storing the necessary program instructions and data for the decoding terminal, the transmission terminal, and the production terminal. The chip system may consist of a chip alone or may include a chip and other discrete components.

[0105] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0106] In one embodiment of the present application, an audio bitstream is decoded to obtain audio optimization metadata, basic audio metadata, and M decoded audio data, wherein the audio optimization metadata includes first metadata of a first optimized listening zone and first decoded mixing parameters corresponding to the first optimized listening zone, where M is a positive integer; the M first decoded audio data are rendered based on a current position of a user and the basic audio metadata to obtain M rendered audio data; when the current position of the user is in the first optimized listening zone, first mixing is performed on the M rendered audio data based on the first decoded mixing parameters to obtain M first mixed audio data; and the M first mixed audio data are mixed to obtain mixed audio data corresponding to the first optimized listening zone. In the embodiment of the present application, metadata of the first optimized listening zone and first decoded mixing parameters corresponding to the first optimized listening zone can be obtained. M first decoded audio data are rendered based on the user's current location and the basic audio metadata to obtain M rendered audio data. Next, it is determined that the user's current location is within the first optimized listening zone. First mixing is performed on the M rendered audio data based on the first decoded mixing parameters to obtain M first mixed audio data. Finally, the M first mixed audio data are mixed to obtain mixed audio data corresponding to the first optimized listening zone. Therefore, in the embodiment of the present application, when the user's current location is within the first optimized listening zone, both mixing and data mixing are performed using the audio data corresponding to the first optimized listening zone. This provides audio optimized metadata suitable for when the user freely moves to the first optimized listening zone, thereby improving the user's listening experience during free movement.

[0107] In another embodiment of the present application, audio optimization metadata, basic audio metadata, and M first audio data are received, where the audio optimization metadata includes: first metadata for a first optimized listening zone and first mixing parameters corresponding to the first optimized listening zone, where M is a positive integer; the audio optimization metadata, basic audio metadata, and M first audio data are compressed and encoded to obtain an audio stream; and the audio stream is transmitted. In this embodiment of the present application, the audio optimization metadata, basic audio metadata, and M first audio data are first received, where the audio optimization metadata includes metadata for the first optimized listening zone and first mixing parameters for the first optimized listening zone. This provides audio optimization metadata suitable for when a user freely moves to the first optimized listening zone, thereby improving the user's listening experience during free movement.

[0108] In another embodiment of the present application, basic audio metadata and metadata for N optimized listening zones are obtained, where N is a positive integer and the N optimized listening zones include a first optimized listening zone. M to-be-processed audio data are rendered based on the first optimized listening zone and the basic audio metadata to obtain M rendered audio data corresponding to the first optimized listening zone, where M is a positive integer. A first mixing process is performed on the M rendered audio data to obtain M first mixed audio data and first mixing parameters corresponding to the first optimized listening zone. Audio optimized metadata is generated based on the first metadata and first mixing parameters for the first optimized listening zone, where the audio optimized metadata includes the first metadata and the first mixing parameters. Because the audio optimized metadata in this embodiment of the present application includes the first metadata and the first mixing parameters for the first optimized listening zone, it can provide audio optimized metadata suitable for when a user freely moves to the first optimized listening zone, thereby improving the user's listening experience during free movement. BRIEF DESCRIPTION OF THE DRAWINGS

[0109] FIG1 is a schematic diagram of the structure of an audio processing system provided in an embodiment of the present application;

[0110] FIG2 is a schematic diagram of an interaction process between a production terminal, a transmission terminal, and a decoding terminal according to an embodiment of the present application;

[0111] FIG3 is a schematic diagram of a flow chart of stream data processing in a virtual reality streaming service system according to an embodiment of the present application;

[0112] FIG4 is an end-to-end flow chart of a 6DoF VR music scene provided by an embodiment of the present application;

[0113] FIG5 is a schematic diagram of a VR concert scene supporting 6DoF provided in an embodiment of the present application;

[0114] FIG6 is an end-to-end flow chart of another 6DoF VR music scene provided in an embodiment of the present application;

[0115] FIG7 is a schematic diagram of the structure of a decoding terminal provided in an embodiment of the present application;

[0116] FIG8 is a schematic diagram of the structure of a transmission terminal provided in an embodiment of the present application;

[0117] FIG9 is a schematic diagram of the structure of a production terminal provided in an embodiment of the present application;

[0118] FIG10 is a schematic diagram of the structure of another decoding terminal provided in an embodiment of the present application;

[0119] FIG11 is a schematic diagram of the structure of another transmission terminal provided in an embodiment of the present application;

[0120] FIG12 is a schematic diagram of the composition structure of another production terminal provided in an embodiment of the present application. DETAILED DESCRIPTION

[0121] Embodiments of the present application provide an audio processing method and a terminal for improving the listening experience of a user when he or she is moving freely.

[0122] The embodiments of the present application are described below with reference to the accompanying drawings.

[0123] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, and this is merely a way of distinguishing the objects of the same attributes when describing them in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0124] The production process of a musical work includes the following steps: writing lyrics and composing music, arranging, recording, mixing, mastering, etc. Among them, mixing is an indispensable step in music production, and the quality of mixing will determine the success or failure of a musical work.

[0125] Mixing is the process of integrating sounds from multiple sources into a single-channel track, stereo track, or multi-channel track. These sound sources may be violins, drums, vocals, or other recorded sounds. During the mixing process, the frequency, dynamics, sound quality, positioning, and sound field of each original signal need to be adjusted individually to optimize the signal of each track; finally, the frequency and dynamics of the mixed signal are adjusted to optimize the auditory effect of the mixed signal. Mixing effects include equalizers, compressors, reverberators, etc. Through mixing, it is possible to produce musical effects with delicate sound quality and distinct layers that ordinary listeners cannot hear in live recordings, making the music more expressive.

[0126] With the development of virtual reality (VR), augmented reality (AR), and mixed reality (MR), VR technology has gradually been applied to the music field. Various VR music scenarios have emerged, such as VR music videos, VR live concerts, and VR music variety shows. Compared to traditional music, these VR music scenarios combine 3D spatial music effects with VR visual experiences, providing a more vivid and immersive experience, significantly enhancing the user's music experience. In most current VR music scenarios, the music effects of 3DoF scenes only support the user's head movement effects and do not support six degrees of freedom (6DoF) scenes.

[0127] As VR hardware matures, users' demands for a superior music experience are also increasing. Therefore, VR music scenes supporting 6DoF will be a future trend in the music industry. In traditional music creation, the production side typically assumes the user is in the sweet spot during mixing, and their position remains constant. Mixing is completed before the music signal is transmitted, and the transmitted music signal is a mixed signal. On the user side (i.e., the audio decoder), the audio renderer only needs to adapt to the user's playback device to ensure a perfect musical experience. In VR music scenes supporting 6DoF, users can move freely within the scene. For example, the volume, timbre, and reverberation of a violin sound heard by a user at 3 meters (m) from the violin differ significantly from that heard at 0.5 meters. Because the user's position constantly changes, it's impossible to determine their actual location during mixing. Therefore, traditional music production methods struggle to ensure a perfect musical experience for users as they move freely, significantly impacting the user experience in VR music scenes supporting 6DoF.

[0128] The embodiment of the present application can improve the listening effect of the user when he moves freely, for example, when the user moves freely in a virtual reality scene or an augmented reality scene, the listening effect of the user is improved. The embodiment of the present application is described in detail below. As shown in Figure 1, the embodiment of the present application provides an audio processing system 100, including: a production terminal 101, a transmission terminal 102 and a decoding terminal 103. Among them, the production terminal 101 can select one or more optimized listening areas in the virtual scene. The optimized listening area can also be called an "optimal listening area". The optimized listening area is a listening area pre-selected from the virtual scene. The production terminal 101 configures audio optimization metadata for each optimized listening area, that is, it can generate a set of corresponding mixing parameters for each optimized listening area to ensure that the listener can hear the music effect of the mixed music signal in the optimized listening area, thereby improving the music experience of the user in the 6DoF music scene.

[0129] The production terminal 101 can communicate with the transmission terminal 102, and the transmission terminal 102 can communicate with the decoding terminal 103. The transmission terminal 102 can receive the audio optimization metadata of each optimized listening area from the production terminal 101, and the transmission terminal 102 compresses and encodes the audio optimization metadata to obtain an audio code stream. The transmission terminal 102 can send the audio code stream to the decoding terminal 103. The decoding terminal 103 can obtain the audio optimization metadata of each optimized listening area, and the decoding terminal 103 selects an optimized listening area that matches the current position of the user (for example, the matched optimized listening area is called the first optimized listening area) and then uses the first decoding mixing parameter corresponding to the first optimized listening area for mixing, so that the user can hear the musical effect of the mixed music signal, thereby improving the user's music experience in the 6DoF music scene.

[0130] In an embodiment of the present application, the production terminal may include VR music software with 6DoF audio, a 3D audio engine, etc., and the production terminal can be applied to VR terminal devices, chips and wireless network devices.

[0131] The transmission terminal can be applied to various terminal devices requiring audio communication, wireless devices requiring transcoding, and core network devices. For example, the transmission terminal can be an audio encoder in the aforementioned terminal devices, wireless devices, or core network devices. For example, the audio encoder can include a media gateway, transcoding device, media resource server, mobile terminal, fixed network terminal, etc. in the wireless access network or core network. The audio encoder can also be an audio encoder used in virtual reality streaming services.

[0132] Similarly, the decoding terminal can be applied to various terminal devices with audio communication needs, wireless devices and core network devices with transcoding needs. For example, the decoding terminal can be the audio decoder of the above terminal devices, wireless devices or core network devices.

[0133] First, an audio processing method provided in an embodiment of the present application is introduced. The audio processing method is implemented based on the audio processing system described in Figure 1. As shown in Figure 2, it is a schematic diagram of an interaction process between the production terminal, transmission terminal and decoding terminal provided in an embodiment of the present application. The production terminal can communicate with the transmission terminal, and the transmission terminal can communicate with the decoding terminal, wherein the production terminal executes the following steps 201 to 204, the transmission terminal executes the following steps 205 to 207, and the decoding terminal executes the following steps 208 to 211.

[0134] 201. A production terminal obtains basic audio metadata and metadata of N optimized listening areas, where N is a positive integer and the N optimized listening areas include a first optimized listening area.

[0135] Basic audio metadata refers to the basic metadata required for creating VR music scenes. There are no restrictions on the components and content of basic audio metadata. For example, as shown in Table 1, basic audio metadata includes at least one of the following: sound source metadata, physical model metadata, acoustic metadata, moving object metadata, interaction metadata, and resource metadata.

[0136] Table 1 is the composition structure of basic audio metadata

[0137]

[0138] Specifically, sound source metadata is used to describe the properties of the sound source. For example, sound source metadata can include object audio metadata, multi-channel audio metadata, and scene audio metadata. Object audio metadata and multi-channel audio metadata include information such as the sound source's reference coordinate system, position, gain, volume and shape, directivity, attenuation mode, and playback control. Scene audio metadata includes information such as the position and reference coordinate system of the scene microphone, the gain of the scene audio, the effective area, the type of degrees of freedom supported for playback (0 / 3 / 6DoF), attenuation mode, and playback control.

[0139] Physical model metadata includes spherical models, cylindrical models, cube models, and triangular mesh models. The spherical models, cylindrical models, and cube models are used to describe the shapes of objects in a virtual room, while the triangular mesh model can be used to describe rooms of any shape and irregular objects in the scene.

[0140] Acoustic metadata includes acoustic material metadata and acoustic environment metadata. Acoustic material metadata is used to describe the acoustic properties of objects and room surface materials in the scene, while acoustic environment metadata is used to describe the reverberation information of the room in the VR scene, etc.

[0141] Moving object metadata is used to describe the motion information of sound sources, objects, etc. in the scene; interaction metadata is used to describe the interaction behavior between users and VR scenes.

[0142] Resource metadata is used to describe resource information required for VR scenes.

[0143] The metadata used in most VR music scenes can be specifically covered by the metadata in Table 1 above.

[0144] In addition, in an embodiment of the present application, in addition to obtaining basic audio metadata, the production terminal can also obtain N optimized listening areas from the virtual scene. The value of N is not limited; for example, N can be equal to 1 or greater than 1. The production terminal obtains metadata for these N optimized listening areas. The metadata for an optimized listening area includes configuration parameters for the optimized listening area, such as the size, shape, and center position of the optimized listening area. The configuration parameters included in the metadata for the optimized listening area are not limited.

[0145] For example, the N optimized listening zones may cover different positions of the user, and the N optimized listening zones include a first optimized listening zone, where the first optimized listening zone may refer to an optimized listening zone that matches the current position of the user.

[0146] In some embodiments of the present application, the first optimized listening zone may be any one of N optimized listening zones. The first metadata of the first optimized listening zone includes at least one of the following: a reference coordinate system of the first optimized listening zone, coordinates of a center position of the first optimized listening zone, and a shape of the first optimized listening zone.

[0147] Specifically, the metadata of the first optimized listening zone may include a reference coordinate system, or the metadata of the first optimized listening zone may not include a reference coordinate system. For example, the first optimized listening zone adopts a default coordinate system.

[0148] The metadata of the first optimized listening area may include descriptive information describing the first optimized listening area, such as the center coordinates of the first optimized listening area, the shape of the first optimized listening area, etc. In the embodiment of the present application, the first optimized listening area may have various shapes, such as a sphere, a cube, a cylinder, or any other arbitrary shape.

[0149] In some embodiments of the present application, step 201 of making the terminal obtain metadata of N optimized listening areas includes:

[0150] A1. The production terminal obtains video image metadata and video image data. The video image metadata includes: video metadata and image metadata. The video image data includes: video data and image data.

[0151] The production terminal can also obtain video image metadata and video image data in the virtual scene. Video image metadata can also be called video and image metadata, and video image data can also be called video and image data. Video image data includes the data content of the video and image. Video image metadata is information used to describe the attributes of the video and image content.

[0152] A2. The production terminal renders the video image data according to the video image metadata to obtain video scene information.

[0153] The production terminal uses the video image metadata to perform video scene rendering processing on the video image data to obtain video scene information. For example, the video scene may be a virtual scene.

[0154] A3. The production terminal obtains metadata of N optimized listening areas based on the video scene information.

[0155] Among them, the production terminal configures N optimized listening areas according to the generated video scene information, so as to generate metadata of the N optimized listening areas. Since the metadata of the N optimized listening areas are generated using the video scene information, the optimized listening area that better matches the video scene can be selected.

[0156] 202. The production terminal renders the M audio data to be processed according to the first optimized listening area and the basic audio metadata to obtain M rendered audio data corresponding to the first optimized listening area, where M is a positive integer.

[0157] The production terminal obtains M first audio data to be processed, and the M first audio data to be processed are audio data that need to be sent to the decoding terminal. The value of M is not limited, for example, M can be equal to 1, or M can be greater than 1.

[0158] After acquiring the M audio data to be processed, the production terminal performs rendering processing on each optimized listening area, thereby obtaining rendered audio data corresponding to each optimized listening area. For example, the production terminal performs rendering processing on the M audio data to be processed based on the first optimized listening area among the N optimized listening areas and the basic audio metadata, thereby obtaining M rendered audio data corresponding to the first optimized listening area among the N optimized listening areas.

[0159] It should be noted that the rendered second audio data may be a single-channel signal or a binaural rendered signal. The N optimized listening areas have a total of N*M rendered second audio data, where * represents a multiplication operator.

[0160] It should be noted that, in addition to the first optimized listening area, the N optimized listening areas may also include a second optimized listening area. The method provided in the embodiment of the present application may further include:

[0161] The production terminal renders the M audio data to be processed according to the second optimized listening area and the basic audio metadata to obtain M rendered audio data corresponding to the second optimized listening area, where M is a positive integer.

[0162] The rendering process performed by the production terminal based on the second optimized listening area is similar to the rendering process performed based on the first optimized listening area in step 201 and will not be repeated here. Similarly, the subsequent steps 203 and 204 are also performed for the first optimized listening area. Similar processes as those in steps 203 and 204 can also be performed for the second optimized listening area and will not be repeated here.

[0163] 203. The production terminal performs a first mixing process on the M rendered audio data to obtain M first mixed audio data and first mixing parameters corresponding to the first optimized listening area.

[0164] Among them, after the production terminal obtains the M rendered audio data corresponding to the first optimized listening area, it can also perform a first mixing process on the M rendered audio data corresponding to the first optimized listening area for the first optimized listening area to obtain M first mixed audio data and first mixing parameters corresponding to the first optimized listening area. Among them, the first mixing parameters are used to record the mixing parameters used in the first mixing process, and the mixing parameters can also be called "mixing metadata". The above mixing processing steps can be completed by the VR music scene production terminal or by the mixing terminal, and are not limited here.

[0165] It should be noted that the M first mixed audio data in step 203 are audio data obtained by the production terminal performing the first mixing process, and are different from the M first mixed audio data obtained by the subsequent decoding terminal performing the first mixing process.

[0166] In some embodiments of the present application, the first mixing parameter includes at least one of the following: an identifier of the first mixed audio data, an equalizer parameter, a compressor parameter, and a reverberator parameter.

[0167] Among them, the first mixing parameters may include the identifier of the first mixed audio data, for example, including the identifiers of M first mixed audio data. The first mixing parameters may also include equalization parameters, for example, the equalization parameters may include the equalization parameter identifier, the gain value and Q value of each frequency band, wherein the Q value is a parameter of the equalization filter, which represents the quality factor of the equalization filter and can be used to describe the bandwidth of the equalization filter. The first mixing parameters may also include compressor parameters, for example, the compressor parameters may include the compressor identifier, threshold value, compression ratio, attack time, release time, and gain compensation value. The first mixing parameters may also include reverberator parameters, for example, the reverberator parameters may include the reverberation type, reverberation time, delay time, and direct-to-reverberation ratio.

[0168] 204. The production terminal generates audio optimization metadata according to the first metadata and the first mixing parameters of the first optimized listening area. The audio optimization metadata includes: the first metadata and the first mixing parameters.

[0169] After obtaining the first mixing parameters corresponding to the first optimized listening zone, the production terminal may generate audio optimization metadata for the first optimized listening zone. This audio optimization metadata, different from the aforementioned basic audio metadata, includes: first metadata for the first optimized listening zone and first mixing parameters corresponding to the first optimized listening zone. The audio optimization metadata is used to optimize the music signal heard by a user currently located within the first optimized listening zone to enhance the musical effect of the music signal.

[0170] In some embodiments of the present application, the audio processing method that can be executed by the production terminal further includes the following steps:

[0171] B1. The production terminal mixes the M first mixed audio data to obtain mixed audio data corresponding to the first optimized listening area;

[0172] B2. Perform a second mixing process on the mixed audio data to obtain second mixed audio data corresponding to the first optimized listening area and second mixing parameters corresponding to the first optimized listening area.

[0173] Specifically, after the production terminal performs the first mixing process in step 203, in order to further improve the mixing effect of the audio data, the production terminal can also perform steps B1 to B2. For the first optimized listening area, the production terminal can mix the M first mixed audio data to obtain mixed audio data corresponding to the first optimized listening area, and then perform a second mixing process on the mixed audio data to obtain second mixed audio data corresponding to the first optimized listening area and second mixing parameters corresponding to the first optimized listening area. The above mixing processing steps can be completed by the VR music scene production terminal or by the mixing terminal, and are not limited here.

[0174] In the implementation scenario of executing steps B1 to B2, the aforementioned step 204 in which the production terminal generates audio optimization metadata based on the first metadata of the first optimized listening area and the first mixing parameters includes:

[0175] The production terminal generates audio optimization metadata according to the first metadata, the first mixing parameter, and the second mixing parameter.

[0176] When the production terminal performs double mixing, the audio optimization metadata obtained by the production terminal may include first metadata for the first optimized listening area, first mixing parameters corresponding to the first optimized listening area, and second mixing parameters corresponding to the first optimized listening area. After the decoding terminal obtains this audio optimization metadata, the decoding terminal also needs to perform double mixing. This double mixing process can further improve the user's listening experience.

[0177] In some embodiments of the present application, the second mixing parameters include: an identifier of the second mixed audio data, an equalizer parameter, a compressor parameter, and a reverberator parameter.

[0178] The second mixing parameters may include an identifier of the second mixed audio data. The second mixing parameters may also include equalization parameters, for example, the equalization parameters may include an equalization parameter identifier, a gain value, and a Q value of the first frequency band. The second mixing parameters may also include compressor parameters, for example, the compressor parameters may include a compressor identifier, a threshold value, a compression ratio, an attack time, a release time, and a gain compensation value. The second mixing parameters may also include reverberator parameters, for example, the reverberator parameters may include a reverberation type, a reverberation time, a delay time, and a direct-to-reverberation ratio.

[0179] In some embodiments of the present application, the audio processing method that can be executed by the production terminal further includes the following steps:

[0180] C1. The production terminal sends M first audio data, audio basic metadata and audio optimization metadata to the transmission terminal.

[0181] The production terminal sends the M first audio data, basic audio metadata, and audio optimization metadata together to the transmission terminal, or the production terminal may send the M first audio data, basic audio metadata, and audio optimization metadata separately to the transmission terminal. The specific sending method is not limited here. The transmission terminal may then send the M first audio data, basic audio metadata, and audio optimization metadata to the decoding terminal, so that the decoding terminal receives the M first audio data, basic audio metadata, and audio optimization metadata.

[0182] In some embodiments of the present application, the production terminal may also adjust the basic audio metadata. Specifically, the aforementioned step 202 of rendering the M audio data to be processed according to the first optimized listening area and the basic audio metadata includes the following steps:

[0183] D1. The production terminal adjusts the basic audio metadata according to the mixed audio data corresponding to the first optimized listening area to obtain adjusted basic audio metadata, where the mixed audio data is obtained by mixing M first mixed audio data;

[0184] D2. The production terminal renders the M audio data to be processed according to the first optimized listening area and the adjusted basic audio metadata.

[0185] Specifically, in D1, for the first optimized listening area, the production terminal can mix and process the M first mixed audio data, so as to obtain the mixed audio data corresponding to the first optimized listening area. The production terminal adjusts the audio basic metadata according to the mixed audio data corresponding to the first optimized listening area to obtain the adjusted audio basic metadata. For example, the frequency response of one or more audio signals in the audio data or the position, gain and other parameters of the audio basic metadata audio signal can be adjusted, so that the position, gain and other parameters of the audio data can be adjusted to adjust the musical effect of the music signal heard by the end user. In D2, the production terminal uses the first optimized listening area and the adjusted audio basic metadata to render the M audio data to be processed. By adjusting the audio basic metadata, the user's listening effect can be further improved.

[0186] The production terminal can obtain the audio optimization metadata by executing the above steps 201 to 204, and then send the audio optimization metadata to the transmission terminal. The transmission terminal executes the subsequent steps 205 to 207.

[0187] 205. The transmission terminal receives audio optimization metadata, basic audio metadata, and M first audio data. The audio optimization metadata includes: first metadata of a first optimized listening area and first mixing parameters corresponding to the first optimized listening area. M is a positive integer.

[0188] Steps 201 to 204 executed by the production terminal describe in detail the process of generating audio optimization metadata. After generating the audio optimization metadata, the production terminal may also send the audio optimization metadata to the transmission terminal, which then receives the audio optimization metadata from the production terminal. Furthermore, the production terminal may also send basic audio metadata and M first audio data to the transmission terminal, which then receives the basic audio metadata and M first audio data from the production terminal.

[0189] 206. The transmission terminal compresses and encodes the audio optimization metadata, the audio basic metadata, and the M first audio data to obtain an audio code stream.

[0190] After receiving the audio optimization metadata, the basic audio metadata, and the M first audio data, the transmission terminal can use a preset encoding algorithm to compress and encode the audio optimization metadata, the basic audio metadata, and the M first audio data to obtain an audio code stream. The embodiments of this application do not limit the encoding algorithm used.

[0191] 207. The transmission terminal sends an audio code stream.

[0192] The transmission terminal sends the audio code stream using a transmission channel between the transmission terminal and the decoding terminal.

[0193] In some embodiments of the present application, the audio processing method that can be executed by the transmission terminal further includes the following steps:

[0194] E1. The transmission terminal receives video image metadata and video image data from the production terminal. The video image metadata includes: video metadata and image metadata. The video image data includes: video data and image data.

[0195] E2, the transmission terminal compresses and encodes the video image metadata and the video image data to obtain a video image code stream;

[0196] E3. The transmission terminal sends the video image code stream to the decoding terminal.

[0197] The production terminal can also send video image metadata and video image data to the transmission terminal. After receiving the video image metadata and video image data, the transmission terminal can generate a video image code stream that carries the video image metadata and video image data. Therefore, after receiving the video image code stream from the transmission terminal, the decoding terminal can obtain the video image metadata and video image data.

[0198] In some embodiments of the present application, the audio optimization metadata further includes: a second mixing parameter change identifier, wherein:

[0199] The second mixing parameter change identifier is used to indicate whether the second mixing parameter corresponding to the first audio data of the current frame has changed relative to the second mixing parameter corresponding to the first audio data of the previous frame.

[0200] The transmitting terminal may set a second mixing parameter change identifier in the audio optimization metadata. The second mixing parameter change identifier is used to indicate whether the second mixing parameter corresponding to the first optimized listening zone has changed, thereby enabling the decoding terminal to determine whether the second mixing parameter corresponding to the first optimized listening zone has changed based on the second mixing parameter change identifier. For example, when the second mixing parameter of the first audio data of the current frame has changed relative to the second mixing parameter corresponding to the first audio data of the previous frame, the second mixing parameter change identifier is true. The transmitting terminal may also send change information about the second mixing parameter corresponding to the first audio data. The decoding terminal receives the change information about the second mixing parameter corresponding to the first audio data and, based on the change information, obtains the changed second mixing parameter corresponding to the first audio data of the current frame.

[0201] In some embodiments of the present application, the audio optimization metadata further includes: second mixing parameters corresponding to the first optimized listening area.

[0202] When the production terminal performs double mixing, the audio optimization metadata obtained by the production terminal may include first metadata for the first optimized listening area, first mixing parameters corresponding to the first optimized listening area, and second mixing parameters corresponding to the first optimized listening area. After the decoding terminal obtains this audio optimization metadata, the decoding terminal also needs to perform double mixing, which can improve the user's listening experience.

[0203] In some embodiments of the present application, the audio optimization metadata includes: a second mixing parameter corresponding to one of N optimized listening zones, and a difference parameter of N-1 second mixing parameters of N-1 optimized listening zones other than the first optimized listening zone among the N optimized listening zones relative to the second mixing parameters corresponding to the first optimized listening zone;

[0204] The difference parameter is a parameter representing the difference between the N-1 second mixing parameters of the N-1 optimized listening zones excluding the first optimized listening zone and the second mixing parameters corresponding to the first optimized listening zone. The difference parameter is not the N-1 second mixing parameter of the N-1 optimized listening zone. For example, the second mixing parameters corresponding to the first optimized listening zone include parameter 1, parameter 2, and parameter 3. If each of the N-1 second mixing parameters corresponding to the N-1 optimized listening zones includes parameter 1, parameter 2, and parameter 4, then the difference parameter between the N-1 second mixing parameters corresponding to the N-1 optimized listening zone and the second mixing parameters corresponding to the first optimized listening zone includes parameter 4. By carrying the difference parameter in the audio optimization metadata, the data size of the audio optimization metadata can be reduced, thereby improving data transmission efficiency and decoding efficiency.

[0205] In some embodiments of the present application, the second mixing parameters include: an identifier of the first audio data, equalization parameters, compressor parameters, and reverberator parameters.

[0206] The second mixing parameters may include an identifier of the first audio data, such as the first audio data identifier. The second mixing parameters may also include equalization parameters, such as an equalization parameter identifier, a gain value for each frequency band, and a Q value. The second mixing parameters may also include compressor parameters, such as a compressor identifier, a threshold value, a compression ratio, an attack time, a release time, and a gain compensation value. The second mixing parameters may also include reverberator parameters, such as a reverberation type, a reverberation time, a delay time, and a direct-to-reverberation ratio.

[0207] In some embodiments of the present application, the audio optimization metadata further includes: N-1 difference parameters of N-1 first mixing parameters of N-1 optimized listening zones other than the first optimized listening zone among the N optimized listening zones relative to the first mixing parameters corresponding to the first optimized listening zone.

[0208] The difference parameter is a parameter representing the difference between the N-1 first mixing parameters of the N-1 optimized listening zones excluding the first optimized listening zone and the first mixing parameters corresponding to the first optimized listening zone. The difference parameter is not the N-1 first mixing parameter of the N-1 optimized listening zone. For example, the first mixing parameters corresponding to the first optimized listening zone include parameter 1, parameter 2, and parameter 3. If each of the N-1 first mixing parameters corresponding to the N-1 optimized listening zones includes parameter 1, parameter 2, and parameter 4, then the difference parameter between the N-1 first mixing parameters corresponding to the N-1 optimized listening zone and the first mixing parameters corresponding to the first optimized listening zone includes parameter 4. By carrying the difference parameter in the audio optimization metadata, the data size of the audio optimization metadata can be reduced, thereby improving data transmission efficiency and decoding efficiency.

[0209] In some embodiments of the present application, the first mixing parameters include: an identifier of the first audio data, equalization parameters, compressor parameters, and reverberator parameters.

[0210] The first mixing parameters may include an identifier of the first audio data, for example, including identifiers of M first audio data. The first mixing parameters may also include equalization parameters, for example, the equalization parameters may include an equalization parameter identifier, a gain value for each frequency band, and a Q value. The first mixing parameters may also include compressor parameters, for example, the compressor parameters may include a compressor identifier, a threshold value, a compression ratio, an attack time, a release time, and a gain compensation value. The first mixing parameters may also include reverberator parameters, for example, the reverberator parameters may include a reverberation type, a reverberation time, a delay time, and a direct-to-reverberation ratio.

[0211] In some embodiments of the present application, the metadata of the first optimized listening zone includes at least one of the following: a reference coordinate system of the first optimized listening zone, center position coordinates of the first optimized listening zone, and a shape of the first optimized listening zone.

[0212] Specifically, the metadata of the first optimized listening zone may include a reference coordinate system, or the metadata of the first optimized listening zone may not include a reference coordinate system. For example, the first optimized listening zone adopts a default coordinate system.

[0213] The metadata of the first optimized listening area may include descriptive information describing the first optimized listening area, such as the center coordinates of the first optimized listening area, the shape of the first optimized listening area, etc. In the embodiment of the present application, the first optimized listening area may have various shapes, such as a sphere, a cube, a cylinder, or any other arbitrary shape.

[0214] In some embodiments of the present application, the audio optimization metadata further includes: the center position coordinates of a first optimized listening zone among the N optimized listening zones, and the position offsets of the center position coordinates of N-1 optimized listening zones among the N optimized listening zones other than the first optimized listening zone relative to the center position coordinates of the first optimized listening zone.

[0215] This position offset is the offset between the center coordinates of the N-1 optimized listening zones (excluding the first optimized listening zone) and the center coordinates of the first optimized listening zone, rather than the center coordinates of the N-1 optimized listening zones. By including the position offset in the audio optimization metadata, the data size of the audio optimization metadata can be reduced, improving data transmission and decoding efficiency.

[0216] In some embodiments of the present application, the audio optimization metadata further includes: an optimized listening area change identifier, and / or a first mixing parameter change identifier, wherein:

[0217] An optimized listening area change identifier, used to indicate whether the first optimized listening area has changed;

[0218] The first mixing parameter change identifier is used to indicate whether the first mixing parameter corresponding to the first audio data of the current frame has changed relative to the first mixing parameter corresponding to the first audio data of the previous frame.

[0219] Among them, the transmission terminal can set a first mixing parameter change identifier in the audio optimization metadata, and the first mixing parameter change identifier is used to indicate whether the first mixing parameter corresponding to the first audio data of the current frame has changed relative to the first mixing parameter corresponding to the first audio data of the previous frame, so that the decoding terminal determines whether the first mixing parameter corresponding to the first optimized listening area has changed according to the first mixing parameter change identifier. In addition, the transmission terminal can set an optimized listening area change identifier in the audio optimization metadata, and the optimized listening area change identifier is used to indicate whether the optimized listening area determined by the production terminal has changed, so that the decoding terminal determines whether the optimized listening area has changed according to the optimized listening area change identifier. An example is given below, in which the optimized listening area metadata change identifier and the first mixing parameter change identifier are added to the encoded 6DoF audio optimization metadata to improve the transmission efficiency of the 6DoF audio optimization metadata. When the VR music scene is initialized, the initial audio optimization metadata is transmitted. When the VR scene changes and the position and shape information of the optimized listening area changes, the optimized listening area position change identifier is true, and the change information of the optimized listening area is transmitted. When the first mixing parameter of the current frame changes, the first mixing parameter change identifier is true, and the change information of the first mixing parameter is transmitted.

[0220] The transmitting terminal executes the aforementioned steps 205 to 207, and the decoding terminal executes subsequent steps 208 to 211. It is understood that the audio processing process executed by the decoding terminal is similar to the audio processing process executed by the production terminal, and the audio processing process executed by the decoding terminal is described below.

[0221] 208. The decoding terminal decodes the audio bitstream to obtain audio optimization metadata, basic audio metadata, and M decoded audio data, where the audio optimization metadata includes first metadata of a first optimized listening area and first decoded mixing parameters corresponding to the first optimized listening area, and M is a positive integer.

[0222] The audio code stream generation process is described in detail in steps 205 to 207 performed by the transmission terminal. The transmission terminal sends the audio code stream to the decoding terminal, which receives the audio code stream from the transmission terminal to obtain M first decoded audio data, audio optimization metadata, and audio basic metadata. The M decoded audio data correspond to the M unprocessed audio data on the production terminal side. The description of the M decoded audio data, audio optimization metadata, and audio basic metadata is detailed in the previous embodiment and will not be repeated here.

[0223] 209. The decoding terminal renders the M first decoded audio data according to the current location of the user and the basic audio metadata to obtain M rendered audio data.

[0224] Among them, after the decoding terminal obtains M first decoded audio data, audio optimization metadata and audio basic metadata, the decoding terminal renders the M first decoded audio data according to the user's current location and audio basic metadata to obtain M rendered audio data.

[0225] It should be noted that the M rendered audio data in step 209 are audio data obtained by the decoding terminal performing the rendering process, and are different from the M rendered audio data obtained by the production terminal performing the rendering process.

[0226] 210. When the current position of the user is within the first optimized listening area, the decoding terminal performs a first mixing process on the M rendered audio data according to the first decoding mixing parameter to obtain M first mixed audio data.

[0227] Among them, the decoding terminal obtains an optimized listening area that matches the current position from the N optimized listening areas according to the current position of the user, and the optimized listening area that matches the current position is called the first optimized listening area. Since the decoding terminal obtains the audio optimization metadata in step 208, and the audio optimization metadata includes the first decoding mixing parameters corresponding to the first optimized listening area, the first decoding mixing parameters corresponding to the first optimized listening area can be obtained from the audio optimization metadata. The decoding terminal performs a first mixing process on the M rendered audio data according to the first decoding mixing parameters corresponding to the first optimized listening area to obtain M first mixed audio data corresponding to the first optimized listening area. The first decoding mixing parameters correspond to the first mixing parameters on the production terminal side, and the first mixing parameters are used to record the mixing parameters used by the production terminal when performing the first mixing process. The above-mentioned mixing processing steps can be completed by the VR music scene production terminal or by the mixing terminal, which is not limited here.

[0228] 211. The decoding terminal performs mixing processing on the M first mixed audio data to obtain mixed audio data corresponding to the first optimized listening area.

[0229] Among them, after the decoding terminal obtains the M first mixed audio data corresponding to the first optimized listening area, the decoding terminal mixes the M first mixed audio data corresponding to the first optimized listening area to obtain mixed audio data corresponding to the first optimized listening area. Since the first optimized listening area is an optimized listening area that includes the user's current position, the decoding terminal mixes the M first mixed audio data corresponding to the first optimized listening area to obtain the mixed audio data corresponding to the first optimized listening area. The first optimized listening area can be adapted to the actual position of the user, and therefore can provide audio optimization metadata suitable for when the user freely moves to the first optimized listening area, thereby improving the user's listening effect when freely moving.

[0230] It should be noted that the mixed audio data can be directly used for playback, and when the mixed audio data is played, it can improve the user's listening experience.

[0231] In some embodiments of the present application, the audio optimization metadata further includes: second decoded mixing parameters corresponding to the first optimized listening area.

[0232] The second decoded audio mixing parameters correspond to the second audio mixing parameters on the production terminal side, and the second audio mixing parameters are used to record the audio mixing parameters used in the second audio mixing process.

[0233] The audio processing method that can be executed by the decoding terminal also includes the following steps:

[0234] F1. The decoding terminal performs a second mixing process on the mixed audio data according to the second decoding mixing parameter to obtain second mixed audio data corresponding to the first optimized listening area.

[0235] Among them, after obtaining the second decoding mixing parameters, the decoding terminal can also perform a second mixing process on the mixed audio data according to the second decoding mixing parameters to obtain second mixed audio data corresponding to the first optimized listening area. The second mixed audio data can be obtained through the above-mentioned second mixing process, and when the second mixed audio data is played, it can improve the user's listening effect. The above-mentioned mixing processing step can be completed by the VR music scene production terminal or by the mixing terminal, which is not limited here.

[0236] When the production terminal performs double mixing, the audio optimization metadata obtained by the production terminal may include first metadata for the first optimized listening area, first mixing parameters corresponding to the first optimized listening area, and second mixing parameters corresponding to the first optimized listening area. After the decoding terminal obtains this audio optimization metadata, the decoding terminal also needs to perform double mixing. This double mixing process can improve the user's listening experience.

[0237] In some embodiments of the present application, the second decoded mixing parameters include: an identifier of the second mixed audio data, equalization parameters, compressor parameters, and reverberator parameters.

[0238] The second decoded mixing parameters may include an identifier of the second mixed audio data. The second decoded mixing parameters may also include equalization parameters, for example, the equalization parameters may include an equalization parameter identifier, a gain value for each frequency band, and a Q value. The second decoded mixing parameters may also include compressor parameters, for example, the compressor parameters may include a compressor identifier, a threshold value, a compression ratio, an attack time, a release time, and a gain compensation value. The second decoded mixing parameters may also include reverberator parameters, for example, the reverberator parameters may include a reverberation type, a reverberation time, a delay time, and a direct-to-reverberation ratio.

[0239] In some embodiments of the present application, the audio optimization metadata further includes: N-1 difference parameters of N-1 second decoded mixing parameters corresponding to N-1 optimized listening zones excluding the first optimized listening zone among the N optimized listening zones relative to the second decoded mixing parameters corresponding to the first optimized listening zone, where N is a positive integer.

[0240] The difference parameter is a parameter representing the difference between the N-1 second decoded mix parameters of the N-1 optimized listening zones excluding the first optimized listening zone, relative to the second decoded mix parameters corresponding to the first optimized listening zone. The difference parameter is not the N-1 second decoded mix parameters of the N-1 optimized listening zones. For example, the second decoded mix parameters corresponding to the first optimized listening zone include parameter 1, parameter 2, and parameter 3. If each of the N-1 second decoded mix parameters corresponding to the N-1 optimized listening zones includes parameter 1, parameter 2, and parameter 4, then the difference parameter between the N-1 second decoded mix parameters corresponding to the N-1 optimized listening zone and the second decoded mix parameters corresponding to the first optimized listening zone includes parameter 4. By carrying the difference parameter in the audio optimization metadata, the data size of the audio optimization metadata can be reduced, thereby improving data transmission efficiency and decoding efficiency.

[0241] In some embodiments of the present application, the first decoded mixing parameter includes at least one of the following: an identifier of the rendered audio data, an equalizer parameter, a compressor parameter, and a reverberator parameter.

[0242] The first decoded mixing parameters may include an identifier of the rendered audio data, for example, including identifiers of M rendered audio data. The first decoded mixing parameters may also include equalization parameters, for example, the equalization parameters may include an equalization parameter identifier, a gain value for each frequency band, and a Q value. The first decoded mixing parameters may also include compressor parameters, for example, the compressor parameters may include a compressor identifier, a threshold value, a compression ratio, an attack time, a release time, and a gain compensation value. The first decoded mixing parameters may also include reverberator parameters, for example, the reverberator parameters may include a reverberation type, a reverberation time, a delay time, and a direct-to-reverberation ratio.

[0243] In some embodiments of the present application, the audio optimization metadata includes: N-1 difference parameters of N-1 first decoded mixing parameters corresponding to N-1 optimized listening zones other than the first optimized listening zone among N optimized listening zones relative to the first decoded mixing parameters corresponding to the first optimized listening zone, where N is a positive integer.

[0244] The difference parameter is a parameter representing the difference between the N-1 first decoded mix parameters of the N-1 optimized listening zones excluding the first optimized listening zone, relative to the first decoded mix parameters corresponding to the first optimized listening zone. The difference parameter is not the N-1 first decoded mix parameters of the N-1 optimized listening zones. For example, the first decoded mix parameters corresponding to the first optimized listening zone include parameter 1, parameter 2, and parameter 3. If each of the N-1 first decoded mix parameters corresponding to the N-1 optimized listening zones includes parameter 1, parameter 2, and parameter 4, then the difference parameter between the N-1 first decoded mix parameters corresponding to the N-1 optimized listening zone and the first decoded mix parameters corresponding to the first optimized listening zone includes parameter 4. By carrying the difference parameter in the audio optimization metadata, the data size of the audio optimization metadata can be reduced, thereby improving data transmission efficiency and decoding efficiency.

[0245] In some embodiments of the present application, the audio processing method that can be performed by the decoding terminal further includes the following steps:

[0246] G1. The decoding terminal decodes the video image code stream to obtain decoded video image data and video image metadata, where the video image metadata includes: video metadata and image metadata;

[0247] G2. The decoding terminal performs rendering processing on the decoded video image data according to the video image metadata to obtain rendered video image data;

[0248] G3, the decoding terminal establishes a virtual scene based on the rendered video image data;

[0249] G3. The decoding terminal identifies a first optimized listening area in the virtual scene according to the rendered video image data and the audio optimization metadata.

[0250] The transmission terminal can generate a video image code stream based on the video image metadata and video image data. The video image code stream carries the video image metadata and video image data. This allows the decoding terminal to obtain the video image metadata and decoded video image data after receiving the video image code stream from the transmission terminal. The decoding terminal then renders the decoded video image data based on the video image metadata to obtain rendered video image data. The decoding terminal then uses the rendered video image data to establish a virtual scene. Finally, the decoding terminal identifies a first optimized listening area in the virtual scene based on the rendered video image data and the audio optimization metadata. This first optimized listening area is then displayed in the virtual scene on the decoding terminal side, guiding the user to experience music within the optimized listening area and improving the user's listening experience.

[0251] As an example, the decoding terminal identifies a first optimized listening area in the virtual scene based on the rendered video image data and the audio optimization metadata, and adopts a similar processing method to the first optimized listening area. The decoding terminal can also identify N optimized listening areas in the virtual scene. The decoding terminal generates an audio experience route based on the N optimized listening areas identified in the virtual scene, guiding the user to better experience 6DoF music.

[0252] As can be seen from the examples of the aforementioned embodiments, the decoding terminal can receive audio optimization metadata. Since the audio optimization metadata in the embodiment of the present application includes metadata of the first optimized listening area and first mixing parameters corresponding to the first optimized listening area, the first optimized listening area is determined according to the current position of the user. Therefore, the first decoding mixing parameters corresponding to the first optimized listening area can be obtained for mixing. Therefore, audio optimization metadata suitable for when the user freely moves to the first optimized listening area can be provided, thereby improving the listening effect of the user when freely moving.

[0253] To facilitate a better understanding and implementation of the above solutions in the embodiments of the present application, the following examples are given for specific explanation using corresponding application scenarios.

[0254] In the embodiment of the present application, the optimized listening area can be expressed as the optimal listening area. In the audio processing method in the embodiment of the present application, the production terminal selects one or several optimal listening areas in the VR scene, and optimizes the musical effect of the rendered music signal heard by the user in each optimal listening area, thereby improving the music experience of the user in the 6DoF music scene.

[0255] Specifically, a 6DoF virtual music scene production method is executed on the production terminal side. It is assumed that the VR video scene production, audio acquisition, 6DoF audio basic metadata production and other processes have been completed in the 6DoF music scene. In the embodiment of the present application, several optimal listening areas are selected in the VR scene. The optimal listening area should be as consistent as possible with the listening area of ​​interest to the user; for each optimal listening area, a rendering signal of each audio signal at the center of the optimal listening area is generated based on the audio signal and the 6DoF audio basic metadata, and then each rendering signal is mixed to adjust the frequency, dynamics, sound quality, positioning and sound field, etc., and the mixing parameters of each mixing step corresponding to each audio signal are retained; finally, the mixed audio signal is mixed to retain the mixing parameters of each mixing step corresponding to the mixed signal.

[0256] The 6DoF audio optimization metadata generated by the production terminal includes optimal listening area metadata and mixing parameters corresponding to each optimal listening area. The optimal listening area metadata includes information such as the center coordinates and shape of each optimal listening area. Each optimal listening area metadata corresponds to a set of mixing metadata, including the mixing parameters of the mixing parameter signal corresponding to a single audio signal. The production terminal sends the 6DoF audio optimization metadata to the transmission terminal, which generates an audio stream based on the 6DoF audio optimization metadata and sends the audio stream to the decoding terminal.

[0257] Optionally, an optimal listening area position change identifier and a mixing parameter change identifier are added to the 6DoF audio optimization metadata. When the VR music scene is initialized, the initial audio optimization metadata is transmitted. When the VR scene changes and the optimal listening area position and shape information changes, the optimal listening area position change identifier is true, and the optimal listening area change information is transmitted. When the mixing metadata of the current frame changes, the mixing parameter change identifier is true, and the mixing metadata change information is transmitted.

[0258] The decoding terminal may include a video renderer and an audio renderer, and the decoding terminal may execute a 6DoF music scene audio rendering method. Specifically, the video renderer identifies the optimal listening area based on the decoded optimal listening area metadata, and guides the user to reasonably experience 6DoF music; when the user's current position is within a certain optimal listening area, the audio renderer renders the audio signal based on the user's position information, 6DoF audio basic metadata and 6DoF audio optimization metadata, giving the user a better music experience. Among them, the audio optimization metadata has a certain range of action, and this range of action is determined by the shape of the current optimal listening area, and the shape is predetermined according to the music production scene. When the user's current position is outside the optimal listening area, the audio renderer renders the audio signal based on the user's position information and 6DoF audio basic metadata.

[0259] The embodiments of this application are applicable to scene creation, audio metadata data transmission, and user-side audio and video rendering in VR, AR, or MR applications. The terminal in the embodiments of this application is specifically used for VR music software and 3D audio engines that include 6DoF audio. For example, the terminal may include a VR terminal device, chip, and wireless network equipment.

[0260] As shown in Figure 3, a flow chart of stream data processing of a virtual reality streaming service system provided by an embodiment of the present application is provided. The embodiment of the present application is applicable to the 6DoF audio rendering module (audio binaural rendering) in applications such as AR and VR, and is specifically applied to the audio data preprocessing module, audio data encoding module, audio data decoding module and audio rendering module in Figure 3. The end-to-end processing flow of the audio signal is as follows: the audio signal is preprocessed (audio preprocessing) after passing through the VR scene acquisition or production module. The preprocessing operation includes filtering out low-frequency components below 50Hz in the signal, extracting 6DoF audio metadata (including 6DoF audio basic metadata, 6DoF audio optimization metadata, etc.), and then encoding (audio encoding) and packaging (file / segment encapsulation) before sending (delivery) to the decoding end. The decoding end first unpacks (file / segment decapsulation), then decodes (audio decoding), and renders (audio rendering) the decoded audio signal. The rendered signal is mapped to the speaker or the listener's headphone device. The headphone device can be an independent headphone or a headphone on a glasses device.

[0261] Figure 4 shows an end-to-end flow chart of a 6DoF VR music scene provided by an embodiment of this application. This flow chart primarily includes a production terminal, a transmission terminal, and a decoding terminal. The following sections provide examples from different terminal sides.

[0262] The processes executed by the production terminal include: VR video scene and metadata production, audio data and 6DoF audio basic metadata production, optimal listening area selection, mixing processing and mixing parameter extraction, audio optimization metadata production, etc.

[0263] The production terminal includes: VR video and image data module, VR video and image metadata module, VR audio data module, 6DoF audio basic metadata module, video renderer rendering module, optimal listening area acquisition module, audio renderer pre-rendering module, mixing processing module and audio optimization metadata module.

[0264] Among them, the VR video and image data module is used to obtain the video and image data to be transmitted.

[0265] The VR video and image metadata module is used to obtain the video and image metadata of the completed VR scene.

[0266] The VR audio data module is used to obtain audio data to be transmitted. Each audio data can be object audio data (object audio), multi-channel audio data (channel-based audio) or scene audio data (scene-based audio).

[0267] The 6DoF audio basic metadata module is used to obtain the 6DoF audio basic metadata of the VR scene. For example, the metadata types that the 6DoF audio basic metadata may include may be one or more of the metadata types listed in Table 1 above.

[0268] The video renderer rendering module is used to render according to the VR video and image data and the VR video and image metadata to generate a first VR video scene.

[0269] The optimal listening area acquisition module is used to obtain information about the optimal listening area within the VR scene. The optimal listening area is selected based on the rendered VR video scene, and there can be one or N optimal listening areas. The optimal listening area information includes information such as the reference coordinate system, center position coordinates, and shape. Optionally, the optimal listening area information also includes center position coordinates and shape information. The optimal listening area shape can be spherical, cubic, cylindrical, or any other arbitrary shape.

[0270] The audio renderer pre-rendering module is used to perform a first rendering process on M audio data (referred to as first audio data) for each optimal listening area based on the optimal listening area center coordinates, VR audio data, and 6DoF audio basic metadata, to obtain M rendered audio data (referred to as second audio data). The audio signal after the first rendering process can be a single-channel signal or a binaural rendering signal. N optimal listening areas have a total of N*M first-rendered audio signals.

[0271] The mixing processing module is used to perform a first mixing process on each audio signal after the first rendering process, and extract the parameters of each mixing processing step of each audio signal during the mixing process, which are recorded as first mixing parameters. The audio data after the mixing process is recorded as third audio data. The audio signals in the third audio data are mixed to obtain a fourth audio signal, and the fourth audio signal is subjected to a second mixing process, and the parameters of each mixing processing step are retained, which are recorded as second mixing parameters. The mixing processing step can be completed by the VR music scene production terminal or by the mixing terminal.

[0272] The audio optimization metadata module is used to obtain the optimal listening area information, the first mixing parameter and the second mixing parameter, and generate the audio optimization metadata according to a certain data structure.

[0273] The processes executed by the transmission terminal include: compression encoding and transmission of video scenes and metadata, compression encoding and transmission of audio data, compression encoding and transmission of 6DoF audio basic metadata, and compression encoding and transmission of audio optimization metadata.

[0274] The transmission terminal includes: video and image metadata compression and transmission module, video compression transmission module, image compression transmission module, audio optimization metadata compression transmission module, audio compression transmission module and 6DoF audio basic metadata compression transmission module.

[0275] Video and image metadata compression and transmission module, used to compress and encode video and image metadata and transmit the generated code stream;

[0276] Video compression and transmission module, used to compress and encode video data in VR scenes and transmit code streams;

[0277] Image compression and transmission module, used to compress and encode image data in VR scenes and transmit code streams;

[0278] An audio optimization metadata compression and transmission module, used to compress and encode the audio optimization metadata proposed in the embodiment of the present application and transmit the code stream;

[0279] Audio compression and transmission module, used to compress and encode audio data in VR scenes and transmit code streams;

[0280] The 6DoF audio basic metadata compression and transmission module is used to compress and encode the 6DoF audio basic metadata and transmit the code stream.

[0281] The processes executed by the decoding terminal (i.e., the user end) include: obtaining the user's 6DoF position information, 6DoF video rendering, and 6DoF audio rendering, among other processes. Among them, the embodiment of the present application uses the decoded audio optimization metadata for 6DoF video rendering and 6DoF audio rendering.

[0282] The decoding terminal includes: audio and video decoders, video renderer and audio renderer.

[0283] Audio and video decoders, used to decode the bitstream to obtain decoded VR video and image data, video and image metadata, audio data, 6DoF audio basic metadata, and audio optimization metadata;

[0284] The video renderer is used to render the VR video scene based on the decoded video and image data, video and image metadata, and user position information.

[0285] Optionally, the video renderer identifies the optimal listening area based on the optimal listening area information in the decoded audio optimization metadata, and identifies a recommended 6DoF music experience route to guide the user to better experience 6DoF music. The experience route can be a line connecting the optimal listening areas, etc., which is not limited in this embodiment of the application.

[0286] Similar to the pre-rendering and mixing process of the production terminal, the audio renderer is used to determine whether the user is within the optimal listening area based on the user's location information and the optimal listening area information in the audio optimization metadata;

[0287] If the user's current position is within a certain optimal listening area, the audio renderer is used to render each audio signal based on the 6DoF audio basic metadata and the user's position information to obtain a rendered signal, and then mix the rendered signal according to the mixing parameters corresponding to each decoded audio signal; after all audio signals are mixed, all mixed audios are mixed, and finally mixing is performed based on the mixing parameters of the mixed signals, and the processed audio signals are sent to the user's headphones or other audio devices.

[0288] If the user is not in the optimal listening area, the audio renderer is used to render each audio signal based on the 6DoF audio basic metadata and user location information, directly mix all rendered audios, and generate the final binaural signal for playback.

[0289] Next, the audio processing method of the embodiment of the present application is described in detail with two specific embodiments.

[0290] Example 1

[0291] Figure 5 shows a schematic diagram of a 6DoF-supported VR concert scene provided by an embodiment of the present application. Taking a typical 6DoF-supported VR concert scene as an example, the technical solutions of the present embodiment are described in detail. This concert scene consists of two parts: a stage area and an audience area. There are four types of sound sources: vocals, violins, cellos, and drums. Assuming all sound sources are stationary, the positions of the sound sources within the VR scene are shown in Figure 5.

[0292] In this embodiment, the end-to-end process of the VR concert scene from the production terminal, the transmission terminal to the decoding terminal, the specific process of embodiment 1 mainly includes:

[0293] Among them, steps S01-S05 are the production terminal process of the VR music scene, step S06 is the transmission terminal process of the VR music scene, and steps S07-S08 are the decoding terminal process of the VR music scene.

[0294] S01. The production terminal obtains VR video and image data, VR video and image metadata, VR scene audio data, and 6DoF audio basic metadata.

[0295] Among them, the above-mentioned VR video and image data, VR video and image metadata, VR scene audio data, and 6DoF audio basic metadata are pre-produced by the VR music scene.

[0296] S02. The production terminal obtains optimal listening area metadata.

[0297] The production terminal renders the VR scene based on the VR video and image data and VR video and image metadata. It then selects the optimal listening area within the VR scene and records the center coordinates and shape information of each optimal listening area. There can be N optimal listening areas, and the optimal listening area selected by the production terminal should match the user's listening area of ​​interest. Finally, the production terminal converts the optimal listening area information into optimal listening area metadata according to a specific data structure.

[0298] The optimal listening area metadata includes the reference coordinate system, center position coordinates, and shape information of the optimal listening area. An example of a data structure of the optimal listening area metadata is as follows:

[0299]

[0300] In addition, the shape of each optimal listening area can be spherical, cylindrical, or any shape composed of a triangular mesh. The optimal listening area metadata includes the reference coordinate system and center position coordinates of the optimal listening area. The shape information uses the default shape information on both the production terminal and the decoding terminal. Another example of the data structure of the optimal listening area metadata is as follows:

[0301]

[0302]

[0303] The data and data structure included in the optimal listening area metadata are not limited to the above two categories. For example, the center position information of the optimal listening area 2-N may be position information relative to the optimal listening area 1.

[0304] S03. For each optimal listening area, the production terminal performs a first rendering process on M audio data (recorded as first audio data) one by one according to the optimal listening area metadata, VR audio data and 6DoF audio basic metadata to obtain M rendered audio data (recorded as second audio data). The audio signal after the first rendering process can be a single-channel signal or a binaural rendering signal. There are N*M audio signals after the first rendering process in the N optimal listening areas, which are recorded as second audio data. Each first audio signal can be an object signal, a multi-channel audio signal or a scene audio signal.

[0305] S04. The production terminal obtains each optimal listening area, performs mixing processing on each rendered audio signal, obtains first mixing parameters, and performs second mixing processing on the final mixed signal.

[0306] The production terminal performs a first mixing process on each second audio signal, extracts parameters of each mixing process step of each audio signal during the mixing process, and records them as first mixing parameters. The audio data after the mixing process is recorded as third audio data.

[0307] Optionally, the audio signals in the third audio data are mixed to obtain a fourth audio signal, the fourth audio signal is subjected to a second mixing process, and the parameters of each mixing process step are retained and recorded as second mixing parameters. The two mixing process steps can be completed by the VR music scene production terminal.

[0308] The first and second mixing parameters include the audio signal identification number, equalization parameters, compressor parameters, and reverberator parameters. The equalization parameters include frequency band, gain value, and Q value. The Q value is a parameter of the equalization filter that represents the quality factor of the equalization filter and can be used to describe the bandwidth of the equalization filter. The compressor parameters include threshold, compression ratio, attack time, release time, and gain compensation value. The reverberator parameters include reverberation time, delay time, and direct-to-reverberation ratio.

[0309] Optionally, mixing parameters of important mixing steps may be retained according to specific application scenarios. The first mixing parameter and the second mixing parameter may include different mixing parameter types.

[0310] S05: The production terminal generates 6DoF audio optimization metadata using a specific data structure, combining the optimal listening area metadata and the mixing parameters corresponding to each optimal listening area. The optimal listening area metadata and the mixing parameters from step S04 can be stored and transmitted as independent data structures. The data structure of the optimal listening area metadata is as shown in step S02. An example data structure of the mixing parameters is as follows:

[0311]

[0312]

[0313]

[0314] It should be noted that the identifier of the optimal listening area N has the same data structure as the mixing parameter data of the optimal listening area 1.

[0315] In the above data structure, the types of mixing parameters in the optimal listening zones 1-N are completely consistent.

[0316] Optionally, the parameter type stored in the optimal listening area 1 is the same as the above data structure, and the mixing parameters of the optimal listening areas 2-N are difference parameters from the mixing parameters of the optimal listening area 1, thereby reducing the number of parameters in the 6DoF audio optimization metadata.

[0317] Optionally, the data structure of the optimal listening area metadata and the data structure of the mixing parameters are integrated into the same data structure, thereby reducing the number of parameters in the 6DoF audio optimization metadata.

[0318] S06. At the transmission terminal, in addition to encoding and transmitting VR video and image data, VR video and image metadata, audio data, and 6DoF audio basic metadata, the above-mentioned 6DoF audio optimization metadata also needs to be encoded and transmitted.

[0319] Optionally, an optimal listening area metadata change identifier and a mixing parameter change identifier are added to the encoded 6DoF audio optimization metadata to improve the transmission efficiency of the 6DoF audio optimization metadata. When the VR music scene is initialized, the initial audio optimization metadata is transmitted. When the VR scene changes and the optimal listening area position and shape information changes, the optimal listening area position change identifier is true, and the optimal listening area change information is transmitted. When the mixing metadata of the current frame changes, the mixing parameter change identifier is true, and the mixing metadata change information is transmitted.

[0320] S07. At the decoding terminal, the user's VR headset obtains the user's 6DoF position information. The video renderer renders the video based on the decoded VR video and image data, VR video and image metadata, and the user's position information. At the same time, the optimal listening area is identified based on the decoded optimal listening area metadata. Optionally, a recommended 6DoF music experience route can also be identified to guide the user to better experience 6DoF music.

[0321] S08. At the decoding terminal, the user's VR headset, etc., obtains the user's 6DoF position information. The audio decoder decodes the audio stream to obtain first decoded audio data, decoded 6DoF audio basic metadata, and decoded 6DoF audio optimization metadata. The audio renderer determines whether the user is within the optimal listening zone based on the user's position information and the decoded optimal listening zone metadata.

[0322] If the user's current position is within the optimal listening area, each first decoding end audio signal is rendered separately based on the 6DoF audio basic metadata and the user position information to obtain M rendered audio signals (recorded as second decoding end audio signals), and then each second decoding end audio signal is mixed according to the decoded first mixing parameters to obtain M third decoding end audio signals after processing; based on the 6DoF audio basic metadata, the M third decoding end audio signals are mixed, and if there are decoded second mixing parameters, the mixed signal is subjected to second mixing processing to obtain the final music signal. When the user's current position is within the optimal listening area, the user can be provided with the best immersive music experience.

[0323] If the user's current position is outside the optimal listening area, the audio renderer renders each first-decoding-end audio signal separately based on the 6DoF audio basic metadata and user position information, and obtains M rendered audio signals (recorded as second-decoding-end audio signals), and mixes the M second-decoding-end audio signals to obtain the final music signal.

[0324] Optionally, a transition distance is set for each optimal listening area, and a smoothing algorithm is used to ensure that the music signal heard can transition naturally when the user's current position moves freely in and out of the optimal listening area. This embodiment of the application does not limit the smoothing algorithm.

[0325] For example, an area a certain distance from the edge of the optimal listening area (i.e., the transition distance) can be set as a transition zone. Within the transition zone, the various parameters of the 6DoF audio optimization metadata gradually become zero, so that the music effect heard by the user can transition naturally.

[0326] Example 2

[0327] As shown in FIG6 , it is another end-to-end flow chart of a 6DoF VR audio scenario provided in an embodiment of the present application.

[0328] The main difference between Example 2 and Example 1 is that the production process for the 6DoF music scene is different. During the mixing process in step S04 of Example 1, the mixing process is performed while maintaining the already produced VR video metadata, VR video data, audio data, and 6DoF audio basic metadata unchanged, and the mixing metadata is extracted.

[0329] In the audio mixing process of step S04, the prepared VR video metadata, VR video data, audio data, and 6DoF audio basic metadata may be adjusted and optimized, and audio mixing metadata may be extracted. For example, during the audio mixing process of step S04, the VR scene production terminal may adjust the audio frequency response and gain, the position of the object sound source, or the acoustic parameters of the room.

[0330] Compared with Example 1, the mixing metadata in Example 2 will be less than that in Example 1, and the 3D immersive music effect after mixing may also be better than that in Example 1. The production process shown in Example 1 does not modify the 6DoF audio basic metadata, only newly added audio optimization metadata is generated; while Example 2 adjusts the 6DoF audio basic metadata. For example, if the sound of a musical instrument in front of you sounds inconsistent, the position of the instrument may be adjusted in the VR video scene, and the sound source position information corresponding to the instrument in the 6DoF audio basic metadata may be modified.

[0331] Optionally, by adjusting the acoustic parameters of the room in the 6DoF audio basic metadata, etc., the reverberation effect of the music signal heard by the user is adjusted, and the first mixing parameters and the second mixing parameters in Example 1 may not include reverberator parameters.

[0332] Optionally, the effect of the music signal heard by the end user is adjusted by adjusting the frequency response of one or more audio signals in the audio data in Figure 6 or the position, gain and other parameters of the 6DoF audio basic metadata audio signal. The first mixing parameters in Example 1 may not include the equalization parameters corresponding to these signals.

[0333] As can be seen from the examples in the aforementioned embodiments, the embodiments of the present application propose a method for producing, transmitting, and rendering a 6DoF virtual music scene. The decoding terminal can guide users to experience the 6DoF music scene more reasonably, effectively conveying their personal aesthetic appreciation of music to users, allowing users to hear a more perfect 3D immersive music effect within each optimal listening zone, and having a different music experience in each optimal listening zone. The embodiments of the present application also propose that an optimal listening zone position change identifier and a mixing parameter change identifier can be added to the 6DoF audio optimization metadata, which can effectively improve the transmission efficiency of the 6DoF audio optimization metadata.

[0334] It should be noted that for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that this application is not limited by the order of the actions described, because according to this application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by this application.

[0335] In order to better implement the above-mentioned solutions of the embodiments of the present application, relevant devices for implementing the above-mentioned solutions are also provided below.

[0336] 7 , a decoding terminal 700 provided in an embodiment of the present application may include: a decoding module 701, a rendering module 702, a mixing module 703, and a mixing module 704, wherein:

[0337] a decoding module, configured to decode the audio bitstream to obtain audio optimization metadata, basic audio metadata, and M decoded audio data, wherein the audio optimization metadata includes first metadata of a first optimized listening area and first decoded mixing parameters corresponding to the first optimized listening area, and M is a positive integer;

[0338] a rendering module, configured to render the M first decoded audio data according to the current location of the user and the basic audio metadata to obtain M rendered audio data;

[0339] a mixing module, configured to perform a first mixing process on the M rendered audio data according to the first decoded mixing parameter to obtain the M first mixed audio data when the current position is within the first optimized listening zone;

[0340] A mixing module is configured to perform mixing processing on the M first mixed audio data to obtain mixed audio data corresponding to the first optimized listening area.

[0341] In the above-mentioned embodiment of the present application, since the audio optimization metadata in the embodiment of the present application includes the first metadata of the first optimized listening zone and the first mixing parameters corresponding to the first optimized listening zone, the first optimized listening zone is determined according to the current position of the user, and therefore the first decoded mixing parameters corresponding to the first optimized listening zone can be obtained for mixing. Therefore, audio optimization metadata suitable for when the user freely moves to the first optimized listening zone can be provided, thereby improving the listening experience of the user when freely moving.

[0342] As shown in FIG8 , a transmission terminal 800 provided in an embodiment of the present application may include: a receiving module 801 , an encoding module 802 , and a sending module 803 , wherein:

[0343] a receiving module, configured to receive audio optimization metadata, basic audio metadata, and the M first audio data, wherein the audio optimization metadata includes: first metadata for a first optimized listening area and first mixing parameters corresponding to the first optimized listening area, and M is a positive integer;

[0344] an encoding module, configured to compress and encode the audio optimization metadata, the audio basic metadata, and the M first audio data to obtain an audio code stream;

[0345] The sending module is used to send the audio code stream.

[0346] In the above-mentioned embodiment of the present application, audio optimization metadata is first received from the production terminal, an audio code stream is generated based on the audio optimization metadata, and the audio code stream is sent to the decoding terminal. The decoding terminal can obtain the audio optimization metadata through the audio code stream. The audio optimization metadata includes first metadata of the first optimized listening zone and first mixing parameters corresponding to the first optimized listening zone. Therefore, audio optimization metadata suitable for when the user freely moves to the first optimized listening zone can be provided, thereby improving the user's listening experience when freely moving.

[0347] As shown in FIG9 , a production terminal 900 provided in an embodiment of the present application may include: an acquisition module 901 , a rendering module 902 , a mixing module 903 and a generation module 904 , wherein:

[0348] an acquisition module, configured to acquire basic audio metadata and metadata of N optimized listening areas, where N is a positive integer and the N optimized listening areas include a first optimized listening area;

[0349] a rendering module, configured to perform rendering processing on M pieces of to-be-processed audio data according to the first optimized listening area and the basic audio metadata, so as to obtain M pieces of rendered audio data corresponding to the first optimized listening area, where M is a positive integer;

[0350] a mixing module, configured to perform a first mixing process on the M rendered audio data to obtain M first mixed audio data and first mixing parameters corresponding to the first optimized listening area;

[0351] A generating module is configured to generate audio optimization metadata according to first metadata of the first optimized listening area and the first mixing parameters, wherein the audio optimization metadata includes: the first metadata and the first mixing parameters.

[0352] In the above-mentioned embodiment of the present application, metadata of N optimized listening zones can be obtained, and the N optimized listening zones include the first optimized listening zone. Therefore, the M first audio data can be rendered and mixed for the first optimized listening zone, and finally, audio optimization metadata can be generated. The audio optimization metadata includes the first metadata of the first optimized listening zone and the first mixing parameters corresponding to the first optimized listening zone. Therefore, audio optimization metadata suitable for when the user freely moves to the first optimized listening zone can be provided, thereby improving the listening effect of the user when freely moving.

[0353] It should be noted that the information interaction, execution process, etc. between the modules / units of the above-mentioned device are based on the same concept as the method embodiment of the present application, and the technical effects they bring are the same as those of the method embodiment of the present application. For specific contents, please refer to the description in the method embodiment shown above in the present application, and no further details will be given here.

[0354] An embodiment of the present application further provides a computer storage medium, wherein the computer storage medium stores a program, and the program executes some or all of the steps recorded in the above method embodiment.

[0355] Next, another decoding terminal provided by an embodiment of the present application is introduced. Referring to FIG10 , the decoding terminal 1000 includes:

[0356] Receiver 1001, transmitter 1002, processor 1003, and memory 1004 (wherein the number of processors 1003 in the decoding terminal 1000 may be one or more, and FIG10 uses one processor as an example). In some embodiments of the present application, the receiver 1001, transmitter 1002, processor 1003, and memory 1004 may be connected via a bus or other means, wherein FIG10 uses a bus connection as an example.

[0357] The memory 1004 may include a read-only memory and a random access memory, and provides instructions and data to the processor 1003. A portion of the memory 1004 may also include non-volatile random access memory (NVRAM). The memory 1004 stores an operating system and operating instructions, executable modules, or data structures, or a subset or an extension thereof. The operating instructions may include various operating instructions for implementing various operations. The operating system may include various system programs for implementing various basic services and processing hardware-based tasks.

[0358] Processor 1003 controls the operation of the decoding terminal and may also be referred to as a central processing unit (CPU). In specific applications, the various components of the decoding terminal are coupled together via a bus system. In addition to a data bus, the bus system may also include a power bus, a control bus, and a status signal bus. However, for clarity, all of these buses are collectively referred to as a bus system in the figure.

[0359] The methods disclosed in the above embodiments of the present application can be applied to or implemented by processor 1003. Processor 1003 can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by hardware integrated logic circuits or software instructions in processor 1003. The above processor 1003 can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The methods, steps, and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of the present application can be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software modules can be located in storage media such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other well-known storage media in the art. The storage medium is located in the memory 1004 , and the processor 1003 reads the information in the memory 1004 and completes the steps of the above method in combination with its hardware.

[0360] The receiver 1001 can be used to receive input digital or character information, and generate signal input related to the relevant settings and function control of the decoding terminal. The transmitter 1002 can include a display device such as a display screen, and the transmitter 1002 can be used to output digital or character information through an external interface.

[0361] In the embodiment of the present application, the processor 1003 is configured to execute the method performed by the decoding terminal shown in FIG. 2 of the aforementioned embodiment.

[0362] Next, another transmission terminal provided in an embodiment of the present application is introduced. Referring to FIG11 , the transmission terminal 1100 includes:

[0363] Receiver 1101, transmitter 1102, processor 1103, and memory 1104 (wherein the number of processors 1103 in the transmission terminal 1100 may be one or more, and FIG11 uses one processor as an example). In some embodiments of the present application, the receiver 1101, transmitter 1102, processor 1103, and memory 1104 may be connected via a bus or other means, wherein FIG11 uses a bus connection as an example.

[0364] Memory 1104 may include read-only memory and random access memory, and provides instructions and data to processor 1103. A portion of memory 1104 may also include non-volatile random access memory (NVRAM). Memory 1104 stores an operating system and operating instructions, executable modules, or data structures, or subsets or extensions thereof. The operating instructions may include various operating instructions for implementing various operations. The operating system may include various system programs for implementing various basic services and processing hardware-based tasks.

[0365] Processor 1103 controls the operation of the transmission terminal and may also be referred to as a central processing unit (CPU). In specific applications, the various components of the transmission terminal are coupled together via a bus system. In addition to a data bus, the bus system may also include a power bus, a control bus, and a status signal bus. However, for clarity, all bus systems are referred to as a bus system in the figure.

[0366] The methods disclosed in the above embodiments of the present application can be applied to or implemented by processor 1103. Processor 1103 can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by hardware integrated logic circuits or software instructions in processor 1103. The above processor 1103 can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The methods, steps, and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of the present application can be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software modules can be located in storage media well-known in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in the memory 1104 , and the processor 1103 reads the information in the memory 1104 and completes the steps of the above method in combination with its hardware.

[0367] The receiver 1101 can be used to receive input digital or character information, and generate signal input related to the relevant settings and function control of the transmission terminal. The transmitter 1102 may include a display device such as a display screen, and the transmitter 1102 can be used to output digital or character information through an external interface.

[0368] In the embodiment of the present application, the processor 1103 is configured to execute the method performed by the transmission terminal shown in FIG. 2 of the aforementioned embodiment.

[0369] Next, another production terminal provided in an embodiment of the present application is introduced. Referring to FIG. 12 , the production terminal 1200 includes:

[0370] Receiver 1201, transmitter 1202, processor 1203, and memory 1204 (wherein the number of processors 1203 in the production terminal 1200 may be one or more, and FIG12 uses one processor as an example). In some embodiments of the present application, the receiver 1201, transmitter 1202, processor 1203, and memory 1204 may be connected via a bus or other means, wherein FIG12 uses a bus connection as an example.

[0371] Memory 1204 may include read-only memory and random access memory, and provides instructions and data to processor 1203. A portion of memory 1204 may also include NVRAM. Memory 1204 stores an operating system and operating instructions, executable modules, or data structures, or subsets or extensions thereof. The operating instructions may include various operating instructions for implementing various operations. The operating system may include various system programs for implementing various basic services and processing hardware-based tasks.

[0372] Processor 1203 controls the operation of the production terminal and may also be referred to as a CPU. In specific applications, the various components of the production terminal are coupled together via a bus system. In addition to a data bus, the bus system may also include a power bus, a control bus, and a status signal bus. However, for clarity, all of these buses are collectively referred to as a bus system in the figure.

[0373] The methods disclosed in the above embodiments of the present application can be applied to or implemented by processor 1203. Processor 1203 can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by hardware integrated logic circuits in processor 1203 or by software instructions. Processor 1203 can be a general-purpose processor, DSP, ASIC, FPGA, or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component. It can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of this application. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software modules can be located in a storage medium well-known in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. This storage medium is located in memory 1204. Processor 1203 reads information from memory 1204 and, in conjunction with its hardware, completes the steps of the above method.

[0374] In the embodiment of the present application, the processor 1203 is used to execute the audio processing method performed by the production terminal shown in Figure 2 of the aforementioned embodiment.

[0375] In another possible design, when the decoding terminal, transmission terminal, or production terminal is a chip within the terminal, the chip includes: a processing unit and a communication unit. The processing unit may be, for example, a processor, and the communication unit may be, for example, an input / output interface, a pin, or a circuit. The processing unit may execute computer-executable instructions stored in the storage unit to enable the chip within the terminal to perform the audio processing method of any one of the first to third aspects described above. Optionally, the storage unit is a storage unit within the chip, such as a register, a cache, etc. The storage unit may also be a storage unit within the terminal located outside the chip, such as a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM), etc.

[0376] The processor mentioned in any of the above places may be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits for controlling the execution of the program of the above-mentioned first to third aspects.

[0377] It should also be noted that the device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment. In addition, in the drawings of the device embodiments provided in this application, the connection relationship between the modules indicates that there is a communication connection between them, which can be specifically implemented as one or more communication buses or signal lines.

[0378] Through the description of the above embodiments, it is clear to those skilled in the art that the present application can be implemented by means of software plus necessary general-purpose hardware, and of course it can also be implemented by means of dedicated hardware including application-specific integrated circuits, dedicated CPUs, dedicated memories, dedicated components, etc. In general, all functions performed by computer programs can be easily implemented with corresponding hardware, and the specific hardware structures used to implement the same function can also be various, such as analog circuits, digital circuits, or dedicated circuits, etc. However, for the present application, software program implementation is a better implementation method in most cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer's floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0379] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0380] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, a computer, a server, or a data center by wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) mode to another website, a computer, a server, or a data center. The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a server or a data center that includes one or more available media integrations. The available medium can be a magnetic medium, (such as a floppy disk, a hard disk, a magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid-state drive (SSD)).

Claims

1. An audio processing method, characterized in that: include: Decoding the audio bitstream to obtain audio optimization metadata, basic audio metadata, and M decoded audio data, wherein the audio optimization metadata includes first metadata of a first optimized listening area and first decoded mixing parameters corresponding to the first optimized listening area, and M is a positive integer; Rendering the M first decoded audio data according to the current location of the user and the basic audio metadata to obtain M rendered audio data; When the current position is in the first optimized listening area, performing a first mixing process on the M rendered audio data according to the first decoded mixing parameter to obtain M first mixed audio data; The M first mixed audio data are mixed to obtain mixed audio data corresponding to the first optimized listening area.

2. The method according to claim 1, characterized in that The audio optimization metadata further includes: second decoded mixing parameters corresponding to the first optimized listening area; The method further comprises: A second mixing process is performed on the mixed audio data according to the second decoded mixing parameters to obtain second mixed audio data corresponding to the first optimized listening area.

3. The method according to claim 2, characterized in that The second decoded mixing parameters include at least one of the following: an identifier of the second mixed audio data, an equalizer parameter, a compressor parameter, and a reverberator parameter.

4. The method according to claim 2 or 3, characterized in that The audio optimization metadata further includes: N-1 difference parameters of N-1 second decoded mixing parameters corresponding to N-1 optimized listening zones excluding the first optimized listening zone among the N optimized listening zones relative to the second decoded mixing parameters corresponding to the first optimized listening zone, where N is a positive integer.

5. The method according to any one of claims 1 to 4, characterized in that The first decoded mixing parameter includes at least one of the following: an identifier of the rendered audio data, an equalizer parameter, a compressor parameter, and a reverberator parameter.

6. The method according to any one of claims 1 to 5, characterized in that The method further comprises: Decoding the video image code stream to obtain decoded video image data and video image metadata, wherein the video image metadata includes: video metadata and image metadata; performing rendering processing on the decoded video image data according to the video image metadata to obtain rendered video image data; establishing a virtual scene according to the rendered video image data; The first optimized listening zone is identified in the virtual scene according to the rendered video image data and the audio optimization metadata.

7. The method according to any one of claims 1 to 6, characterized in that The first metadata includes at least one of the following: a reference coordinate system of the first optimized listening zone, center position coordinates of the first optimized listening zone, and a shape of the first optimized listening zone.

8. The method according to any one of claims 1 to 7, characterized in that The audio optimization metadata includes: N-1 difference parameters of N-1 first decoded mixing parameters corresponding to N-1 optimized listening zones other than the first optimized listening zone among N optimized listening zones relative to the first decoded mixing parameters corresponding to the first optimized listening zone, where N is a positive integer.

9. The method according to any one of claims 1 to 8, characterized in that The audio optimization metadata further includes: center position coordinates of a first optimized listening zone among the N optimized listening zones, and position offsets of center position coordinates of N-1 optimized listening zones among the N optimized listening zones excluding the first optimized listening zone relative to the center position coordinates of the first optimized listening zone, where N is a positive integer.

10. An audio processing method, characterized in that: include: receiving audio optimization metadata, audio basic metadata, and M first audio data, wherein the audio optimization metadata includes: first metadata of a first optimized listening area and first mixing parameters corresponding to the first optimized listening area, and M is a positive integer; compressing and encoding the audio optimization metadata, the audio basic metadata, and the M first audio data to obtain an audio code stream; Send the audio code stream.

11. The method according to claim 10, characterized in that The audio optimization metadata further includes: a second mixing parameter change identifier, wherein, The second mixing parameter change identifier is used to indicate whether the second mixing parameter corresponding to the first audio data of the current frame has changed relative to the second mixing parameter corresponding to the first audio data of the previous frame.

12. The method according to claim 10 or 11, characterized in that The audio optimization metadata further includes: second mixing parameters corresponding to the first optimized listening area.

13. The method according to any one of claims 10 to 12, characterized in that The audio optimization metadata further includes: second mixing parameters corresponding to a first optimized listening zone among the N optimized listening zones, and N-1 difference parameters of N-1 second mixing parameters corresponding to N-1 optimized listening zones among the N optimized listening zones excluding the first optimized listening zone relative to the second mixing parameters corresponding to the first optimized listening zone, where N is a positive integer.

14. The method according to any one of claims 11 to 13, characterized in that The second mixing parameters include at least one of the following: an identifier of the first audio data, an equalizer parameter, a compressor parameter, and a reverberator parameter.

15. The method according to any one of claims 10 to 14, characterized in that The audio optimization metadata further includes: N-1 difference parameters of N-1 first mixing parameters corresponding to N-1 optimized listening zones other than the first optimized listening zone among the N optimized listening zones relative to the first mixing parameters corresponding to the first optimized listening zone.

16. The method according to any one of claims 10 to 15, characterized in that The first mixing parameters include at least one of the following: an identifier of the first audio data, an equalizer parameter, a compressor parameter, and a reverberator parameter.

17. The method according to any one of claims 10 to 16, characterized in that The first metadata of the first optimized listening zone includes at least one of the following: a reference coordinate system of the first optimized listening zone, center position coordinates of the first optimized listening zone, and a shape of the first optimized listening zone.

18. The method according to any one of claims 10 to 16, characterized in that The audio optimization metadata further includes: center position coordinates of a first optimized listening zone among the N optimized listening zones, and position offsets of center position coordinates of N-1 optimized listening zones among the N optimized listening zones excluding the first optimized listening zone relative to the center position coordinates of the first optimized listening zone, where N is a positive integer.

19. The method according to any one of claims 10 to 18, characterized in that The audio optimization metadata further includes: an optimized listening area change identifier, and / or a first mixing parameter change identifier, wherein: The optimized listening area change identifier is used to indicate whether the first optimized listening area has changed; The first mixing parameter change identifier is used to indicate whether the first mixing parameter corresponding to the first audio data of the current frame has changed relative to the first mixing parameter corresponding to the first audio data of the previous frame.

20. An audio processing method, characterized in that: include: Acquire basic audio metadata and metadata of N optimized listening areas, where N is a positive integer and the N optimized listening areas include a first optimized listening area; Rendering the M audio data to be processed according to the first optimized listening area and the basic audio metadata to obtain M rendered audio data corresponding to the first optimized listening area, where M is a positive integer; Performing a first mixing process on the M rendered audio data to obtain M first mixed audio data and first mixing parameters corresponding to the first optimized listening area; Audio optimization metadata is generated according to the first metadata of the first optimized listening area and the first mixing parameters, where the audio optimization metadata includes: the first metadata and the first mixing parameters.

21. The method according to claim 20, characterized in that The method further includes: performing a mixing process on the M first mixed audio data to obtain mixed audio data corresponding to the first optimized listening area; performing a second mixing process on the mixed audio data to obtain second mixed audio data corresponding to the first optimized listening area and second mixing parameters corresponding to the first optimized listening area; Generating audio optimization metadata according to the first metadata of the first optimized listening area and the first mixing parameter includes: The audio optimization metadata is generated according to the first metadata, the first mixing parameter, and the second mixing parameter.

22. The method according to claim 21, characterized in that The second mixing parameters include: an identifier of the second mixed audio data, an equalizer parameter, a compressor parameter, and a reverberator parameter.

23. The method according to any one of claims 20 to 22, characterized in that The first mixing parameters include: an identifier of the first mixed audio data, equalizer parameters, compressor parameters, and reverberator parameters.

24. The method according to any one of claims 20 to 23, characterized in that The obtaining of metadata of N optimized listening areas includes: Acquire video image metadata and video image data, wherein the video image metadata includes: video metadata and image metadata, and the video image data includes: video data and image data; Rendering the video image data according to the video image metadata to obtain video scene information; The metadata of the N optimized listening areas are obtained according to the video scene information.

25. The method according to any one of claims 20 to 24, characterized in that The rendering processing of the M audio data to be processed according to the first optimized listening area and the basic audio metadata includes: adjusting the audio basic metadata according to the mixed audio data corresponding to the first optimized listening area to obtain adjusted audio basic metadata, wherein the mixed audio data is obtained by mixing the M first mixed audio data; The rendering process is performed on the M to-be-processed audio data according to the first optimized listening area and the adjusted basic audio metadata.

26. The method according to any one of claims 20 to 25, characterized in that The first metadata of the first optimized listening zone includes at least one of the following: a reference coordinate system of the first optimized listening zone, coordinates of a center position of the first optimized listening zone, and a shape of the first optimized listening zone.

27. A decoding terminal, characterized in that: include: a decoding module, configured to decode the audio bitstream to obtain audio optimization metadata, basic audio metadata, and M decoded audio data, wherein the audio optimization metadata includes first metadata of a first optimized listening area and first decoded mixing parameters corresponding to the first optimized listening area, and M is a positive integer; a rendering module, configured to render the M first decoded audio data according to the current location of the user and the basic audio metadata to obtain M rendered audio data; a mixing module, configured to perform a first mixing process on the M rendered audio data according to the first decoded mixing parameter to obtain the M first mixed audio data when the current position is within the first optimized listening zone; A mixing module is configured to perform mixing processing on the M first mixed audio data to obtain mixed audio data corresponding to the first optimized listening area.

28. A transmission terminal, characterized in that: include: a receiving module, configured to receive audio optimization metadata, basic audio metadata, and the M first audio data, wherein the audio optimization metadata includes: first metadata for a first optimized listening area and first mixing parameters corresponding to the first optimized listening area, and M is a positive integer; an encoding module, configured to compress and encode the audio optimization metadata, the audio basic metadata, and the M first audio data to obtain an audio code stream; The sending module is used to send the audio code stream.

29. A production terminal, characterized in that: include: an acquisition module, configured to acquire basic audio metadata and metadata of N optimized listening areas, where N is a positive integer and the N optimized listening areas include a first optimized listening area; a rendering module, configured to perform rendering processing on M pieces of to-be-processed audio data according to the first optimized listening area and the basic audio metadata, so as to obtain M pieces of rendered audio data corresponding to the first optimized listening area, where M is a positive integer; a mixing module, configured to perform a first mixing process on the M rendered audio data to obtain M first mixed audio data and first mixing parameters corresponding to the first optimized listening area; A generating module is configured to generate audio optimization metadata according to first metadata of the first optimized listening area and the first mixing parameters, wherein the audio optimization metadata includes: the first metadata and the first mixing parameters.

30. A terminal device, characterized in that: The terminal device includes: a processor and a memory; the processor and the memory communicate with each other; The memory is used to store instructions; The processor is configured to execute the instructions in the memory to perform the method according to any one of claims 1 to 9, or 10 to 19, or 20 to 26.

31. A computer-readable storage medium comprising instructions, which, when executed on a computer, causes the computer to perform the method according to any one of claims 1 to 9, or 10 to 19, or 20 to 26.

32. A computer program product comprising instructions which, when run on a computer, cause the computer to perform the method according to any one of claims 1 to 9, or 10 to 19, or 20 to 26.

Images