Audio processing method and device

Abstract

The invention discloses an audio processing method and device, and relates to the field of coding and decoding. The decoding apparatus acquires an audio stream including metadata for describing object information of an object, adjusts the metadata of the audio stream in real time in conjunction with a change of the metadata by a user when decoding the audio stream, thereby controlling an audio output of the audio stream based on the metadata adjusted in real time. Thus, a user does not need to modify the metadata of the object in the audio stream in the code stream making process, and can adjust the sound channel and other attributes of any object in the audio stream in real time according to the requirement, so that audio output meeting the user requirement is realized, and the flexibility of audio output is improved.

Description

Technical Field

[0001] The present application relates to the field of coding and decoding, and in particular to an audio processing method and device. Background Art

[0002] Object-based audio streaming decomposes the audio signal into multiple independent objects in the audio stream, such as the voice of a singer in a concert, the engine sound of a car on a track, etc. Each object contains part of the information of the audio signal, such as the location, direction, and distance of the sound source. These objects can be encoded, transmitted, and processed independently, thus achieving more flexible and efficient audio processing and transmission.

[0003] When outputting an object-based audio stream, the object will be rendered to different channels so that speakers of different channels can be used to play multi-channel audio. However, the object rendering method in the audio stream is fixed during the audio stream code production process, and users cannot dynamically adjust the channel and other properties of the object in the audio stream based on their subjective feelings, resulting in poor flexibility in audio stream output. Summary of the invention

[0004] The present application provides an audio processing method and device, which solves the problem that users cannot adjust the object properties of object-based audio streams and the audio stream output flexibility is poor.

[0005] In a first aspect, an audio processing method is provided, which is applied to a codec system, such as the audio processing method is executed by a decoding device in the codec system. The audio processing method includes: first, the decoding device obtains an audio stream, the audio stream includes metadata, and the metadata is used to describe the object information in the audio stream. Then, the decoding device displays a control group on an interactive interface based on the metadata, and generates metadata change information in response to the operation of the control group. When decoding the audio stream, the decoding device changes the metadata of the audio stream according to the metadata change information to obtain a decoding result, and the decoding result includes the changed metadata and the audio data carried by the audio stream. Finally, the decoding device outputs the audio corresponding to the audio data according to the changed metadata.

[0006] In this embodiment, the decoding device provides an interactive interface to the user to display the control group, and changes the metadata of the object in the decoding process of the audio stream according to the user's operation on the control group. In this way, the user does not need to modify the metadata of the object in the audio stream during the code stream production process, and can also adjust the attributes such as the channel of any object in the audio stream in real time according to the needs, so as to achieve audio output that meets the user's needs, thereby improving the flexibility of audio output.

[0007] As a possible implementation, the audio stream includes at least one object. The metadata includes at least one object parameter, and an object parameter is used to describe object information of an object. The object parameter includes at least one of an object name, an object volume, a mute state, and an object sound direction. The control group includes a first control group and a second control group.

[0008] Optionally, when displaying the interactive interface, the decoding device displays a first control group based on the name of at least one object parameter, then determines the target object parameter in response to a first operation on the first control group, and then displays a second control group based on the target object parameter. One control in the second control group is used to adjust the object volume, mute state, or object sound direction in response to the second operation, and generate metadata change information.

[0009] For example, the first control group includes at least one control, one control is displayed as the name of an object parameter, and each control is used to determine the target object parameter selected by the user in response to the user's first operation.

[0010] For another example, the second control group includes at least one control, each control being used to adjust the object volume, mute state or object sound direction corresponding to the target object parameter in response to the user's second operation, and generate metadata change information.

[0011] In the above implementation, the decoding device provides the user with controls for each object in the audio stream and different types of object parameters of each object in the interactive interface, thereby improving the user's flexibility in adjusting the audio stream based on the object.

[0012] As a possible implementation manner, taking the target object parameter in the at least one object parameter as an example, the metadata change information includes first metadata change information, and the first metadata change information includes the changed target object parameter.

[0013] Optionally, when decoding an audio stream, the decoding device parses the audio stream into object information, sound bed information, and metadata information, and then replaces the target object parameters in the metadata information with the changed target object parameters to obtain the changed metadata encoding, and finally outputs the decoding result. Among them, the changed metadata information is used to represent the changed metadata, and the decoding result includes object information, sound bed information, and changed metadata information. In this way, when decoding the audio stream, the encoding device changes the metadata obtained by decoding the audio stream according to the metadata change information, and the user does not need to modify the metadata of the objects in the audio stream in the code stream production process, which reduces the professional requirements for the user and improves the versatility of the method. At the same time, compared with encapsulating the metadata change information into the audio stream to indicate the parsing of the audio stream, the metadata of the objects in the audio stream is adjusted in the parsing process of the audio stream, and there is no need to encapsulate the metadata change information, which reduces the bandwidth and computing resources required for audio stream parsing.

[0014] Optionally, before changing the metadata information, the decoding device determines the target object parameters in the metadata information according to the index of the target object parameters, thereby achieving precise adjustment for the object.

[0015] As a possible implementation manner, the decoding device can enable the user to adjust different objects based on the grouping of objects, thereby providing the user with more audio output modes.

[0016] Optionally, the object parameter also includes the group to which the object belongs and the group members. The decoding device displays a third control group on the interactive interface based on the metadata, and in response to a third operation of the user on the third control group, determines the target group and generates second metadata change information, so that when decoding the audio stream, the metadata of the audio stream is also changed according to the second metadata change information. The second metadata change information includes at least one object parameter after the change, and the object volume of the object parameter belonging to the object outside the target group in the at least one object parameter after the change is the minimum value.

[0017] In the above implementation, the decoding device batch adjusts the object parameters of multiple objects in the audio stream based on the user's selection of the group, providing the user with a simple switching method for different audio output modes and reducing the operation complexity.

[0018] As a possible implementation, the object parameters also include mutually exclusive objects. After the decoding device determines the target object parameters in response to the first operation on the first control group, the object volume of the object parameters of the mutually exclusive objects of the target object parameters is set to the minimum value. In this way, when the user chooses to output the audio of the target object among multiple objects, the object drainage of the mutually exclusive objects that interfere with the target object is set to the minimum value, and the user does not need to manually select and adjust the parameters such as the object volume of the mutually exclusive objects, thereby simplifying the user operation.

[0019] As a possible implementation method, the decoding device updates the interactive interface in real time according to different frames of the audio stream, so that the user can adjust the object parameters more accurately and quickly according to the changes in the preset object parameters of the object in the audio stream, thereby improving the real-time performance of the object parameter adjustment.

[0020] As a possible implementation method, when the decoding device outputs audio according to the decoding result, it mixes the object information to the corresponding sound bed according to the constraints of the changed metadata to obtain the original multi-channel encoded data, and then renders the original multi-channel encoded data with sound effects to obtain the rendered multi-channel encoded data, and finally outputs the audio according to the rendered multi-channel encoded data.

[0021] As a possible implementation manner, the object sound direction is represented by three-dimensional coordinates.

[0022] In a second aspect, an audio processing device is provided, which includes a module for implementing the audio processing method provided by any one of the implementation methods in the first aspect. Exemplarily, the audio processing device includes a code stream module, a display module, an adjustment module, and a rendering module. The code stream module is used to obtain an audio stream, and the audio stream includes metadata, and the metadata is used to describe the object information in the audio stream. The display module is used to display a control group on an interactive interface based on the metadata, and generate metadata change information in response to an operation on the control group. The adjustment module is used to change the metadata according to the metadata change information when decoding the audio stream, and obtain a decoding result, and the decoding result includes the changed metadata and the audio data carried by the audio stream. The rendering module is used to output the audio corresponding to the audio data according to the changed metadata.

[0023] As a possible implementation, the audio stream includes at least one object, the metadata includes at least one object parameter, an object parameter is used to describe object information of an object, and the object parameter includes at least one of an object name, an object volume, a mute state, and an object sound direction.

[0024] Optionally, the display module is specifically used to: display a first control group based on the name of at least one object parameter; determine the target object parameters in response to a first operation on the first control group; display a second control group based on the target object parameters, and a control in the second control group is used to adjust the object volume, mute state or object sound direction in response to the second operation, and generate metadata change information.

[0025] As a possible implementation manner, the metadata change information includes first metadata change information, and the first metadata change information includes changed target object parameters.

[0026] Optionally, the adjustment module is specifically used to: when decoding an audio stream, parse the audio stream into object information, sound bed information and metadata information; replace the target object parameters in the metadata information with the changed target object parameters to obtain the changed metadata information, and the changed metadata information is used to represent the changed metadata; output the decoding result, which includes the object information, sound bed information and the changed metadata information.

[0027] As a possible implementation, the first metadata change information further includes an index of the target object parameter. The adjustment module is further configured to: determine the target object parameter in the metadata information according to the index of the target object parameter.

[0028] As a possible implementation, the object parameters also include the group to which the object belongs and the group members. The display module is further used to: display a third control group on the interactive interface based on the metadata; in response to a third operation on the third control group, determine a target group and generate second metadata change information, the second metadata change information including at least one object parameter after the change, wherein the object volume of the object parameter outside the target group in the at least one object parameter after the change is the minimum value.

[0029] As a possible implementation manner, the object parameters also include mutually exclusive objects, and the adjustment module is further used to: set the object volume of the object parameter of the mutually exclusive object of the target object parameter to a minimum value.

[0030] As a possible implementation method, the display module is also used to: when the metadata of the second frame of the audio stream is different from the metadata of the first frame, update the control group based on the metadata of the second frame, the second frame is the next n frames of the first frame, and n is a positive integer.

[0031] As a possible implementation method, the rendering module is specifically used to: mix the object information to the corresponding sound bed according to the constraints of the changed metadata to obtain original multi-channel encoded data; render the original multi-channel encoded data with sound effects to obtain rendered multi-channel encoded data; and output audio according to the rendered multi-channel encoded data.

[0032] As a possible implementation manner, the object sound direction is represented by three-dimensional coordinates.

[0033] In a third aspect, an electronic device is provided, comprising a memory and a processor, wherein the memory is used to store a set of computer instructions, and when the processor executes the set of computer instructions, it is used to execute the operating steps of the audio processing method in the first aspect or any possible implementation of the first aspect.

[0034] In a fourth aspect, a coding and decoding system is provided, comprising a coding device and a decoding device, wherein the coding device is communicatively connected to the decoding device, the coding device is used to generate an audio stream, the audio stream comprises metadata, and the metadata is used to describe object information in the audio stream, and the decoding device is used to execute the operating steps of the audio processing method in the first aspect or any possible implementation of the first aspect.

[0035] In addition, the technical effects of the audio processing device described in the second aspect, the electronic device described in the third aspect, and the encoding and decoding system described in the fourth aspect can refer to the technical effects of the obstacle detection method described in the first aspect, and will not be repeated here.

[0036] In a fifth aspect, a readable storage medium is provided, which includes: a computer program or instructions; when the computer program or instructions are executed on a computer, the computer executes the audio processing method described in any possible implementation of the first aspect.

[0037] In a sixth aspect, a computer program product is provided, which includes a computer program or instructions, and when the computer program or instructions are executed on a computer, the computer executes the audio processing method described in any possible implementation manner in the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 A schematic diagram of the principle of Dolby Atmos;

[0039] Figure 2 This is a schematic diagram of the principle of immersive sound;

[0040] Figure 3 A schematic diagram of the structure of an audio transmission system provided by this application;

[0041] Figure 4 A schematic diagram of an audio codec system provided by the present application;

[0042] Figure 5 A flowchart of an audio processing method provided by this application;

[0043] Figure 6 A schematic diagram of an interactive interface provided for this application;

[0044] Figure 7 A schematic diagram of a group interaction interface provided by this application;

[0045] Figure 8a A schematic diagram of another group interaction interface provided by this application;

[0046] Figure 8bA schematic diagram of another group interaction interface provided by the present application;

[0047] Fig. 9 A schematic diagram of an object provided by the present application that does not support sound direction adjustment;

[0048] Fig.10 A schematic diagram of an object supporting sound direction adjustment provided in the present application;

[0049] Fig.11 A schematic diagram of another method not supporting sound direction adjustment provided by the present application;

[0050] Fig.12 A schematic diagram of another object supporting sound direction adjustment provided by the present application;

[0051] Fig.13 A schematic diagram of the structure of an audio processing device provided by the present application;

[0052] Fig.14 A structural schematic diagram of another audio processing device provided by the present application;

[0053] Fig.15 A schematic diagram of the structure of an electronic device provided in this application. DETAILED DESCRIPTION

[0054] The audio processing method and device provided in the embodiments of the present application can be applied to multi-channel spatial audio playback scenarios, and a brief introduction to the practical technology of the present application is given.

[0055] (1) Spatial Audio

[0056] Spatial audio can refer to the audio specifications of three-dimensional sound. Compared with traditional multi-channel, spatial audio has more channels than stereo, and can also accommodate object-based sound information, which accurately records the position (static) and displacement (dynamic) of the occurrence point relative to the listener in the form of coordinates. On the playback device, in conjunction with the local or cloud codec, the amplitude-frequency response (timbre) and phase-frequency response (spatial sense) of the object in the channel or sound bed can be accurately calculated and played back.

[0057] Channels usually refer to independent audio signal channels collected or played back at different spatial locations in audio production and playback. Sound bed is the term corresponding to the production end and channel. For example, when a 7.1.4-channel audio system uses object mixing, any audio track of the object is assigned to the 7.1.4 sound bed channel according to the three-dimensional coordinates of the object, thereby controlling the direction of the sound image in three-dimensional space.

[0058] Objects are audio and video information recorded in the form of three-dimensional coordinates, and each object contains part of the information of the audio signal, such as the cry of a child, the commentary in a football game, the cheers of the audience in a basketball game, etc.

[0059] Spatial audio also includes metadata of objects. Metadata describes the information of objects in the audio stream, including: the volume of the object, the orientation and angle of the object, etc. These metadata can be adjusted during the bitstream production process to achieve better audio-visual effects.

[0060] Spatial audio technology also includes a rendering part, that is, the renderer renders the objects in the audio. During the rendering process, the objects are rendered to speakers of different channels according to the metadata of the objects, restoring the intentions of the bitstream producer or the on-site listening experience of the sound recording, allowing users to have an immersive audio experience.

[0061] (2) Codec

[0062] Codec is a general term for encoders and decoders. A codec is a device or program that can transform a signal or a data stream. The transformation here refers to both encoding a signal or data stream (usually for transmission, storage or encryption) or extracting a coded stream, and also includes the operation of restoring a form suitable for observation or operation from this coded stream for observation or processing.

[0063] For example, an audio encoder is used to compress a digital audio signal sampled one by one into a relatively small data stream for easy transmission or storage. Another example is an audio decoder is used to restore the data stream compressed by the audio encoder into the original digital audio signal for easy playback and processing.

[0064] Existing object-based audio stream processing includes standards such as Dolby Atmos and DTS:X.

[0065] Dolby Atmos is a widely used object-based spatial audio technology that can support playback systems with up to 128 independent audio tracks and 64 speaker combinations. Dolby Atmos is applied to the sound bed and introduces the concept of sound objects. Each sound object has its own metadata information to record the X, Y, and Z axis coordinates of the object and the sound size information of the object.

[0066] like Figure 1 As shown in the figure, Dolby Atmos includes the steps of encoding, decoding / rendering, rendering / post-processing and speaker output.

[0067] However, after the Dolby Atmos bitstream is produced, the metadata information remains fixed. Users cannot dynamically adjust the properties of the sound object based on their subjective feelings. There are also strict requirements on the number and arrangement of hardware such as speakers. It is necessary to match the metadata requirements during bitstream production to better restore the multi-channel playback effect.

[0068] Like Dolby Atmos, immersive sound is object-based, but the number of speakers is not strictly calibrated. The playback rendering can be freely rendered according to the number and position of speakers. Figure 2 As shown, the user can change the metadata configuration of the code stream in the source device through a user setting device such as a remote control, send the adjusted metadata through the source device to the sink device for reception, and then output it through the speaker after decoding and rendering by the sink device.

[0069] Although the immersive audio system allows users to modify the properties of objects, users need to re-encode the bitstream to achieve the modification of the object's properties. The re-encoding operation is usually implemented on the production side, which is complex and highly professional, and places high demands on the user's professionalism. At the same time, the re-encoding operation results in a long time span for the entire process from modifying the object's properties to the speaker output, which is not conducive to real-time adjustment of the audio object's properties.

[0070] The present application provides an audio processing method, specifically an audio processing method that "supports real-time modification of object metadata". First, a decoding device obtains an audio stream, and the audio stream includes metadata, and the metadata is used to describe the object information in the audio stream. Then, the decoding device displays a control group on an interactive interface based on the metadata, and generates metadata change information in response to the operation of the control group. When decoding the audio stream, the decoding device changes the metadata of the audio stream according to the metadata change information to obtain a decoding result, and the decoding result includes the changed metadata. Finally, the decoding device outputs the decoding result in audio according to the changed metadata. In this way, the user can change the metadata of the object in real time during the decoding process of the audio stream based on the interactive interface, without modifying the metadata of the object in the audio stream in the code stream production process, and can also adjust the attributes such as the channel of any object in the audio stream in real time according to the needs, so as to achieve audio output that meets the user's needs, thereby improving the flexibility of audio output.

[0071] The implementation of the embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0072] Figure 3 A schematic diagram of the structure of an audio transmission system provided in this application.

[0073] Figure 3Schematic diagram of an audio transmission system provided in an embodiment of the present application, the audio processing process includes audio acquisition, audio encoding, audio transmission, audio decoding and playback process, and the audio transmission system includes multiple terminal devices (such as Figure 3 The terminal devices 311 to 315 are shown and a network, wherein the network can realize the function of audio transmission, and the network may include one or more network devices, which may be routers or switches, etc.

[0074] Figure 3 The terminal device shown may be, but is not limited to, a user equipment (UE), a mobile station (MS), a mobile terminal (MT), etc. The terminal device may be a mobile phone (such as Figure 3 ), a tablet computer, a computer with wireless transceiver function (such as the terminal device 314 shown in Figure 3 ), a virtual reality (VR) terminal device (such as the terminal device 315 shown in Figure 3 The terminal device 313 shown in the figure), augmented reality (AR) terminal device, wireless terminal in industrial control, wireless terminal in self driving, wireless terminal in smart city, wireless terminal in smart home, etc.

[0075] like Figure 3 As shown, in different audio processing processes, the terminal devices are different. For example, in the audio acquisition process, the terminal device 311 can be a speaker with an audio recording function, a camera (such as a video camera, a camera, etc.), a mobile phone, a tablet computer or a smart wearable device with an audio acquisition function, etc. For another example, in the audio encoding process, the terminal device 312 can be a server or a data center, and the data center can include one or more physical devices with encoding functions, such as a server, a mobile phone, a tablet computer or other encoding devices. For another example, in the audio decoding and playback process, the terminal device 313 can be a TV, and the user can modify the metadata of the object in the audio stream by selecting the control group of the interactive interface through the remote control; the terminal device 314 can be a mobile phone, and the user can control the control group of the interactive interface through touch operation or air operation to modify the metadata of the object in the audio stream; the terminal device 315 can be a personal computer, and the user can control the control group of the interactive interface through input devices such as a mouse or keyboard to modify the metadata of the object in the audio stream.

[0076] It can be understood that audio is a general term, and audio includes a plurality of continuous frames of audio sequences, and one frame corresponds to a fixed length of audio data. For example, one frame corresponds to a 26 millisecond audio data.

[0077] Figure 3 This is just a schematic diagram. The audio transmission system may also include other devices. Figure 3 The embodiments of the present application do not limit the number and type of terminal devices included in the system.

[0078] exist Figure 3 The audio transmission system shown is based on Figure 4 A schematic diagram of an audio codec system provided in an embodiment of the present application, the audio codec system 400 includes an encoding device 410 and a decoding device 420 , and the encoding device 410 establishes a communication connection with the decoding device 420 via a communication channel 430 .

[0079] The above-mentioned encoding device 410 can realize the function of audio encoding, such as Figure 3 As shown, the encoding device 410 may be the terminal device 312, and the encoding device 410 may also be a data center with audio encoding capabilities, for example, the data center includes multiple servers.

[0080] The encoding device 410 may include a data source 411 , a pre-processing module 412 , an encoder 413 , and a communication interface 414 .

[0081] The data source 411 may include or may be any type of electronic device for collecting audio, and / or any type of source audio generating device, such as a computer sound card for generating computer animation scenes or any type of device for obtaining and / or providing source audio, or computer-generated source audio. The data source 411 may be any type of memory or storage for storing the above-mentioned source audio. The above-mentioned source audio may include multiple audio streams collected by multiple audio collection devices (such as microphones, speakers, etc.), etc.

[0082] The preprocessing module 412 is used to receive source audio and preprocess the source audio to obtain multi-channel audio (such as spatial audio). For example, the preprocessing performed by the preprocessing module 412 may include audio format conversion, audio splicing, etc.

[0083] The encoder 413 is used to receive audio and encode the audio to obtain encoded data. For example, the encoder 413 encodes the audio to obtain a code stream (also called an audio stream). In some optional situations, the encoded code stream may also be called a bit stream.

[0084] The communication interface 414 in the encoding device 410 can be used to receive encoded data and send the encoded data (or a version of the encoded data after any other processing) to another device such as the decoding device 420 or any other device through the communication channel 430 for storage, playback or direct reconstruction.

[0085] Optionally, the encoding device 410 includes a bitstream buffer, which is used to store bitstreams corresponding to one or more encoding units.

[0086] The above-mentioned decoding device 420 can realize the function of audio decoding, such as Figure 3 As shown, the decoding device 420 may be Figure 3 Any one of the terminal devices 313 to 315 is shown.

[0087] The decoding device 420 may include a playback device 421 , a post-processing module 422 , a decoder 423 and a communication interface 424 .

[0088] The communication interface 424 in the decoding device 420 is used to receive the encoded data (or a version of the encoded data after any other processing is performed on the encoded data) from the encoding device 410 or from any other encoding device such as a storage device.

[0089] Communication interface 414 and communication interface 424 can be used to send or receive encoded data through a direct communication link between encoding device 410 and decoding device 420, such as a direct wired or wireless connection, or through any type of network, such as a wired network, a wireless network or any combination thereof, any type of private network and public network or any type of combination thereof.

[0090] The communication interface 424 corresponds to the communication interface 414, for example, and can be used to receive transmission data and process the transmission data using any type of corresponding transmission decoding or processing and / or decapsulation to obtain encoded data.

[0091] The communication interface 424 and the communication interface 414 can be configured as follows: Figure 4 A unidirectional communication interface, or a bidirectional communication interface, indicated by an arrow pointing from the encoding device 410 to the corresponding communication channel 430 of the decoding device 420, and can be used to send and receive messages, etc. to establish a connection, confirm and exchange any other information related to the communication link, or data transmission such as encoded compressed data transmission, etc.

[0092] The decoder 423 is used to receive the encoded data and decode the encoded data to obtain decoded data (audio, etc.).

[0093] The post-processing module 422 is used to post-process the decoded data to obtain post-processed data (such as audio to be played). The post-processing performed by the post-processing module 422 may include sound bed rendering, sound effect rendering, etc., or any other processing for generating data for playback by the playback device 421, etc.

[0094] The playback device 421 is used to receive the post-processed data to play to the user or viewer. The playback device 421 can be or include any type of display screen and speaker, for example, a display screen with an integrated speaker. For example, the display screen may include a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display, a projector, a micro LED display, a liquid crystal on silicon (LCoS), a digital light processor (DLP), or any other type of display screen. For another example, the speaker may include a multi-channel speaker of an electrodynamic, electrostatic, electromagnetic, piezoelectric, or other type.

[0095] As an optional implementation, the encoding device 410 and the decoding device 420 may transmit the encoded data via a data forwarding device, such as a router or a switch.

[0096] The implementation method of the present application is described in detail below with reference to the accompanying drawings.

[0097] Here, the audio processing method of the embodiment of the present application is as follows: Figure 4 The decoding device 420 shown is used as an example for explanation. Figure 5 A flowchart of an audio processing method provided in an embodiment of the present application is provided. The audio processing method includes the following steps 510 to 550.

[0098] Step 510: The decoding device 420 obtains an audio stream.

[0099] In one possible example, an audio stream includes data of at least one object. The audio stream also includes metadata, which is used to describe object information of multiple objects in the audio stream. The objects here may refer to but are not limited to sound objects, such as multiple independent objects obtained by decomposing an audio signal in an audio stream, such as the voice of a singer in a concert, the engine sound of a car on a track, etc. Each object contains part of the information of the audio signal, such as the location, direction, distance, etc. of the sound source. These objects can be encoded, transmitted and processed independently, thereby achieving more flexible and efficient audio processing and transmission.

[0100] As a possible implementation, the metadata includes at least one object parameter, and an object parameter is used to describe object information of an object. The object parameter includes at least one field of object name, object volume, mute state, and object sound direction.

[0101] The object volume is used to indicate the volume of the object. The mute state is used to indicate whether the object is in a mute state. The volume of the object in a mute state is the minimum value. The object sound direction is used to indicate the sound position of the object. For example, the object sound direction uses a Cartesian coordinate system to indicate the sound position of the object.

[0102] This application does not limit the format of the audio stream. For example, the audio stream can be in avs3-p3 format, etc.

[0103] Step 520: The decoding device 420 displays the control group on the interactive interface based on the metadata.

[0104] In a possible example, the control group includes a first control group and a second control group, the first control group is used to determine the target object parameters in response to a first operation of the user, and the second control group is used to adjust the value of at least one field in the target object parameters in response to a second operation of the user to generate metadata change information.

[0105] As a possible implementation, the decoding device 420 displays a first control group based on the name of at least one object parameter, determines a target object parameter in response to a first operation of the user on the first control group, and then displays a second control group based on the target object parameter.

[0106] Optionally, the target object parameter is carried by metadata of the audio stream, and the metadata may be obtained by parsing the audio stream by the decoder 423 .

[0107] For example Figure 6 As shown, the first area of ​​the interactive interface displays the object to be selected, and the decoding device 420 uses the object name of the object to which at least one object parameter belongs as a display identifier, displays the first control group name and the controls corresponding to each object in the first area, such as three controls whose object names are Object 1, Object 2 and Object 3 respectively.

[0108] Continuing to take the control of object 1 as an example, the control of object 1 includes a selected state and an unselected state, the selected state is used to indicate that the object parameter corresponding to the control of object 1 is in a selected state, and the unselected state is used to indicate that the object parameter corresponding to the control of object 1 is in an unselected state. When the user triggers the control of object 1 in an unselected state by touching the screen or selecting with a remote controller, the decoding device 420 switches the object parameter corresponding to the control of object 1 to a selected state in response to the user operation, and determines that the target object parameter is the object parameter of object 1.

[0109] In the scenario of a football match broadcast, object 1 may be commentary, object 2 may be the atmosphere of the match, and object 3 may be the broadcast of the match. The commentary may include Chinese commentary, English commentary, and the like.

[0110] The second area of ​​the interactive interface displays the name of the second control group and the second control group for adjusting the target object parameters, and the second control group includes controls for adjusting any field value of the target parameter object. The number and type of controls contained in the second control group are determined by the fields contained in the target object parameters. For example, when the target object parameter is object 1, the second control group includes three controls for indicating the mute state, object volume, and object sound direction of the object parameter. The control corresponding to the mute state can be displayed as a mute mark, including a selected state and an unselected state, and the selected state is used to indicate that the object volume of object 1 is set to the minimum value. The control corresponding to the object volume includes a sliding bar, which adjusts the object volume in response to user operation, and increases the object volume when the sliding bar slides in a first direction (for example, toward the 16 direction), and reduces the object drainage when sliding in a second direction (for example, toward the 0 direction), and the second direction is the opposite direction of the first direction.

[0111] Optionally, the interactive interface can also display the second control group in more areas. For example, the third area of ​​the interactive interface displays the controls corresponding to the object sound direction in the second control group. The controls corresponding to the object sound direction include multiple sliders, and the number of sliders is related to the position dimension of the sound direction. For example, if the object sound direction is represented in three dimensions in a Cartesian coordinate system, the controls corresponding to the object sound direction may include three sliders, and the three sliders respectively respond to user operations to adjust the object sound direction to move left and right, front and back, or up and down. For example, the minimum value of the three sliders is -10, and the maximum value is 10. The slider for adjusting the object sound direction to move left and right is marked as left and right, respectively displayed on the left and right sides of the slider, the slider for adjusting the object sound direction to move back and forth is marked as back and front, respectively displayed on the left and right sides of the slider, and the slider for adjusting the object sound direction to move up and down is marked as bottom and top, respectively displayed on the left and right sides of the slider.

[0112] As a possible implementation manner, the decoding device 420 may update the control group displayed on the interactive interface based on changes in metadata of different frames of the audio stream.

[0113] Optionally, when the metadata of the second frame of the audio stream is different from the metadata of the first frame, the decoding device 420 updates the control group based on the metadata of the second frame, wherein the second frame is n frames after the first frame, and n is a positive integer.

[0114] Step 530: The decoding device 420 generates metadata change information in response to the operation on the control group.

[0115] As a possible example, the decoding device 420 generates metadata change information in response to the user's operation on the first control group and the second control group. The metadata change information includes target object parameters after one or more fields are changed.

[0116] For example, in response to the user adjusting the object volume of object 1 based on the interactive interface, the decoding device 420 increases the value of the object volume from 6 to 8, and the object volume of the changed target object parameter in the metadata change information generated by the decoding device 420 is set to 8, and the values ​​of other fields remain unchanged. For another example, in response to the user adjusting the object sound direction of object 1 based on the interactive interface, the coordinate value of the object sound direction of the changed target object parameter in the metadata change information generated by the decoding device 420 is an updated value. Among them, the direction value (such as the left and right movement value, the front and back movement value, or the up and down movement value) in the control corresponding to the object sound direction is mapped to the coordinate system of the object sound direction in the metadata, so as to obtain the coordinate value of the object sound direction of the changed target object parameter according to the direction value.

[0117] Step 540: When decoding the audio stream, the decoding device 420 changes the metadata according to the metadata change information to obtain a decoding result.

[0118] As a possible example, the decoding device 420 replaces the target object parameters in the metadata information with the changed target object parameters in the process of using the decoder 423 to decode the audio stream into object information, sound bed information and metadata information to obtain a decoding result.

[0119] The decoding result includes the changed metadata information and the audio data carried by the audio stream, and the changed metadata information is used to represent the changed metadata.

[0120] Optionally, in addition to the metadata information, the audio stream also carries audio data, including object information and sound bed information. The object information, sound bed information and metadata information may be in pulse code modulation (PCM) format, and thus may also be referred to as object coding, sound bed coding and metadata coding.

[0121] As a possible implementation manner, the decoding device 420 determines the target object parameter in the metadata information according to the index of the target object parameter, thereby changing the target object parameter.

[0122] Step 550: The decoding device 420 outputs the decoding result in audio according to the modified metadata.

[0123] As a possible example, the decoding device 420 mixes the object information to the corresponding sound bed according to the constraints of the changed metadata to obtain original multi-channel encoded data, then renders the original multi-channel encoded data with sound effects to obtain rendered multi-channel encoded data, and outputs audio based on the rendered multi-channel encoded data.

[0124] Optionally, the sound rendering may be implemented by using algorithms such as dynamic range compression (DRC) and audio quality detection (AQ).

[0125] As a possible implementation manner, the decoding device 420 outputs the rendered multi-channel encoded data, which can be transmitted to an external speaker via a network or the like for playback, or can be played via the playback device 421 of the decoding device 420 .

[0126] In a possible embodiment of the present application, the metadata types supported by the above decoding device 420 and the method of performing audio output can refer to the international standard content of the audio definition model (ADM), which will not be repeated here.

[0127] Based on the above steps 510 to 550, the decoding device 420 provides an interactive interface to the user to display the control group, and changes the metadata of the object in the decoding process of the audio stream according to the user's operation on the control group, so that the user can visually indicate the modification of the metadata based on the control group of the interactive interface. In this way, whether it is an online playback scenario or a local playback scenario, the user does not need to re-encode the audio stream in the bitstream production process to modify the metadata of the object, and can also adjust the attributes such as the channel of any object in the audio stream in real time according to the needs to achieve audio output that meets the user's needs, thereby improving the flexibility of audio output. At the same time, the user does not need to re-encode the audio stream, which reduces the overhead of computing resources and the complexity of operations.

[0128] Combination of the above Figure 5 The overall process of the audio processing method is described. The audio processing method realizes the individual adjustment of the metadata of the object. However, in scenarios such as sports event broadcasts and live concerts, users often need to adjust the metadata of multiple related objects at a time. Therefore, the audio processing method of the present application can also divide at least one object in the audio stream into groups, and one group contains object parameters of multiple related objects, thereby providing users with batch object adjustment functions of different groups (or modes).

[0129] Combine the following Figure 7 A detailed description is given on how to implement the group function.

[0130] When the metadata change information of the object parameters of a single object is referred to as the first metadata change information, the decoding device 420 displays a third control group in the interactive interface based on the group in the metadata, determines the target group and generates second metadata change information in response to a third operation on the third control group.

[0131] For example Figure 7 As shown, the fourth area of ​​the interactive interface displays the third control group. The decoding device 420 uses the group name of multiple groups composed of at least one object as a display identifier, and displays the controls corresponding to each group in the third control group in the fourth area, such as three controls with object names Group 1, Group 2 and Group 3.

[0132] Continuing to take the controls of group 1 as an example, the controls of group 1 include a selected state and an unselected state, the selected state is used to indicate that the object parameter corresponding to the controls of group 1 is in a selected state, and the unselected state is used to indicate that the object parameter corresponding to the controls of group 1 is in an unselected state. When the user triggers the controls of group 1 in an unselected state by touching the screen or selecting with a remote controller, the decoding device 420 switches the object parameter corresponding to the controls of group 1 to a selected state in response to the user operation, determines that the target group is the objects included in group 1, and displays the first control group and the second control group corresponding to the objects included in group 1 on the interactive interface. Figure 7 The circular control in front of the group indicates the selected or unselected state of the group. For example, a filled circle indicates that Group 1 is selected, and an unfilled circle indicates that Group 2 and Group 3 are unselected.

[0133] In the scenario of a football match broadcast, group 1 may be a standard group, group 2 may be a home and away atmosphere group, and group 3 may be a live atmosphere group. For example, when a user selects the home and away atmosphere group by touching the screen or selecting with a remote control, the decoding device 420 displays controls corresponding to objects such as commentary, stadium atmosphere, stadium broadcast, fans of team A, and fans of team B included in the home and away atmosphere group in the first area of ​​the interactive interface.

[0134] like Figure 8a As shown, the home and away game range groups include Chinese commentary, English commentary, stadium atmosphere, stadium broadcast, fans of team A, fans of team B, and other objects.

[0135] like Figure 8b As shown, the on-site atmosphere group includes objects such as stadium atmosphere and stadium broadcast.

[0136] Considering that different groups usually represent different modes, when a user selects a target group to adjust metadata of an object, it is necessary to mute the objects not included in the target group. For the convenience of user operation, after the decoding device 420 determines the target group in response to the operation of the third control group, it generates second metadata change information, and the second metadata change information includes at least one object parameter after the change, and the object volume of the object parameter that does not belong to the target group in the at least one object parameter after the change is the minimum value.

[0137] As a possible implementation, when the commentary displayed in the first area of ​​the interactive interface includes Chinese commentary and English commentary, it is considered that if the user selects the Chinese commentary and the English commentary respectively for adjustment, the sounds of the Chinese commentary and the English commentary are played at the same time, which will interfere with the user's listening experience. The object parameters of the present application may also include mutually exclusive object fields for representing mutually exclusive objects of object parameters. When the decoding device 420 determines the target object parameter in response to the user's first operation on the first control group, the value of the object volume field of the object parameter of the mutually exclusive object of the target object parameter is set to the minimum value.

[0138] Optionally, when displaying mutually exclusive objects of a certain object, the decoding device 420 displays a group of mutually exclusive objects in a selection box.

[0139] For example, the mutually exclusive object of Chinese commentary includes English commentary. When the decoding device 420 determines that the marked object parameter is Chinese commentary in response to the user's first operation on the first control group, it queries the object parameter of English commentary in the metadata according to the object index of the mutually exclusive object English commentary, and sets the value of the object volume field of the object parameter of English commentary to the minimum value.

[0140] For another example, the mutually exclusive objects of Team A fans include Team B fans. When the decoding device 420 determines that the target object parameter is Team A fans in response to the user's first operation on the first control group, it queries the object parameters of Team B fans in the metadata based on the object index of the mutually exclusive object Team B fans, and sets the value of the object volume field of the object parameters of Team B fans to the minimum value.

[0141] As a possible implementation method, for any object parameter contained in the metadata, the values ​​of all fields of the object parameter may support change, or the values ​​of some fields of the object parameter may not support change. In this case, the decoding device 420 sets the controls corresponding to the some fields that do not support change to an inoperable state in the interactive interface.

[0142] For example Fig. 9 As shown, the English commentary object does not support the adjustment of the object volume, and the decoding device 420 sets the control corresponding to the object volume in the interactive interface to an inoperable state. Fig.10As shown, the English commentary object supports the adjustment of the object volume, and the decoding device 420 sets the control corresponding to the object volume to an operable state in the interactive interface.

[0143] For example Fig.11 As shown, the English commentary object does not support the adjustment of the object sound direction, and the decoding device 420 sets the control corresponding to the object sound direction in the interactive interface to an inoperable state. Fig.12 As shown, the Chinese commentary object supports the adjustment of the object sound direction, and the decoding device 420 sets the control corresponding to the object sound direction to an operable state in the interactive interface.

[0144] Above Figure 9-12 In the example, low grayscale is used to represent controls in an inoperable state in the interactive interface, and high grayscale is used to represent controls in an operable state in the interactive interface.

[0145] It is understandable that, in order to implement the functions in the above embodiments, the encoding device and the decoding device include hardware structures and / or software modules corresponding to the execution of each function. It should be easily appreciated by those skilled in the art that, in combination with the units and method steps of each example described in the embodiments disclosed in this application, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application scenario and design constraints of the technical solution.

[0146] Fig.13 Schematic diagram of the structure of an audio processing device provided in an embodiment of the present application. The audio processing device can be used to implement the function of the decoding device in the above method embodiment, and thus can also achieve the beneficial effects of the above audio processing method embodiment. In an embodiment of the present application, the audio processing device can be as follows: Figure 4 The decoding device 420 shown may also be a module (such as a chip) applied to a decoding device.

[0147] like Fig.13 As shown, the audio processing device 1300 includes: a code stream module 1301, a display module 1302, an adjustment module 1303 and a rendering module 1304. The code stream module 1301 is used to obtain an audio stream, and the audio stream includes metadata, and the metadata is used to describe the object information in the audio stream. The display module 1302 is used to display a control group on an interactive interface based on the metadata, and generate metadata change information in response to an operation on the control group. The adjustment module 1303 is used to change the metadata according to the metadata change information when decoding the audio stream, and obtain a decoding result, and the decoding result includes the changed metadata. The rendering module 1304 is used to output the audio corresponding to the audio data according to the changed metadata.

[0148] As a possible implementation, the audio stream includes at least one object, the metadata includes at least one object parameter, an object parameter is used to describe object information of an object, and the object parameter includes at least one of an object name, an object volume, a mute state, and an object sound direction.

[0149] The audio processing device 1300 can be used to implement the operation steps corresponding to each method in the above-mentioned audio processing method embodiment, and thus can also achieve corresponding beneficial effects, which will not be described in detail here. For more information about the bit stream module 1301, the display module 1302, the adjustment module 1303 and the rendering module 1304, please refer to the content of the above audio processing method embodiment, which will not be described in detail here.

[0150] When the encoding device or decoding device implements any of the methods shown in the above figures by software, the encoding device or decoding device and its various units may also be software modules. The above method is implemented by calling the software module by a processor. The processor may be a central processing unit (CPU) or other application specific integrated circuit (ASIC), or a programmable logic device (PLD), and the above PLD may be a complex programmable logical device (CPLD), a field programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof.

[0151] For a more detailed description of the above-mentioned encoding device or decoding device, please refer to the relevant description in the embodiments shown in the above-mentioned figures, which will not be repeated here. It can be understood that the encoding device or decoding device shown in the above-mentioned figures is only an example provided by this embodiment, and the encoding device or decoding device may include more or fewer units, which is not limited in this application. Fig.14 As shown, the audio processing device 1400 includes an interaction module 1401, a player module 1402, a metadata management module 1403, an audio decoder module 1404, a metadata adjustment module 1405, an object rendering module 1406, a sound effect rendering module 1407 and a playback module 1408. Each module is obtained by splitting or combining the modules of the audio processing device 1300. For the functions of each module, please refer to Fig.14 The description of the functional steps 1-9 in will not be repeated here.

[0152] When the audio processing device 1300 is implemented by hardware, the hardware can be implemented by a processor or a chip. The chip includes an interface circuit and a control circuit. The interface circuit is used to receive data from other devices outside the processor and transmit it to the control circuit, or send data from the control circuit to other devices outside the processor.

[0153] The control circuit and the interface circuit are used to implement any possible implementation method in the above embodiments through logic circuits or execution code instructions. The beneficial effects can be found in the description of any aspect of the above embodiments, which will not be repeated here.

[0154] In addition, the audio processing device 1300 can also be implemented by an electronic device, where the electronic device may refer to the decoding device in the aforementioned embodiment, or, when the electronic device is a chip or a chip system applied to a server, the audio processing device 1300 can also be implemented by the chip or the chip system.

[0155] like Fig.15 As shown, Fig.15 The electronic device 1500 is a schematic diagram of a structure of an electronic device provided in an embodiment of the present application, and includes: a processor 1510, a bus 1520, a memory 1530, a memory unit 1550 (also referred to as a main memory unit), and a communication interface 1540. The processor 1510, the memory 1530, the memory unit 1550, and the communication interface 1540 are connected via the bus 1520.

[0156] It should be understood that in this embodiment, the processor 1510 may be a CPU, and the processor 1510 may also be other general-purpose processors, DSPs, ASICs, FPGAs or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0157] The communication interface 1540 is used to implement communication between the electronic device 1500 and an external device or component. In this embodiment, the communication interface 1540 is used to perform data exchange with other electronic devices.

[0158] The bus 1520 may include a path for transmitting information between the above components (such as the processor 1510, the memory unit 1550, and the memory 1530). In addition to the data bus, the bus 1520 may also include a power bus, a control bus, and a status signal bus. However, for the sake of clarity, various buses are labeled as bus 1520 in the figure. The bus 1520 may be a Peripheral Component Interconnect Express (PCIe) bus, or an extended industry standard architecture (EISA) bus, a unified bus (Ubus or UB), a compute express link (CXL), a cache coherent interconnect for accelerators (CCIX), etc.

[0159] As an example, electronic device 1500 may include multiple processors. The processor may be a multi-core (multi-CPU) processor. The processor here may refer to one or more devices, circuits, and / or computing units for processing data (e.g., computer program instructions). Processor 1510 may call computer program instructions stored in memory 1530 to execute the following steps: Figure 5 Steps 510 to 550 are shown.

[0160] It is worth mentioning that Fig.15 In the example, an electronic device 1500 including one processor 1510 and one memory 1530 is taken as an example. Here, the processor 1510 and the memory 1530 are respectively used to indicate a type of device or equipment. In a specific embodiment, the number of each type of device or equipment can be determined according to business requirements.

[0161] The memory unit 1550 may correspond to the storage medium for storing information such as pre-upgrade data in the above method embodiment. The memory unit 1550 may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0162] The memory 1530 is used to store bit streams or coding units to be encoded, etc., and can be a solid state drive or a mechanical hard disk.

[0163] It should be understood that the electronic device 1500 may be a DPU. The electronic device 1500 according to this embodiment may correspond to the encoding device or the decoding device in this embodiment, and may correspond to the device for executing the Figure 5 The above and other operations and / or functions of each module in the encoding device or decoding device are respectively for implementing the corresponding processes in the aforementioned method embodiment, and for the sake of brevity, they will not be repeated here.

[0164] The method steps in this embodiment can be implemented by hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable ROM (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, mobile hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC. In addition, the ASIC can be located in a computing device. Of course, the processor and the storage medium can also exist as discrete components in a network device or a terminal device.

[0165] The present application also provides a chip system, which includes a processor for implementing the functions of the encoding device or the decoding device in the above method. In one possible design, the chip system also includes a memory for storing program instructions and / or data. The chip system can be composed of a chip, or can include a chip and other discrete devices.

[0166] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instruction is loaded and executed on a computer, the process or function described in the embodiment of the present application is executed in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user device or other programmable device. The computer program or instruction may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer program or instruction may be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired or wireless means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium, for example, a floppy disk, a hard disk, a tape; it may also be an optical medium, for example, a digital video disc (DVD); it may also be a semiconductor medium, for example, a solid state drive (SSD).

[0167] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. An audio processing method, characterized in that: include: Acquire an audio stream, wherein the audio stream includes metadata, and the metadata is used to describe object information in the audio stream; Displaying a control group on an interactive interface based on the metadata; generating metadata change information in response to an operation on the control group; When decoding the audio stream, the metadata is changed according to the metadata change information to obtain a decoding result, wherein the decoding result includes the changed metadata and the audio data carried by the audio stream; The audio corresponding to the audio data is output according to the modified metadata in the decoding result.

2. The method according to claim 1, characterized in that The audio stream includes at least one object, the metadata includes at least one object parameter, an object parameter is used to describe object information of an object, the object parameter includes at least one of an object name, an object volume, a mute state, and an object sound direction, the control group includes a first control group and a second control group, and displaying the control group on the interactive interface based on the metadata includes: displaying the first group of controls based on the name of the at least one object parameter; In response to a first operation on the first control group, determining a target object parameter; A second control group is displayed based on the target object parameter, and a control in the second control group is used to adjust the object volume, mute state or object sound direction in response to a second operation to generate the metadata change information.

3. The method according to claim 2, characterized in that The metadata change information includes first metadata change information, the first metadata change information includes changed target object parameters, and changing the metadata according to the metadata change information to obtain a decoding result includes: Parsing the audio stream into object information, sound bed information and metadata information; Replacing the target object parameter in the metadata information with the changed target object parameter to obtain the changed metadata information, wherein the changed metadata information is used to represent the changed metadata; The decoding result is output, where the decoding result includes the object information, the sound bed information, and the modified metadata information.

4. The method according to claim 3, characterized in that The first metadata change information also includes an index of the target object parameter, and the method further includes: The target object parameter is determined in the metadata information according to an index of the target object parameter.

5. The method according to any one of claims 2 to 4, characterized in that: The object parameters also include the group to which the object belongs and the group members, and the method further includes: Displaying a third control group on the interactive interface based on the metadata; In response to a third operation on the third control group, a target group is determined and second metadata change information is generated, wherein the second metadata change information includes at least one of the changed object parameters, wherein the object volume of the object parameters outside the target group in the at least one changed object parameter is a minimum value.

6. The method according to any one of claims 2 to 5, characterized in that: The object parameters also include mutually exclusive objects, and after determining the target object parameters in response to the first operation on the first control group, the method further includes: Sets the object volume of the object parameter of the mutually exclusive object of the target object parameter to the minimum value.

7. The method according to any one of claims 1 to 6, characterized in that The method further comprises: When metadata of a second frame of the audio stream is different from metadata of a first frame, the control group is updated based on metadata of the second frame, where the second frame is n frames after the first frame, and n is a positive integer.

8. The method according to claim 3, characterized in that The step of outputting the audio corresponding to the audio information according to the modified metadata in the decoding result includes: Mixing the object information to the corresponding sound bed according to the constraints of the modified metadata to obtain original multi-channel encoded data; Performing sound effect rendering on the original multi-channel encoded data to obtain rendered multi-channel encoded data; Output audio according to the rendered multi-channel encoded data.

9. The method according to any one of claims 2 to 8, characterized in that: The object sound direction is represented by three-dimensional coordinates.

10. An audio processing device, characterized in that: The device comprises a module for executing the operation steps of any one of the methods of claims 1-9.

11. An electronic device, characterized in that: The electronic device comprises a memory and a processor, wherein the memory is used to store a group of computer instructions; when the processor executes the group of computer instructions, the operation steps of any one of the methods described in claims 1-9 are performed.

12. A coding and decoding system, characterized in that: The invention comprises an encoding device and a decoding device, wherein the encoding device is communicatively connected to the decoding device, the encoding device is used to generate an audio stream, the audio stream comprises metadata, and the metadata is used to describe object information in the audio stream, and the decoding device is used to implement the method described in any one of claims 1 to 9.

13. A readable storage medium, characterized in that: The readable storage medium includes a computer program or instructions. When the computer program or instructions are executed on a computer, the computer is enabled to execute the operation steps of any one of the methods described in claims 1 to 9.

Images