Audio information processing method, system and equipment for live broadcast
By obtaining and encoding the metadata of audio signals in live broadcast technology, the problem that live broadcast audiences cannot personalize audio signals is solved, and the user experience is improved.
Patent Information
- Application Number
- CN202310639537.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-31
- Publication Date
- 2024-12-03
AI Technical Summary
In live broadcast technology, live broadcast viewers cannot personalize the received audio signals according to their own preferences, resulting in poor user experience.
By obtaining the audio signal of the sound source in the live broadcast scene, determining that the metadata of the audio signal is associated with the sound source, and encoding it according to the preset audio encoding method, an editable encoded audio information stream is generated, allowing the push streaming client and the playback client to be allowed to Decode and edit audio signals.
It realizes that live broadcast viewers can personalize audio signals according to their own preferences, improving user experience.
Smart Images

Figure CN119071560A_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present disclosure relate to the technical fields of digital signal processing and audio production, and in particular to an audio information processing method, system and device for live broadcasting. Background Art
[0002] Live streaming, as a technology that can synchronously produce and broadcast the occurrence and development of on-site events, is becoming more and more widely used. Correspondingly, live streaming technology has also made great progress, and various live streaming technologies have emerged, such as low-latency live streaming and interactive live streaming.
[0003] The inventors found that the processing of sound in live broadcasts is still based on traditional channel technology to encode audio information. Traditional channels include mono, stereo, 2.1 channels, 5.1 channels and 7.1 channels. The audio information processed by the live streaming client cannot be changed. In this way, the live streaming client pushes the encoded live audio information to the streaming media server and then distributes it to the client of the live audience. Different live audiences hear the same sound. For live audiences, it is impossible to personalize the audio signal received in the live broadcast according to their own preferences, resulting in a poor user experience. Summary of the invention
[0004] The embodiments of the present disclosure provide an audio information processing method, system and device for live broadcasting, so as to overcome the problem in the existing live broadcasting technology that live broadcasting viewers cannot personalize the received live broadcasting audio data.
[0005] In a first aspect, an embodiment of the present disclosure provides an audio information processing method for live broadcast, which is applied to a streaming client, and includes: obtaining an audio signal of a sound source in a live broadcast scene; determining first metadata of the audio signal, and associating the first metadata with the audio signal of the sound source; encoding the associated first metadata and the audio signal of the sound source according to a preset audio encoding method to obtain an encoded audio information stream; and sending the encoded audio information stream, wherein the associated first metadata and audio signal are editable after the encoded audio information stream is decoded.
[0006] In a second aspect, an embodiment of the present disclosure provides an audio information processing method for live broadcast, which is applied to a playback client, comprising: receiving an encoded audio information stream, wherein the encoded audio information stream is obtained by encoding the associated first metadata and the audio signal of the sound source by the streaming client according to a preset audio encoding method, and the sound source is the sound source in the live broadcast scene; decoding the encoded information stream based on a preset audio decoding method to obtain the audio signal and the first metadata associated with the audio signal; rendering the audio signal frame by frame based on the first metadata or the second metadata, and the second metadata is obtained by performing a second metadata editing operation on the audio signal according to a second editing instruction.
[0007] In a third aspect, an embodiment of the present disclosure provides an audio information processing system for live broadcasting, comprising a streaming client and a playback client; wherein the streaming client is used to obtain an audio signal of a sound source in a live broadcasting scene; determine first metadata of the audio signal, and associate the first metadata with the audio signal of the sound source; encode the associated first metadata and the audio signal of the sound source according to a preset audio encoding method to obtain an encoded audio information stream; and send the encoded audio information stream, wherein the associated first metadata and audio signal after the encoded audio information stream is decoded are editable;
[0008] A playback client is used to receive an encoded audio information stream, wherein the encoded audio information stream is obtained by encoding the associated first metadata and the audio signal of the sound source by the push streaming client according to a preset audio encoding method, and the sound source is the sound source in the live broadcast scene; the encoded information stream is decoded based on a preset audio decoding method to obtain the audio signal and the first metadata associated with the audio signal; the audio signal is rendered frame by frame based on the first metadata or the second metadata, and the second metadata is obtained by performing a second metadata editing operation on the audio signal according to a second editing instruction.
[0009] In a fourth aspect, an embodiment of the present disclosure provides an audio information processing device for live broadcast, which is applied to a streaming client, including: an acquisition unit, used to acquire an audio signal of a sound source in a live broadcast scene; a determination unit, used to determine first metadata of the audio signal, and associate the first metadata with the audio signal of the sound source; an encoding unit, used to encode the associated first metadata and the audio signal of the sound source according to a preset audio encoding method to obtain an encoded audio information stream; a sending unit, used to send the encoded audio information stream, wherein the associated first metadata and audio signal after decoding the encoded audio information stream are editable.
[0010] In a fifth aspect, an embodiment of the present disclosure provides an audio information processing device for live broadcast, which is applied to a playback client and includes: a receiving unit, which is used to receive an encoded audio information stream, wherein the encoded audio information stream is obtained by encoding the associated first metadata and the audio signal of the sound source by the streaming client according to a preset audio encoding method; a decoding unit, which is used to decode the encoded information stream based on a preset audio decoding method to obtain the audio signal and the first metadata associated with the audio signal; a rendering unit, which is used to render the audio signal frame by frame based on the first metadata or the second metadata, wherein the second metadata is obtained by performing a second editing operation on the metadata of the target audio signal according to a second editing instruction.
[0011] In a sixth aspect, an embodiment of the present disclosure provides an electronic device, comprising: a processor and a memory; the memory stores computer execution instructions; the processor executes the computer execution instructions stored in the memory, so that the at least one processor executes the first aspect or the second aspect above and various possible audio information processing methods for live broadcast of the first aspect or the second aspect.
[0012] In the seventh aspect, an embodiment of the present disclosure provides a computer-readable storage medium, in which computer execution instructions are stored. When a processor executes the computer execution instructions, the first aspect or the second aspect above and various possible audio information processing methods for live broadcast of the first aspect or the second aspect are implemented.
[0013] In an eighth aspect, an embodiment of the present disclosure provides a computer program product, including a computer program, which, when executed by a processor, implements the first or second invention as well as various possible audio information processing methods for live broadcasting of the first or second aspect.
[0014] The audio information processing method, system and device for live broadcast provided in this embodiment obtain the audio signal of the sound source in the live broadcast scene; determine the first metadata of the audio signal, and associate the first metadata with the audio signal of the sound source; encode the associated first metadata and the audio signal of the sound source according to a preset audio encoding method to obtain an encoded audio information stream; send the encoded audio information stream, wherein the associated first metadata and audio signal after decoding of the encoded audio information stream are editable, so that each audio signal after decoding of the encoded information stream provided by the streaming client can be edited separately, which is conducive to the live broadcast audience to personalize the audio signal received in the live broadcast according to their own preferences, and can improve the user experience. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0016] Figure 1 A flowchart of an audio information processing method for live broadcasting provided by an embodiment of the present disclosure;
[0017] Figure 2 A flowchart of an audio information processing method for live broadcasting provided by an embodiment of the present disclosure;
[0018] Figure 3 A schematic block diagram of an audio information processing system for live broadcasting provided by an embodiment of the present disclosure;
[0019] Figure 4 A structural block diagram of an audio information processing device for live broadcasting provided by an embodiment of the present disclosure;
[0020] Figure 5 A structural block diagram of an audio information processing device for live broadcasting provided by an embodiment of the present disclosure;
[0021] Figure 6 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION
[0022] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.
[0023] Please refer to Figure 1 , Figure 1 A schematic diagram of an audio information processing method for live broadcasting provided by an embodiment of the present disclosure. The method can be applied to a streaming client. Figure 1 As shown, the method comprises the following steps:
[0024] S101: Acquire an audio signal of a sound source in a live broadcast scene.
[0025] The live streaming clients here include but are not limited to computers, mobile terminals, tablet computers, IoT devices, wearable devices, smart devices, embedded terminals, etc. The live streaming clients can support running various known or future-to-be-developed operating systems.
[0026] The live broadcast scene here can be various scenes, such as a game scene, a live interview scene, etc. In the live broadcast scene, at least one sound source can be included. The sound source here can be various objects that can emit audio signals. Taking the game scene as an example, the sound source can include players, background music playback equipment, live instruments, commentators, spectators, etc. Usually, one sound source can correspond to one audio signal.
[0027] The audio signal includes at least one of the following: a mono audio signal, a stereo audio signal, a surround sound audio signal, and an audio signal with spatial structure information.
[0028] Audio signals with spatial structures include but are not limited to Wanos, Atmos, FOA (First Order Ambisonic) signals, HOA (Higher Order Ambisonics) signals, A-Format signals, B-Format signals, etc.
[0029] As an implementation method, the audio signal of the live broadcast scene can be obtained from an audio signal collector set in the live broadcast scene.
[0030] It is understandable that for the same sound source, the audio collectors of the sound source can be respectively set at different angles. Each sound source can correspond to at least one audio signal collector.
[0031] At least one audio signal collector can be set at the sound source in the live broadcast scene, and the audio signal collector can be connected to the above-mentioned streaming client via wired or wireless communication. The above-mentioned streaming client can receive the audio signal of the sound source in the live broadcast scene collected by the above-mentioned audio signal collector in real time.
[0032] In some application scenarios, audio signal collectors of different sound sources can send collected audio signals to the streaming client respectively. The audio signals of different sound sources collected by different audio signal collectors can be frame synchronized according to the time axis.
[0033] In some application scenarios, the above audio signal may be an audio signal that a streaming client monitors in real time in a live broadcast scene. The monitoring environment includes but is not limited to spatial audio headphones, a 5.1.4 monitoring environment, a 7.1.4 monitoring environment, and the like.
[0034] S102: Determine first metadata of the audio signal, and associate the first metadata with the audio signal of the sound source.
[0035] The metadata of the audio signal may be descriptive data of the audio signal. The metadata may be a set of information describing the audio signal and the artistic intent used to transfer the original or coded audio signal to a reproduction system.
[0036] The description data of the audio signal in the present disclosure includes, but is not limited to, data describing one or more of the position, number of channels, propagation direction, and gain of the audio signal. That is, the metadata of the audio signal refers to at least one or more of the position metadata, the number of channels metadata, the propagation direction metadata, and the gain metadata.
[0037] The above-mentioned position metadata can be used to describe the position of the audio signal in the audio scene. The channel number metadata is used to describe the number of channels. The sound source object metadata may include, for example, description data of the sound source object. The sound source object metadata may include, for example, description data indicating that the sound source is spatial noise, description data indicating that the sound source is commentary, description data indicating that the sound source is a player, and description data indicating that the sound source is environmental noise. The gain metadata is used to describe the gain of the audio signal.
[0038] In some embodiments, the above step S102 includes: extracting initial metadata of the audio signal from the audio file, and using the extracted initial metadata group as the first metadata.
[0039] In these embodiments, the audio file may carry initial metadata of the audio signal. The initial metadata here may be initial metadata that is previously edited and saved in the audio file by a human or a device. After receiving the audio file, the streaming client may extract metadata information from the audio file and extract the initial metadata from the audio file.
[0040] In these embodiments, the initial metadata extracted from the audio file may be used as the first metadata, so that the initial metadata originally carried in the audio file can also be subjected to subsequent metadata editing operations on the playback client.
[0041] In some other embodiments, the above step 102 includes: receiving metadata input by a user for the audio signal, and using the metadata as the first metadata.
[0042] In these embodiments, the audio signal may be an audio signal collected from an audio signal collector in a live broadcast scene. After receiving the audio signal, the user may input metadata of the audio signal in the streaming client, and use the metadata as the first metadata of the audio signal.
[0043] In these embodiments, the first metadata may be added to the audio signal without metadata according to the user's operation, so as to facilitate the subsequent editing operation of the first metadata by the playback client.
[0044] In some other embodiments, the step 102 includes: first metadata generated by an audio processing device before the streaming client. The audio processing device may include processing such as editing and associating audio signals. The editing of the audio signal includes editing the amplitude, frequency, etc. The associating of the audio signal includes associating the virtual audio signal with the audio signal of the live scene, etc. The audio processing device may be, for example, a mixing console or a device running audio processing software.
[0045] The above-mentioned audio processing device may generate corresponding metadata when performing processing operations such as editing and associating audio signals, and the above-mentioned metadata may be used as the first metadata of the audio signal.
[0046] In some other embodiments, the above step S102 includes: performing a first metadata editing operation on the initial metadata extracted from the audio file according to the received first editing instruction to obtain the first metadata.
[0047] In these embodiments, the streaming client may display a metadata editing interface for the audio signal. The user may perform a first metadata editing operation on the initial metadata in the editing interface. The first metadata is determined according to the user's editing operation on the initial metadata.
[0048] The first metadata editing operation includes at least one of the following: deletion, addition, modification and merging.
[0049] That is, the user can modify or delete at least part of the initial metadata in the editing interface in the streaming client; can also input new metadata in the editing interface; can also merge part of the metadata, etc. Through the first metadata editing operation, the metadata finally obtained is used as the first metadata.
[0050] After the audio signals corresponding to different sound sources are acquired, the first metadata editing operation can be performed on the metadata of the different audio signals according to the first editing instruction.
[0051] The first editing instruction may be issued by a user of a live streaming client or by a live streaming server.
[0052] As an implementation method, when the metadata of the audio signal is edited, a metadata editing interface can be displayed on the live streaming client. The metadata editing interface can include multiple metadata editing items that can be operated. The user of the live streaming client can edit the metadata through the metadata editing items.
[0053] In some application scenarios, the editing operation may be a modification of the position metadata of the audio signal. As an illustrative example, the editing operation may be a modification of the coordinates of the audio signal.
[0054] In some other application scenarios, the audio signal may include multiple channels. The above-mentioned editing operation on the metadata of the audio signal may be, for example, an editing operation on the number of channels of the audio signal. As an illustrative example, the above-mentioned editing operation may be adding the number of channels of the audio signal, or may be deleting the number of channels of the audio signal. For example, adding or deleting the number of speakers used for playing back the audio signal.
[0055] Channels refer to independent audio signals collected or played back at different spatial locations when recording or playing sounds. The number of channels refers to the number of sound sources when recording sounds or the corresponding number of speakers when playing them back.
[0056] In some other application scenarios, the editing operation on the audio signal metadata may be an editing operation on the gain metadata. The editing operation on the gain metadata may be, for example, an increase or decrease in the gain value.
[0057] After the first metadata is obtained, a method for establishing an association relationship between metadata and an audio signal in related technologies may be used to establish an association relationship between the first metadata and the corresponding audio signal.
[0058] In some embodiments, the audio signal in the live broadcast scene includes audio signals of multiple sound sources; the first metadata includes first metadata corresponding to the audio signals of at least some of the multiple sound sources; and the above-mentioned associating the first metadata with the audio signal of the sound source includes:
[0059] For each sound source of the at least part of the sound sources, an audio signal of the sound source is associated with first metadata corresponding to the sound source.
[0060] That is to say, for the audio signal of each sound source, the audio signal can be associated with the first metadata of the audio signal. In this way, different audio signals can be associated with respective first metadata.
[0061] By associating the audio signal of each sound source with the first metadata of the audio signal, each audio signal and the first metadata of the audio signal can be edited separately on the playback client.
[0062] S103: Encode the associated first metadata and the audio signal of the sound source according to a preset audio encoding method to obtain an encoded audio information stream.
[0063] The associated metadata and audio signal may be encoded using a preset audio encoding method.
[0064] The above-mentioned preset audio encoding method can be compressed format encoding or uncompressed format encoding, such as WAV, MP3, AAC, AC3, WANOS / ATMOS, etc. The encoded audio signal includes a sound bed track signal and an object track signal, etc.
[0065] It can be understood that, for live broadcast application scenarios, the audio signal may include multiple continuous audio frames, and the first metadata may be acquired frame by frame, or acquired from a certain audio frame, and then the first metadata of the audio frame is applied to the audio frame and other subsequent audio frames. During encoding, the audio signal may be encoded frame by frame to obtain an encoded audio information stream.
[0066] S104: Sending a coded audio information stream, wherein the first metadata and the audio signal associated with the coded audio information stream after decoding are editable.
[0067] The live streaming client can send the encoded audio information stream according to a preset transmission protocol. The preset transmission protocol includes but is not limited to: Real Time Messaging Protocol (RTMP), Real-Time Communication Protocol (RTC), Secure Reliable Transport Protocol (SRT), etc.
[0068] The above-mentioned coded audio information stream is decoded by the playback client after being received by the playback client. The decoded associated audio signal and the first metadata are editable, that is, they can be edited again on the playback client.
[0069] The audio information processing method for live broadcast provided in this embodiment obtains the audio signal of the sound source in the live broadcast scene; determines the first metadata of the audio signal, and associates the first metadata with the audio signal of the sound source; encodes the audio signal of the sound source with the associated first metadata according to a preset audio encoding method to obtain an encoded audio information stream; and sends the encoded audio information stream, wherein the associated first metadata and audio signal after decoding the encoded audio information stream are editable, so that each audio signal after decoding the provided encoded information stream can be edited separately, which is beneficial for the live broadcast audience to personalize the received audio signal in the live broadcast according to their own preferences, and can improve the user experience.
[0070] In some embodiments, the above step 101 includes:
[0071] Analyze the audio signal corresponding to the sound source from the acquired audio file of the live broadcast scene; or
[0072] The received encoded audio file is decoded to obtain the audio signal of the sound source.
[0073] In some application scenarios, the streaming client can receive an audio file of a live broadcast scene. The audio file can be an audio file generated based on the audio signal of the live broadcast scene collected by the audio signal collector. In these application scenarios, the above step 101 can include parsing the audio signal corresponding to the sound source from the acquired audio file of the live broadcast scene.
[0074] The streaming client can parse the audio file, for example, split the audio signals corresponding to different sound sources from the audio file according to the frequency spectrum of the audio file, and identify the audio signals of the above sound sources from the split results.
[0075] The audio file may be an audio file sent by other electronic devices. The audio file includes a directly audible audio file. The other electronic devices may include, for example, a mixing console.
[0076] In some other application scenarios, the audio file received by the streaming client may be an encoded audio file of a live broadcast scene sent by other electronic devices. In these application scenarios, step 101 includes: decoding the received encoded audio file to obtain an audio signal of the sound source.
[0077] If the audio file is an encoded audio file of a live broadcast scene, the streaming client can first decode the audio file, and separate the decoded audio file into audio signals corresponding to different sound sources from the audio file according to the spectrum. The above decoding of the audio file includes decoding the audio file using an audio decoder, or decoding the audio file using a sound card, etc.
[0078] The above signal may be a multi-channel mixed audio signal. In these application scenarios, the streaming client may split the audio information to separate audio signals corresponding to different sound sources from the audio information.
[0079] In these embodiments, the audio signal of the live scene can be acquired through various audio signal acquisition methods, so that audio signals can be extracted from audio files of the live scene from different sources, and metadata editing operations can be performed on the audio signals.
[0080] In some optional implementations, the above-mentioned audio information processing method for live broadcast also includes: rendering the audio signal according to the first metadata.
[0081] In these optional implementations, the streaming client may render the audio signal according to the first metadata, thereby pre-evaluating the rendering effect of the audio signal.
[0082] Furthermore, if the pre-evaluation result of the audio signal rendered according to the first metadata is not good, the streaming user can continue to perform editing operations on the first metadata on the streaming client.
[0083] Please refer to Figure 2 , Figure 2 The following is a schematic diagram of a method for processing audio information for live broadcasting provided by an embodiment of the present disclosure. The method can be applied to a playback client. Figure 2 As shown, the method comprises the following steps:
[0084] S201: Receive an encoded audio information stream, wherein the encoded audio information stream is obtained by encoding the associated first metadata and the audio signal of the sound source by the streaming client according to a preset audio encoding method, and the sound source is a sound source in a live broadcast scene.
[0085] The live broadcast clients include but are not limited to computers, mobile terminals, tablet computers, IoT devices, wearable devices, smart devices, embedded terminals, etc. The above-mentioned broadcast clients can support running various known or future-to-be-developed operating systems.
[0086] The above-mentioned encoded audio information stream can be sent by a streaming client.
[0087] The streaming client and the playback client may be located in different spatial locations, or in the same or similar spatial locations.
[0088] In some application scenarios, the streaming client may send the encoded audio information stream to the streaming server, and the streaming server forwards the encoded audio information stream.
[0089] The streaming media server can send the encoded audio information stream to multiple playback clients. The multiple playback clients can receive the encoded audio information stream via wired communication or wireless communication. In this embodiment, one playback client is used as an example for description.
[0090] S202: Decode the encoded information stream based on a preset audio decoding method to obtain an audio signal and first metadata associated with the audio signal.
[0091] The above-mentioned preset audio encoding method can be a method for decoding compressed format encoded data or uncompressed format encoded data, such as WAV, MP3, AAC, AC3, WANOS / ATMOS, etc.
[0092] The preset decoding method here may be a decoding method that matches the preset encoding method used by the streaming client.
[0093] After decoding the encoded audio information stream, the associated audio signal and the first metadata may be obtained.
[0094] The decoded audio signal may be a multi-channel signal, may be a plurality of sound objects, or may be a combination of the two.
[0095] The decoded data of each audio signal may include the audio signal and first metadata associated with the audio signal.
[0096] The first metadata of the audio signal may include description data of the audio signal. The metadata may be a set of information representing artistic intent for describing the audio signal and for transferring the original or encoded audio signal to a reproduction system. The decoded audio signal and the first metadata may be edited.
[0097] S203: Rendering the audio signal frame by frame based on the first metadata or the second metadata, wherein the second metadata is obtained by performing a second metadata editing operation on the audio signal according to a second editing instruction.
[0098] The second editing instruction may be an instruction sent by a user of the playback client to the playback client, or may be an instruction sent by the live broadcast server to the playback client.
[0099] When the playback client does not receive the metadata editing instruction for the audio signal, that is, when the playback client does not generate the second metadata of the audio signal, the audio signal can be rendered using the first metadata of the audio signal.
[0100] After the playback client receives the metadata editing instruction for the audio signal and generates the second metadata of the audio signal, the second metadata can be used to render the audio signal.
[0101] That is, for each audio signal, it can be determined during rendering whether the audio signal has the second metadata generated by the playback client. If so, the second metadata is used to render the audio signal. Otherwise, the first metadata of the audio signal is used to render the audio signal.
[0102] In some application scenarios, the second metadata may be obtained by a user performing a second metadata editing operation on the first metadata of the audio signal on a playback client.
[0103] In these application scenarios, the user performs a second metadata editing operation on the first metadata of the audio signal on the playback client to obtain the second metadata of the audio signal required by the user.
[0104] The second metadata editing operation includes at least one of the following: deletion, addition, modification and merging.
[0105] In some optional implementations, the above step S203 includes:
[0106] First, information of at least one audio signal is displayed.
[0107] The playback client can display information of multiple audio signals. The displayed audio signal information may include information of the sound source of the audio signal, such as commentary, background music, narration, etc.
[0108] The live broadcast audience can perform a selection operation on the information of at least one audio signal. The live broadcast client can determine an audio signal to be edited according to the selection operation.
[0109] Secondly, the metadata editing interface of the audio signal is displayed according to the triggering instructions of the live audience.
[0110] After determining the audio signal to be edited, the user of the live client can issue a trigger instruction to display the metadata editing interface, such as clicking or touching the metadata editing control in the display interface of the audio signal.
[0111] The displayed metadata editing interface may include multiple data interaction windows.
[0112] Finally, a second editing instruction issued by the user of the live broadcast client in the metadata editing interface is received, and a second editing operation is performed on the metadata of the audio signal to obtain second metadata of the audio signal.
[0113] The live broadcast audience can perform input and modification operations in the data interaction window of the metadata editing interface. The operations of the live broadcast audience can be regarded as the second editing instructions. The playback client can perform editing operations on the first metadata according to the second editing instructions.
[0114] In these optional implementations, the live broadcast client facilitates the live broadcast audience to edit the metadata of the audio signal by providing a visual interface, thereby reducing the complexity of the live broadcast audience editing the metadata of the audio signal.
[0115] In some application scenarios, the second editing operation includes: setting and / or modifying position information of the audio signal.
[0116] In these application scenarios, the decoded audio signal may include an audio signal and first metadata describing the audio signal. The first metadata may include position metadata describing spatial position information of the audio signal. Second metadata for describing the position information of the audio signal may be determined by setting and / or modifying the position metadata. This allows the spatial effect of the audio object to be adjusted during rendering.
[0117] In some other application scenarios, the second editing operation includes: adding and deleting the number of channels.
[0118] In these application scenarios, the first metadata of the decoded audio signal may include channel metadata describing the audio signal channel. The second editing operation may be, for example, adding or deleting the number of channels, for example, adding or deleting the number of speakers used to generate the audio signal.
[0119] In some other application scenarios, the second editing operation includes: increasing and decreasing the gain.
[0120] In these application scenarios, the decoded first metadata associated with the audio signal may be gain metadata, and the second editing operation may be, for example, increasing or decreasing the gain of the audio signal.
[0121] After obtaining the second metadata, the audio signal may be rendered according to the second metadata. The audio signal here may be, for example, commentary, background music, and narration.
[0122] The above-mentioned rendering of the audio signal according to the first metadata or the second metadata may include format conversion of the audio signal to be rendered, local reverberation processing, binaural rendering, speaker rendering, etc. Specifically, reference may be made to related audio signal rendering technologies.
[0123] As a scenario description, the decoded audio signal information stream may include the audio signal and the first metadata corresponding to the commentary in the game scene, the audio signal and the first metadata corresponding to the background music, and the audio signal and the first metadata corresponding to the narration. The live audience (the user of the playback client) can select the commentary audio signal in the displayed audio information interface, and then display the display interface of the commentary audio signal according to the user's selection. The metadata editing interface can be displayed in the display interface of the commentary audio signal. The live audience can perform corresponding operations on the first metadata in the metadata editing interface, thereby sending a second editing instruction to the playback client. For example, a trigger operation is performed on the gain increase control corresponding to the gain metadata, and for example, metadata is directly entered in the above-mentioned gain metadata editing window. The playback client performs a corresponding second editing operation on the first metadata according to the above-mentioned second editing instruction. Thereby, the live audience realizes the re-editing of the commentary audio signal. When the commentary audio signal is re-edited, there is no need to edit other audio signals.
[0124] In some application scenarios, the first metadata and the second metadata of the audio signal may exist simultaneously in the playback client, and the live broadcast audience may select one of the two metadata to render the audio signal as needed.
[0125] Through the above steps of this embodiment, for the same encoded audio information stream, different live viewers' clients may have different second metadata, thus producing different rendering effects. Different live viewers may hear different sounds.
[0126] The audio information processing method for live broadcast provided in this embodiment receives an encoded audio information stream, wherein the encoded audio information stream is obtained by encoding the associated first metadata and the audio signal of the sound source by a streaming client according to a preset audio encoding method, and the sound source is the sound source in the live broadcast scene; the encoded information stream is decoded based on a preset audio decoding method to obtain the audio signal and the first metadata associated with the audio signal; the audio signal is rendered frame by frame based on the first metadata or the second metadata, and the second metadata is obtained by performing a second metadata editing operation on the audio signal according to a second editing instruction, thereby realizing that the metadata of the audio signal is edited on the playback client to obtain the second metadata, and the audio signal is rendered according to the second metadata, so that the live broadcast audience can personalize the received audio signal in the live broadcast according to their own preferences, and bring an immersive and interactive audio experience to the live broadcast audience.
[0127] Please refer to Figure 3 , Figure 3 Schematic diagram of an audio information processing system for live broadcasting provided by an embodiment of the present disclosure. Figure 3As shown, the audio information processing system applied to live broadcast includes a streaming client and a playback client.
[0128] A streaming client can be used to obtain an audio signal of a sound source in a live broadcast scene; determine first metadata of the audio signal, and associate the first metadata with the audio signal of the sound source; encode the audio signal of the sound source associated with the first metadata according to a preset audio encoding method to obtain an encoded audio information stream; and send the encoded audio information stream, wherein the associated first metadata and audio signal are editable after the encoded audio information stream is decoded;
[0129] A playback client can be used to receive an encoded audio information stream, wherein the encoded audio information stream is obtained by encoding the associated first metadata and the audio signal of the sound source by the push streaming client according to a preset audio encoding method, and the sound source is the sound source in the live broadcast scene; the encoded information stream is decoded based on a preset audio decoding method to obtain the audio signal and the first metadata associated with the audio signal; the audio signal is rendered frame by frame based on the first metadata or the second metadata, and the second metadata is obtained by performing a second metadata editing operation on the audio signal according to a second editing instruction.
[0130] like Figure 3 As shown, the audio signal of the sound source (ie, the audio source, including multiple audio sources Audio1, Audio2, and Audio3) can be collected by the sound source audio signal collection device and sent to the streaming client. In addition, the above-mentioned sound source audio signal can also be an audio file sent by other devices.
[0131] The streaming client may include a production system and a live broadcast system. The production system may be used to determine first metadata for the audio signal of each sound source. Figure 3 As shown, the production system can determine the first metadata for the audio signal Audio1, and associate the first metadata with the audio signal Audio1.
[0132] The above method for determining the first metadata for the audio signal can refer to Figure 1 The description of the relevant parts of the illustrated embodiment will not be repeated here.
[0133] The live broadcast system can encode the first metadata and audio signal associated with each other, and then transmit them according to the preset audio and video transmission protocol. For specific instructions, please refer to Figure 1 Relevant parts of the illustrated embodiment.
[0134] In some application scenarios, the system may further include a streaming media server disposed between the streaming client and the playback client. The streaming media server is used to receive the encoded audio information stream sent by the live streaming client and forward the received encoded audio information stream to multiple playback clients.
[0135] In some optional implementations, the streaming client obtains the audio signal of the sound source in the live broadcast scene, including:
[0136] Analyze the audio signal corresponding to the sound source from the acquired audio file of the live broadcast scene;
[0137] The audio signal collector collects the audio signal of the sound source; or
[0138] The received encoded audio file is decoded to obtain the audio signal of the sound source.
[0139] In some optional implementations, the streaming client obtains the first metadata of the audio signal, including:
[0140] Determining first metadata of the audio signal includes at least one of the following:
[0141] Extracting initial metadata of the audio signal from the audio file, and using the extracted initial metadata group as first metadata;
[0142] performing a first metadata editing operation on initial metadata extracted from the audio file according to the received first editing instruction to obtain first metadata;
[0143] receiving metadata input by a user for an audio signal, and using the metadata as first metadata;
[0144] The metadata generated by the audio processing device before the streaming client is used as the first metadata.
[0145] In some optional implementations, the first editing operation performed by the streaming client includes one of the following:
[0146] Delete, add, modify and merge.
[0147] In some optional implementations, the first metadata of the streaming client includes at least one of the following:
[0148] Position metadata, channel count metadata, direction metadata, gain metadata.
[0149] In some optional implementations, the audio signal of the sound source in the live broadcast scene includes audio signals of multiple sound sources; the first metadata includes first metadata corresponding to the audio signals of at least some of the multiple sound sources; and
[0150] The associating the first metadata with the audio signal of the sound source includes:
[0151] For each sound source of at least some of the sound sources, an audio signal of the sound source is associated with first metadata corresponding to the sound source.
[0152] In some optional implementations, the streaming client encodes the associated first metadata and the audio signal of the sound source according to a preset audio encoding method to obtain an encoded audio information stream, including:
[0153] The associated first metadata and audio signal are encoded using a compressed format audio sound encoding method or an uncompressed format audio encoding method to obtain an encoded audio information stream.
[0154] In some optional implementations, the streaming client renders the audio signal according to the first metadata.
[0155] In some optional implementations, the audio signal includes at least one of the following: a mono audio signal, a stereo audio signal, a surround sound audio signal, and an audio signal with spatial structure information.
[0156] In some optional implementations, playing the second editing operation that can be performed by the client includes one of the following:
[0157] Delete, add, modify and merge.
[0158] In some optional implementations, the playing client performing the second metadata editing operation on the audio signal according to the second editing instruction includes:
[0159] displaying information of at least one audio signal;
[0160] Displays the metadata editing interface of the audio signal according to the triggering instructions of the live audience;
[0161] A second editing instruction issued by a live broadcast viewer on the metadata editing interface is received, and a second metadata editing operation is performed on the audio signal to obtain second metadata of the audio signal.
[0162] Please refer to Figure 4 , Figure 4 The structure diagram of an audio information processing device for live broadcast provided by the embodiment of the present disclosure is shown. The audio information processing device 40 for live broadcast is applied to a streaming client. For ease of explanation, only the parts related to the embodiment of the present disclosure are shown. Figure 4 As shown, the audio information processing device 40 for live broadcasting includes: an acquisition unit 401, a determination unit 402, an encoding unit 403 and a sending unit 404.
[0163] An acquisition unit 401 is used to acquire an audio signal of a sound source in a live broadcast scene;
[0164] A determining unit 402 is used to determine first metadata of the audio signal and associate the first metadata with the audio signal of the sound source;
[0165] The encoding unit 403 is used to encode the associated first metadata and the audio signal of the sound source according to a preset audio encoding method to obtain an encoded audio information stream;
[0166] The sending unit 404 is used to send the encoded audio information stream, wherein the first metadata and the audio signal associated with the encoded audio information stream after decoding are editable.
[0167] In some optional implementations, the acquiring unit 401 is further configured to:
[0168] Analyze the audio signal corresponding to the sound source from the acquired audio file of the live broadcast scene;
[0169] The audio signal collector collects the audio signal of the sound source; or
[0170] The received encoded audio file is decoded to obtain the audio signal of the sound source.
[0171] In some optional implementations, the determining unit 402 is further configured to determine the first metadata of the audio signal according to at least one of the following:
[0172] Extracting initial metadata of the audio signal from the audio file, and using the extracted initial metadata group as first metadata;
[0173] performing a first metadata editing operation on initial metadata extracted from the audio file according to the received first editing instruction to obtain first metadata;
[0174] receiving metadata input by a user for an audio signal, and using the metadata as first metadata;
[0175] The metadata generated by the audio processing device before the streaming client is used as the first metadata.
[0176] In some optional implementations, the first metadata editing operation includes at least one of the following:
[0177] Delete, add, modify and merge.
[0178] In some optional implementations, the first metadata includes at least one of the following:
[0179] Position metadata, channel count metadata, direction metadata, gain metadata.
[0180] In some optional implementations, the audio signal of the sound source in the live broadcast scene includes audio signals of multiple sound sources; the first metadata includes first metadata corresponding to the audio signals of at least some of the multiple sound sources, and the determination unit 402 is further used to: for each sound source of at least some of the sound sources, associate the audio signal of the sound source with the first metadata corresponding to the sound source.
[0181] In some optional implementations, the encoding unit 403 is further used to: use a compressed format audio sound encoding method or a non-compressed format audio encoding method to encode the associated first metadata and the audio signal to obtain an encoded audio information stream.
[0182] In some optional implementations, the device 40 further includes a pre-rendering unit (not shown in the figure), and the pre-rendering unit is used to render the audio signal according to the first metadata.
[0183] In some optional implementations, the audio signal includes at least one of the following: a mono audio signal, a stereo audio signal, a surround sound audio signal, and an audio signal with spatial structure information.
[0184] Please refer to Figure 5 , Figure 5 The structure diagram of an audio information processing device for live broadcast provided by the embodiment of the present disclosure is shown. The audio information processing device 50 for live broadcast is applied to a playback client. For ease of explanation, only the parts related to the embodiment of the present disclosure are shown. Figure 5 As shown, the audio information processing device 50 for live broadcasting includes: a receiving unit 501, a decoding unit 502 and a rendering unit 503.
[0185] The receiving unit 501 is used to receive a coded audio information stream, wherein the coded audio information stream is obtained by encoding the associated first metadata and the audio signal of the sound source by the streaming client according to a preset audio encoding method;
[0186] A decoding unit 502, configured to decode the encoded information stream based on a preset audio decoding method to obtain an audio signal and first metadata associated with the audio signal;
[0187] The rendering unit 503 is used to render the audio signal frame by frame based on the first metadata or the second metadata, where the second metadata is obtained by performing a second metadata editing operation on the metadata of the target audio signal according to the second editing instruction.
[0188] In some optional implementations, the second metadata editing operation includes one of the following:
[0189] Delete, add, modify and merge.
[0190] In some optional implementations, the device 50 further includes an editing unit (not shown), which is configured to determine the second metadata based on the following operations:
[0191] displaying information of at least one audio signal;
[0192] Displays the metadata editing interface of the audio signal according to the triggering instructions of the live audience;
[0193] A second editing instruction issued by a live broadcast viewer on the metadata editing interface is received, and a second metadata editing operation is performed on the audio signal to obtain second metadata of the audio signal.
[0194] In order to implement the above embodiment, the embodiment of the present disclosure also provides an electronic device.
[0195] refer to Figure 6 , which shows a schematic diagram of the structure of an electronic device 600 suitable for implementing an embodiment of the present disclosure, and the electronic device may be an electronic device for a streaming client or a playback client. The electronic device 600 may be a head-mounted AR device, a terminal device, or a server. Among them, the terminal device may include but is not limited to mobile terminals such as mobile phones, laptops, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (Portable Android Devices, PADs), portable multimedia players (PMPs), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 6 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.
[0196] like Figure 6 As shown, the electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program (computer execution instruction) loaded from a storage device 608 to a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the electronic device 600 are also stored. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.
[0197] Typically, the following devices may be connected to the I / O interface 605: input devices 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 608 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 609. The communication device 609 may allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 6 The electronic device 600 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.
[0198] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 609, or installed from a storage device 608, or installed from a ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.
[0199] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program (computer-executable instructions), which may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.
[0200] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.
[0201] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the method shown in the above embodiment.
[0202] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0203] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0204] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware. The name of a unit does not limit the unit itself in some cases. For example, the acquisition unit may also be described as a "unit for acquiring an audio signal of a sound source in a live scene".
[0205] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.
[0206] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0207] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.
[0208] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.
[0209] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.
Claims
1. A method for processing audio information for live broadcast, applied to a streaming client, comprising: Obtain the audio signal of the sound source in the live broadcast scene; determining first metadata of the audio signal, and associating the first metadata with the audio signal of the sound source; Encoding the associated first metadata and the audio signal of the sound source according to a preset audio encoding method to obtain an encoded audio information stream; The encoded audio information stream is sent, wherein the first metadata and the audio signal associated with the encoded audio information stream after decoding are editable.
2. The method according to claim 1, characterized in that The obtaining of the audio signal of the sound source in the live broadcast scene includes at least one of the following: Parsing the audio signal corresponding to the sound source from the acquired audio file of the live scene; The audio signal collector collects the audio signal of the sound source; The received encoded audio file is decoded to obtain the audio signal of the sound source.
3. The method according to claim 1, characterized in that The determining of the first metadata of the audio signal comprises at least one of the following: Extracting initial metadata of the audio signal from the audio file, and using the extracted initial metadata group as first metadata; performing a first metadata editing operation on initial metadata extracted from the audio file according to the received first editing instruction to obtain first metadata; receiving metadata input by a user for the audio signal, and using the metadata as first metadata; The metadata generated by the audio processing device before the streaming client is used as the first metadata.
4. The method according to claim 3, characterized in that The first metadata editing operation includes at least one of the following: Delete, add, modify and merge.
5. The method according to claim 1, characterized in that The first metadata includes at least one of the following: Position metadata, channel count metadata, direction metadata, gain metadata.
6. The method according to claim 1, characterized in that The audio signal of the sound source in the live broadcast scene includes audio signals of multiple sound sources; the first metadata includes first metadata corresponding to the audio signals of at least some of the multiple sound sources; as well as The associating the first metadata with the audio signal of the sound source comprises: For each sound source of the at least part of the sound sources, an audio signal of the sound source is associated with first metadata corresponding to the sound source.
7. The method according to claim 1, characterized in that The step of encoding the associated first metadata and the audio signal of the sound source according to a preset audio encoding method to obtain an encoded audio information stream includes: The associated first metadata and audio signal are encoded using a compressed format audio sound encoding method or an uncompressed format audio encoding method to obtain an encoded audio information stream.
8. The method according to any one of claims 1 to 7, characterized in that The method further comprises rendering the audio signal according to the first metadata.
9. The method according to any one of claims 1 to 7, characterized in that The audio signal includes at least one of: a mono audio signal, a stereo audio signal, a surround sound audio signal, and an audio signal with spatial structure information.
10. A method for processing audio information for live broadcast, applied to a playback client, comprising: Receiving an encoded audio information stream, wherein the encoded audio information stream is obtained by encoding the associated first metadata and an audio signal of a sound source by a streaming client according to a preset audio encoding method, and the sound source is a sound source in a live broadcast scene; Decoding the encoded information stream based on a preset audio decoding method to obtain an audio signal and first metadata associated with the audio signal; The audio signal is rendered frame by frame based on the first metadata or the second metadata, and the second metadata is obtained by performing a second metadata editing operation on the audio signal according to a second editing instruction.
11. The method according to claim 10, characterized in that The second metadata editing operation includes at least one of the following: Delete, add, modify and merge.
12. The method according to claim 10, characterized in that The performing a second metadata editing operation on the audio signal according to the second editing instruction comprises: displaying information of at least one audio signal; Displays the metadata editing interface of the audio signal according to the triggering instructions of the live audience; A second editing instruction issued by a live broadcast viewer in the metadata editing interface is received, and a second metadata editing operation is performed on the audio signal to obtain second metadata of the audio signal.
13. An audio information processing system applied to live broadcast, including a streaming client and a playback client; The streaming client is used to obtain the audio signal of the sound source in the live broadcast scene; determining first metadata of the audio signal, and associating the first metadata with the audio signal of the sound source; Encoding the associated first metadata and the audio signal of the sound source according to a preset audio encoding method to obtain an encoded audio information stream; Sending the encoded audio information stream, wherein the first metadata and the audio signal associated with the encoded audio information stream after decoding are editable; A playback client is used to receive an encoded audio information stream, wherein the encoded audio information stream is obtained by encoding the associated first metadata and the audio signal of the sound source by the push streaming client according to a preset audio encoding method, and the sound source is the sound source in the live broadcast scene; the encoded information stream is decoded based on a preset audio decoding method to obtain the audio signal and the first metadata associated with the audio signal; the audio signal is rendered frame by frame based on the first metadata or the second metadata, and the second metadata is obtained by performing a second metadata editing operation on the audio signal according to a second editing instruction.
14. An audio information processing device for live broadcast, applied to a streaming client, comprising: An acquisition unit, used to acquire an audio signal of a sound source in a live broadcast scene; a determining unit, configured to determine first metadata of the audio signal, and associate the first metadata with the audio signal of the sound source; An encoding unit, configured to encode the associated first metadata and the audio signal of the sound source according to a preset audio encoding method to obtain an encoded audio information stream; The sending unit is used to send the encoded audio information stream, wherein the first metadata and the audio signal associated with the encoded audio information stream after decoding are editable.
15. An audio information processing device for live broadcast, applied to a playback client, comprising: A receiving unit, configured to receive a coded audio information stream, wherein the coded audio information stream is obtained by encoding the associated first metadata and the audio signal of the sound source by a streaming client according to a preset audio encoding method; A decoding unit, configured to decode the encoded information stream based on a preset audio decoding method to obtain an audio signal and first metadata associated with the audio signal; The rendering unit is used to render the audio signal frame by frame based on the first metadata or the second metadata, where the second metadata is obtained by performing a second metadata editing operation on the audio signal according to a second editing instruction.
16. An electronic device, characterized in that: include: Processor and memory; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor executes the audio information processing method for live broadcast as described in any one of claims 1 to 9, or executes the audio information processing method for live broadcast as described in any one of claims 10 to 12.
17. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer execution instructions. When the processor executes the computer execution instructions, it implements the audio information processing method for live broadcast as described in any one of claims 1 to 9, or executes the audio information processing method for live broadcast as described in any one of claims 10 to 12.
18. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, it implements the audio information processing method for live broadcast as described in any one of claims 1 to 9, or executes the audio information processing method for live broadcast as described in any one of claims 10 to 12.
