Audio data processing method and device, equipment and storage medium

Abstract

The embodiment of the invention discloses an audio data processing method and device, equipment and a storage medium, and the method comprises the steps: adding first mute data with a first set size to a created audio coding sending queue when a switching instruction from a multi-person audio mode to a single-person audio mode is received; obtaining first audio data from the created coding synchronization management queue, adding the first audio data to an audio coding sending queue, and adding second audio data collected relative to a single-person audio mode to the audio coding sending queue; and controlling the audio encoder to switch from the first encoding format to a second encoding format, and encoding the audio data packet transmitted from the audio encoding sending queue in the second encoding format. By means of the method, higher link priority determination flexibility is achieved, the situation that a network or an access point related to communication of the first terminal and the access end changes can be better adapted, and a proper transmission link can be better selected.

Description

Technical Field

[0001] This disclosure relates to the field of audio processing technology, and in particular to an audio data processing method, apparatus, device, and storage medium. Background Technology

[0002] In live streaming applications, the broadcaster encodes the captured audio data using an encoder and uploads it to the server. The server then distributes the encoded audio as a stream to the viewer via a content delivery network. Finally, the viewer decodes the audio stream to enable playback of the audio data in the live streaming application.

[0003] The audio encoding format used by the broadcaster varies depending on the live streaming business model. For example, in multi-person audio mode (such as multi-person live chat mode), the collected audio data can be considered live audio data, and in this case, Advanced Audio Coding (AAC) should be used to encode the live audio data. However, in single-person audio mode (broadcaster-only mode), the collected audio data can be considered live chat audio data, and in this case, an encoding format with low latency and weak network resistance (such as Opus encoding, Opus is an audio codec format) should be used to encode the live chat audio data.

[0004] Currently, the audience-side player is primarily a traditional open-source player, which only supports decoding AAC encoded audio data. To achieve effective playback of the live chat audio data to the audience, the server needs to merge the audio data from multiple live chat participants, then re-encode the merged audio data into AAC audio data using an encoder before sending it to the audience. The live AAC audio data, directly encoded by the broadcaster's encoder, can be directly sent to the audience. Therefore, the live chat AAC audio data and the live AAC audio data sent to the audience originate from different encoders.

[0005] When a live stream switches between single-person and multi-person audio modes, or vice versa, the audio data played on the viewer's end will also switch accordingly, from AAC audio data from a live chat session to AAC audio data from the live stream session, or vice versa. Because this switching involves audio data from different encoders, it can cause brief audio playback stutters due to the different encoder sources, thus affecting the viewer's listening experience. Summary of the Invention

[0006] This disclosure provides an audio data processing method, apparatus, device, and storage medium that can effectively resume audio data generated when switching between different business modes in live streaming applications, thereby avoiding audio data stuttering issues.

[0007] In a first aspect, embodiments of this disclosure provide an audio data processing method, which is applied to a first terminal and includes:

[0008] When a switching instruction from multi-person audio mode to single-person audio mode is received, a first set size of first silence data is added to the created audio encoding sending queue;

[0009] The system retrieves first audio data from the created encoding synchronization management queue, adds the first audio data to the audio encoding sending queue, and adds second audio data acquired relative to the single-person audio mode to the audio encoding sending queue. The first audio data is audio data that has not been encoded in the first encoding format relative to the multi-person audio mode.

[0010] The audio encoder is controlled to switch from the first encoding format to the second encoding format, and after the encoding initialization is completed, the audio data packets received from the audio encoding transmission queue are encoded in the second encoding format.

[0011] Secondly, this disclosure also provides an audio data processing method applied to a second terminal, comprising:

[0012] When the encoding format of the received encoded audio packet is detected to have switched from the first encoding format to the second encoding format, the received encoded audio packet is decoded and the first decoded audio packet is discarded. The receiving, decoding and discarding operations are repeated until the number of discarded first decoded audio packets reaches the set value.

[0013] If the received encoded audio packet is the first encoded audio packet after the discard operation ends, then the first encoded audio packet is decoded, and the decoded second decoded audio packet is re-encoded using the second encoding format;

[0014] The encoded audio package after secondary encoding is obtained and sent to the content distribution network for distribution to the business client.

[0015] Thirdly, this disclosure also provides an audio data processing device configured in a first terminal, the device comprising:

[0016] The first switching module is used to add a first set size of first mute data to the created audio encoding sending queue when it receives a switching instruction from multi-person audio mode to single-person audio mode;

[0017] The first caching module is used to obtain first audio data from the created encoding synchronization management queue, add the first audio data to the audio encoding sending queue, and add second audio data collected relative to the single-person audio mode to the audio encoding sending queue. The first audio data is audio data that has not been encoded in the first encoding format relative to the multi-person audio mode.

[0018] The first encoding module is used to control the audio encoder to switch from the first encoding format to the second encoding format, and to encode the audio data packets received from the audio encoding sending queue in the second encoding format after the encoding initialization is completed.

[0019] Fourthly, embodiments of this disclosure also provide an audio data processing device configured in a second terminal, the device comprising:

[0020] The first packet loss module is used to decode the received encoded audio packet when the encoding format of the received encoded audio packet is switched from the first encoding format to the second encoding format, and discard the first decoded audio packet after decoding. The receiving, decoding and discarding operations are repeated until the number of discarded first decoded audio packets reaches a set value.

[0021] The first encoding / decoding module is used to decode the first encoded audio packet when the received encoded audio packet is the first encoded audio packet after the discard operation ends, and then encode the decoded second decoded audio packet again using the second encoding format.

[0022] The first sending module is used to obtain the encoded audio packet after secondary encoding and send it to the content distribution network so that it can be distributed to the service client through the content distribution network.

[0023] Fifthly, embodiments of this disclosure also provide a computer device, the computer device comprising:

[0024] One or more processors;

[0025] Storage device for storing one or more programs.

[0026] When the one or more programs are executed by the one or more processors, the one or more processors implement the audio data processing method provided in any embodiment of this disclosure.

[0027] Sixthly, embodiments of this disclosure also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the audio data processing method provided in any embodiment of this disclosure.

[0028] In a seventh aspect, embodiments of this disclosure also provide a computer program product, including a computer program that, when executed by a processor, implements the audio data processing method provided in any embodiment of this disclosure.

[0029] The technical solution of this disclosure embodiment specifically discloses an audio data processing method, apparatus, device, and storage medium. The method first involves a first terminal receiving a switching instruction from a multi-person audio mode to a single-person audio mode, adding first silence data of a first predetermined size to a created audio encoding transmission queue; obtaining first audio data from a created encoding synchronization management queue and adding the first audio data to the audio encoding transmission queue; and adding second audio data acquired relative to the single-person audio mode to the audio encoding transmission queue. The first audio data is audio data that was not encoded in the first encoding format relative to the multi-person audio mode. The method then controls the audio encoder to switch from the first encoding format to the second encoding format and processes the audio data in the second encoding format. The audio data packets received from the audio encoding transmission queue are encoded. For the second terminal, when it detects that the encoding format of the received encoded audio packet has switched from the first encoding format to the second encoding format, it decodes the received encoded audio packet and discards the first decoded audio packet. The receiving, decoding, and discarding operations are repeated until the number of discarded first decoded audio packets reaches a set value. If the received encoded audio packet is the first encoded audio packet after the discarding operation ends, the first encoded audio packet is decoded, and the second decoded audio packet is then re-encoded using the second encoding format. The re-encoded audio packet is obtained and sent to the content delivery network for distribution to the service client. In the above-described technical solution of this embodiment, the first terminal can add silence data to the audio encoding transmission queue and add the audio data that has not been encoded in the multi-person audio mode back to the audio encoding transmission queue when the service mode is switched, such as from multi-person audio mode to single-person audio mode. Combined with the second terminal's packet loss handling of silence data and secondary encoding of the unencoded audio data, the continuity between audio data corresponding to different service modes is ensured when the service mode is switched. This continuity can fill the problem of audio data disconnection caused by different encoding formats corresponding to different service modes, thereby avoiding the problem of audio data stuttering caused by audio data disconnection. In this way, the auditory experience of audio data in real-time call applications is improved by eliminating audio data stuttering. Attached Figure Description

[0030] To more clearly illustrate the technical solutions of the exemplary embodiments of this disclosure, the accompanying drawings used in describing the embodiments are briefly introduced below. Obviously, the accompanying drawings described are only a portion of the embodiments to be described in this disclosure, and not all of them. For those skilled in the art, other drawings can be obtained from these drawings without any creative effort.

[0031] Figure 1 A flowchart illustrating an audio data processing method provided in an embodiment of this disclosure;

[0032] Figure 2 An example diagram of the existing processing flow of audio data in live streaming application scenarios, including the live chat mode and the live streaming mode, is provided.

[0033] Figure 3 An example diagram showing the audio data contained in the audio encoding and transmission queue in the audio data processing method provided in this embodiment is given;

[0034] Figure 4 The audio encoder in the audio data processing method provided in this embodiment is given. Figure 3 The image shown is an example of the audio data after encoding the audio data in the audio encoding and sending queue.

[0035] Figure 5 Another example of the audio data contained in the audio encoding and transmission queue in the audio data processing method provided in this embodiment is shown in the figure.

[0036] Figure 6 The audio encoder in the audio data processing method provided in this embodiment is given. Figure 5 The image shown is an example of the audio data after encoding the audio data in the audio encoding and sending queue.

[0037] Figure 7 A flowchart illustrating an audio data processing method provided in an embodiment of this disclosure;

[0038] Figure 8 A schematic diagram of the logical implementation of the audio data processing method provided in this embodiment when the encoding format of the encoded audio packet received by the second terminal is switched from the first encoding format to the second encoding format is given;

[0039] Figure 9 A schematic diagram of the logic implementation when the encoding format of the encoded audio packet received by the second terminal is switched from the second encoding format to the first encoding format in the audio data processing method provided in this embodiment is given;

[0040] Figure 10 This is a schematic diagram of the structure of an audio data processing device provided in an embodiment of the present disclosure;

[0041] Figure 11 This is a schematic diagram of the structure of an audio data processing device provided in an embodiment of the present disclosure;

[0042] Figure 12 This is a schematic diagram of the structure of a computer device provided in an embodiment of this disclosure. Detailed Implementation

[0043] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0044] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0045] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0046] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules, or units, and are not used to limit the order of functions performed by these devices, modules, or units or their interdependencies. It should also be noted that the modifications of "a" and "a plurality of" mentioned in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0047] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0048] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0049] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0050] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0051] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0052] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0053] Figure 1 This is a flowchart illustrating an audio data processing method provided in an embodiment of the present disclosure. This embodiment is applicable to the processing of audio data in live streaming applications. The method can be executed by an audio data processing device, which can be implemented by software and / or hardware. It can be configured in a terminal and / or server to implement the audio data processing method in this embodiment of the present disclosure, and preferably, the terminal and / or server can be used as the first terminal to execute the audio data processing method provided in this embodiment.

[0054] It should be noted that a live streaming application scenario can be used as one application scenario in this embodiment. For example, Figure 2 This document presents example diagrams illustrating the existing audio data processing workflows in live streaming application scenarios, specifically in multi-user and live streaming modes. Figure 2As shown, when the live streaming application is running, if the host client 21 is in a multi-person audio mode, it will use an encoding format, such as Opus encoding, to encode the generated multi-person audio data. The encoded multi-person Opus audio data will be uploaded to the live streaming server 22 and merged with the multi-person Opus audio data uploaded by the connected guests. Then, the live streaming server 22 will decode the merged multi-person Opus audio data and encode it again using another encoding format, such as AAC encoding. The resulting multi-person AAC audio data will be sent to the content delivery network for viewers to retrieve.

[0055] Following the above description, as Figure 2 As shown, when the live streaming application is running, if the host client 21 is in live streaming mode, it will directly sample AAC encoding to encode the generated indicator audio data, and the encoded live AAC audio data will be directly sent to the content distribution network through the live streaming server 22 for the audience client to retrieve.

[0056] pass Figure 2 It can be seen that the AAC audio data generated in the co-hosting mode and the AAC audio data generated in the live streaming mode come from different encoders. As a result, when switching from co-hosting mode to live streaming mode, or vice versa, there will be a problem of discontinuity between the co-hosting AAC audio data and the live streaming AAC audio data. On the viewer's end, this will result in audio playback stuttering when switching modes. At the same time, since the audio data of the two modes are encoded into AAC audio data through different encoders, and there is an encoding delay during encoding initialization, this encoding delay will also cause audio playback stuttering.

[0057] Based on this, this embodiment provides an audio data processing method that enables seamless continuation of corresponding multi-person audio data and single-person audio data when switching between single-person audio mode (such as the single-broadcast mode in a live streaming application) and multi-person audio mode (such as a multi-person live chat mode). It should be noted that the method provided in this embodiment is not limited to live streaming applications; it can be applied to any real-time call application involving switching between single-person and multi-person audio output. Specifically, for example... Figure 1 As shown, an audio data processing method provided in this embodiment may include:

[0058] S101. When a switching instruction from multi-person audio mode to single-person audio mode is received, first silence data of a first set size is added to the created audio encoding sending queue.

[0059] In this embodiment, the switching instruction from multi-person audio mode to single-person audio mode can be generated after the operator performs the mode switching operation, and this step can receive the generated switching instruction. In this embodiment, the audio encoding and sending queue can be understood as a queue that buffers the audio data collected by the first terminal before encoding.

[0060] It should be noted that the first terminal can be understood as the terminal that generates audio data during a real-time call. This could be a terminal used by a broadcaster in a live streaming application, or a terminal used by a teacher in online real-time teaching. In this embodiment, the audio data collected by the first terminal needs to be cached in an audio encoding and transmission queue before encoding. Then, the audio data in the audio encoding and transmission queue is transmitted to the encoder in frames as audio data packets for encoding processing. The encoder also transmits the encoded audio data packets in frames through the second terminal to the content delivery network, or, after being decoded and encoded again by the second terminal, it is sent to the content delivery network. The second terminal can be considered as the terminal participating in the audio data encoding and processing generated by the first terminal, such as a live streaming server in a live streaming application.

[0061] It is known that when an encoder encodes audio data according to a certain encoding format, an encoding delay is generated during the encoding initialization phase. Therefore, the encoder experiences an encoding delay when encoding the incoming audio data packets. This encoding delay generated during the encoder initialization phase is also encoded as silence data by the encoder. However, the silence data involved in encoding cannot constitute one or more complete frame-length encoded silence data packets. The incomplete frame-length encoded silence data will combine with subsequent audio data involved in encoding to form a complete frame-length encoded audio data packet. Therefore, the first frame of encoded audio data packet formed based on this encoding method contains encoded silence data that cannot be discarded.

[0062] As described above, when a real-time call application switches from multi-person audio mode to single-person audio mode, the encoding format also needs to be switched. After the encoding format is switched, the encoder needs to be re-initialized. This means that the encoder generates a segment of undiscardable encoded silence data every time it switches. The encoded silence data generated during the switch can be perceived as audio stuttering on the viewer's client.

[0063] In this embodiment, when switching from a multi-person audio mode to a single-person audio mode, this step pushes a set size of silence data into the audio encoding sending queue. In this embodiment, this set size is denoted as the first set size, and the silence data is denoted as the first silence data. In this embodiment, the first silence data of the first set size is added to combine with the incomplete frame-length silence data generated during the encoding process to form a complete frame-length encoded silence data packet.

[0064] The first set size can be represented as the duration, and its specific value can be determined based on the encoding delay duration of the switched encoding format. Specifically, given the encoding delay duration and the frame length of an audio data packet, it is possible to determine how much more time is needed to complete a frame. For example, when switching from multi-person audio mode to single-person audio mode, the encoding format needs to be switched to the second encoding format corresponding to single-person audio mode. After determining the encoding delay duration of the second encoding format, the first set size can be determined.

[0065] S102. Obtain first audio data from the created encoding synchronization management queue, add the first audio data to the audio encoding sending queue, and add second audio data collected relative to the single-person audio mode to the audio encoding sending queue. The first audio data is audio data that has not been encoded in the first encoding format relative to the multi-person audio mode.

[0066] In this embodiment, the first silence data added through the above steps can be used to supplement the encoded silence data generated by the encoding delay, so as to discard the encoded silence data without discarding valid single-person audio data. It should also be noted that the encoder's encoding delay is also manifested in the fact that a portion of unencoded audio data will remain in the encoder. If the encoder continuously encodes audio data in one encoding format, the unencoded audio data will be combined with a portion of the subsequently received audio data to continue participating in the encoding process.

[0067] However, if a live streaming application switches from multi-person audio mode to single-person audio mode, the encoder's encoding format also needs to change. Since encoder initialization is required during this process, some data remaining in the encoder is directly cleared and cannot participate in subsequent encoding. This is equivalent to audio data loss, making it impossible to effectively continue the audio data between multi-person and single-person modes during the switch, which will also cause stuttering in the audio playback on the viewer's client.

[0068] Based on this, this embodiment can use this step to identify audio data that has not been fully encoded in the encoder and add it back to the audio encoding transmission queue so that the unencoded audio data can be re-encoded in the switched encoding format. Specifically, this embodiment pre-creates an encoding synchronization management queue, which can realize the synchronous input and synchronous output of audio data packets contained in the encoder. Thus, the encoding synchronization management queue can be regarded as a queue for synchronizing and managing the unencoded audio data packets contained in the encoder.

[0069] As described above, the encoding synchronization management queue contains audio data that has not been encoded by the encoder. Therefore, this embodiment can directly retrieve the unencoded audio data from the encoding synchronization management queue through this step when a switching instruction from multi-person audio mode to single-person audio mode is received. This embodiment refers to this unencoded audio data as the first audio data, and it can be known that this first audio data specifically refers to audio data collected in multi-person audio mode that has not been encoded in the first encoding format. The first encoding format can be considered the encoding format corresponding to the multi-person audio mode, such as the Opus encoding format.

[0070] In this embodiment, the first audio data obtained from the encoding synchronization management queue in this step can be passed to the audio encoding transmission queue again. Simultaneously, it is known that when switching from multi-person audio mode to single-person audio mode, the first terminal will also correspondingly collect the single-person audio data generated in single-person audio mode. This step can also normally add this single-person audio data to the audio encoding transmission queue for transmission to the encoder for encoding processing. In this embodiment, this single-person audio data can be recorded as the second audio data.

[0071] For example, Figure 3 An example diagram illustrating the audio data contained in the audio encoding transmission queue in the audio data processing method provided in this embodiment is given. When a real-time call application switches from a multi-person audio mode to a single-person audio mode, after the execution of the above-described S101 and this step, the audio encoding transmission queue contains audio data such as... Figure 3 The audio data content shown. Specifically, as... Figure 3 As shown, the first silence data 31 of a first set size is added to the head of the audio encoding and sending queue, followed by the first audio data 32 obtained from the encoding synchronization management queue and the second audio data 33 acquired relative to the single-person audio mode.

[0072] S103. Control the audio encoder to switch from the first encoding format to the second encoding format, and encode the audio data packets received from the audio encoding sending queue in the second encoding format after the encoding initialization is completed.

[0073] It should be noted that the above steps of the method provided in this embodiment can all be considered as execution steps triggered when a switching instruction from multi-person audio mode to single-person audio mode is received. While adding the first silence data to the audio encoding sending queue through S101, the first audio data obtained from the encoding synchronization management queue can also be added to the audio encoding sending queue through S102. While executing the above steps, the encoding format of the audio encoder can also be switched from the first encoding format to the second encoding format through this step, and the audio data packets passed from the audio encoding sending queue can be encoded using the second encoding format.

[0074] The first encoding format can be considered as the encoding format used to encode audio data in multi-person audio mode, and can be the Opus encoding format. The second encoding format can be considered as the encoding format used to encode audio data in single-person audio mode, and can be the AAC encoding format.

[0075] Meanwhile, it is known that as long as audio data exists in the audio encoding transmission queue, it can be formed into audio data packets in frames and sent to the audio encoder. For the audio encoder, if the encoding format does not change, it can continuously encode the incoming audio data packets in the set encoding format. If the encoding format changes, at the moment of encoding format switching, encoding initialization will first be performed, and then the incoming audio data packets will be encoded in the second encoding format after the switch. The encoding initialization process can be equated to the encoding process of silence data.

[0076] For example, Figure 4 The audio encoder in the audio data processing method provided in this embodiment is given. Figure 3 The image shown is an example of the audio data after encoding the audio data in the audio encoding transmission queue. After switching the encoding format through this step, the audio encoder first performs encoding initialization, as follows... Figure 4 As shown, the encoded audio data generated by the audio encoder first includes the first initial encoded silence data 41 formed during the encoding initialization phase, followed by... Figure 3 The first mute data 31 shown is encoded to form the first mute packet 42, which is then processed by encoding. Figure 3 The first audio data shown is encoded to form the first audio packet 43. Furthermore, the audio encoder will continue to process the first audio data in subsequent encoding processes. Figure 3 The second audio data 33 shown is encoded to form the corresponding encoded audio data.

[0077] This embodiment provides an audio data processing method to avoid the problem of brief stuttering in audio data playback when the audience client switches from multi-person audio mode to single-person audio mode. When switching from multi-person audio mode to single-person audio mode, the first terminal acting as the streaming end adds a segment of silence data to the audio encoding transmission queue, and also re-adds the incompletely encoded audio data from the multi-person audio mode to the audio encoding transmission queue. This silence data fills in the silence data generated during encoding initialization after the audio encoder switches encoding formats. The re-added incompletely encoded audio data can be re-entered into the audio encoder for encoding again in the switched encoding format, thus achieving seamless continuation of multi-person audio data and single-person audio data during mode switching. Through the silence filling and seamless connection of audio data between the two modes in this embodiment, the problem of brief stuttering in audio data that occurs when the audience client switches from multi-person audio mode to single-person audio mode can be better avoided.

[0078] As a first optional embodiment of this example, based on the above optimization, further optimization can be made after encoding the audio data packets received from the audio encoding transmission queue in the second encoding format, including: transmitting the second encoded audio packet output by the audio encoder to the second terminal, wherein the second encoded audio packet contains encoded audio data in the second encoding format.

[0079] It should be noted that the audio encoder also encodes audio data frame by frame, and the encoded audio data will also be output frame by frame. In this embodiment, after switching to single-person audio mode, the audio output data after the audio encoder encodes the incoming audio data packet is recorded as the second encoded audio packet. This second encoded audio packet can be considered as an audio packet formed by the audio encoder using the second encoding format corresponding to single-person audio mode, specifically containing the encoded audio data of the second encoding format.

[0080] In this embodiment, the first terminal can transmit the second encoded audio packet output by the audio encoder to the second terminal. It is understood that after the audio encoder switches to the second encoding format, the initial second encoded audio packet output in the second encoding format can be an encoded silence data packet, subsequent second encoded audio packets can be encoded audio data packets of the first audio data, and still subsequent second encoded audio packets can be encoded audio data packets of the second audio data. The first terminal can transmit the second encoded audio packets sequentially to the second terminal according to the encoding sequence of the audio encoder.

[0081] This embodiment, specifically the first optional embodiment described above, provides a logical implementation for transmitting the second encoded audio packet output by the audio encoder on the first terminal to the second terminal. This technical solution ensures that the viewer client can retrieve the encoded audio data from the first terminal.

[0082] As a second optional embodiment of this example, based on the above optimizations, the method provided in this example can be further optimized to include:

[0083] a1) When a switching instruction from the single-person audio mode to the multi-person audio mode is received, the third audio data is obtained from the encoding synchronization management queue. The third audio data is audio data that has not been encoded in the second encoding format relative to the single-person audio mode.

[0084] It is known that in real-time call applications, single-person audio mode and multi-person audio mode can be switched not only from multi-person audio mode to single-person audio mode, but also from single-person audio mode to multi-person audio mode. This optional embodiment provides the execution logic when switching from single-person audio mode to multi-person audio mode.

[0085] In this embodiment, a corresponding switching command can also be generated when a participant in a real-time call switches from single-person audio to multi-person audio. Upon receiving the switching command, this embodiment can trigger the execution of steps a1) to c1).

[0086] In combination with the above Figure 2 The illustrated live streaming application scenario allows for analysis of the differences between switching from single-person audio mode to multi-person audio mode and switching back to single-person audio mode in a live streaming application. Specifically, when switching from multi-person audio mode to single-person audio mode, since encoding only occurs once on the first terminal in single-person audio mode, and the second terminal does not perform decoding and encoding processing, the delay caused by encoder initialization due to the encoding format switch needs to be compensated for by muting on the first terminal. However, when switching from single-person audio mode to multi-person audio mode, since the audio data in multi-person audio mode is encoded once on the first terminal and then again on the second terminal, and considering that the audio data encoded on the first terminal is also decoded on the second terminal, the delay caused by encoder initialization due to the encoding format switch on the first terminal does not need to be compensated for on the first terminal.

[0087] Based on the above description, this step can directly obtain audio data that has not been encoded in the corresponding second encoding format in the single-person audio mode from the encoding synchronization management queue when a switching instruction from single-person audio mode to multi-person audio mode is received. In this embodiment, the audio data can be recorded as third audio data, and the obtained third audio data can be used for seamless continuation of the corresponding audio data when switching between the two modes.

[0088] b1) Add the third audio data to the audio encoding and sending queue, and add the fourth audio data acquired relative to the multi-person audio mode to the audio encoding and sending queue.

[0089] In this embodiment, the obtained third audio data can be re-added to the head of the audio encoding and sending queue. The audio encoding and sending queue can be cleared during mode switching. By re-adding the third audio data to the audio encoding and sending queue, seamless continuation of audio data corresponding to the two modes when switching from single-person audio mode to multi-person audio mode can be achieved.

[0090] Meanwhile, it is known that when switching from single-person audio mode to multi-person audio mode, the first terminal will also collect the multi-person audio data generated in multi-person audio mode. This step can also add the multi-person audio data to the audio encoding and sending queue normally for transmission to the encoder for encoding processing. In this embodiment, the single-person audio data can be recorded as the fourth audio data.

[0091] Figure 5 Another example diagram illustrating the audio data contained in the audio encoding transmission queue in the audio data processing method provided in this embodiment is given. This allows for the execution of the steps provided in this optional embodiment, when a real-time call application switches from a single-person audio mode to a multi-person audio mode, to ensure that the audio encoding transmission queue contains audio data such as... Figure 5 The audio data content shown. Specifically, as... Figure 5 As shown, the head of the audio encoding and sending queue is added with the third audio data 51 obtained from the encoding synchronization management queue and the fourth audio data 52 acquired from the relatively multi-person audio mode.

[0092] c1) Control the audio encoder to switch from the second encoding format to the first encoding format, and encode the audio data packets received from the audio encoding transmission queue in the first encoding format.

[0093] Based on the above description, it can be understood that the steps described in this second optional embodiment can all be considered as execution steps triggered when a switching instruction from single-person audio mode to multi-person audio mode is received. While adding the third and fourth audio data to the online audio encoding and sending queue through the above steps, the encoding format of the audio encoder can also be switched from the second encoding format to the first encoding format through this step, and the first encoding format can be used to encode the audio data packets passed from the audio encoding and sending queue.

[0094] Similarly, after the audio encoder switches from the second encoding format to the first encoding format, it first performs encoding initialization, and then encodes the incoming audio data packets using the switched first encoding format. Considering that the first encoding format corresponds to a multi-person audio mode, it can be seen that the audio encoder can encode the incoming audio data packets to form encoded audio data packets with the first encoding format.

[0095] For example, Figure 6 The audio encoder in the audio data processing method provided in this embodiment is given. Figure 5 The image shown is an example of the audio data after encoding the audio data in the audio encoding transmission queue. After switching the encoding format through this step, the audio encoder first performs encoding initialization, as follows... Figure 6 As shown, the encoded audio data generated by the audio encoder first includes the second encoded silence data 61 formed during the encoding initialization phase, followed by... Figure 5 The encoded third audio packet 62 is formed by encoding the third audio data shown. Furthermore, the audio encoder will continue to process the data in subsequent encoding processes. Figure 5 The fourth audio data shown is encoded to form the corresponding encoded audio data.

[0096] In the second optional embodiment described above, to avoid the problem of brief pauses in audio data playback when the viewer's client switches from single-person audio mode to multi-person audio mode, the first terminal, acting as the streaming end, will re-add the incompletely encoded audio data from the single-person audio mode to the audio encoding and transmission queue. This re-added, incompletely encoded audio data can then be re-encoded in the audio encoder using the new encoding format, thus achieving seamless transition between single-person and multi-person audio data during mode switching. Through the seamless connection of audio data between the two modes in the above technical solution of this embodiment, the problem of brief audio data pauses that occur when the viewer's client switches from single-person audio mode to multi-person audio mode can be better avoided.

[0097] As a third optional embodiment of this embodiment, based on the second optional embodiment described above, the audio data packets transmitted from the audio encoding transmission queue can be encoded in the first encoding format, and further optimized by: transmitting the first encoded audio packet output by the audio encoder to the second terminal, wherein the first encoded audio packet contains audio data in the first encoding format.

[0098] In this optional embodiment, after switching to multi-person audio mode, the audio output data after the audio encoder encodes the incoming audio data packet can be recorded as the first encoded audio packet. This first encoded audio packet can be considered as an audio packet encoded by the audio encoder in the first encoding format corresponding to the multi-person audio mode, specifically containing the encoded audio data in the first encoding format.

[0099] In this embodiment, the first terminal can transmit the first encoded audio packet output by the audio encoder to the second terminal. It is understood that after the audio encoder switches to the first encoding format, the first encoded audio packet initially output in the first encoding format can be the encoded silence data packet corresponding to the encoding initialization phase. Subsequent first encoded audio packets can be encoded audio packets of the third audio data, and even later, they can be encoded audio packets of the fourth audio data. The first terminal can transmit the first encoded audio packets sequentially to the second terminal according to the encoding timing of the audio encoder.

[0100] The third optional embodiment described above provides a logical implementation for transmitting the first encoded audio packet output by the audio encoder on the first terminal to the second terminal. This optional embodiment also ensures that the viewer client can retrieve the encoded audio data from the first terminal.

[0101] As a fourth optional embodiment of this embodiment, based on the above embodiment, the encoding synchronization management queue can be cleared after obtaining the first audio data from the created encoding synchronization management queue, or after obtaining the third audio data from the encoding synchronization management queue.

[0102] It should be noted that the encoding synchronization management queue can be considered as a queue for synchronizing the audio data being encoded by the incoming audio encoder. That is, the encoding synchronization management queue is mainly used to buffer audio data that has been passed to the audio encoder but has not yet been encoded.

[0103] As described above, during the switching between multi-person audio mode and single-person audio mode in a real-time call application, or between single-person audio mode and multi-person audio mode, audio data that has been passed to the audio encoder but has not yet been encoded by the audio encoder is cleared. Since the aforementioned unencoded audio data also exists synchronously in the encoding synchronization management queue, clearing the unencoded audio data in the encoder also requires clearing the audio data in the encoding synchronization management queue.

[0104] As a fifth optional embodiment of this embodiment, based on the fourth optional embodiment described above, the method provided in this embodiment can be further optimized to include the following steps:

[0105] a2) While pulling audio data packets from the audio encoding transmission queue and transmitting them to the audio encoder, the pulled audio data packets are transmitted to the encoding synchronization management queue.

[0106] In this optional embodiment, this step can be used to synchronize the encoding synchronization management queue with the audio data in the audio encoder. Specifically, audio data packets pulled from the audio encoding transmission queue can be synchronously copied to the encoding synchronization management queue.

[0107] b2) After detecting that the audio encoder has completed encoding the audio data packet, delete the audio data packet from the encoding synchronization management queue.

[0108] Similarly, the encoding synchronization management queue only caches unencoded audio data. This step allows us to delete the audio data cached in the encoding synchronization management queue relative to the encoded audio data packet once the encoded audio data packet is output from the audio encoder.

[0109] The above technical solution in this embodiment provides a synchronous update operation for the encoding synchronization management queue, so as to effectively obtain the remaining unencoded audio data in the encoder through the encoding synchronization management queue, and provide basic data support for subsequent audio data continuation.

[0110] In this specific embodiment, Figure 7 This is a flowchart illustrating an audio data processing method provided in an embodiment of this disclosure. This embodiment is applicable to situations involving the processing of audio data. The method can be executed by an audio data processing device, which can be implemented through software and / or hardware. This device can be configured in a terminal and / or server to implement the audio data processing method in this embodiment, and preferably, the terminal and / or server can be used as a second terminal to execute the audio data processing method provided in this embodiment.

[0111] In this specific embodiment, the method provided in this embodiment can be considered as the response given by the second terminal after the first terminal executes the method provided in the above embodiment.

[0112] Specifically, such as Figure 7 As shown, this embodiment provides an audio data processing method, which may include:

[0113] S701. When the encoding format of the received encoded audio packet is detected to have switched from the first encoding format to the second encoding format, the received encoded audio packet is decoded and the first decoded audio packet is discarded. The receiving, decoding and discarding operations are repeated until the number of discarded first decoded audio packets reaches the set value.

[0114] It can be seen that, through the above Figure 2 The scenario diagram shows that the second terminal needs to re-encode the multi-person audio data uploaded by the first terminal to form an audio data format that can be played on the viewer's client.

[0115] It is also known that when the real-time calling application is in multi-person audio mode, the second terminal can continuously receive encoded multi-person audio data uploaded by the first terminal. This encoded multi-person audio data has the Opus encoding format. The second terminal can decode this Opus-encoded multi-person audio data, then re-encode the decoded multi-person audio data in the AAC encoding format, and then send the encoded multi-person audio data to the content delivery network for the playback terminal to retrieve the audio data.

[0116] Furthermore, it can be known that when the real-time calling application is always in single-person audio mode, the second terminal can continuously receive the encoded single-person audio data uploaded by the first terminal. This encoded single-person audio data has an AAC encoding format, and the second terminal can directly send the received encoded single-person audio data to the content delivery network.

[0117] As described above, when the real-time call application on the first terminal switches from multi-person audio mode to single-person audio mode, the encoding format of the encoded audio data transmitted from the first terminal to the second terminal will also change. Specifically, the second terminal can execute this step and subsequent steps of this embodiment when it detects that the encoding format of the received encoded audio packet has switched from the first encoding format to the second encoding format. Since the first terminal sends the encoded audio data in frames, this embodiment can refer to the encoded audio data sent by the first terminal as an encoded audio packet.

[0118] In this embodiment, the first encoding format can be considered to correspond to the multi-person audio mode, and the second encoding format can be considered to correspond to the single-person audio mode. When the encoded audio data packet of the first encoding format is received to switch to the second encoding format, the encoded audio data packet of the second encoding format can be considered to be the encoded audio data formed by the first terminal after switching from the multi-person audio mode to the single-person audio mode.

[0119] In the existing execution of the second terminal, the encoded audio data packets of the second encoding format are not processed; instead, they are directly sent to the content delivery network. Unlike the existing logic, in this embodiment, the second terminal decodes the received encoded audio packets in this step. The decoded audio data packets can be recorded as the first decoded audio packets, and the discarding operation of the first decoded audio packets can continue. Considering that the second terminal can continuously receive the encoded audio packets sent by the first terminal, the second terminal can continuously decode and discard the received encoded audio packets until the number of discarded first decoded audio packets reaches a set value.

[0120] The purpose of the second terminal performing the above-mentioned operation in this step can be described as follows: the audio data encoded by the audio encoder on the first terminal after switching to the second encoding format includes the first silence data and the first audio data that has not been encoded in the first encoding format. Considering the encoding initialization of the audio encoder when encoding in the second encoding format, the audio encoder will also form a segment of encoded silence data before encoding the first silence data.

[0121] Correspondingly, when the encoding format of the encoded audio data packets received by the second terminal is switched to the second encoding format, it is equivalent to first receiving the encoded silence data packet for encoding initialization and the encoded silence data packet for the first silence data. Therefore, the second terminal can discard the encoded audio packets with the initial set value after the encoding format is switched to the second encoding format. This set value can be determined based on the initialization duration of the audio encoder. Knowing the duration of one frame of audio data packets, this set value can be determined based on the total initialization duration and the duration of a single frame. This set value can be the rounded-up ratio of the initialization duration to the duration of a single frame.

[0122] S702. If the received encoded audio packet is the first encoded audio packet after the discard operation ends, the first encoded audio packet is decoded, and the decoded second decoded audio packet is re-encoded using the second encoding format.

[0123] It is known that the second terminal can continuously receive encoded audio packets. This step can determine whether the received encoded audio packet is the first encoded audio packet after the discard operation ends. If so, the first encoded audio packet can be decoded in this step to obtain the decoded second decoded audio packet. This step can then re-encode the second decoded audio packet using the second encoding format.

[0124] The purpose of the second terminal performing the above-mentioned operation can be described as follows: After the audio encoder on the first terminal switches to the second encoding format and completes the encoding of the silence data, it will continue to encode the first audio data that has not yet completed the encoding of the first encoding format into the second encoding format. To solve the audio playback stuttering problem caused by mode switching on the observation client, the second terminal needs to concatenate the encoded audio data packet corresponding to the first audio data packet with the encoded audio data packet with the first encoding format received before the mode switch.

[0125] As described above, it should be noted that when decoding the encoded audio packet of the second encoding format, it is necessary to use the decoding association data in the previous frame of the encoded audio packet. Therefore, it is necessary to ensure that the last frame of the encoded audio data packet of the first encoding format can include the decoding association data required for decoding the first frame of the encoded audio packet after the first audio data is encoded.

[0126] This step combines the decoded association data from the last coded mute packet to be discarded. First, the first coded audio packet is decoded. After discarding the last coded mute packet, the resulting second decoded audio packet is essentially a continuation of the last frame of decoded audio data with the first encoding format before the mode switch. Therefore, after this continuation, the second decoded audio packet needs to be encoded again to form an encoded audio packet that can be distributed to the content delivery network.

[0127] S703. Obtain the encoded audio packet after secondary encoding and send it to the content distribution network so that it can be distributed to the service client through the content distribution network.

[0128] In this embodiment, the encoded audio packet after the secondary encoding completed in the above steps can be obtained through this step. This encoded audio packet can be regarded as the first encoded audio packet carrying valid audio data after the multi-person audio mode is switched to the single-person audio mode. It can be continued after the last frame encoded audio packet in the multi-person audio mode before the mode switch and sent to the content delivery network. Finally, the business client used for audio data playback can pull the sent audio data from the content delivery network.

[0129] The technical solution described in this embodiment can effectively continue the audio data between the multi-person audio mode before the switch and the single-person audio mode after the switch by discarding the received encoded mute packet and decoding and re-encoding the first encoded audio packet received after the discard. This avoids the problem of brief audio stuttering caused by mode switching on the viewer's client side.

[0130] As an optional embodiment of the above-described scheme in this disclosure, the method provided in this embodiment may further include: if the received encoded audio packet is a subsequent encoded audio packet of the first encoded audio packet, then the subsequent encoded audio packet is directly sent to the content delivery network.

[0131] It is understood that the second terminal continuously receives the encoded audio packets from the first terminal. After completing the continuation of the last frame of encoded audio packets corresponding to the first encoding format and the first non-silent encoded audio packet corresponding to the second encoding format through the above steps, the second terminal can directly send the encoded audio packets (referred to as the subsequent encoded audio packets in this embodiment) after the first encoded audio packet (i.e., the first non-silent encoded audio packet transmitted by the first terminal) to the content delivery network through the execution logic of this optional embodiment, without the need for additional decoding and encoding operations.

[0132] For example, Figure 8 A schematic diagram illustrating the logical implementation of the audio data processing method provided in this embodiment when the encoding format of the encoded audio packet received by the second terminal is switched from the first encoding format to the second encoding format is given. For example... Figure 8 As shown, assuming that the first three frames after switching from the first encoding format to the second encoding format are determined to be encoded mute data packets (including the first initial mute encoded data and the encoded first mute packet), and the fourth frame is the first encoded audio packet corresponding to the first audio data (multi-person audio data in multi-person audio mode that has not been encoded in the first encoding format), then the first three frames, which are encoded mute data packets, can be discarded. Decoding the first encoded audio packet of the fourth frame after encoding can achieve effective continuation with the last frame of the multi-person audio mode's live chat encoded audio packet. Furthermore, if... Figure 8 As shown, starting from the 5th received frame, no decoding and re-encoding is required; the data can be directly sent to the content delivery network.

[0133] The above-described technical solution in this embodiment enables the normal transmission of single-person audio data to the audience client after switching to single-person audio mode.

[0134] Similarly, as another optional embodiment of the above-described solution in this disclosure, the method provided in this embodiment may further include:

[0135] a3) When the encoding format of the received encoded audio packet is detected to have switched from the second encoding format to the first encoding format, the received encoded audio packet is decoded and the second silence data in the third decoded audio packet formed after decoding is discarded.

[0136] In this optional embodiment, the scenario can be considered as the real-time call application on the first terminal switching from a single-person audio mode to a multi-person audio mode. Therefore, the encoding format of the encoded audio data transmitted from the first terminal to the second terminal will also change. Specifically, the second terminal can trigger the execution of the method steps provided in this optional embodiment when it detects that the encoding format of the received encoded audio packet has switched from the second encoding format to the first encoding format.

[0137] Specifically, the second terminal can operate on the encoded audio packet formed by the first terminal in the first encoding format after switching to multi-person audio mode through this step. First, it can decode the encoded audio packet and know whether the decoded audio packet (referred to as the third decoded audio packet in this embodiment) contains silence data (referred to as the second silence data in this embodiment), and can discard the contained second silence data.

[0138] The reason why silence data exists in the decoded audio packet is that the audio encoder of the first terminal also needs to perform encoding initialization before encoding in the first encoding format. The silence generated during the encoding initialization process can be encoded as encoded silence data and included in the encoded audio packet for transmission to the second terminal. Therefore, the second terminal can discard the second silence data through this step.

[0139] b3) The third decoded audio packet is supplemented with the third silence data of the second set size to form a silence supplemented audio packet, and the audio encoder is controlled to perform secondary encoding on the silence supplemented audio packet in the second encoding format.

[0140] As described above, the second terminal needs to re-encode the received audio packet with the first encoding format using the second encoding format. After discarding the second silence data through the above steps, the received first encoded audio packet essentially contains the encoded audio data of the third audio data (single-person audio data that was not encoded using the second encoding format when the first terminal switched to multi-person audio mode). In this embodiment, this first encoded audio packet needs to be concatenated with the encoded audio packet of the last frame of live data before switching to multi-person audio mode.

[0141] Since the received encoded audio packet after the second mute data is discarded needs to be encoded again in the second encoding format, and the audio encoder also has encoding initialization problems, this step is needed to solve the encoding mute packet problem caused by encoding initialization before continuing the encoding audio packet of the first encoded audio packet and the last frame of live data before switching to multi-person audio mode.

[0142] Specifically, this step can use silence data of a second predetermined size to supplement the third decoded audio packet determined in the previous steps. The second predetermined size can also be determined based on the encoding initialization duration of the audio encoder on the second terminal under the second encoding format, ensuring that the silence data of the second predetermined size can be used to form a complete audio data packet with the silence data generated during encoding initialization. This step denotes the third decoded audio packet with the supplemented silence data as the silence supplement audio packet. This silence supplement audio packet can then be encoded again using this step.

[0143] Furthermore, as another optional embodiment of the above-described solution in this disclosure, the method provided in this embodiment may also include:

[0144] a4) Obtain the secondary encoded audio packet output by the audio encoder, and determine whether the secondary encoded audio packet is a silent audio packet. If yes, proceed to step b4); otherwise, proceed to step c4).

[0145] It is known that the audio encoder of the second terminal can perform secondary encoding on the aforementioned silence-filled audio packet, and it can also perform secondary encoding on the third decoded audio packet after the second terminal has completed decoding. During the secondary encoding process, the third silence data in the silence-filled audio packet can combine with the silence data generated during the encoding initialization phase to form a complete silence audio packet.

[0146] This step can detect the secondary encoded audio packet output by the audio encoder to determine whether the secondary encoded audio packet is a silent audio packet. If it is, the silent audio packet can be discarded through step b4) below; if it is not, it can be distributed to the content delivery network through step c4) below.

[0147] b4) Discard the silence data packet and return to the operation of obtaining the secondary encoded audio packet.

[0148] It is understood that the audio encoder continuously outputs secondary encoded audio packets. This optional embodiment can repeatedly execute the secondary encoded audio packets and determine and discard silent audio packets as well as send non-silent packets.

[0149] c4) Send the secondary encoded audio packet to the content delivery network and return to perform the secondary encoded audio packet acquisition operation.

[0150] Understandably, after all silent audio packets are discarded, the first secondary encoded audio packet sent can seamlessly continue from the last encoded audio packet in single-person audio mode before switching to multi-person audio mode.

[0151] For example, Figure 9A schematic diagram illustrating the logical implementation of the audio data processing method provided in this embodiment when the encoding format of the encoded audio packet received by the second terminal is switched from the second encoding format to the first encoding format is given. For example... Figure 9 As shown, the second terminal decodes the received encoded audio packet 91, forming a third decoded audio packet 92. If there is silence data in the third decoded audio packet 92, the silence data is discarded, and then the third silence data is added to form silence supplement audio data 93. The silence supplement audio data 93 is encoded, including the third silence data included after encoding and the silence data with the initial encoding delay to form a complete silence packet and non-silent secondary encoded audio data 94. The secondary encoded audio data 94 realizes the continuation of the last frame encoded audio packet 95 in single-person audio mode in terms of timing.

[0152] The technical solution described in this embodiment can effectively continue the audio data between the single-person audio mode before switching and the multi-person audio mode after switching by discarding the received encoded mute packet, supplementing the mute data, and discarding the mute audio packet formed after secondary encoding. This avoids the problem of brief audio stuttering caused by mode switching on the audience's client side.

[0153] Figure 10 This is a schematic diagram of an audio data processing device provided in an embodiment of the present disclosure. This embodiment is applicable to the processing of audio data. The device can be implemented by software and / or hardware, and can be configured in a terminal and / or server as a first terminal to implement the audio data processing method in this embodiment of the present disclosure. Specifically, the device may include: a first switching module 101, a first buffer module 102, and a first encoding module 103.

[0154] The first switching module 101 is used to add a first set size of first silence data to the created audio encoding sending queue when it receives a switching instruction from multi-person audio mode to single-person audio mode.

[0155] The first cache module 102 is used to obtain first audio data from the created encoding synchronization management queue, add the first audio data to the audio encoding sending queue, and add second audio data collected relative to the single-person audio mode to the audio encoding sending queue. The first audio data is audio data that has not been encoded in the first encoding format relative to the multi-person audio mode.

[0156] The first encoding module 103 is used to control the audio encoder to switch from the first encoding format to the second encoding format, and to encode the audio data packets passed from the audio encoding sending queue in the second encoding format after the encoding initialization is completed.

[0157] This embodiment provides an audio data processing device to avoid the problem of brief stuttering in audio data playback when the audience client switches from multi-person audio mode to single-person audio mode. When this happens, the first terminal, acting as the streaming end, adds a segment of silence data to the audio encoding transmission queue, and also re-adds the incompletely encoded audio data from the multi-person audio mode to the audio encoding transmission queue. This silence data fills in the silence data generated during encoding initialization after the audio encoder switches encoding formats. The re-added incompletely encoded audio data can then be re-entered into the audio encoder for encoding in the new encoding format, thus achieving seamless continuation between multi-person and single-person audio data during mode switching. Through the silence filling and seamless connection of audio data between the two modes in this embodiment, the problem of brief stuttering in audio data that occurs when the audience client switches from multi-person audio mode to single-person audio mode can be better avoided.

[0158] Furthermore, the device also includes: a first transmission module, configured to transmit a second encoded audio packet output by the audio encoder to a second terminal after encoding the audio data packet received from the audio encoding transmission queue in a second encoding format, wherein the second encoded audio packet contains encoded audio data in the second encoding format.

[0159] Furthermore, the device also includes:

[0160] The second receiving module is used to obtain third audio data from the encoding synchronization management queue when it receives a switching instruction from the single-person audio mode to the multi-person audio mode. The third audio data is audio data that has not been encoded in the second encoding format relative to the single-person audio mode.

[0161] The second caching module is used to add the third audio data to the audio encoding and sending queue, and to add the fourth audio data acquired relative to the multi-person audio mode to the audio encoding and sending queue.

[0162] The second encoding module is used to control the audio encoder to switch from the second encoding format to the first encoding format, and to encode the audio data packets received from the audio encoding transmission queue in the first encoding format.

[0163] Furthermore, the device may also include: a second transmission module, configured to transmit a first encoded audio packet output by the audio encoder to a second terminal after encoding the audio data packet received from the audio encoding transmission queue in the first encoding format, wherein the first encoded audio packet contains audio data in the first encoding format.

[0164] Furthermore, the device may also include a clearing module for clearing the encoding synchronization management queue after obtaining first audio data from the created encoding synchronization management queue, or after obtaining third audio data from the encoding synchronization management queue.

[0165] Furthermore, the device may also include: a queue management module, used to simultaneously pull audio data packets from the audio encoding transmission queue and transmit them to the audio encoder, and to transmit the pulled audio data packets to the encoding synchronization management queue; and to delete the audio data packets from the encoding synchronization management queue after detecting that the audio encoder has completed encoding the audio data packets.

[0166] The above-described apparatus can execute the methods provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects for executing the methods.

[0167] Figure 11 This is a schematic diagram of an audio data processing device provided in an embodiment of the present disclosure. This embodiment is applicable to the processing of audio data. The device can be implemented by software and / or hardware, and can be configured in a terminal and / or server as a second terminal to implement the audio data processing method in this embodiment of the present disclosure. Specifically, the device may include: a first packet loss module 111, a first encoding / decoding module 112, and a first sending module 113.

[0168] The first packet loss module 111 is used to decode the received encoded audio packet when the encoding format of the received encoded audio packet is switched from the first encoding format to the second encoding format, and discard the first decoded audio packet after decoding. The receiving, decoding and discarding operations are repeated until the number of discarded first decoded audio packets reaches a set value.

[0169] The first encoding / decoding module 112 is used to decode the first encoded audio packet when the received encoded audio packet is the first encoded audio packet after the discard operation ends, and then encode the decoded second decoded audio packet again in the second encoding format.

[0170] The first sending module 113 is used to obtain the encoded audio packet after secondary encoding and send it to the content distribution network so that it can be distributed to the service client through the content distribution network.

[0171] This embodiment provides an audio data processing device that can effectively continue audio data between the multi-person audio mode before switching and the single-person audio mode after switching by discarding the received encoded mute packet and decoding and re-encoding the first encoded audio packet received after the discard. This avoids the problem of brief audio stuttering caused by mode switching on the viewer's client side.

[0172] Furthermore, the device also includes a second transmitting module, configured to directly transmit the subsequently encoded audio packet to the content delivery network if the received encoded audio packet is a subsequent encoded audio packet of the first encoded audio packet.

[0173] Furthermore, the device may also include:

[0174] The second packet loss module is used to detect when the encoding format of the received encoded audio packet is switched from the second encoding format to the first encoding format, decode the received encoded audio packet, and discard the second silence data in the third decoded audio packet formed after decoding.

[0175] The second encoding / decoding module is used to supplement the third decoded audio packet with third silence data of a second set size to form a silence supplemented audio packet, and to control the audio encoder to perform secondary encoding on the silence supplemented audio packet in the second encoding format.

[0176] Furthermore, the device may also include: a determination execution module, configured to acquire the secondary encoded audio packet output by the audio encoder, and determine whether the secondary encoded audio packet is a silent audio packet; if so, discard the silent data packet and return to the acquisition operation of the secondary encoded audio packet; if not, send the secondary encoded audio packet to the content delivery network and return to the acquisition operation of the secondary encoded audio packet.

[0177] It is worth noting that the various units and modules included in the above-mentioned device are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be realized; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the protection scope of the embodiments of this disclosure.

[0178] Figure 12 This is a schematic diagram of the structure of a computer device provided in an embodiment of this disclosure. Reference is made below. Figure 12 It illustrates a computer device suitable for implementing embodiments of the present disclosure (e.g., Figure 12 The diagram below shows the structure of the terminal device or server 120. The terminal device in this embodiment may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), and vehicle terminals (e.g., vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 12 The computer device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0179] like Figure 12As shown, the computer device 120 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 121, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 122 or a program loaded from a storage device 128 into a random access memory (RAM) 1212. The RAM 1212 also stores various programs and data required for the operation of the computer device 120. The processing unit 121, ROM 122, and RAM 1212 are interconnected via a bus 125. An edit / output (I / O) interface 124 is also connected to the bus 125.

[0180] Typically, the following devices can be connected to I / O interface 124: input devices 1212 including, for example, a touchscreen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; output devices 127 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 128 including, for example, magnetic tape, hard disk, etc.; and communication devices 129. Communication device 129 allows computer device 120 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 12 A computer device 120 with various devices is shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have instead.

[0181] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 129, or installed from storage device 128, or installed from ROM 122. When the computer program is executed by processing device 121, it performs the functions defined in the methods of embodiments of this disclosure.

[0182] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0183] The computer device provided in this embodiment and the audio data processing method provided in the above embodiments belong to the same inventive concept. Technical details not described in detail in this embodiment can be found in the above embodiments, and this embodiment has the same beneficial effects as the above embodiments.

[0184] This disclosure provides a computer storage medium storing a computer program that, when executed by a processor, implements the audio data processing method provided in the above embodiments.

[0185] It should be noted that the computer-readable medium described above in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0186] In this disclosure, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0187] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0188] The aforementioned computer-readable medium may be included in the aforementioned computer device; or it may exist independently and not assembled into the computer device.

[0189] The aforementioned computer-readable medium carries one or more programs that, when executed by the computer device, cause the computer device to:

[0190] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0191] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0192] The units described in the embodiments of this disclosure can be implemented in software or in hardware. The name of a unit does not necessarily limit the unit itself; for example, the first acquisition unit can also be described as "a unit that acquires at least two Internet Protocol addresses".

[0193] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0194] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0195] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0196] Furthermore, although the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while some specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0197] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. An audio data processing method, characterized in that, Applied to the first terminal, including: When a switching instruction from multi-person audio mode to single-person audio mode is received, a first set size of first silence data is added to the created audio encoding sending queue; The system retrieves first audio data from the created encoding synchronization management queue, adds the first audio data to the audio encoding sending queue, and adds second audio data acquired relative to the single-person audio mode to the audio encoding sending queue. The first audio data is audio data that has not been encoded in the first encoding format relative to the multi-person audio mode. The audio encoder is controlled to switch from the first encoding format to the second encoding format, and after the encoding initialization is completed, the audio data packets received from the audio encoding transmission queue are encoded in the second encoding format.

2. The method according to claim 1, characterized in that, After encoding the audio data packets received from the audio encoding transmission queue in the second encoding format, the method further includes: The second encoded audio packet output by the audio encoder is transmitted to the second terminal, and the second encoded audio packet contains encoded audio data in the second encoding format.

3. The method according to claim 1, characterized in that, Also includes: When a switching instruction from the single-person audio mode to the multi-person audio mode is received, third audio data is obtained from the encoding synchronization management queue. The third audio data is audio data that has not been encoded in the second encoding format relative to the single-person audio mode. The third audio data is added to the audio encoding and sending queue, and the fourth audio data acquired relative to the multi-person audio mode is added to the audio encoding and sending queue; The audio encoder is controlled to switch from the second encoding format to the first encoding format, and the audio data packets received from the audio encoding transmission queue are encoded using the first encoding format.

4. The method according to claim 3, characterized in that, After encoding the audio data packets received from the audio encoding transmission queue in the first encoding format, the method further includes: The first encoded audio packet output by the audio encoder is transmitted to the second terminal, and the first encoded audio packet contains audio data in the first encoding format.

5. The method according to claim 1 or 3, characterized in that, After obtaining the first audio data from the created encoding synchronization management queue, or after obtaining the third audio data from the encoding synchronization management queue, the method further includes: Clear the encoding synchronization management queue.

6. The method according to claim 5, characterized in that, Also includes: While pulling audio data packets from the audio encoding transmission queue and transmitting them to the audio encoder, the pulled audio data packets are also transmitted to the encoding synchronization management queue; After detecting that the audio encoder has completed encoding the audio data packet, the audio data packet is deleted from the encoding synchronization management queue.

7. An audio data processing method, characterized in that, Applied to the second terminal, including: When the encoding format of the received encoded audio packet is detected to have switched from the first encoding format to the second encoding format, the received encoded audio packet is decoded and the first decoded audio packet is discarded. The receiving, decoding and discarding operations are repeated until the number of discarded first decoded audio packets reaches the set value. If the received encoded audio packet is the first encoded audio packet after the discard operation ends, then the first encoded audio packet is decoded, and the decoded second decoded audio packet is re-encoded using the second encoding format; The encoded audio package after secondary encoding is obtained and sent to the content distribution network for distribution to the business client.

8. The method according to claim 7, characterized in that, Also includes: If the received encoded audio packet is a subsequent encoded audio packet of the first encoded audio packet, then the subsequent encoded audio packet is directly sent to the content delivery network.

9. The method according to claim 7, characterized in that, Also includes: When the encoding format of the received encoded audio packet is detected to have switched from the second encoding format to the first encoding format, the received encoded audio packet is decoded and the second silence data in the third decoded audio packet formed after decoding is discarded. The third decoded audio packet is supplemented with third silence data of a second set size to form a silence supplemented audio packet, and the audio encoder is controlled to perform secondary encoding on the silence supplemented audio packet in the second encoding format.

10. The method according to claim 9, characterized in that, Also includes: Obtain the secondary encoded audio packet output by the audio encoder, and determine whether the secondary encoded audio packet is a silent audio packet; If so, discard the silence data packet and return to the operation of obtaining the secondary encoded audio packet; If not, the secondary encoded audio packet is sent to the content delivery network, and the process of obtaining the secondary encoded audio packet is returned.

11. An audio data processing apparatus, characterized in that, Configured on the first terminal, including: The first switching module is used to add a first set size of first mute data to the created audio encoding sending queue when it receives a switching instruction from multi-person audio mode to single-person audio mode; The first caching module is used to obtain first audio data from the created encoding synchronization management queue, add the first audio data to the audio encoding sending queue, and add second audio data collected relative to the single-person audio mode to the audio encoding sending queue. The first audio data is audio data that has not been encoded in the first encoding format relative to the multi-person audio mode. The first encoding module is used to control the audio encoder to switch from the first encoding format to the second encoding format, and to encode the audio data packets received from the audio encoding sending queue in the second encoding format after the encoding initialization is completed.

12. An audio data processing device, characterized in that, Configured on the second terminal, including: The first packet loss module is used to decode the received encoded audio packet when the encoding format of the received encoded audio packet is switched from the first encoding format to the second encoding format, and discard the first decoded audio packet after decoding. The receiving, decoding and discarding operations are repeated until the number of discarded first decoded audio packets reaches a set value. The first encoding / decoding module is used to decode the first encoded audio packet when the received encoded audio packet is the first encoded audio packet after the discard operation ends, and then encode the decoded second decoded audio packet again using the second encoding format. The first sending module is used to obtain the encoded audio packet after secondary encoding and send it to the content distribution network so that it can be distributed to the service client through the content distribution network.

13. A computer device, characterized in that, The computer device includes: One or more processors; a storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the audio data processing method as described in any one of claims 1-10.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the audio data processing method as described in any one of claims 1-10.

15. A computer program product comprising a computer program that, when executed by a processor, implements the audio data processing method according to any one of claims 1-10.

Images