Audio coding method, device, equipment and medium

Abstract

This disclosure provides an audio encoding method, apparatus, device, and medium, relating to the field of computer science. The audio encoding method includes: determining the number of users currently watching a live stream; if the number of users is greater than or equal to a first user number threshold and less than a second user number threshold, determining the device load of the live streaming terminal; if the device load is greater than the device load threshold, determining a first audio type of the live audio according to a first audio encoding mode, and a first target encoding bitrate corresponding to the first audio type, and encoding the live audio using the first target encoding bitrate. During the audio encoding process, the audio encoding quality can be improved while reducing the waste of bandwidth resources.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to an audio encoding method, an audio encoding apparatus, an electronic device, and a computer-readable storage medium. Background Technology

[0002] With the maturity of internet technology, live streaming has become a common media tool. In a live streaming scenario, users can watch events unfolding in real time through their devices. Typically, the audio in a live stream can include three categories: music, speech, and background noise.

[0003] In related technologies, different bitrates can usually be selected to encode different types of audio data. However, if the encoding bitrate is determined solely based on the audio type, the encoding of audio data will still result in wasted bandwidth resources, or insufficient bandwidth allocation will lead to live streaming stuttering, delays, and other issues, affecting the user's live streaming viewing experience. Summary of the Invention

[0004] This disclosure provides an audio encoding method, apparatus, device, and medium that can improve the audio encoding quality of live streaming in different live streaming scenarios, ensuring the normal operation of the live streaming terminal while reducing the waste of bandwidth resources. The technical solution of this disclosure is as follows:

[0005] According to a first aspect of the present disclosure, an audio encoding method is provided, comprising:

[0006] Determine the number of user accounts currently watching the live stream;

[0007] If the number of user accounts is greater than or equal to the first user account number threshold and less than the second user account number threshold, then the device load of the live streaming terminal is determined.

[0008] If the device load is greater than the device load threshold, the first audio type of the live audio is determined according to the first audio encoding mode, and the first target encoding bitrate corresponding to the first audio type is determined, and the live audio is encoded using the first target encoding bitrate.

[0009] Optionally, the method further includes:

[0010] If the number of user accounts is less than the first user account number threshold, then the device load of the live streaming terminal is determined.

[0011] If the device load is less than or equal to the device load threshold, then the first audio type of the live audio and the first target encoding bitrate corresponding to the first audio type are determined according to the first audio encoding mode, and the live audio is encoded using the first target encoding bitrate.

[0012] Optionally, determining the first audio type of the live audio according to the first audio encoding mode, and the first target encoding bitrate corresponding to the first audio type, includes:

[0013] Get the first currently live audio segment of the first preset duration;

[0014] The first current live audio segment is input into a pre-trained audio type recognition model for audio type recognition to obtain the first audio type of the first current live audio segment;

[0015] In a pre-established first correspondence table between audio type and encoding bitrate, determine the first target encoding bitrate corresponding to the first audio type;

[0016] Encoding the live audio using the first target encoding bitrate includes:

[0017] The next live audio segment of the first preset duration is encoded using the first target encoding bitrate.

[0018] Optionally, the method further includes:

[0019] If the device load is greater than the device load threshold, then the second audio type of the live audio is determined according to the second audio encoding mode, and the second target encoding bitrate corresponding to the second audio type is determined, and the live audio is encoded using the second target encoding bitrate.

[0020] Optionally, determining the second audio type of the live audio according to the second audio encoding mode, and the second target encoding bitrate corresponding to the second audio type, includes:

[0021] The energy of the current frame's live audio is compared with the energy of the background noise to determine the second audio type of the current frame's live audio.

[0022] In a pre-established second correspondence table between audio type and coding bitrate, determine the second target coding bitrate corresponding to the second audio type;

[0023] Encoding the live audio using the second target coding bitrate includes:

[0024] The current frame of live audio is encoded using the second target encoding bitrate.

[0025] Optionally, the method further includes:

[0026] If the device load is less than or equal to the device load threshold, then the third audio type of the live audio is determined according to the third audio encoding mode, and the third target encoding bitrate corresponding to the third audio type is determined, and the live audio is encoded using the third target encoding bitrate.

[0027] Optionally, determining the third audio type of the live audio according to the third audio encoding mode, and the third target encoding bitrate corresponding to the third audio type, includes:

[0028] Obtain the second currently live audio segment with a second preset duration;

[0029] The second current live audio segment is input into a pre-trained audio type recognition model for audio type recognition to obtain the fourth audio type of the second current live audio segment;

[0030] In the next live audio segment of the second preset duration, the energy of the first frame of live audio is compared with the background noise energy to obtain the fifth audio type of the first frame of live audio in the next live audio segment;

[0031] Based on the fourth audio type and the fifth audio type, determine the third audio type of the first frame of live audio;

[0032] In a pre-established third correspondence table between audio types and coding bitrates, a third target coding bitrate corresponding to the third audio type is determined;

[0033] Encoding the live audio using the third target coding bitrate includes:

[0034] The third target encoding bitrate is used to encode the next live audio segment of a preset duration.

[0035] Optionally, the method further includes:

[0036] If the number of user accounts is greater than or equal to the second user account number threshold, the live audio is encoded according to the fourth target encoding bitrate.

[0037] According to a second aspect of the present disclosure, an audio encoding apparatus is provided, comprising:

[0038] The first determining module is configured to determine the number of user accounts currently watching the live stream;

[0039] The second determining module is configured to determine the device load of the live streaming terminal if the number of user accounts is greater than or equal to the first user account number threshold and less than the second user account number threshold.

[0040] The first encoding module is configured to determine the first audio type of the live audio and the first target encoding bitrate corresponding to the first audio type according to the first audio encoding mode if the device load is greater than the device load threshold, and to encode the live audio using the first target encoding bitrate.

[0041] Optionally, the device further includes a second encoding module configured to:

[0042] If the number of user accounts is less than the first user account number threshold, then the device load of the live streaming terminal is determined.

[0043] If the device load is less than or equal to the device load threshold, then the first audio type of the live audio and the first target encoding bitrate corresponding to the first audio type are determined according to the first audio encoding mode, and the live audio is encoded using the first target encoding bitrate.

[0044] Optionally, the first encoding module or the second encoding module is configured as follows:

[0045] Get the first currently live audio segment of the first preset duration;

[0046] The first current live audio segment is input into a pre-trained audio type recognition model for audio type recognition to obtain the first audio type of the first current live audio segment;

[0047] In a pre-established first correspondence table between audio type and encoding bitrate, determine the first target encoding bitrate corresponding to the first audio type;

[0048] Encoding the live audio using the first target encoding bitrate includes:

[0049] The next live audio segment of the first preset duration is encoded using the first target encoding bitrate.

[0050] Optionally, the device further includes a third encoding module configured to:

[0051] If the device load is greater than the device load threshold, then the second audio type of the live audio is determined according to the second audio encoding mode, and the second target encoding bitrate corresponding to the second audio type is determined, and the live audio is encoded using the second target encoding bitrate.

[0052] Optionally, the third encoding module is configured as follows:

[0053] The energy of the current frame's live audio is compared with the energy of the background noise to determine the second audio type of the current frame's live audio.

[0054] In a pre-established second correspondence table between audio type and coding bitrate, determine the second target coding bitrate corresponding to the second audio type;

[0055] Encoding the live audio using the second target coding bitrate includes:

[0056] The current frame of live audio is encoded using the second target encoding bitrate.

[0057] Optionally, the device further includes a fourth encoding module configured to:

[0058] If the device load is less than or equal to the device load threshold, then the third audio type of the live audio is determined according to the third audio encoding mode, and the third target encoding bitrate corresponding to the third audio type is determined, and the live audio is encoded using the third target encoding bitrate.

[0059] Optionally, the fourth encoding module is configured as follows:

[0060] Obtain the second currently live audio segment with a second preset duration;

[0061] The second current live audio segment is input into a pre-trained audio type recognition model for audio type recognition to obtain the fourth audio type of the second current live audio segment;

[0062] The energy of the first frame of live audio in the next live audio segment of a preset duration is compared with the energy of the background noise to obtain the fifth audio type of the first frame of live audio in the next live audio segment.

[0063] Based on the fourth audio type and the fifth audio type, determine the third audio type of the first frame of live audio;

[0064] In a pre-established third correspondence table between audio types and coding bitrates, a third target coding bitrate corresponding to the third audio type is determined;

[0065] Encoding the live audio using the third target coding bitrate includes:

[0066] The third target encoding bitrate is used to encode the next live audio segment of a preset duration.

[0067] Optionally, the device further includes a fifth encoding module, configured to:

[0068] If the number of user accounts is greater than or equal to the second user account number threshold, the live audio is encoded according to the fourth target encoding bitrate.

[0069] According to a third aspect of the present disclosure, an electronic device is provided, comprising:

[0070] processor;

[0071] Memory used to store the processor's executable instructions;

[0072] The processor is configured to execute the instructions to implement the audio encoding method as described in the first aspect.

[0073] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided that, when instructions in the computer-readable storage medium are executed by a processor of an electronic device, enables the electronic device to perform the audio encoding method as described in the first aspect.

[0074] According to a fifth aspect of the present disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the audio encoding method as described in the first aspect.

[0075] The technical solutions provided by the embodiments of this disclosure have at least the following beneficial effects:

[0076] The audio encoding method, apparatus, device, and medium disclosed herein can, on the one hand, select an audio encoding mode that will not affect the normal operation of the live streaming terminal under the actual situation of the current live streaming scenario, based on the number of users and the device load, and determine the encoding bitrate for the live audio. This can improve the encoding quality of the live audio while ensuring the normal operation of the live streaming terminal. On the other hand, in the process of determining the encoding bitrate for encoding the live audio, the audio model needs to first determine the audio type of the live audio and select the encoding bitrate applicable to the audio type of the live audio. This can save bandwidth resources and prevent the waste of bandwidth resources while ensuring the normal operation of the live streaming terminal and improving the encoding quality of the live audio.

[0077] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0078] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0079] Figure 1 This is a schematic diagram illustrating the structure of an audio encoding system according to an exemplary embodiment;

[0080] Figure 2This is a flowchart illustrating an audio encoding method according to an exemplary embodiment;

[0081] Figure 3 This is a flowchart illustrating a method for determining a third target coding rate according to an exemplary embodiment;

[0082] Figure 4 This is a flowchart illustrating another audio encoding method according to an exemplary embodiment;

[0083] Figure 5 This is a block diagram illustrating an audio encoding apparatus according to an exemplary embodiment;

[0084] Figure 6 This is a block diagram illustrating an electronic device according to an exemplary embodiment. Detailed Implementation

[0085] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0086] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0087] In related technologies, different types of audio data can be encoded at different bitrates in live streaming scenarios. For example, for speech audio data, the encoding bitrate corresponding to the speech audio data is selected for encoding; for background noise data, the encoding bitrate corresponding to the background noise data is selected for encoding.

[0088] However, in determining the encoding bitrate of audio data, only considering the type of audio data to select an appropriate encoding bitrate can lead to wasted bandwidth resources during the encoding process, or insufficient bandwidth resource allocation can cause live streaming stuttering, delays, and other issues, affecting the user's live streaming viewing experience.

[0089] In view of the above problems, the exemplary embodiments of this disclosure provide an audio encoding method. The application scenarios of this audio encoding method include, but are not limited to: during a live webcast, determining the number of user accounts currently watching the live stream; if the number of user accounts is greater than or equal to a first user account number threshold and less than a second user account number threshold, determining the device load of the live streaming terminal; if the device load is greater than the device load threshold, determining a first audio type of the live stream audio according to a first audio encoding mode, and a first target encoding bitrate corresponding to the first audio type, and encoding the live stream audio using the first target encoding bitrate. By combining the number of user accounts watching the live stream, the device load of the live streaming terminal, and the type of the live stream audio, a target encoding bitrate for encoding the live stream audio can be selected. This can improve the adaptability of the determined target encoding bitrate to the current actual situation in the live stream, reducing bandwidth resource waste while improving the encoding quality of the live stream audio and enhancing the user's live stream viewing experience.

[0090] To implement the above-described text processing method, an exemplary embodiment of this disclosure provides an audio encoding system. Figure 1 A schematic architecture diagram of the audio encoding system is shown. Figure 1 As shown, the audio encoding system 100 may include a server 110, a live streaming terminal 120, and a user terminal 130. The server 110 is a backend server deployed by the online live streaming service provider; the live streaming terminal 120 is the terminal device used by users performing live streaming; and the user terminal 130 is the terminal device corresponding to the user account watching the live stream. More specifically, the terminal device may be a smartphone, personal computer, tablet computer, etc. The server 110 establishes connections with the live streaming terminal 120 and the user terminal 130 via a network.

[0091] It should be understood that server 110 can be a single server or a cluster of multiple servers. This disclosure does not limit the specific architecture of server 110.

[0092] In one alternative implementation, the live streaming terminal 120 may send a live streaming room establishment request to the server 110 in response to the live streaming initiation operation.

[0093] Server 110 can respond to a live streaming room creation request to create a live streaming room for live streaming terminal 120 and send live streaming data to user terminal 130 in real time;

[0094] User terminal 130 can respond to the live streaming data sent by the server and watch the live streaming content.

[0095] During the live broadcast, the live broadcast terminal 120 can determine the number of user accounts currently watching the live broadcast. If the number of user accounts is greater than or equal to a first user account number threshold and less than a second user account number threshold, the device load of the live broadcast terminal is determined. If the device load is greater than the device load threshold, the first audio type of the live broadcast audio and the first target encoding bitrate corresponding to the first audio type are determined according to the first audio encoding mode, and the live broadcast audio is encoded using the first target encoding bitrate.

[0096] Figure 2 This is a flowchart illustrating an audio encoding method according to an exemplary embodiment, such as... Figure 2 As shown, the audio encoding method can be applied to a live streaming terminal, including steps S201 to S203:

[0097] Step S201: Determine the number of user accounts currently watching the live stream;

[0098] Step S202: If the number of user accounts is greater than or equal to the first user account number threshold and less than the second user account number threshold, then determine the device load of the live streaming terminal.

[0099] Step S203: If the device load is greater than the device load threshold, the first audio type of the live audio and the first target encoding bitrate corresponding to the first audio type are determined according to the first audio encoding mode, and the live audio is encoded using the first target encoding bitrate.

[0100] In summary, the audio encoding method provided in this disclosure has two advantages. First, it can select an audio encoding mode that will not affect the normal operation of the live streaming terminal based on the actual situation of the current live streaming scenario, such as the number of user accounts and device load, and determine the encoding bitrate for the live audio. This can improve the encoding quality of the live audio while ensuring the normal operation of the live streaming terminal. Second, in the process of determining the encoding bitrate for encoding the live audio, the audio model needs to first determine the audio type of the live audio and select the encoding bitrate applicable to the audio type. This can save bandwidth resources and prevent the waste of bandwidth resources while ensuring the normal operation of the live streaming terminal and improving the encoding quality of the live audio.

[0101] The following is about Figure 2 The specific steps in the process will be explained in detail.

[0102] In step S201 above, the live streaming terminal can determine the number of user accounts currently watching the live stream.

[0103] In this embodiment of the disclosure, after determining its live streaming status, the live streaming terminal can determine the number of user accounts currently watching the live stream in real time, so as to select the encoding bitrate to encode the live audio based on the number of user accounts.

[0104] In one optional implementation, the process by which the live streaming terminal determines the number of user accounts currently watching the live stream may include: in response to the live streaming terminal being in a live streaming state, determining the number of user accounts in the live stream at the current moment, and obtaining the number of user accounts currently watching the live stream.

[0105] In step S202 above, if the number of user accounts is greater than or equal to the first user account number threshold and less than the second user account number threshold, then the device load of the live streaming terminal is determined.

[0106] In this embodiment, multiple audio encoding modes are provided to determine the target encoding bitrate for encoding live audio. However, the hardware resource consumption and the compatibility between the determined target encoding bitrate and the live audio differ in the process of determining the target encoding bitrate for each audio encoding mode. Considering the actual situation such as the number of user accounts currently watching the live stream and the device load of the live streaming terminal, in order to ensure the quality of live audio encoding, improve the user's live streaming viewing experience, and reduce bandwidth waste, the audio encoding mode that best guarantees the quality of live audio encoding and improves the user's live streaming viewing experience can be selected from multiple audio encoding modes for live streaming scenarios with different numbers of user accounts and device loads.

[0107] It should be noted that, in this embodiment of the disclosure, the live streaming scenario may include the following five types. The first type of live streaming scenario is that the number of user accounts currently watching the live stream is at a high level, that is, the number of user accounts currently watching the live stream is greater than or equal to the second user account number threshold.

[0108] The second live streaming scenario is that the number of user accounts currently watching the live stream is at a medium level, and the device load of the live streaming terminal is high. That is, the number of user accounts currently watching the live stream is greater than or equal to the first user account number threshold and less than the second user account number threshold, and the device load is greater than the device load threshold.

[0109] The third live streaming scenario is: the number of user accounts currently watching the live stream is at a low level, and the device load of the live streaming terminal is low, that is, the number of user accounts currently watching the live stream is less than the first user account number threshold, and the device load is less than or equal to the device load threshold.

[0110] The fourth live streaming scenario is: the number of user accounts currently watching the live stream is at a low level, and the device load of the live streaming terminal is high, that is, the number of user accounts currently watching the live stream is less than the first user account number threshold, and the device load is greater than the device load threshold.

[0111] The fifth live streaming scenario is: the number of user accounts currently watching the live stream is at a medium level, and the device load of the live streaming terminal is low. That is, the number of user accounts currently watching the live stream is greater than or equal to the first user account number threshold and less than the second user account number threshold, and the device load is less than or equal to the device load threshold.

[0112] It is understood that the first and second user account number thresholds can be determined based on actual needs, and this disclosure does not limit this. For example, the first user account number threshold is 5000, and the second user account number threshold is 10000. Device load can refer to the usage of hardware resources of the live streaming terminal, such as CPU utilization and memory usage. The device load threshold can also be determined based on actual needs, and this disclosure does not limit this.

[0113] In this embodiment, the audio encoding mode can include four types. The first audio encoding mode is based on a pre-trained audio type recognition model. This mode consumes moderate amounts of hardware resources from the live streaming terminal and produces moderate audio quality. The second audio encoding mode is based on audio frame processing. This mode consumes the second least amount of hardware resources from the live streaming terminal and produces the lowest audio quality. The third audio encoding mode is based on both a pre-trained audio type recognition model and audio frame processing. This mode consumes the most hardware resources from the live streaming terminal and produces the second highest audio quality. The fourth audio encoding mode consumes the least amount of hardware resources from the live streaming terminal and is considered a high-quality encoding mode, but it consumes a large amount of bandwidth resources and produces the highest audio quality among the four modes.

[0114] Understandably, among the four audio encoding modes, the encoding quality from highest to lowest is as follows: the fourth audio encoding mode is higher than the third audio encoding mode, the third audio encoding mode is higher than the first audio encoding mode, and the first audio encoding mode is higher than the second audio encoding mode. During the operation of the four audio encoding modes, the hardware resource consumption of the live streaming terminal from highest to lowest is as follows: the third audio encoding mode is higher than the first audio encoding mode, the first audio encoding mode is higher than the second audio encoding mode, and the second audio encoding mode is higher than the fourth audio encoding mode.

[0115] Therefore, based on five live streaming scenarios and four audio encoding modes, a correspondence table between live streaming scenarios and audio encoding modes can be established in advance. In the first live streaming scenario, since the number of user accounts is relatively high, the encoding quality is the main consideration. Four audio encoding modes can be selected to determine the encoding bitrate of the live audio and encode the live audio.

[0116] In the second live streaming scenario, when selecting the audio encoding mode, since the number of user accounts watching the live stream is at a moderate level, a certain level of audio encoding quality is required. At the same time, due to the high device load, there are not many hardware resources reserved for audio encoding. Therefore, it is advisable to choose an audio encoding mode with moderate encoding quality and moderate hardware resource consumption of the live streaming terminal. In this case, the first audio encoding mode can be selected.

[0117] In the third live streaming scenario, when selecting the audio encoding mode, since the device load of the live streaming terminal is relatively small, an encoding mode that consumes more hardware resources of the live streaming terminal can be selected. However, since the number of user accounts watching the live stream is at a low level, there is no need for excessively high encoding quality, so the first audio encoding mode can be selected.

[0118] In the fourth live streaming scenario, when selecting the audio encoding mode: Since the live streaming terminal has a high device load, an encoding mode that consumes less hardware resources of the live streaming terminal can be selected. However, since the number of user accounts watching the live stream is at a low level, there is no need for excessively high encoding quality, so the second audio encoding mode can be selected.

[0119] In the fifth live streaming scenario, when selecting an audio encoding mode, since the number of user accounts watching the live stream is at a moderate level, a certain level of audio encoding quality is required. At the same time, since the device load is relatively low, there are still many hardware resources reserved for audio encoding. Therefore, it is advisable to choose an audio encoding mode with higher encoding quality, even if it consumes more hardware resources for the live streaming terminal. In this case, the third audio encoding mode can be selected.

[0120] In one alternative implementation, if the live streaming terminal determines that the number of user accounts is greater than or equal to a first user account number threshold and less than a second user account number threshold, it can be determined that there is no need to use a high-quality encoding mode to encode the live audio, and the device load of the live streaming terminal can be further determined.

[0121] In step S203 above, if the live streaming terminal determines that the device load is greater than the device load threshold, it determines the first audio type of the live audio and the first target encoding bitrate corresponding to the first audio type according to the first audio encoding mode, and encodes the live audio using the first target encoding bitrate.

[0122] In this embodiment of the disclosure, if the number of user accounts is greater than or equal to the first user account number threshold and less than the second user account number threshold, and the device load is greater than the device load threshold, it indicates that the number of user accounts currently watching the live broadcast is moderate, but the load of the live broadcast terminal is high. In this case, it can be determined that an audio encoding mode with moderate hardware resource consumption and moderate encoding quality should be selected, namely the first audio encoding mode.

[0123] In one optional implementation, the process by which the live streaming terminal determines the first audio type of the live audio and the first target encoding bitrate corresponding to the first audio type according to the first audio encoding mode may include: obtaining a first current live audio segment of a first preset duration; inputting the first current live audio segment into a pre-trained audio type recognition model for audio type recognition to obtain the first audio type of the first current live audio segment; determining the first target encoding bitrate corresponding to the first audio type in a pre-established first correspondence table between audio types and encoding bitrates; and encoding the live audio using the first target encoding bitrate, including: encoding the next live audio segment of the first preset duration using the first target encoding bitrate. The first preset duration can be determined based on actual needs, and this embodiment does not limit it. For example, the first preset duration may be 1 second, 2 seconds, or 3 seconds. In live streaming scenarios with a moderate number of user accounts but high device load, an audio encoding mode with moderate encoding quality and moderate hardware resource consumption can be selected. This involves using a pre-trained audio type recognition model to determine the encoding bitrate for live audio encoding. This can improve encoding quality, save bandwidth resources, and prevent excessive pursuit of high-quality encoding from putting pressure on the hardware resources of the live streaming terminal, which could lead to abnormal operation of the live streaming terminal.

[0124] Table 1 shows a pre-established first correspondence table between audio type and encoding bitrate, which is used to represent the encoding bitrate corresponding to different channels when the first audio type is music audio, speech audio and background noise, in kbps.

[0125] Based on Table 1, the process of determining the first target coding bitrate corresponding to the first audio type may include: if the first audio type is music audio, the first target coding bitrate is 64kbps in stereo audio mode and 56kbps in mono audio mode; if the first audio type is speech audio, the first target coding bitrate is 48kbps in stereo audio mode and 40kbps in mono audio mode; if the first audio type is background noise, the first target coding bitrate is 32kbps in stereo audio mode and 32kbps in mono audio mode.

[0126] Table 1

[0127] Music Audio Voice audio Background noise dual channel 64 48 32 Mono 56 40 32

[0128] In one optional implementation, if the live streaming terminal determines that the number of user accounts currently watching the live stream is less than a first user account number threshold, then the device load of the live streaming terminal is determined; if the device load is determined to be less than or equal to the device load threshold, it can be determined that the number of user accounts currently watching the live stream is small, and the load of the live streaming terminal is also small. It is necessary to select an audio encoding mode with moderate hardware resource consumption and moderate encoding quality. Then, the first audio type of the live audio and the first target encoding bitrate corresponding to the first audio type are determined according to the first audio encoding mode, and the live audio is encoded using the first target encoding bitrate.

[0129] The process of determining the first audio type of the live audio according to the first audio encoding mode, the first target encoding bitrate corresponding to the first audio type, and encoding the live audio using the first target encoding bitrate can be referred to the above embodiments, and will not be elaborated on in this embodiment. In live streaming scenarios with a small number of user accounts and low device load, an audio encoding mode with medium encoding quality and medium hardware resource consumption can be selected, and the encoding bitrate can be determined for live audio encoding. This can ensure encoding quality, save bandwidth resources, and reduce excessive consumption of hardware resources even if the live streaming terminal has sufficient hardware resources.

[0130] In one optional implementation, when the live streaming terminal determines that the number of user accounts is less than a first user account number threshold, if it also determines that the device load is greater than a device load threshold, it can be determined that the number of user accounts currently watching the live stream is small, but the load on the live streaming terminal is high. An audio encoding mode that consumes less hardware resources and can balance encoding quality can be selected. Then, a second audio type and a second target encoding bitrate corresponding to the second audio type can be determined according to the second audio encoding mode, and the live audio can be encoded using the second target encoding bitrate. In live streaming scenarios where the number of user accounts is small but the device load is high, selecting an audio encoding mode with certain encoding quality and low hardware resource consumption to determine the encoding bitrate for live audio encoding can prevent pressure on the normal operation of the live streaming terminal when hardware resources are insufficient. This ensures the normal operation of the live stream while improving the encoding quality of the live audio and saving bandwidth resources.

[0131] The process of determining the second audio type and the corresponding second target encoding bitrate of the live audio according to the second audio encoding mode includes: comparing the energy of the current frame's live audio with the background noise energy to determine the second audio type of the current frame's live audio; determining the second target encoding bitrate corresponding to the second audio type in a pre-established second correspondence table of audio types and encoding bitrates; and encoding the live audio using the second target encoding bitrate, which includes encoding the current frame's live audio using the second target encoding bitrate. The background noise energy can be determined based on a noise estimation algorithm, such as the minimum value tracking method. In live streaming scenarios with a small number of user accounts but a high device load, selecting the audio frame processing method to determine the live audio type and corresponding encoding bitrate for live audio encoding can ensure encoding quality, save bandwidth resources, and determine the encoding bitrate for live audio encoding with relatively low hardware resource consumption, preventing abnormal operation of the live streaming terminal.

[0132] In this embodiment of the disclosure, based on the encoding mode of audio frame processing, music audio and speech audio are classified into the same audio type. The process of comparing the energy of the current frame live audio with the background noise energy to determine the second audio type of the current frame live audio may include: determining the energy ratio of the current frame live audio energy to the background noise energy; if the energy ratio is greater than the energy ratio threshold, the second audio type of the current frame live audio is determined to be music / speech audio; if the energy ratio is less than or equal to the energy ratio threshold, the second audio type of the current frame live audio is determined to be background noise.

[0133] Table 2 shows a pre-established second correspondence table between audio type and encoding bitrate. The second correspondence table is used to represent the encoding bitrate corresponding to different channels when the first audio type is music speech audio and background noise, in kbps.

[0134] Table 2

[0135] Music, voice, and audio Background noise dual channel 60 32 Mono 56 32

[0136] In Table 2, if the second audio type is music / speech audio, the second target coding bitrate is 60kbps in stereo audio mode and 56kbps in mono audio mode; if the second audio type is background noise, the second target coding bitrate is 32kbps in stereo audio mode and 32kbps in mono audio mode.

[0137] In one optional implementation, when the indicator terminal determines that the number of user accounts is greater than or equal to a first user account number threshold and less than a second user account number threshold, if the device load of the live streaming terminal is determined to be less than or equal to a device load threshold, it can be determined that the number of user accounts currently watching the live stream is moderate and the load of the live streaming terminal is low. Therefore, an audio encoding mode that consumes more hardware resources but improves encoding quality can be selected. Then, a third audio type for the live audio and a third target encoding bitrate corresponding to the third audio type can be determined according to the third audio encoding mode, and the live audio can be encoded using the third target encoding bitrate. In live streaming scenarios with a moderate number of user accounts but low device load, selecting an audio encoding mode that consumes more hardware resources but improves encoding quality can fully utilize the hardware resources of the live streaming terminal, ensuring the normal operation of the live streaming terminal while further improving encoding quality and saving bandwidth resources.

[0138] Among them, such as Figure 3 As shown, the process of determining the third audio type of the live audio according to the third audio encoding mode, and the third target encoding bitrate corresponding to the third audio type, may include steps S301 to S304:

[0139] Step S301: Obtain the second current live audio segment with a second preset duration;

[0140] The second preset duration can be determined based on actual needs, and this embodiment does not limit it. For example, the second preset duration can be 1 second, 2 seconds, or 3 seconds.

[0141] Step S302: Input the second current live audio segment into the pre-trained audio type recognition model to identify the audio type and obtain the fourth audio type of the second current live audio segment.

[0142] Step S303: Compare the energy of the first frame of live audio in the next live audio segment of the second preset duration with the background noise energy to obtain the fifth audio type of the first frame of live audio in the next live audio segment.

[0143] In step S303, the process of comparing the energy of the first frame of live audio in the next live audio segment of a preset duration with the background noise energy to obtain the fifth audio type of the first frame of live audio in the next live audio segment can refer to the process of comparing the energy of the current frame of live audio with the background noise energy to determine the second audio type of the current frame of live audio in the above embodiment. This disclosure embodiment will not elaborate on this.

[0144] Step S304: Determine the third audio type of the first frame of live audio based on the fourth and fifth audio types;

[0145] In step S304, the process of determining the third audio type of the first frame of live audio based on the fourth and fifth audio types may include: determining the third audio type of the first frame of live audio based on the fourth and fifth audio types in a pre-established audio type classification table.

[0146] Table 3 shows a pre-established audio type classification table, and the second correspondence table is used to represent the combination relationship for determining the third audio type based on the combination pattern. Based on Table 3, the process of determining the third audio type of the first frame of live audio according to the fourth and fifth audio types may include: if the fourth audio type is music audio and the fifth audio type is either music audio or speech audio, then the third audio type can be determined to be music audio; if the fourth audio type is music audio and the fifth audio type is background noise, then the third audio type can be determined to be music audio; other combination results can be referred to Table 3, and are not listed here.

[0147] Table 3

[0148]

[0149] Step S305: In the pre-established third correspondence table between audio type and coding bitrate, determine the third target coding bitrate corresponding to the third audio type;

[0150] If the third correspondence table is the same as the first correspondence table, then in step S305, the process of determining the third target coding bitrate corresponding to the third audio type in the pre-established first correspondence table of audio type and coding bitrate may include: determining the third target coding bitrate corresponding to the third audio type based on Table 1; wherein, if the third audio type is music audio, the third target coding bitrate is 64kbps in stereo audio mode and 56kbps in mono audio mode; if the third audio type is speech audio, the third target coding bitrate is 48kbps in stereo audio mode and 40kbps in mono audio mode; if the third audio type is background noise, the third target coding bitrate is 32kbps in stereo audio mode and 32kbps in mono audio mode.

[0151] In one optional implementation, the process of encoding live audio using a third target encoding bitrate by the live streaming terminal may include: encoding the next live audio segment of a preset duration using the third target encoding bitrate. In live streaming scenarios with a moderate number of user accounts but low device load, selecting a combined audio encoding mode—which consumes significant hardware resources but improves encoding quality—can fully utilize the hardware resources of the live streaming terminal, ensuring its normal operation while further improving encoding quality and saving bandwidth resources.

[0152] In one optional implementation, if the live streaming terminal determines that the number of user accounts is greater than or equal to a second user account number threshold, indicating that the number of user accounts currently watching the live stream is at a high level, it selects the audio encoding mode with the highest encoding quality, and then encodes the live audio using the fourth audio encoding mode, followed by encoding the live audio according to the fourth target encoding bitrate. The fourth target encoding bitrate can be determined based on actual needs, and this embodiment does not limit it; for example, the fourth target encoding bitrate can be 64kbps. When there are a large number of user accounts, the audio encoding mode with the highest encoding quality but the lowest hardware resource consumption can be selected, prioritizing the encoding quality of the live audio while ensuring the normal operation of the live streaming terminal, thus guaranteeing the live streaming viewing experience for user accounts.

[0153] Figure 4 This is a flowchart illustrating an audio encoding method according to an exemplary embodiment, such as... Figure 4 As shown, the audio encoding method can be applied to a live streaming terminal, including steps S401 to S409:

[0154] Step S401: Determine the number of user accounts currently watching the live stream, and determine whether the number of user accounts is greater than or equal to the second user account number threshold.

[0155] Step S402: If the number of user accounts is greater than or equal to the second user account number threshold, then the live audio is encoded according to the fourth target encoding bitrate.

[0156] Step S403: If the number of user accounts is not greater than or equal to the second user account number threshold, then determine whether the number of user accounts is greater than or equal to the first user account number threshold and less than the second user account number threshold.

[0157] Step S404: If the number of user accounts is greater than or equal to the first user account number threshold and less than the second user account number threshold, then determine whether the device load of the live streaming terminal is greater than the device load threshold.

[0158] Step S405: If the device load is greater than the device load threshold, then determine the first audio type of the live audio and the first target encoding bitrate corresponding to the first audio type according to the first audio encoding mode, and encode the live audio using the first target encoding bitrate.

[0159] Step S406: If the device load is less than or equal to the device load threshold, the third audio type of the live audio and the third target encoding bitrate corresponding to the third audio type are determined according to the third audio encoding mode, and the live audio is encoded using the third target encoding bitrate.

[0160] Step S407: If the number of user accounts is less than the first user account number threshold, then determine whether the device load of the live streaming terminal is greater than the device load threshold.

[0161] Step S408: If the device load is less than or equal to the device load threshold, then determine the first audio type of the live audio and the first target encoding bitrate corresponding to the first audio type according to the first audio encoding mode, and encode the live audio using the first target encoding bitrate.

[0162] Step S409: If the device load is greater than the device load threshold, then determine the second audio type of the live audio according to the second audio encoding mode, and the second target encoding bitrate corresponding to the second audio type, and encode the live audio using the second target encoding bitrate.

[0163] Figure 5 This is a block diagram illustrating an audio encoding apparatus according to an exemplary embodiment. (Refer to...) Figure 5 The audio encoding device 500 includes:

[0164] The first determining module 501 is configured to determine the number of user accounts currently watching the live stream;

[0165] The second determining module 502 is configured to determine the device load of the live streaming terminal if the number of user accounts is greater than or equal to the first user account number threshold and less than the second user account number threshold.

[0166] The first encoding module 503 is configured to determine the first audio type of the live audio and the first target encoding bitrate corresponding to the first audio type according to the first audio encoding mode if the device load is greater than the device load threshold, and encode the live audio using the first target encoding bitrate.

[0167] Optional, such as Figure 5 As shown, the audio encoding device 500 also includes a second encoding module 504, configured as follows:

[0168] If the number of user accounts is less than the first threshold for the number of user accounts, then the device load of the live streaming terminal is determined.

[0169] If the device load is less than or equal to the device load threshold, the first audio type of the live audio is determined according to the first audio encoding mode, and the first target encoding bitrate corresponding to the first audio type is determined, and the live audio is encoded using the first target encoding bitrate.

[0170] Optionally, either the first encoding module 503 or the second encoding module 504 is configured as follows:

[0171] Get the first currently live audio segment of the first preset duration;

[0172] The first current live audio segment is input into a pre-trained audio type recognition model to identify the audio type and obtain the first audio type of the first current live audio segment.

[0173] In a pre-established first correspondence table between audio type and coding bitrate, determine the first target coding bitrate corresponding to the first audio type;

[0174] Encoding the live audio using the first target coding bitrate includes:

[0175] The next live audio segment of the first preset duration is encoded using the first target encoding bitrate.

[0176] Optional, such as Figure 5 As shown, the audio encoding device 500 also includes a third encoding module 505, configured as follows:

[0177] If the device load exceeds the device load threshold, the second audio type of the live audio is determined according to the second audio encoding mode, and the second target encoding bitrate corresponding to the second audio type is determined, and the live audio is encoded using the second target encoding bitrate.

[0178] Optionally, the third encoding module 505 is configured as follows:

[0179] The energy of the current frame's live audio is compared with the energy of the background noise to determine the second audio type of the current frame's live audio.

[0180] In a pre-established second correspondence table between audio type and coding bitrate, determine the second target coding bitrate corresponding to the second audio type;

[0181] Encoding live audio using a second target coding bitrate includes:

[0182] The live audio of the current frame is encoded using the second target encoding bitrate.

[0183] Optional, such as Figure 5 As shown, the audio encoding device 500 also includes a fourth encoding module 506, configured as follows:

[0184] If the device load is less than or equal to the device load threshold, the third audio type of the live audio is determined according to the third audio encoding mode, and the third target encoding bitrate corresponding to the third audio type is determined, and the live audio is encoded using the third target encoding bitrate.

[0185] Optionally, the fourth encoding module 506 is configured as follows:

[0186] Obtain the second currently live audio segment with a second preset duration;

[0187] The second current live audio segment is input into a pre-trained audio type recognition model for audio type recognition to obtain the fourth audio type of the second current live audio segment;

[0188] The energy of the first frame of live audio in the next live audio segment of the preset duration is compared with the energy of the background noise to obtain the fifth audio type of the first frame of live audio in the next live audio segment.

[0189] Based on the fourth and fifth audio types, determine the third audio type of the first frame of live audio;

[0190] In the pre-established third correspondence table between audio type and coding bitrate, determine the third target coding bitrate corresponding to the third audio type;

[0191] Encoding live audio using a third target coding bitrate includes:

[0192] The next live audio segment of the preset duration is encoded using the third target encoding bitrate.

[0193] Optional, such as Figure 5 As shown, the audio encoding device 500 also includes a fifth encoding module 507, configured as follows:

[0194] If the number of user accounts is greater than or equal to the second threshold for the number of user accounts, the live audio will be encoded according to the fourth target encoding bitrate.

[0195] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0196] Exemplary embodiments of this disclosure also provide an electronic device, which may be a live streaming terminal. References below... Figure 6 This electronic device will be described. It should be understood that the following references... Figure 6 This electronic device will be described. It should be understood that... Figure 6 The electronic device 600 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.

[0197] like Figure 6 As shown, the electronic device 600 is presented in the form of a general-purpose computing device. The components of the electronic device 600 may include, but are not limited to: at least one processing unit 610, at least one storage unit 620, and a bus 630 connecting different system components (including storage unit 620 and processing unit 610).

[0198] The storage unit stores program code, which can be executed by the processing unit 610 to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of the present invention. For example, the processing unit 610 can perform actions such as... Figures 2 to 4 The methods and steps shown are as follows.

[0199] Storage unit 620 may include volatile storage units, such as random access memory (RAM) 621 and / or cache memory 622, and may further include read-only memory (ROM) 623.

[0200] Storage unit 620 may also include a program / utility 624 having a set (at least one) of program modules 625, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0201] Bus 630 may include a data bus, an address bus, and a control bus.

[0202] Electronic device 600 can also communicate with one or more external devices 700 (e.g., keyboards, pointing devices, Bluetooth devices, etc.) via input / output (I / O) interface 640. Electronic device 600 can also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via network adapter 680. As shown, network adapter 680 communicates with other modules of electronic device 600 via bus 630. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0203] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to exemplary embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0204] Those skilled in the art will understand that various aspects of this disclosure can be implemented as systems, methods, or program products. Therefore, various aspects of this disclosure can be embodied in entirely hardware implementations, entirely software implementations (including firmware, microcode, etc.), or implementations combining hardware and software aspects, collectively referred to herein as “circuit,” “module,” or “system.” Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

[0205] In addition, this disclosure also provides a computer-readable storage medium that, when the instructions in the computer-readable storage medium are executed by the processor of an electronic device, enables the electronic device to perform the audio encoding method provided in the above embodiments.

[0206] In addition, this disclosure also provides a computer program product, including computer instructions, which, when executed on an electronic device, cause the electronic device to perform the audio encoding method provided in the above embodiments.

[0207] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0208] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. An audio encoding method, characterized in that, include: Determine the number of user accounts currently watching the live stream; If the number of user accounts is greater than or equal to the first user account number threshold and less than the second user account number threshold, then the device load of the live streaming terminal is determined. If the device load is greater than the device load threshold, the first audio type of the live audio and the first target encoding bitrate corresponding to the first audio type are determined according to the first audio encoding mode, and the live audio is encoded using the first target encoding bitrate. The first audio encoding mode is an encoding mode based on a pre-trained audio type recognition model. If the number of user accounts is less than the first user account number threshold, then the device load of the live streaming terminal is determined. If the device load is less than or equal to the device load threshold, then the first audio type of the live audio and the first target encoding bitrate corresponding to the first audio type are determined according to the first audio encoding mode, and the live audio is encoded using the first target encoding bitrate. The first audio encoding mode is determined through the following steps: Obtain a pre-established correspondence table between live streaming scenarios and audio encoding modes. The live streaming scenarios are determined based on the number of user accounts and the device load of the live streaming terminal, and the audio encoding modes are determined based on hardware resource consumption and audio encoding quality. A query operation is performed based on the correspondence table between the live streaming scene and the audio encoding mode, and the first audio encoding mode is taken as the audio encoding mode that matches the current live streaming scene. The step of determining the first audio type of the live audio according to the first audio encoding mode, and the first target encoding bitrate corresponding to the first audio type, includes: Get the first currently live audio segment of the first preset duration; The first current live audio segment is input into a pre-trained audio type recognition model for audio type recognition to obtain the first audio type of the first current live audio segment; In a pre-established first correspondence table between audio type and encoding bitrate, determine the first target encoding bitrate corresponding to the first audio type; Encoding the live audio using the first target encoding bitrate includes: The next live audio segment of the first preset duration is encoded using the first target encoding bitrate.

2. The audio encoding method according to claim 1, characterized in that, The method further includes: If the device load is greater than the device load threshold, then the second audio type of the live audio and the second target encoding bitrate corresponding to the second audio type are determined according to the second audio encoding mode, and the live audio is encoded using the second target encoding bitrate. The second audio encoding mode is an encoding mode based on audio frame processing. The step of determining the second audio type of the live audio according to the second audio encoding mode, and the second target encoding bitrate corresponding to the second audio type, includes: The energy of the current frame's live audio is compared with the energy of the background noise to determine the second audio type of the current frame's live audio. In a pre-established second correspondence table between audio type and coding bitrate, determine the second target coding bitrate corresponding to the second audio type; Encoding the live audio using the second target coding bitrate includes: The current frame of live audio is encoded using the second target encoding bitrate.

3. The audio encoding method according to claim 1, characterized in that, The method further includes: If the device load is less than or equal to the device load threshold, the third audio type of the live audio and the third target coding bitrate corresponding to the third audio type are determined according to the third audio coding mode, and the live audio is encoded using the third target coding bitrate. The third audio coding mode is based on a pre-trained audio type recognition model and an audio frame processing coding mode. The step of determining the third audio type of the live audio according to the third audio encoding mode, and the third target encoding bitrate corresponding to the third audio type, includes: Obtain the second currently live audio segment with a second preset duration; The second current live audio segment is input into a pre-trained audio type recognition model for audio type recognition to obtain the fourth audio type of the second current live audio segment; In the next live audio segment of the second preset duration, the energy of the first frame of live audio is compared with the background noise energy to obtain the fifth audio type of the first frame of live audio in the next live audio segment; Based on the fourth audio type and the fifth audio type, determine the third audio type of the first frame of live audio; In a pre-established third correspondence table between audio types and coding bitrates, a third target coding bitrate corresponding to the third audio type is determined; Encoding the live audio using the third target coding bitrate includes: The third target encoding bitrate is used to encode the next live audio segment of a preset duration.

4. The audio encoding method according to claim 1, characterized in that, The method further includes: If the number of user accounts is greater than or equal to the second user account number threshold, the live audio is encoded according to the fourth target encoding bitrate, which is determined by performing a query operation on the correspondence table between the live scene and the audio encoding mode.

5. An audio encoding device, characterized in that, include: The first determining module is configured to determine the number of user accounts currently watching the live stream; The second determining module is configured to determine the device load of the live streaming terminal if the number of user accounts is greater than or equal to the first user account number threshold and less than the second user account number threshold. The first encoding module is configured to determine the first audio type of the live audio and the first target encoding bitrate corresponding to the first audio type according to the first audio encoding mode if the device load is greater than the device load threshold, and encode the live audio using the first target encoding bitrate. The second encoding module is configured to determine the device load of the live streaming terminal if the number of user accounts is less than the first user account number threshold. If the device load is less than or equal to the device load threshold, then the first audio type of the live audio and the first target encoding bitrate corresponding to the first audio type are determined according to the first audio encoding mode, and the live audio is encoded using the first target encoding bitrate. The first encoding module is further configured to: acquire a first current live audio segment of a first preset duration; input the first current live audio segment into a pre-trained audio type recognition model for audio type recognition to obtain a first audio type of the first current live audio segment; determine a first target encoding bitrate corresponding to the first audio type in a pre-established first correspondence table between audio type and encoding bitrate; and encode the next live audio segment of the first preset duration using the first target encoding bitrate. The first audio encoding mode is determined through the following steps: obtaining a pre-established correspondence table between live streaming scenarios and audio encoding modes, wherein the live streaming scenario is determined based on the number of user accounts and the device load of the live streaming terminal, and the audio encoding mode is determined based on hardware resource consumption and audio encoding quality; performing a query operation based on the correspondence table between live streaming scenarios and audio encoding modes, and using the first audio encoding mode as the audio encoding mode that matches the current live streaming scenario.

6. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the audio encoding method as described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is able to perform the audio encoding method as described in any one of claims 1 to 4.

Images