A voice processing method, apparatus, device, storage medium, and program product

Abstract

The present application discloses a voice processing method, apparatus, device, storage medium and program product. When the method is applied to a voice sending terminal, it includes: predicting the total uplink bandwidth; based on the total uplink bandwidth, performing bitrate allocation on the voice stream to be sent to determine at least two allocated bitrates, and sending the at least two allocated bitrates to the server; receiving the reference bitrates corresponding to the respective allocated bitrates sent by the server, where the reference bitrates are obtained by the server according to the received allocated bitrates and the estimated downlink bandwidth; based on the respective allocated bitrates and the corresponding reference bitrates, determining the target bitrates corresponding to the respective allocated bitrates; respectively encoding the voice stream using the target bitrates, and sending the encoded voice stream to the server, which can make full use of the network information of the uplink and downlink, thereby improving the accuracy of traffic control. And it can avoid using mixing for flow control in the server, thereby reducing the project complexity.

Description

Technical Field

[0001] The present application relates to the technical field of data processing, and in particular, to a voice processing method, a voice processing device, a voice processing device, a computer-readable storage medium, and a computer program product. Background Art

[0002] In order to achieve long-distance transmission of images, usually the sending device needs to encode the images to generate bitstream data, and then the sending device transmits the generated bitstream data to the receiving device to save the required transmission bandwidth.

[0003] During a voice call, in order to balance the voice service quality, the audio quality will be improved while ensuring smoothness and latency as much as possible. In the related art, the server can obtain the actual network condition of the receiving party, and then perform mixing processing at different bitrates according to the actual network condition to obtain mixed streams at different bitrates and distribute them to receiving parties in different network states. This solution has many advantages, but there are also some disadvantages. For example, the mixing process itself has a certain degree of complexity, software decoding and encoding will cause a certain delay and consume a large amount of CPU overhead, and as the number of senders, the number of receivers, and the network complexity increase, the more voice streams need to be decoded and mixed, the greater the CPU overhead on the server side. In order to balance performance and efficiency, hardware encoding and decoding are often required for acceleration. Summary of the Invention

[0004] The present application provides a voice processing method, device, equipment, storage medium, and program product to solve the problems in the prior art that when different bitrate voice streams need to be sent, mixing processing in the server leads to complex projects and large CPU overhead on the server.

[0005] According to a first aspect of the present application, there is provided a voice processing method, which is applied to a voice sending terminal, and the method includes:

[0006] Predict the total uplink bandwidth;

[0007] Based on the total uplink bandwidth, perform bitrate allocation on the voice stream to be sent to determine at least two allocated bitrates, and send the at least two allocated bitrates to the server;

[0008] Receive the reference bitrates corresponding to the allocated bitrates sent by the server, where the reference bitrates are obtained by the server according to the received allocated bitrates and the estimated downlink bandwidth;

[0009] Based on each allocated bitrate and the corresponding reference bitrate, determine the target bitrate corresponding to each allocated bitrate;

[0010] Encode the voice stream using the respective target bitrates, and send the encoded voice stream to the server.

[0011] According to a second aspect of the present application, there is provided a voice processing method, which is applied to a server. The method includes:

[0012] Receive at least two allocated bitrates for a voice stream to be sent from a voice sending terminal;

[0013] Determine a voice receiving terminal that will receive the voice stream, and predict the downstream bandwidth of each voice receiving terminal;

[0014] Based on the downstream bandwidth of each voice receiving terminal and the at least two allocated bitrates, determine the reference bitrates corresponding to the respective allocated bitrates;

[0015] Send the reference bitrates to the voice sending terminal, so that the voice sending terminal determines the target bitrates of the allocated bitrates according to the reference bitrates and the corresponding allocated bitrates, and encodes the voice stream using the target bitrates.

[0016] According to a third aspect of the present application, there is provided a voice processing device, which is arranged in a voice sending terminal. The device includes:

[0017] An uplink bandwidth prediction module, configured to predict the total uplink bandwidth;

[0018] A bitrate allocation module, configured to allocate bitrates to a voice stream to be sent based on the total uplink bandwidth, so as to determine at least two allocated bitrates, and send the at least two allocated bitrates to the server;

[0019] A reference bitrate receiving module, configured to receive the reference bitrates corresponding to the respective allocated bitrates sent by the server, where the reference bitrates are obtained by the server according to the received allocated bitrates and the estimated downstream bandwidth;

[0020] A target bitrate determination module, configured to determine the target bitrates corresponding to the respective allocated bitrates based on the respective allocated bitrates and the corresponding reference bitrates;

[0021] An encoding module, configured to encode the voice stream using the respective target bitrates, and send the encoded voice stream to the server.

[0022] According to a fourth aspect of the present application, there is provided a voice processing device, which is arranged in a server. The device includes:

[0023] An allocated bitrate receiving module, configured to receive at least two allocated bitrates for a voice stream to be sent from a voice sending terminal;

[0024] A downlink bandwidth prediction module, configured to determine a voice receiving terminal that receives the voice stream, and predict the downlink bandwidth of each voice receiving terminal;

[0025] A reference code rate determination module, configured to determine a reference code rate corresponding to each allocation code rate according to the downlink bandwidth of each voice receiving terminal and the at least two allocation code rates;

[0026] A reference code rate sending module, configured to send the reference code rate to the voice sending terminal, so that the voice sending terminal determines a target code rate of the allocation code rate according to the reference code rate and the corresponding allocation code rate, and encodes the voice stream using the target code rate.

[0027] According to a fifth aspect of the present application, there is provided a voice processing device, where the voice processing device includes:

[0028] At least one processor; and

[0029] A memory communicatively connected to the at least one processor; wherein,

[0030] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor, so that the at least one processor can execute the method described in the first aspect or the second aspect above.

[0031] According to a sixth aspect of the present application, there is provided a computer-readable storage medium, where the computer-readable storage medium stores computer instructions, and the computer instructions are used to implement the method described in any embodiment of the present application when executed by a processor.

[0032] According to a seventh aspect of the present application, there is provided a computer program product, where the computer program product includes computer-executable instructions, and the computer-executable instructions are used to implement the method described in any embodiment of the present application when executed.

[0033] In this embodiment, the voice sending terminal predicts the total uplink bandwidth, and performs code rate allocation on the voice stream to be sent based on the total uplink bandwidth to determine at least two allocation code rates. Then, the at least two allocation code rates are sent to the server, and the reference code rates corresponding to the allocation code rates sent by the server are received. When determining the reference code rate, the server not only considers the allocation code rate of the voice sending terminal, but also considers the downlink bandwidth of the voice receiving terminal. Therefore, the target code rate determined by the voice sending terminal according to the reference code rate and the allocation code rate can make full use of the network information of the uplink and downlink, so that the useful network information of the downlink can be used by the uplink sender for reference regulation, thereby improving the accuracy of traffic regulation.

[0034] Moreover, in this embodiment, the voice sending terminal generates voice streams with different bitrates according to the uplink and downlink network information, and sends the voice streams with different bitrates to the server, avoiding using mixing for flow control in the server, thereby reducing project complexity, reducing the latency of voice transmission, and reducing the CPU overhead of the server. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] To more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the accompanying drawings required for the description of the embodiments. Obviously, the accompanying drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0036] Figure 1 is a flowchart of a voice processing method provided in Embodiment 1 of the present application;

[0037] Figure 2 is a flowchart of a voice processing method provided in Embodiment 2 of the present application;

[0038] Figure 3 is a flowchart of a voice processing method provided in Embodiment 3 of the present application;

[0039] Figure 4 is a schematic structural diagram of a voice processing device provided in Embodiment 4 of the present application;

[0040] Figure 5 is a schematic structural diagram of a voice processing device provided in Embodiment 5 of the present application;

[0041] Figure 6 is a schematic structural diagram of a voice processing device for implementing the voice processing method in the embodiments of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0042] To enable those skilled in the art to better understand the solutions of the present application, the following clearly and completely describes the technical solutions in the embodiments of the present application in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0043] It should be noted that the terms "first", "second", etc. in the description, claims and the above-mentioned drawings of this application are used to distinguish similar objects, and do not necessarily describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of this application described here can be implemented in an order other than those illustrated or described here. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0044] Embodiment 1

[0045] Figure 1 As shown in the flowchart of a voice processing method provided for Embodiment 1 of this application, this embodiment can be applied to a voice sending terminal.

[0046] As Figure 1 shown, this embodiment may include the following steps:

[0047] Step 101, predict the total uplink bandwidth.

[0048] When implemented, the voice sending terminal can use a congestion controller CC (Congestion Controller, which is used to monitor the network link, predict the bandwidth, and control the rate of sending packets to the network to avoid link deterioration) to perform periodic (the prediction period is an empirical value, for example, it can be one cycle of 100 ms) bandwidth prediction to obtain the total uplink bandwidth.

[0049] Step 102, based on the total uplink bandwidth, perform bitrate allocation on the voice stream to be sent to determine at least two allocated bitrates, and send the at least two allocated bitrates to the server.

[0050] In this step, after the voice sending terminal obtains the total uplink bandwidth, it can start bitrate allocation and divide the total uplink bandwidth into at least two allocated bitrates.

[0051] In one embodiment, Step 102 may further include the following steps:

[0052] Step 102-1, obtain at least two preset bitrate intervals, and each of the bitrate intervals includes a minimum bitrate value and a maximum bitrate value.

[0053] When implemented, at least two preset bitrate ranges can be read from a configuration file. The bitrate ranges defined by each bitrate range are different, and each bitrate range includes a minimum bitrate value and a maximum bitrate value. For example, the bitrate range can be expressed as [16k, 32k], [48k, 128k], etc. According to the minimum bitrate value and the maximum bitrate value of each bitrate range, the sizes of the bitrate ranges can be compared.

[0054] Step 102-2, determine the sum of the minimum bitrate values of each bitrate range.

[0055] For example, if the bitrate ranges include [16k, 32k] and [48k, 128k], then the sum of the minimum bitrate values = 16k + 48k = 64k.

[0056] Another example, if the bitrate ranges include [16k, 32k], [32k, 64k] and [64k, 128k], then the sum of the minimum bitrate values = 16k + 32k + 64k = 112k.

[0057] Step 102-3, if the total upstream bandwidth is greater than the sum of the minimum bitrate values of each bitrate range, then allocate the bitrate to each bitrate range according to its minimum bitrate value, and calculate the first remaining available upstream bandwidth, and use the first remaining available upstream bandwidth to increase the allocated bitrate of each bitrate range.

[0058] For example, if the bitrate ranges include [16k, 32k] and [48k, 128k], and the total upstream bandwidth is 128k, and the total upstream bandwidth exceeds the sum of the minimum bitrate values of the above two bitrate ranges, then allocate 16k of upstream bandwidth to the [16k, 32k] bitrate range, and allocate 48k of upstream bandwidth to the [48k, 128k] bitrate range, and the first remaining available upstream bandwidth = 128k - 16k - 48k = 64k.

[0059] Another example, if the bitrate ranges include [16k, 32k], [32k, 64k] and [64k, 128k], and the total upstream bandwidth is 120k, and the total upstream bandwidth exceeds the sum of the minimum bitrate values of the above three bitrate ranges, then allocate 16k of upstream bandwidth to the [16k, 32k] bitrate range, allocate 32k of upstream bandwidth to the [32k, 64k] bitrate range, allocate 64k of upstream bandwidth to the [64k, 128k] bitrate range, and the first remaining available upstream bandwidth = 120k - 16k - 32k - 64k = 8k.

[0060] When the minimum bitrate value of each bitrate range is satisfied, the first remaining available upstream bandwidth can be used to increase the allocated bitrate of each bitrate range.

[0061] In one embodiment, the step of using the first remaining available uplink bandwidth to increase the allocated bitrates of each bitrate interval may further include the following steps:

[0062] Increase the allocated bitrates of each bitrate interval in ascending order of the bitrate intervals. For the currently required bitrate interval, determine whether the first remaining available uplink bandwidth is greater than the minimum bitrate value of this bitrate interval; if so, directly allocate the bitrate for this bitrate interval according to the minimum bitrate value, update the first remaining available uplink bandwidth, and continue to traverse the next bitrate interval; if not, directly allocate the first remaining available uplink bandwidth to this bitrate interval.

[0063] For example, for the above example, if the bitrate intervals include [16k, 32k] and [48k, 128k], the total uplink bandwidth is 128k, and the first remaining available uplink bandwidth is 64k. When performing bitrate increase, first increase the allocated bitrate of [16k, 32k]. Since the first remaining available uplink bandwidth 64k is greater than the minimum value 16k of this bitrate interval, 16k of bandwidth can be reallocated to this bitrate interval, making the allocated bitrate become 32k, and the first remaining available uplink bandwidth is updated to 64k - 16k = 48k. Then increase the allocated bitrate of [48k, 128k]. Since the first remaining available uplink bandwidth 48k is equal to the minimum value 48k of this bitrate interval, 48k of bandwidth can be reallocated to this bitrate interval, making the allocated bitrate become 96k, and the first remaining available uplink bandwidth is updated to 48k - 48k = 0k, completing the bitrate allocation. Finally, the allocated bitrate of [16k, 32k] is 32k, and the allocated bitrate of [48k, 128k] is 96k.

[0064] Another example, for the above example, if the bitrate intervals include [16k, 32k], [32k, 64k], and [64k, 128k], the total uplink bandwidth is 120k, and the first remaining available uplink bandwidth is 8k. When performing bitrate increase, first increase the allocated bitrate of [16k, 32k]. Since the first remaining available uplink bandwidth 8k is less than the minimum value 16k of this bitrate interval, 8k of bandwidth can be reallocated to this bitrate interval, making the allocated bitrate become 16k + 8k = 24k, and the first remaining available uplink bandwidth is updated to 8k - 8k = 0k, completing the bitrate allocation. Finally, the allocated bitrate of [16k, 32k] is 24k, the allocated bitrate of [32k, 64k] is 32k, and the allocated bitrate of [64k, 128k] is 64k.

[0065] Step 102-4, if the total uplink bandwidth is equal to the sum of the minimum bitrate values of each bitrate interval, then allocate the bitrate for each bitrate interval according to its minimum bitrate value respectively.

[0066] If the total uplink bandwidth is exactly equal to the sum of the minimum bitrate values of each bitrate interval, then the bitrate can be allocated to each bitrate interval according to its minimum bitrate value. For example, if the bitrate intervals include [16k, 32k], [32k, 64k], and [64k, 128k], and the total uplink bandwidth is 112k, and the total uplink bandwidth is equal to the sum of the minimum bitrate values of the above three bitrate intervals, then allocate 16k of uplink bandwidth to the [16k, 32k] bitrate interval, allocate 32k of uplink bandwidth to the [32k, 64k] bitrate interval, and allocate 64k of uplink bandwidth to the [64k, 128k] bitrate interval.

[0067] Step 102-5, if the total uplink bandwidth is less than the sum of the minimum bitrate values of each bitrate interval, then after removing the largest first N bitrate intervals, allocate the bitrate to the remaining bitrate intervals according to their minimum bitrate values, and calculate the second remaining available uplink bandwidth, and use the second remaining available uplink bandwidth to increase the allocated bitrates of the remaining bitrate intervals.

[0068] In one embodiment, the largest first N bitrate intervals are determined in the following manner:

[0069] Remove the largest bitrate interval, and determine whether the sum of the minimum bitrate values of the remaining bitrate intervals is less than or equal to the total uplink bandwidth; if not, then continue to remove the second largest bitrate interval, and determine whether the sum of the minimum bitrate values of the remaining bitrate intervals is less than or equal to the total uplink bandwidth, and so on, until the sum of the minimum bitrate values of the remaining bitrate intervals is less than or equal to the total uplink bandwidth.

[0070] In implementation, the method of allocating the bitrate to the remaining bitrate intervals according to their minimum bitrate values, and calculating the second remaining available uplink bandwidth, and using the second remaining available uplink bandwidth to increase the allocated bitrates of the remaining bitrate intervals is similar to Step 102-3, and the bitrate allocation process of Step 102-3 can be referred to.

[0071] For example, if the bitrate intervals include [16k, 32k], [32k, 64k], and [64k, 128k], the total uplink bandwidth is 100k, and the sum of the minimum bitrate values of the last three bitrate intervals is 112k, exceeding the total uplink bandwidth of 100k. Then, first remove the largest bitrate interval [64k, 128k]. The remaining bitrate intervals are [16k, 32k] and [32k, 64k]. The sum of the minimum bitrate values of the remaining bitrate intervals is 48k, which is less than the total uplink bandwidth of 100k. When performing bitrate allocation, no bitrate is allocated to [64k, 128k]. First, allocate 16k of uplink bandwidth to the bitrate interval [16k, 32k], and allocate 32k of uplink bandwidth to the bitrate interval [32k, 64k]. The second remaining available uplink bandwidth = 100k - 16k - 32k = 52k. Then, perform bitrate increase. First, increase the allocated bitrate of [16k, 32k]. Since the second remaining available uplink bandwidth of 52k is greater than the minimum value of 16k in this bitrate interval, an additional 16k of bandwidth can be allocated to this bitrate interval, making the allocated bitrate become 32k. The second remaining available uplink bandwidth is updated to 52k - 16k = 36k. Next, increase the allocated bitrate of [32k, 64k]. Since the second remaining available uplink bandwidth of 36k is greater than the minimum value of 32k in this bitrate interval, an additional 32k of bandwidth can be allocated to this bitrate interval, making the allocated bitrate become 64k. The second remaining available uplink bandwidth is updated to 36k - 32k = 4k. Continue to perform bitrate increase. First, increase the allocated bitrate of [16k, 32k]. Since the second remaining available uplink bandwidth of 4k is less than the minimum value of 16k in this bitrate interval, all 4k of bandwidth can be allocated to this bitrate interval, making the allocated bitrate become 32k + 4 = 36k. The second remaining available uplink bandwidth is updated to 4k - 4k = 0k, completing the bitrate allocation. Finally, the allocated bitrate of [16k, 32k] is 36k, and the allocated bitrate of [32k, 64k] is 64k.

[0072] Another example, if the bitrate intervals include [16k, 32k], [32k, 64k], and [64k, 128k], the total uplink bandwidth is 30k, and the sum of the minimum bitrate values of the last three bitrate intervals is 112k, exceeding the total uplink bandwidth of 30k. Then, first remove the largest bitrate interval [64k, 128k]. The remaining bitrate intervals are [16k, 32k] and [32k, 64k]. The sum of the minimum bitrate values of the remaining bitrate intervals is 48k, still greater than the total uplink bandwidth of 30k. Then, further remove the second largest bitrate interval [32k, 64k], leaving [16k, 32k]. When performing bitrate allocation, all 30k of the total uplink bandwidth is allocated to this bitrate interval.

[0073] After the voice sending terminal obtains at least two allocated bitrates, it can send the at least two allocated bitrates to the server. For example, it sends the bitrate allocation result with a small stream bitrate of 32k and a large stream bitrate of 96k to the server.

[0074] Step 103, receive the reference bitrates corresponding to the respective allocated bitrates sent by the server, where the reference bitrates are obtained by the server based on the received allocated bitrates and the estimated downstream bandwidth.

[0075] In this step, after the voice sending terminal sends out the allocated bitrates, it will wait for the reference bitrates returned by the server. Among them, the reference bitrates are obtained by the server according to the allocation logic of the reference bitrates, based on the received allocated bitrates and the estimated downstream bandwidth. The determination method of the reference bitrates by the server will be described in the next embodiment.

[0076] Step 104, based on the respective allocated bitrates and the corresponding reference bitrates, determine the target bitrates corresponding to the respective allocated bitrates.

[0077] After the voice sending terminal obtains the reference bitrates corresponding to the respective allocated bitrates, it can then determine whether it is necessary to correct the corresponding allocated bitrates based on the reference bitrates, so as to determine the target bitrates corresponding to the respective allocated bitrates.

[0078] In one embodiment, step 104 may further include the following steps:

[0079] If the allocated bitrate is less than the corresponding reference bitrate, then use the allocated bitrate as the target bitrate; if the allocated bitrate is greater than the corresponding reference bitrate, then use the reference bitrate as the target bitrate, and obtain the remaining bitrate in the allocated bitrate except the reference bitrate; select the bitrate interval where the target bitrate does not reach the corresponding maximum bitrate value as the candidate bitrate interval; update the target bitrates of the respective candidate bitrate intervals according to the remaining bitrate.

[0080] Specifically, if the target bitrate is less than or equal to the reference bitrate, the target bitrate is not corrected. If the target bitrate is greater than the reference bitrate, the target bitrate is corrected.

[0081] In one implementation, the correction method may be: truncate the target bitrate using the reference bitrate, use the bitrate equal to the reference bitrate as the target bitrate, then use the truncated bitrate except the reference bitrate as the remaining bitrate, and re-allocate the remaining bitrate to the bitstreams that can still increase the bitrate to obtain at least two final target bitrates.

[0082] Among them, if the target bitrate of a certain bitrate interval does not reach the maximum bitrate value of this bitrate interval, it means that the target bitrate can still be increased, and this bitrate interval can be called a candidate bitrate interval. After determining one or more candidate bitrate intervals, the remaining bitrate can be allocated to each candidate bitrate interval according to the bitrate increase method mentioned in step 102 to obtain the final target bitrate of each candidate bitrate interval.

[0083] Step 105: Encode the voice stream respectively using the target bitrate, and send the encoded voice stream to the server.

[0084] When the voice sending terminal determines at least two target bitrates, it can encode the voice stream to be sent using the at least two target bitrates to obtain at least two voice streams with different target bitrates, and then send the at least two voice streams to the server, and the server decides which target bitrate of the voice stream needs to be sent to the voice receiving terminal.

[0085] In practice, the at least two target bitrate voice streams include a large bitrate voice stream and a small bitrate voice stream. The large bitrate voice stream is used to provide high-quality audio for voice receivers with good networks as much as possible, and the small bitrate voice stream is to take care of voice receivers with poor networks so that they can obtain a smooth audio experience.

[0086] In this embodiment, the voice sending terminal predicts the total uplink bandwidth, and allocates the bitrate for the voice stream to be sent based on the total uplink bandwidth to determine at least two allocated bitrates. Then send the at least two allocated bitrates to the server, and receive the reference bitrates corresponding to each allocated bitrate sent by the server. When determining the reference bitrate, the server will consider not only the allocated bitrate of the voice sending terminal but also the downlink bandwidth of the voice receiving terminal. Therefore, the target bitrate determined by the voice sending terminal according to the reference bitrate and the allocated bitrate can make full use of the network information of the uplink and downlink, so that the useful network information of the downlink can be used by the uplink sender for reference regulation, thereby improving the accuracy of traffic regulation.

[0087] Moreover, in this embodiment, the voice sending terminal generates voice streams with different bitrates according to the network information of the uplink and downlink, and sends the voice streams with different bitrates to the server, avoiding using mixing for flow control in the server, thereby reducing the project complexity, reducing the delay of voice sending, and reducing the CPU overhead of the server.

[0088] Embodiment Two

[0089] Figure 2 It is a flowchart of a voice processing method provided in the second embodiment of this application. This embodiment can be applied to a server.

[0090] As Figure 2As shown in the figure, this embodiment may include the following steps:

[0091] Step 201: Receive at least two allocation bitrates for the voice stream to be sent from the voice sending terminal.

[0092] Step 202: Determine the voice receiving terminals that will receive the voice stream, and predict the downstream bandwidth of each voice receiving terminal.

[0093] This embodiment can be applied to the scenario of multi-person audio transmission. One or more voice receiving terminals can be determined according to the current voice communication scenario. For example, in the scenario of a multi-person voice conference, the voice receiving terminals can include all user terminals participating in the conference; in the live broadcast scenario, the voice receiving terminals can include all user terminals in the live broadcast room.

[0094] In implementation, the server can use a congestion controller CC to predict the downstream bandwidth of each voice receiving terminal.

[0095] Step 203: Determine the reference bitrates corresponding to each allocation bitrate according to the downstream bandwidth of each voice receiving terminal and the at least two allocation bitrates.

[0096] After the server receives at least two allocation bitrates sent by the voice sending terminal and determines the downstream bandwidth of each voice receiving terminal, it can refer to the downstream bandwidth of each voice receiving terminal to determine whether it is necessary to correct each allocation bitrate, so as to determine the reference bitrates corresponding to each allocation bitrate.

[0097] In one embodiment, step 203 may further include the following steps:

[0098] Step 203-1: Obtain at least two preset bitrate intervals.

[0099] Among them, the bitrate intervals obtained by the server are the same as those obtained by the voice sending terminal in Embodiment 1. In implementation, the server can also read at least two preset bitrate intervals from the configuration file.

[0100] Step 203-2: Classify the at least two allocation bitrates of all received voice sending terminals according to the belonging bitrate intervals to obtain at least two first classification sets. Each first classification set has a corresponding bitrate interval, and each first classification set contains the allocation bitrates of the voice sending terminals belonging to the same bitrate interval.

[0101] For example, assume there are three voice sending terminals, including sender A, sender B, and sender C. Assume each voice sending terminal sends two allocated bitrates to the server, herein referred to as the large bitrate and the small bitrate. The allocated bitrates sent by sender A include the large bitrate A1 and the small bitrate A2. The allocated bitrates sent by sender B include the large bitrate B1 and the small bitrate B2. The allocated bitrates sent by sender C include the large bitrate C1 and the small bitrate C2. The bitrate intervals include the large bitrate interval [48k, 128k] and the small bitrate interval [16k, 32k]. Bitrates belonging to the large bitrate interval [48k, 128k] can be classified into the large bitrate set. For example, the large bitrate set includes A1, B1, and C1. Bitrates belonging to the small bitrate interval [16k, 32k] can be classified into the small bitrate set. For example, the small bitrate set includes A2, B2, and C2.

[0102] Step 203-3: Determine the target downlink bandwidth for each bitrate interval according to the downlink bandwidth of each voice receiving terminal.

[0103] In one embodiment, the server can sort the downlink bandwidths of each voice receiving terminal and take the quantiles according to at least two set quantile values as the target downlink bandwidth corresponding to the quantile values.

[0104] For example, the two set quantile values can include the large bitrate quantile and the small bitrate quantile. The large bitrate quantile can be 50% for example, and the small bitrate quantile can be 20% for example. Then, the bandwidth at the 20% quantile after sorting can be taken as the target downlink bandwidth for the small bitrate, and the bandwidth at the 50% quantile after sorting can be taken as the target downlink bandwidth for the large bitrate.

[0105] Wherein, each bitrate interval corresponds to a quantile value.

[0106] Step 203-4: Determine the total allocated bitrate in each first classification set and compare the total allocated bitrate with the target downlink bandwidth of the corresponding bitrate interval.

[0107] For example, for the above example, for the large bitrate set, calculate the sum of the allocated bitrates A1, B1, and C1 therein as the total large bitrate allocated bitrate, and then compare the total large bitrate allocated bitrate with the target downlink bandwidth of the large bitrate. For the small bitrate set, calculate the sum of the allocated bitrates A2, B2, and C2 therein as the total small bitrate allocated bitrate, and then compare the total small bitrate allocated bitrate with the target downlink bandwidth of the small bitrate.

[0108] Step 203-5: Determine the reference bitrate corresponding to each allocated bitrate according to the comparison result.

[0109] In this step, it is possible to determine whether the downlink bandwidth is sufficient according to the comparison result, and then determine the reference bitrate corresponding to each allocated bitrate according to the sufficiency of the downlink bandwidth.

[0110] In one embodiment, the step of determining the reference bitrate corresponding to each allocated bitrate according to the comparison result may further include the following steps:

[0111] Step 203-5-1, if the target downlink bandwidth is less than the sum of the corresponding allocated bitrates, calculate the difference between the sum of the allocated bitrates and the target downlink bandwidth as the bitrate that needs to be reduced.

[0112] Among them, when the target downlink bandwidth is less than the sum of the corresponding allocated bitrates, it indicates that the downlink bandwidth is very insufficient, and it is necessary to reduce each allocated bitrate according to the target downlink bandwidth to obtain the reference bitrate. First, calculate the bitrate that needs to be reduced.

[0113] For example, for the small bitrate set, the sum of the allocated bitrates A2, B2, and C2 therein is A2 + B2 + C2, and the target downlink bandwidth of the small bitrate is EstimateBitrate. Then, the bitrate that needs to be reduced NeedDecrease can be calculated using the following formula:

[0114] NeedDecrease = A2 + B2 + C2 – EstimateBitrate.

[0115] Step 203-5-2, determine the minimum allocated bitrate in the first classification set corresponding to the sum of the allocated bitrates.

[0116] For example, for the small bitrate set, the allocated bitrates A2, B2, and C2 therein are 18k, 20k, and 23k respectively, then the minimum allocated bitrate is A2.

[0117] Step 203-5-3, calculate the product of the minimum allocated bitrate and the number of allocated bitrates in the first classification set as the total minimum allocated bitrate of the first classification set.

[0118] For example, for the small bitrate set, there are three allocated bitrates, and the minimum allocated bitrate is A2. Then the total minimum allocated bitrate of the small bitrate set is 3 * A2.

[0119] Step 203-5-4, take the difference between the sum of the allocated bitrates and the total minimum allocated bitrate as the bitrate that can be reduced.

[0120] For example, for the small bitrate set, the bitrate that can be reduced = A2 + B2 + C2 – 3 * A2.

[0121] Step 203-5-5: For each allocated bitrate in the first classification set, calculate the reference bitrate corresponding to the allocated bitrate according to the allocated bitrate, the minimum allocated bitrate, the bitrate that can be reduced, and the bitrate that needs to be reduced.

[0122] In implementation, the reference bitrate of a certain allocated bitrate can be calculated using the following formula:

[0123] Reference bitrate = (Allocated bitrate - Minimum allocated bitrate) / Bitrate that can be reduced * Bitrate that needs to be reduced.

[0124] For example, for the small bitrate set, assuming we want to determine the reference bitrate A2' of A2, the following formula can be used: A2' = (A2 – LowBitrateMin) / DecreaseMax * NeedDecrease;

[0125] where LowBitrateMin is the minimum allocated bitrate in the current small bitrate set, and DecreaseMax is the bitrate that can be reduced.

[0126] It should be noted that the above examples are all illustrated with the small bitrate set as an example. In fact, the large bitrate set is also processed in the same way. According to the above formula, the reference bitrates of each allocated bitrate in each first classification set can be calculated. For example, the reference bitrate A2' of A2, the reference bitrate B2' of B2, the reference bitrate C2' of C2, the reference bitrate A1' of A1, the reference bitrate B1' of B1, and the reference bitrate C1' of C1.

[0127] In another embodiment, the step of determining the reference bitrate corresponding to each allocated bitrate according to the comparison result may further include the following steps:

[0128] If the target downlink bandwidth is greater than the sum of the corresponding allocated bitrates, calculate the difference between the target downlink bandwidth and the sum of the allocated bitrates as the excess downlink bandwidth; allocate the excess downlink bandwidth to each allocated bitrate in the first classification set corresponding to the sum of the allocated bitrates to obtain the reference bitrates corresponding to each allocated bitrate.

[0129] Among them, when the target downlink bandwidth is greater than the sum of the corresponding allocated bitrates, it means that the downlink bandwidth is sufficient, and the reference bitrates can be obtained by increasing each allocated bitrate based on the target downlink bandwidth. Specifically, the difference between the target downlink bandwidth and the sum of the allocated bitrates can be calculated as the excess downlink bandwidth. Then, refer to the bitrate increase method in Embodiment 1 to allocate the excess downlink bandwidth to each allocated bitrate in the corresponding set.

[0130] To enable those skilled in the art to better understand how to determine the reference bitrate, the following provides an exemplary illustration through specific examples:

[0131] Assume that there are 2 users on the live stream, and the corresponding allocated bitrates are as follows:

[0132] User on the live stream 1: large stream bitrate 96k, small stream bitrate 32k;

[0133] User on the live stream 2: large stream bitrate 96k, small stream bitrate 32k;

[0134] The server calculates that the target downstream bandwidth at the 20th percentile is 32k. At this time, the small stream bitrate of user on the live stream 1 (32k) + the small stream bitrate of user on the live stream 2 (32k) = 64k, which exceeds the target downstream bandwidth of the current receiver (32k). Through the calculation formulas from step 203-5-1 to step 203-5-5, the server calculates that the total bitrate that all senders need to reduce is 32k + 32k - 32k = 32k. Furthermore, it calculates that the small stream reference codes of user on the live stream 1 and user on the live stream 2 are both 16k. This is the part of the small stream reference bitrate.

[0135] Regarding the large stream reference bitrate, the server calculates that the target downstream bandwidth at the 50th percentile is 300k. The sum of the large stream bitrates of user on the live stream 1 and user on the live stream 2 is 96 + 96 = 192k. Since the current 50th percentile bandwidth (300k) > 192k, there is still room for improvement of 300k - 192k = 108k in the large stream bitrate of each sender. And the maximum bitrate value in the large bitrate range [48k, 128k] is 128k. Therefore, the large stream bitrates of user on the live stream 1 and user on the live stream 2 can both be increased to the upper limit of 128k. That is to say, the reference bitrates of user on the live stream 1 and user on the live stream 2 are both 128k.

[0136] The final result obtained is:

[0137] User on the live stream 1: large stream reference bitrate 128k, small stream bitrate 16k;

[0138] User on the live stream 2: large stream reference bitrate 128k, small stream bitrate 16k.

[0139] It should be noted that in the above examples, two bitrates are used for illustration. However, in fact, the number of bitrates can exceed two, for example, it can be three bitrates. Then the percentile values can be set to 20%, 50%, and 80%. Other calculation processes are similar to those of two bitrates.

[0140] Step 204: Send the reference bitrate to the voice sending terminal so that the voice sending terminal determines the target bitrate of the allocated bitrate according to the reference bitrate and the corresponding allocated bitrate, and encodes the voice stream using the target bitrate.

[0141] In this step, after the server determines the reference bitrates corresponding to the allocated bitrates, it can return the reference bitrates corresponding to the allocated bitrates to the corresponding voice sending terminals. For the voice sending terminals, they can determine the target bitrates according to the process described in Embodiment 1 and encode the voice streams using the target bitrates.

[0142] In this embodiment, after the server receives the allocated bitrates sent by the voice sending terminals, it can obtain the downstream bandwidth of the voice receiving terminals and correct the allocated bitrates based on the downstream bandwidth to determine the reference bitrates corresponding to the allocated bitrates, and return the reference bitrates to the voice sending terminals, so that the reference bitrates comprehensively consider the upstream bandwidth situation and the downstream bandwidth situation, improving the accuracy of the bitrates.

[0143] Embodiment 3

[0144] Figure 3 FIG. 10 is a flowchart of a voice processing method provided in Embodiment 3 of the present application. This embodiment can be applied to a server. On the basis of Embodiment 2, this embodiment describes the scenario of voice distribution. As Figure 3 shown, this embodiment may include the following steps:

[0145] Step 301, receiving at least two allocated bitrates for the voice stream to be sent sent by the voice sending terminal.

[0146] Step 302, determining the voice receiving terminals that will receive the voice stream and predicting the downstream bandwidths of the voice receiving terminals.

[0147] Step 303, determining the reference bitrates corresponding to the allocated bitrates according to the downstream bandwidths of the voice receiving terminals and the at least two allocated bitrates.

[0148] Step 304, sending the reference bitrates to the voice sending terminal so that the voice sending terminal determines the target bitrates of the allocated bitrates according to the reference bitrates and the corresponding allocated bitrates, and encodes the voice stream using the target bitrates.

[0149] Step 305, receiving at least two voice streams sent by the voice sending terminal, where each voice stream has a corresponding target bitrate.

[0150] For example, the at least two voice streams sent by the voice sending terminal include a high-bitrate voice stream of 128 k and a low-bitrate voice stream of 32 k.

[0151] Step 306: Classify at least two target bitrates of all received voice sending terminals according to the belonging bitrate intervals to obtain at least two second classification sets. Each second classification set has a corresponding bitrate interval, and each second classification set contains the target bitrates of each voice sending terminal belonging to the same bitrate interval.

[0152] The generation method of the second classification set is similar to that of the first classification set in the second embodiment above. For example, assume there are three voice sending terminals, including sender A, sender B, and sender C. The voice streams sent by sender A include a high-bitrate voice stream of 128k and a low-bitrate voice stream of 32k. The voice streams sent by sender B include a high-bitrate voice stream of 64k and a low-bitrate voice stream of 28k. The voice streams sent by sender C include a high-bitrate voice stream of 80k and a low-bitrate voice stream of 22k. The bitrate intervals include a high-bitrate interval [48k, 128k] and a low-bitrate interval [16k, 32k]. The target bitrates belonging to the high-bitrate interval [48k, 128k] can be classified into the high-bitrate target bitrate set. For example, the high-bitrate target bitrate set includes 128k, 64k, and 80k. The target bitrates belonging to the low-bitrate interval [16k, 32k] can be classified into the low-bitrate target bitrate set. For example, the low-bitrate target bitrate set includes 32k, 28k, and 22k.

[0153] Step 307: Determine the total target bitrate in each second classification set.

[0154] For example, for the high-bitrate target bitrate set, the total target bitrate obtained by calculating the sum of each target bitrate is 128k + 64k + 80k = 272k. For the low-bitrate target bitrate set, the total target bitrate obtained by calculating the sum of each target bitrate is 32k + 28k + 22k = 82k.

[0155] Step 308: For each voice receiving terminal, compare the downstream bandwidth of the voice receiving terminal with the total target bitrate of each second classification set.

[0156] For example, assume there are voice receivers D, E, and F, and their corresponding downstream bandwidths are 300k, 220k, and 90k respectively. Then for each voice receiver, its downstream bandwidth is compared with the total target bitrate of each second classification set.

[0157] Step 309: Based on the comparison result, select the voice stream with an adapted target bitrate and send it to the voice receiving terminal.

[0158] Among them, the purpose of comparing the downstream bandwidth with the total target bitrate of the second classification set is to determine the bandwidth situation of the current voice receiver and decide which bitrate of voice stream to send to this voice receiver.

[0159] In one embodiment, step 309 may further include the following steps:

[0160] Step 309-1: Search for the total target bitrate that is close to and smaller than the downstream bandwidth of the current voice receiving terminal. If the total target bitrate is found, execute step 309-2; if the total target bitrate is not found, execute step 309-3.

[0161] For example, assume there are voice receivers D, E, and F, with corresponding downstream bandwidths of 300k, 220k, and 90k respectively. The total target bitrate of the large-bitrate target bitrate set is 272k; the total target bitrate of the small-bitrate target bitrate set is 82k. Then, for voice receiver D, the closest total target bitrate is 272k; for voice receiver E, the closest total target bitrate is not found; for voice receiver F, the closest total target bitrate is 82k.

[0162] Step 309-2: Use the second classification set corresponding to the found total target bitrate as the target second classification set, and send the voice streams with the corresponding target bitrates of each voice sending terminal to the voice receiving terminal according to the target second classification set.

[0163] For example, for voice receiver F, the closest total target bitrate found is 82k, which is the total target bitrate of the small-bitrate target bitrate set. The small-bitrate target bitrate set includes 32k, 28k, and 22k. That is, send the small-bitrate voice streams of senders A, B, and C to voice receiver F respectively, namely, send the 32k voice stream of sender A, the 28k voice stream of sender B, and the 22k voice stream of sender C to voice receiver F respectively.

[0164] Another example, for voice receiver D, the closest total target bitrate found is 272k, which is the total target bitrate of the large-bitrate target bitrate set. The large-bitrate target bitrate set includes 128k, 64k, and 80k. That is, send the large-bitrate voice streams of senders A, B, and C to voice receiver D respectively, namely, send the 128k voice stream of sender A, the 64k voice stream of sender B, and the 80k voice stream of sender C to voice receiver D respectively.

[0165] Step 309-3: Select the adapted bitrates of each voice sending terminal from all the target bitrates of all voice sending terminals using a preset dynamic programming algorithm, and send the voice streams with the adapted bitrates of each voice sending terminal to the voice receiving terminal, where the sum of the adapted bitrates of each voice sending terminal is close to the downstream bandwidth of the current voice receiving terminal.

[0166] For example, for the voice receiver E, the closest total target bitrate is not found. In this case, according to the downstream bandwidth of 220k of the voice receiver E, a dynamic programming algorithm can be used to select a combination of a part of high-bitrate voice streams and a part of low-bitrate voice streams to adapt to a bitrate close to the downstream bandwidth. For example, in the above example, after traversing using the dynamic programming algorithm, it is found that the combination 128k + 64k + 22k = 214k is close to the downstream bandwidth of 220k of the voice receiver E and has the highest bandwidth utilization rate. Therefore, the 128k high-bitrate voice stream of the sender A, the 64k high-bitrate voice stream of the sender B, and the 22k low-bitrate voice stream of the sender C are respectively sent to the voice receiver E.

[0167] In this embodiment, no specific limitation is imposed on the dynamic programming algorithm, as long as it can traverse the target bitrate combination closest to the current downstream bandwidth.

[0168] It should be noted that to avoid frequent switching of voice streams with different bitrates from affecting the sound quality transformation at the receiving end, after obtaining the target bitrate combination each time, the target bitrate combination can be compared with the target bitrate combination obtained last time. If the bitrate difference between the two exceeds a set ratio (e.g., 10%), the target bitrate combination is switched to the currently obtained target bitrate combination; otherwise, the target bitrate combination obtained last time is continued to be used.

[0169] In this embodiment, when the server distributes voice streams, it can select voice streams with adapted bitrates for sending according to the downstream bandwidth of the voice receiving terminal. For example, when the network of most receivers is very poor, the server can select voice streams with low bitrates for sending. When the network conditions of the receivers vary, the server can send high-bitrate voice streams to users with good networks to provide them with a better sound quality experience, and send low-bitrate voice streams to users with poor networks to provide them with a smooth experience, thereby improving the bandwidth utilization rate.

[0170] Embodiment 4

[0171] Figure 4 FIG. is a schematic structural diagram of a voice processing device provided in Embodiment 4 of the present application. The voice processing device is arranged in a voice sending terminal and may include the following modules:

[0172] An uplink bandwidth prediction module 401, configured to predict the total uplink bandwidth;

[0173] A bitrate allocation module 402, configured to perform bitrate allocation on the voice streams to be sent based on the total uplink bandwidth to determine at least two allocated bitrates, and send the at least two allocated bitrates to the server;

[0174] A reference bit rate receiving module 403 is configured to receive the reference bit rates corresponding to the respective allocated bit rates sent by the server, where the reference bit rates are obtained by the server according to the received allocated bit rates and the estimated downlink bandwidth;

[0175] A target bit rate determining module 404 is configured to determine the target bit rates corresponding to the respective allocated bit rates based on the respective allocated bit rates and the corresponding reference bit rates;

[0176] An encoding module 405 is configured to encode the voice stream respectively using the target bit rates, and send the encoded voice stream to the server.

[0177] In one embodiment, the bit rate allocation module 402 is further configured to:

[0178] Obtain at least two preset bit rate intervals, each of the bit rate intervals including a minimum bit rate value and a maximum bit rate value;

[0179] Determine the sum of the minimum bit rate values of each bit rate interval;

[0180] If the total uplink bandwidth is greater than the sum of the minimum bit rate values of each bit rate interval, then perform bit rate allocation for each bit rate interval according to its minimum bit rate value, calculate the first remaining available uplink bandwidth, and use the first remaining available uplink bandwidth to increase the allocated bit rates of each bit rate interval;

[0181] If the total uplink bandwidth is equal to the sum of the minimum bit rate values of each bit rate interval, then perform bit rate allocation for each bit rate interval according to its minimum bit rate value;

[0182] If the total uplink bandwidth is less than the sum of the minimum bit rate values of each bit rate interval, then after removing the top N largest bit rate intervals, perform bit rate allocation for the remaining bit rate intervals according to their minimum bit rate values, calculate the second remaining available uplink bandwidth, and use the second remaining available uplink bandwidth to increase the allocated bit rates of the remaining bit rate intervals.

[0183] In one embodiment, the bit rate allocation module 402 is further configured to:

[0184] Perform bit rate increase on the allocated bit rates of each bit rate interval in ascending order of the bit rate intervals. For the bit rate interval that currently requires bit rate allocation, determine whether the first remaining available uplink bandwidth is greater than the minimum bit rate value of this bit rate interval;

[0185] If so, directly perform bit rate allocation for this bit rate interval according to the minimum bit rate value, update the first remaining available uplink bandwidth, and continue to traverse the next bit rate interval;

[0186] If not, directly allocate the first remaining available uplink bandwidth to this bit rate interval.

[0187] In one embodiment, the largest N code rate intervals are determined in the following manner:

[0188] Remove the largest code rate interval, and determine whether the sum of the minimum code rate values of the remaining code rate intervals is less than or equal to the total uplink bandwidth;

[0189] If not, continue to remove the second largest code rate interval, and determine whether the sum of the minimum code rate values of the remaining code rate intervals is less than or equal to the total uplink bandwidth, and so on, until the sum of the minimum code rate values of the remaining code rate intervals is less than or equal to the total uplink bandwidth.

[0190] In one embodiment, the target code rate determination module 404 is further configured to:

[0191] If the allocated code rate is less than the corresponding reference code rate, use the allocated code rate as the target code rate;

[0192] If the allocated code rate is greater than the corresponding reference code rate, use the reference code rate as the target code rate, and obtain the remaining code rate in the allocated code rate except the reference code rate;

[0193] Select the code rate intervals whose target code rates do not reach the corresponding maximum code rate values as candidate code rate intervals;

[0194] Update the target code rates of the candidate code rate intervals according to the remaining code rate.

[0195] A voice processing device provided by an embodiment of the present application can execute a voice processing method provided by Embodiment 1 of the present application, and has corresponding functional modules and beneficial effects for executing the method.

[0196] Embodiment 5

[0197] Figure 5 FIG. is a schematic structural diagram of a voice processing device provided by Embodiment 5 of the present application. The voice processing device is arranged in a server and may include the following modules:

[0198] An allocated code rate receiving module 501, configured to receive at least two allocated code rates for a voice stream to be sent sent by a voice sending terminal;

[0199] A downlink bandwidth prediction module 502, configured to determine a voice receiving terminal for receiving the voice stream and predict the downlink bandwidth of each voice receiving terminal;

[0200] A reference code rate determination module 503, configured to determine a reference code rate corresponding to each allocated code rate according to the downlink bandwidth of each voice receiving terminal and the at least two allocated code rates; <>

[0201] The reference bitrate sending module 504 is configured to send the reference bitrate to the voice sending terminal, so that the voice sending terminal determines the target bitrate of the allocated bitrate according to the reference bitrate and the corresponding allocated bitrate, and encodes the voice stream using the target bitrate.

[0202] In one embodiment, the reference bitrate determining module 503 is further configured to:

[0203] Obtain at least two preset bitrate intervals;

[0204] Classify at least two allocated bitrates of all received voice sending terminals according to the belonging bitrate intervals to obtain at least two first classification sets, each first classification set has a corresponding bitrate interval, and each first classification set contains the allocated bitrates of each voice sending terminal belonging to the same bitrate interval;

[0205] Determine the target downstream bandwidth of each bitrate interval according to the downstream bandwidth of each voice receiving terminal;

[0206] Determine the sum of the allocated bitrates in each first classification set, and compare the sum of the allocated bitrates with the target downstream bandwidth of the corresponding bitrate interval;

[0207] Determine the reference bitrate corresponding to each allocated bitrate according to the comparison result.

[0208] In one embodiment, the reference bitrate determining module 503 is further configured to:

[0209] If the target downstream bandwidth is less than the sum of the corresponding allocated bitrates, calculate the difference between the sum of the allocated bitrates and the target downstream bandwidth as the bitrate that needs to be reduced;

[0210] Determine the minimum allocated bitrate in the first classification set corresponding to the sum of the allocated bitrates;

[0211] Calculate the product of the minimum allocated bitrate and the number of allocated bitrates in the first classification set as the sum of the minimum allocated bitrates in the first classification set;

[0212] Take the difference between the sum of the allocated bitrates and the sum of the minimum allocated bitrates as the bitrate that can be reduced;

[0213] For each allocated bitrate in the first classification set, calculate the reference bitrate corresponding to the allocated bitrate according to the allocated bitrate, the minimum allocated bitrate, the bitrate that can be reduced, and the bitrate that needs to be reduced.

[0214] In one embodiment, the reference bitrate determining module 503 is further configured to:

[0215] If the target downlink bandwidth is greater than the sum of the corresponding allocated bitrates, calculate the difference between the target downlink bandwidth and the sum of the allocated bitrates as the redundant downlink bandwidth.

[0216] Allocate the redundant downlink bandwidth to each of the allocated bitrates in the first classification set corresponding to the sum of the allocated bitrates to obtain the reference bitrates corresponding to each allocated bitrate.

[0217] In one embodiment, the apparatus may further include the following modules:

[0218] A voice stream receiving module, configured to receive at least two voice streams sent by a voice sending terminal, where each voice stream has a corresponding target bitrate.

[0219] A bitrate classification module, configured to classify at least two target bitrates of all voice sending terminals received according to the belonging bitrate intervals to obtain at least two second classification sets, each second classification set having a corresponding bitrate interval, and each second classification set includes the target bitrates of the voice sending terminals belonging to the same bitrate interval.

[0220] A bitrate sum determination module, configured to determine the sum of the target bitrates in each second classification set.

[0221] A comparison module, configured to compare the downlink bandwidth of each voice receiving terminal with the sum of the target bitrates of each second classification set for each voice receiving terminal.

[0222] A voice stream selection module, configured to select and send the voice stream with the adapted target bitrate to the voice receiving terminal based on the comparison result.

[0223] In one embodiment, the voice stream selection module is further configured to:

[0224] Search for the sum of the target bitrates that is close to and smaller than the downlink bandwidth of the current voice receiving terminal.

[0225] If the sum of the target bitrates is found, use the second classification set corresponding to the sum of the target bitrates found as the target second classification set, and send the voice streams with the corresponding target bitrates of each voice sending terminal to the voice receiving terminal according to the target second classification set.

[0226] If the sum of the target bitrates is not found, select the adapted bitrates of each voice sending terminal from all the target bitrates of all voice sending terminals by using a preset dynamic programming algorithm, and send the voice streams with the adapted bitrates of each voice sending terminal to the voice receiving terminal, where the sum of the adapted bitrates of each voice sending terminal is close to the downlink bandwidth of the current voice receiving terminal.

[0227] In one embodiment, the reference bitrate determination module 503 is further configured to:

[0228] Sort the downstream bandwidths of each voice receiving terminal, and take quantiles according to at least two set quantile values as the target downstream bandwidths corresponding to the quantile values.

[0229] A voice processing device provided by an embodiment of the present application can execute a voice processing method provided by Embodiment 1 of the present application, and has functional modules and beneficial effects corresponding to the execution of the method.

[0230] Embodiment Six

[0231] Figure 6 FIG. shows a schematic structural diagram of a voice processing device 10 that can be used to implement the method embodiment of the present application. As Figure 6 shown, the voice processing device 10 includes at least one processor 11 and a storage device communicatively connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc. Among them, the storage device stores one or more computer programs executable by the at least one processor. The processor 11 can execute various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 into the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the voice processing device 10 can also be stored.

[0232] In some embodiments, a voice processing method can be implemented as a computer program, which is tangibly included in a computer-readable storage medium, such as the storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed onto the voice processing device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the voice processing method described above can be executed.

[0233] In some embodiments, a voice processing method can be implemented as a computer program product, which includes computer-executable instructions that are used to execute one or more steps of the voice processing method described above when executed.

Claims

1. A voice processing method, characterized in that, Applied to a voice sending terminal, the method includes: Predict the total uplink bandwidth; Based on the total uplink bandwidth, perform bitrate allocation on the voice stream to be sent to determine at least two allocated bitrates, and send the at least two allocated bitrates to the server; Receive the reference bitrates corresponding to each allocated bitrate sent by the server, where the reference bitrates are obtained by the server according to the received allocated bitrates and the estimated downlink bandwidth; Based on each allocated bitrate and the corresponding reference bitrate, determine the target bitrate corresponding to each allocated bitrate; Encode the voice stream using the target bitrates respectively, and send the encoded voice stream to the server.

2. The method according to claim 1, wherein The performing bitrate allocation on the voice stream to be sent based on the total uplink bandwidth to determine at least two allocated bitrates includes: Obtain at least two preset bitrate intervals, each of the bitrate intervals including a minimum bitrate value and a maximum bitrate value; Determine the sum of the minimum bitrate values of each bitrate interval; If the total uplink bandwidth is greater than the sum of the minimum bitrate values of each bitrate interval, perform bitrate allocation on each bitrate interval according to its minimum bitrate value, calculate the first remaining available uplink bandwidth, and use the first remaining available uplink bandwidth to increase the allocated bitrates of each bitrate interval; If the total uplink bandwidth is equal to the sum of the minimum bitrate values of each bitrate interval, perform bitrate allocation on each bitrate interval according to its minimum bitrate value; If the total uplink bandwidth is less than the sum of the minimum bitrate values of each bitrate interval, after removing the largest first N bitrate intervals, perform bitrate allocation on the remaining bitrate intervals according to their minimum bitrate values, calculate the second remaining available uplink bandwidth, and use the second remaining available uplink bandwidth to increase the allocated bitrates of the remaining bitrate intervals.

3. The method according to claim 2, wherein The using the first remaining available uplink bandwidth to increase the allocated bitrates of each bitrate interval includes: increasing the bitrates of the allocated bitrates of each bitrate interval in ascending order of the bitrate intervals. For the current bitrate interval that needs bitrate allocation, determine whether the first remaining available uplink bandwidth is greater than the minimum bitrate value of this bitrate interval; If so, directly perform bitrate allocation on this bitrate interval according to the minimum bitrate value, update the first remaining available uplink bandwidth, and continue to traverse the next bitrate interval; If not, directly allocate the first remaining available uplink bandwidth to this bitrate interval.

4. The method according to claim 2, characterized in that, The largest first N bitrate intervals are determined in the following way: Remove the largest bitrate interval, and determine whether the sum of the minimum bitrate values of the remaining bitrate intervals is less than or equal to the total uplink bandwidth; If not, continue to remove the second largest bitrate interval, and determine whether the sum of the minimum bitrate values of the remaining bitrate intervals is less than or equal to the total uplink bandwidth, and so on, until the sum of the minimum bitrate values of the remaining bitrate intervals is less than or equal to the total uplink bandwidth.

5. The method according to any one of claims 2-4, characterized in that, The determining the target bitrate corresponding to each allocated bitrate based on each allocated bitrate and the corresponding reference bitrate includes: If the allocated bitrate is less than the corresponding reference bitrate, use the allocated bitrate as the target bitrate; If the allocated bit rate is greater than the corresponding reference bit rate, then use the reference bit rate as the target bit rate, and obtain the remaining bit rate in the allocated bit rate except the reference bit rate; Select a bit rate interval where the target bit rate does not reach the corresponding maximum bit rate value as the candidate bit rate interval; Update the target bit rate of each candidate bit rate interval according to the remaining bit rate.

6. A voice processing method, characterized in that, When applied to a server, the method includes: Receiving at least two allocated bit rates for a voice stream to be sent from a voice sending terminal; Determining a voice receiving terminal for receiving the voice stream, and predicting the downstream bandwidth of each voice receiving terminal; Determining the reference bit rate corresponding to each allocated bit rate according to the downstream bandwidth of each voice receiving terminal and the at least two allocated bit rates; Sending the reference bit rate to the voice sending terminal, so that the voice sending terminal determines the target bit rate of the allocated bit rate according to the reference bit rate and the corresponding allocated bit rate, and encodes the voice stream using the target bit rate.

7. The method according to claim 6, wherein The determining the reference bit rate corresponding to each allocated bit rate according to the downstream bandwidth of each voice receiving terminal and the at least two allocated bit rates includes: Obtaining at least two preset bit rate intervals; Classifying the at least two allocated bit rates of all received voice sending terminals according to the belonging bit rate interval, obtaining at least two first classification sets, each first classification set has a corresponding bit rate interval, and each first classification set contains the allocated bit rates of each voice sending terminal belonging to the same bit rate interval; Determining the target downstream bandwidth of each bit rate interval according to the downstream bandwidth of each voice receiving terminal; Determining the total allocated bit rate in each first classification set, and comparing the total allocated bit rate with the target downstream bandwidth of the corresponding bit rate interval; Determining the reference bit rate corresponding to each allocated bit rate according to the comparison result.

8. The method according to claim 7, wherein The determining the reference bit rate corresponding to each allocated bit rate according to the comparison result includes: If the target downstream bandwidth is less than the corresponding total allocated bit rate, then calculate the difference between the total allocated bit rate and the target downstream bandwidth as the bit rate to be reduced; Determining the minimum allocated bit rate in the first classification set corresponding to the total allocated bit rate; Calculating the product of the minimum allocated bit rate and the number of allocated bit rates in the first classification set as the total minimum allocated bit rate of the first classification set; Taking the difference between the total allocated bit rate and the total minimum allocated bit rate as the bit rate that can be reduced; For each allocated bit rate in the first classification set, calculate the reference bit rate corresponding to the allocated bit rate according to the allocated bit rate, the minimum allocated bit rate, the bit rate that can be reduced, and the bit rate to be reduced.

9. The method according to claim 7, wherein The determining the reference bit rate corresponding to each allocated bit rate according to the comparison result includes: If the target downstream bandwidth is greater than the corresponding total allocated bit rate, then calculate the difference between the target downstream bandwidth and the total allocated bit rate as the excess downstream bandwidth; Allocating the excess downstream bandwidth to each allocated bit rate in the first classification set corresponding to the total allocated bit rate to obtain the reference bit rate corresponding to each allocated bit rate.

10. The method according to any one of claims 7-9, characterized in that, The method further includes: Receive at least two voice streams sent by a voice sending terminal, where each voice stream has a corresponding target bit rate; Classify at least two target bit rates of all received voice sending terminals according to the belonging bit rate intervals to obtain at least two second classification sets. Each second classification set has a corresponding bit rate interval, and each second classification set contains the target bit rates of each voice sending terminal belonging to the same bit rate interval; Determine the total target bit rate in each second classification set; For each voice receiving terminal, compare the downlink bandwidth of the voice receiving terminal with the total target bit rate of each second classification set; Based on the comparison result, select a voice stream with an adapted target bit rate and send it to the voice receiving terminal.

11. The method according to claim 10, wherein The step of, based on the comparison result, selecting a voice stream with an adapted target bit rate and sending it to the voice receiving terminal includes: Search for a total target bit rate that is close to the downlink bandwidth of the current voice receiving terminal and smaller than the downlink bandwidth; If the total target bit rate is found, use the second classification set corresponding to the found total target bit rate as the target second classification set, and send the voice streams with the corresponding target bit rates of each voice sending terminal to the voice receiving terminal according to the target second classification set; If the total target bit rate is not found, use a preset dynamic programming algorithm to select the adapted bit rate for each voice sending terminal from all the target bit rates of all voice sending terminals, and send the voice streams with the adapted bit rates of each voice sending terminal to the voice receiving terminal, where the sum of the adapted bit rates of each voice sending terminal is close to the downlink bandwidth of the current voice receiving terminal.

12. The method according to claim 7, wherein The step of determining the target downlink bandwidth of each bit rate interval according to the downlink bandwidth of each voice receiving terminal includes: Sort the downlink bandwidths of each voice receiving terminal, and take quantiles according to at least two set quantile values as the target downlink bandwidths corresponding to the quantile values.

13. A voice processing device, characterized in that, The device is disposed in a voice sending terminal, and includes: An uplink bandwidth prediction module, configured to predict the total uplink bandwidth; A bit rate allocation module, configured to allocate bit rates to the voice streams to be sent based on the total uplink bandwidth to determine at least two allocated bit rates, and send the at least two allocated bit rates to a server; A reference bit rate receiving module, configured to receive the reference bit rates corresponding to the allocated bit rates sent by the server, where the reference bit rates are obtained by the server according to the received allocated bit rates and the estimated downlink bandwidth; A target bit rate determination module, configured to determine the target bit rates corresponding to the allocated bit rates based on the allocated bit rates and the corresponding reference bit rates; An encoding module, configured to encode the voice streams respectively using the target bit rates, and send the encoded voice streams to the server.

14. A voice processing device, characterized in that, The device is disposed in a server, and includes: An allocated bit rate receiving module, configured to receive at least two allocated bit rates for the voice streams to be sent sent by a voice sending terminal; A downlink bandwidth prediction module, configured to determine the voice receiving terminals that receive the voice streams, and predict the downlink bandwidths of each voice receiving terminal; A reference code rate determination module, configured to determine a reference code rate corresponding to each allocated code rate according to the downlink bandwidth of each voice receiving terminal and the at least two allocated code rates; A reference code rate sending module, configured to send the reference code rate to the voice sending terminal, so that the voice sending terminal determines a target code rate of the allocated code rate according to the reference code rate and the corresponding allocated code rate, and encodes the voice stream using the target code rate.

15. A voice processing device, characterized in that, The voice processing device includes: One or more processors; A storage device, configured to store one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1-12.

16. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, the method according to any one of claims 1-12 is implemented.

17. A computer program product, the computer program product includes computer-executable instructions, and the computer-executable instructions are used to implement the method according to any one of claims 1-12 when executed.

Images