Code rate determination method applied to audio and video transmission, electronic device and storage medium

Abstract

The application provides a code rate determination method applied to audio and video transmission, an electronic device and a storage medium. According to the embodiment of the application, quality data fed back by a client in a historical downlink transmission process of audio and video data can be acquired, wherein the quality data comprises at least one of network delay, packet loss rate and response rate in at least two historical downlink transmission processes. An estimated downlink code rate used for downlink transmission of audio and video data is determined according to the quality data, then transmission efficiency evaluation values evaluating transmission quality of the downlink transmission process are respectively determined based on the quality data in the at least two historical downlink transmission processes, and a current code rate adjustment value for the estimated downlink code rate is determined according to the determined transmission efficiency evaluation values. Finally, the estimated downlink code rate is adjusted according to the current code rate adjustment value to determine a target downlink code rate, and audio and video data is downlink transmitted to the client based on the target downlink code rate, so as to control network congestion and reduce audio and video lag and delay.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of network communication, and in particular to a code rate determination method and device applied to audio / video transmission, an electronic device and a storage medium. BACKGROUND

[0002] In recent years, with the development of network communication technology, information delivery through online or offline audio / video transmission has become one of the main means of daily communication. In the scenario of audio / video transmission, especially in the scenario of live broadcast or video conference, the client for receiving audio / video may have a network congestion problem due to insufficient available bandwidth, thereby causing audio / video to appear to be stuck, delayed, and the like, which seriously affects the user experience of the user receiving the audio / video.

[0003] In the scenario of transmitting audio / video data, the sending rate to the client can be controlled by using a downlink code rate suitable for the network transmission condition of the client. However, in actual application, since the audio / video data is transmitted from the server to the client through multiple intermediate devices, the network transmission condition is complex during downlink transmission, and it is very difficult to determine the downlink code rate.

[0004] Therefore, there is an urgent need for a new code rate determination method applied to audio / video transmission to use a more suitable downlink code rate to reduce network congestion and avoid the problem of declining network quality of service (QoS, Quality of Service) due to the inability to timely adjust the downlink code rate when the available bandwidth is insufficient, thereby improving the user experience of the client user. SUMMARY

[0005] Embodiments of the present application provide a code rate determination method applied to audio / video transmission, an electronic device and a storage medium to solve one or more of the above technical problems.

[0006] In a first aspect, the embodiments of the present application provide a code rate determination method applied to audio / video transmission, wherein the method comprises: obtaining quality data fed back by a client for a historical downlink transmission process of audio / video data, the quality data comprising at least one of network delay, packet loss rate and response rate in at least two historical downlink transmission processes; determining an estimated downlink code rate used for the current downlink transmission of audio / video data according to the quality data; determining a transmission efficiency evaluation value evaluating the transmission quality of the downlink transmission process based on the quality data in the at least two historical downlink transmission processes respectively; determining a current code rate adjustment value for the estimated downlink code rate according to the determined transmission efficiency evaluation value; adjusting the estimated downlink code rate according to the current code rate adjustment value to determine a target downlink code rate, and downlink transmitting audio / video data to the client based on the target downlink code rate.

[0007] In a second aspect, an embodiment of the present application provides a code rate determination device applied to audio and video transmission, wherein the device comprises: a data acquisition module configured to acquire quality data fed back by a client for a historical downlink transmission process of audio and video data, the quality data comprising at least one of network delay, packet loss rate and response rate in at least two historical downlink transmission processes; a pre-estimated code rate determination module configured to determine a pre-estimated downlink code rate used for downlink transmission of audio and video data in the current downlink transmission process according to the quality data; an efficiency evaluation value determination module configured to determine a transmission efficiency evaluation value for evaluating transmission quality of the downlink transmission process based on the quality data in the at least two historical downlink transmission processes; a code rate adjustment value determination module configured to determine a current code rate adjustment value for the pre-estimated downlink code rate according to the determined transmission efficiency evaluation value; and a target code rate determination module configured to adjust the pre-estimated downlink code rate to determine a target downlink code rate according to the current code rate adjustment value, and to downlink transmit audio and video data to the client based on the target downlink code rate.

[0008] In a third aspect, an embodiment of the present application provides an electronic device comprising a memory, a processor and a computer program stored in the memory, wherein the processor implements the method of any one of the above aspects when executing the computer program.

[0009] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the method of any one of the above aspects.

[0010] Compared with related technologies, the present application has the following advantages:

[0011] According to the embodiments of the present application, quality data fed back by a client for a historical downlink transmission process of audio and video data is acquired, wherein the quality data comprises at least one of network delay, packet loss rate and response rate in at least two historical downlink transmission processes, and a pre-estimated downlink code rate used for downlink transmission of audio and video data in the current downlink transmission process is determined according to the quality data. Then, a transmission efficiency evaluation value for evaluating transmission quality of the downlink transmission process is determined based on the quality data in the at least two historical downlink transmission processes, and a current code rate adjustment value for the pre-estimated downlink code rate is determined according to the determined transmission efficiency evaluation value. Finally, a target downlink code rate is determined by adjusting the pre-estimated downlink code rate according to the current code rate adjustment value, and audio and video data is downlinked to the client based on the target downlink code rate, so as to control network congestion, reduce audio and video lag and delay, and improve the smoothness of audio and video played by the client.

[0012] In addition, a numerical attribute of a plurality of rate adjustment values including the current rate adjustment value and a historical rate adjustment value determined in history can be obtained. The numerical attribute includes a positive value or a negative value. When the same numerical attribute appears continuously more than a threshold number of times, the absolute value of the current rate adjustment value is increased, and in the case of the same trend of rate conversion, the numerical value of the rate adjustment value is increased, and the rate adjustment speed is accelerated, so as to improve the convergence speed of the target downlink rate, so as to adjust the target downlink rate to a proper value that can improve the smoothness of the audio and video stream more quickly.

[0013] The above description is only a summary of the technical solutions of the present application. In order to enable a clearer understanding of the technical means of the present application, the content of the specification can be implemented, and in order to enable the above and other purposes, features and advantages of the present application to be more obvious and easy to understand, the following specific embodiments of the present application are described. BRIEF DESCRIPTION OF DRAWINGS

[0014] In the drawings, the same reference numerals designate the same or similar elements throughout the several views. The drawings are not necessarily to scale. It is to be understood that these drawings only depict certain embodiments in accordance with the present application and are not to be considered limiting of its scope.

[0015] Figure 1 A scene schematic diagram of a rate determination scheme for audio and video transmission provided in an embodiment of the present application is shown;

[0016] Figure 2 A flowchart of a rate determination method for audio and video transmission provided in an embodiment of the present application is shown;

[0017] Figure 3 A structural block diagram of a rate determination apparatus for audio and video transmission provided in an embodiment of the present application is shown; and

[0018] Figure 4 A block diagram of an electronic device for implementing an embodiment of the present application is shown. DETAILED DESCRIPTION

[0019] In the following, only certain exemplary embodiments are simply described. As those skilled in the art can recognize, the described embodiments can be modified in various different ways without departing from the concept or scope of the present application. Therefore, the drawings and the description are considered to be exemplary in nature, rather than limiting.

[0020] In order to facilitate understanding of the technical solutions of the embodiments of the present application, the related technologies of the embodiments of the present application are described below. The following related technologies can be combined with the technical solutions of the embodiments of the present application in any way as optional solutions, which all belong to the protection scope of the embodiments of the present application.

[0021] Embodiments of the present application relate to determining a code rate used for downlink transmission of audio and video data in real-time communication in a real-time communication (RTC) scenario by analyzing transmission quality in a downlink transmission process. In such a scenario, network congestion occurs when the downlink code rate exceeds the available bandwidth of the client, and even packet loss occurs, which seriously affects the user experience of the client-side user in the real-time communication process.

[0022] In a related technology, a specific processing method is implemented based on a specific network event when solving the related problem of network congestion. However, due to the complexity and randomness of network transmission conditions, for some randomly occurring network events (different from network events caused by specific network conditions), or when the data used to analyze network events has certain errors, the related technology cannot effectively solve the related problem of network congestion.

[0023] Therefore, embodiments of the present application provide a new code rate determination scheme for audio and video transmission to wholly or partially solve the above technical problems. The related concepts and application scenarios involved in the embodiments of the present application are introduced as follows.

[0024] Figure 1 is a schematic diagram of an exemplary application scenario for implementing the method of the embodiments of the present application. In order to distinguish different clients, Figure 1 In the application scenario shown in FIG. 1, the client that sends audio and video data is referred to as a first client, and the client that receives audio and video data is referred to as a second client. In the application scenario shown in FIG. 1, Figure 1 In the application scenario shown in FIG. 1, the user on the first client side communicates with multiple users on the second client side in real time through a network. For example, in the scenario of an online teaching classroom, the user on the first client side can be a teacher. After the teacher triggers the first client to generate a video conference initiation request through a human-computer interaction interface, the first client sends the online teaching classroom initiation request to a server. The server sends prompt information for prompting students (users on the second client side) to participate in the online teaching classroom to at least one second client respectively in response to the online teaching classroom initiation request. The second client generates and displays a trigger control for triggering the second client to enter the online teaching classroom after receiving the prompt information. After the user on the second client side triggers the trigger control through a human-computer interaction interface, the second client can enter the online teaching classroom initiated by the first client.

[0025] In the process of online teaching between the first client and at least one second client, the server continuously receives audio and video data output by the first client and sends the audio and video data to the second client. The second client obtains the audio and video data in real-time communication through the downlink transmission link of the server. In the scenario of audio and video data transmission to multiple second clients, the downlink code rate used for downlink transmission between the server and the second client can be determined for each of the multiple second clients according to the downlink transmission quality. In a possible application example, the server can obtain and analyze the TWCC (Transport Wide Congestion Control) data packet fed back by a single second client to the server based on the accepted audio and video data. The TWCC data packet fed back by the second client includes quality data reflecting the network transmission condition in the downlink transmission process, and the quality data includes network delay, packet loss rate, and response rate.

[0026] The code rate determination apparatus applied to audio and video transmission provided by the embodiment of the application can be deployed in the server. After obtaining the quality data fed back by a second client, the server can analyze the quality data provided by the second client to determine the target downlink code rate (i.e., the target value shown in Figure 1 In the process of determining the target downlink code rate, the estimated downlink code rate (i.e., the estimated value shown in Figure 1 ) used for the downlink transmission of audio and video data in this time can be determined based on the quality data in the historical downlink transmission process. Then, the code rate adjustment value (i.e., the adjustment value shown in Figure 1 ) used for adjusting the estimated downlink code rate can be determined based on the quality data in at least two historical downlink transmission processes. In the determination of the code rate adjustment value, the quality data in the historical downlink transmission process and the estimated downlink code rate in this time can be combined to determine the code rate adjustment value. Finally, the estimated downlink code rate is adjusted using the code rate adjustment value, for example, the sum of the estimated downlink code rate and the code rate adjustment value can be used as the target downlink code rate, and the audio and video data is downlink transmitted to the second client based on the target downlink code rate.

[0027] The application scenario of the embodiment of the application can include online education, video conference, live broadcast, and the like. The specific application scenario is not limited in the application. In actual application, the server can analyze the quality data fed back by multiple second clients as shown in Figure 1 , and determine the target downlink code rate corresponding to the network transmission condition in the downlink transmission process of each second client, to solve the related problems of network congestion in the downlink transmission process to the second client.

[0028] The execution subject of the embodiments of the present application can be an application program, a service, an instance, a functional module in a software form, a virtual machine (VM), a container, or a cloud server, etc., or a hardware device (such as a server or a terminal device) or a hardware chip with a data processing function, such as a central processing unit (CPU), a graphics processing unit (GPU), a field programmable gate array (FPGA), a neural network unit (NPU), an artificial intelligence (AI) acceleration card, or a data processing unit (DPU), etc. The device for realizing the determination of the code rate can be deployed on a computing device of an application party providing a corresponding service or a cloud computing platform providing computing power, storage, and network resources. The mode of providing services by the cloud computing platform can be IaaS (Infrastructure as a Service), PaaS (Platform as a Service), SaaS (Software as a Service), or DaaS (Data as a Service). Taking the platform providing the SaaS (Software as a Service) as an example, the cloud computing platform can utilize its own computing resources to provide the functions of the pre-estimated code rate determination module or the code rate adjustment value determination module, and the specific application architecture can be built according to the service requirements. For example, the platform can provide an architecture building service of a real-time communication application to an application party or an individual using the resources of the platform, and further realize the function of determining the code rate based on the code rate determination request submitted by a related client or server device.

[0029] The method provided by the embodiments of the present application will be described below. It can be understood that the code rate determination method applied to audio and video transmission provided by the embodiments of the present application is mainly used to determine the downlink code rate of downlink transmission of audio and video data to a client. The client involved later is the second client shown in the second embodiment of the present application. Figure 1 The second client is shown in the second embodiment of the present application.

[0030] The embodiments of the present application provide a code rate determination method applied to audio and video transmission, as shown in Figure 2 The flowchart of the code rate determination method 200 applied to audio and video transmission according to an embodiment of the present application is shown in

[0031] In step S201, the quality data fed back by the client for the historical downlink transmission process of audio and video data is obtained. The quality data includes at least one of network latency, packet loss rate and response rate in at least two historical downlink transmission processes.

[0032] The method provided in this application can be applied to a server in audio and video transmission. In this application, the client refers to a client that receives audio and video data. The audio and video data refers to audio or video data transmitted in applications such as live streaming and video conferencing. During the continuous transmission of audio and video data from the server to the client, the client can provide feedback to the server on its data reception status during historical downlink transmissions by sending quality data. The server can determine the downlink bitrate used for transmitting audio and video data to the client in the current transmission by acquiring and analyzing the quality data from at least two historical downlink transmissions. The quality data involved includes at least one of network latency, packet loss rate, and acknowledgment rate. In one application example, the quality data from the at least two historical downlink transmissions may be the quality data fed back by the client based on the two most recent downlink transmissions.

[0033] It is understood that the quality data involved can be obtained from data directly fed back by the client, or through analysis of data packets fed back by the client, or calculated from other relevant data fed back by the client. For example, the specific data for network latency can be calculated from the round-trip time (RTT) fed back by the client. This application does not restrict the specific method of obtaining quality data.

[0034] In one possible implementation, when obtaining the quality data fed back by the client for the historical downlink transmission process of audio and video data, real-time transmission control data packets can be extracted from the client for the historical uplink transmission process corresponding to the historical downlink transmission process at a set time interval, and the quality data obtained by the client analysis can be extracted from the real-time transmission control data packets.

[0035] As mentioned above, in the process of continuously transmitting audio and video data from the server to the client, the client can feed back quality data for the downlink transmission to the server. The client can feed back the quality data to the server by transmitting the implementation transmission control data packet to the server based on the uplink transmission. The implementation transmission control data packet can include the RTP (Real-time Transport Protocol) data packet, the RTCP (Real-time Transport Control Protocol) data packet, and the TWCC (Transport Wide Congestion Control) data packet, etc. The real-time transport control data packet includes the network delay, the packet loss rate, and the response rate, etc. The server can extract the real-time transport control data packet fed back by the client according to the set time interval, and extract the quality data from the obtained real-time transport control data packet, so as to obtain the quality data fed back by the client for the historical downlink transmission process of the audio and video data. In an application example, the time interval can be set according to the average round-trip delay of the downlink transmission. For example, the time interval can be set as half of the average round-trip delay, so as to ensure the effectiveness of the obtained quality data.

[0036] Since the obtained quality data can have errors, in a possible implementation manner, a low-pass filtering algorithm and / or a linear regression algorithm, or other algorithms for correcting data errors can be called to correct the errors of the quality data, so as to eliminate the noise of the obtained quality data and improve the accuracy of the quality data.

[0037] In step S202, the estimated downlink code rate used for the current downlink transmission of the audio and video data is determined according to the quality data.

[0038] In the embodiment of the present application, in order to distinguish the multiple downlink transmissions, the historical downlink transmission is recorded as the historical downlink transmission, and the upcoming downlink transmission is recorded as the current downlink transmission. The application of the embodiment of the present application can determine the target downlink code rate used for the current downlink transmission.

[0039] After the quality data is acquired, first, the estimated downlink code rate used by the current downlink transmission audio and video data is determined according to the acquired quality data. The estimated downlink code rate refers to the downlink code rate suitable for the current client network transmission state analyzed and estimated according to the acquired quality data. In an application example, the estimated downlink code rate can be obtained by using a BBR (Bottleneck Bandwidth and Round-trip propagation time) algorithm, a GCC (a network congestion control algorithm for real-time media communication) or other methods. The embodiments of the present application do not limit the way of obtaining the estimated downlink code rate.

[0040] In step S203, the transmission performance evaluation value for evaluating the transmission quality of the downlink transmission process is determined based on the quality data in at least two historical downlink transmission processes.

[0041] The transmission performance evaluation value is a quantitative value for evaluating the transmission quality of a downlink transmission process. As described above, the quality data can be used to reflect the network transmission status in the downlink transmission process. In the embodiments of the present application, the transmission performance evaluation value is determined based on the quality data. In actual applications, multiple quality data can be comprehensively considered, and the multiple quality data can be converted into the transmission performance evaluation value to reflect the transmission quality of the downlink transmission process.

[0042] In a possible implementation, when the transmission performance evaluation value for evaluating the transmission quality of the downlink transmission process is determined based on the quality data in at least two historical downlink transmission processes, the quality data in the at least two historical downlink transmission processes can be respectively input into a set code rate adjustment strategy function, and then the transmission performance evaluation value for evaluating the transmission quality of the downlink transmission process can be output by the code rate adjustment strategy function.

[0043] The code rate adjustment strategy function can map multiple quality data into a transmission performance evaluation value. The transmission performance control coefficients corresponding to the quality data are respectively configured in the code rate adjustment strategy function, and the transmission performance control coefficients respectively configured for the quality data can be adjusted to obtain a more accurate transmission performance evaluation value. The quality data in the at least two historical downlink transmission processes can be determined by the two latest real-time transmission control data packets fed back by the second client, for example.

[0044] In an application example, the code rate adjustment strategy function can be denoted as U, and based on the network delay (t) and the packet loss rate (l) in the downlink transmission process involved in the embodiments of the present application and the estimated downlink code rate (x) acquired in the historical downlink transmission process, the code rate adjustment strategy function can be determined as follows:

[0045] U(x, t, l) = f(x) + g(t) + k(l)

[0046] wherein f(x) is a monotonically increasing function, g(t) and k(l) are monotonically decreasing functions. In the above f(x), g(t) and k(l), x, t and l are respectively configured with corresponding transmission performance control coefficients. In actual application, the transmission performance control coefficients can be adjusted according to actual application scenarios to improve the accuracy of the transmission efficiency evaluation value. Since the code rate adjustment strategy function U(x, t, l) includes a monotonically increasing function related to the downlink code rate, a monotonically decreasing function related to the network delay and the packet loss rate, the calculation result obtained from U(x, t, l) can be used to evaluate the transmission efficiency evaluation value in the scenario of encouraging the growth of the downlink code rate while punishing the growth of the network delay and the packet loss rate.

[0047] The method provided by the embodiments of the present application controls network congestion by determining the target downlink code rate, and therefore, in actual application, in order to better observe the change of the transmission quality caused by the change of the downlink code rate, f(x) can be set as a function with a faster growth rate, such as an exponential function. It can be understood that the present application example only illustratively shows a possible code rate adjustment strategy function, and in actual application, the code rate adjustment strategy function can also include functions related to other types of quality data, and the embodiments of the present application do not limit the specific setting of the code rate adjustment strategy function and the specific configuration of the transmission performance control coefficients in the code rate adjustment strategy function.

[0048] In step S204, the current code rate adjustment value for the estimated downlink code rate is determined according to the determined transmission efficiency evaluation value.

[0049] The code rate adjustment value is used to adjust the estimated downlink code rate. In the embodiment of the present application, the transmission performance evaluation value is used to evaluate the transmission quality. The greater the transmission performance evaluation value, the better the transmission quality. In actual application, in order to provide better network service quality, the estimated downlink code rate can be adjusted based on the transmission performance evaluation value. That is, after the transmission performance evaluation value is determined, the code rate adjustment value is determined according to the transmission performance evaluation value, so as to adjust the estimated downlink code rate. The adjustment can include adjustment direction and adjustment degree. Specifically, the adjustment direction can be determined by the numerical property (positive or negative) of the code rate adjustment value, and the adjustment degree can be determined by the absolute value of the code rate adjustment value. For example, when the code rate adjustment value is determined, if it is found through further mining of the quality data that the estimated downlink code rate needs to be increased, the numerical property of the code rate adjustment value can be determined as positive. When the estimated downlink code rate needs to be adjusted by a large degree, the absolute value of the code rate adjustment value can be determined as a large value.

[0050] In an application example, the estimated downlink code rate can be first adjusted to be enlarged or reduced respectively, to obtain a plurality of estimated downlink code rates. Then, the quality data determined in the last history is combined respectively to obtain a plurality of transmission performance evaluation values. The code rate adjustment value is determined by comparing the numerical values of the transmission performance evaluation values and the corresponding estimated downlink code rates.

[0051] In a possible implementation, when the code rate adjustment value for the estimated downlink code rate is determined according to the determined transmission performance evaluation value, the gradient change data of the transmission performance evaluation value can be first determined, and then the code rate adjustment value is determined according to the gradient change data of the transmission performance evaluation value. It can be understood that the gradient change data can be used to reflect the change direction and degree of the function, so that a plurality of transmission performance evaluation values can be determined according to a plurality of sets of quality data and estimated downlink code rates, and then the gradient change data of the transmission performance evaluation value is determined according to the plurality of transmission performance evaluation values, and then the code rate adjustment value is determined.

[0052] In a possible implementation, the numerical properties of a plurality of code rate adjustment values can also be counted. The plurality of code rate adjustment values include the current code rate adjustment value and the historical code rate adjustment value determined in the history. The numerical properties include positive or negative. If the same numerical property appears continuously for more than a set number of times, it indicates that the estimated downlink code rate has been adjusted in the same direction for many times, so that the absolute value of the current code rate adjustment value can be increased to speed up the adjustment speed of the estimated downlink code rate in the same direction.

[0053] In a possible implementation, the absolute value of the current rate adjustment value can also be compared with a rate absolute value upper limit. The initial value of the rate absolute value upper limit can be preset. If the absolute value of the current rate adjustment value exceeds the rate absolute value upper limit, the absolute value of the current rate adjustment value is replaced by the rate absolute value upper limit. In actual application, a situation that the current rate adjustment value is excessively large can occur due to a random network event causing a large change in quality data. By setting the rate absolute value upper limit, the adjustment degree of the current rate adjustment value can be limited, the unreasonable current rate adjustment value in the above situation can be avoided to affect normal downlink transmission, the reliability of the current rate adjustment value is improved, and the stability of downlink transmission in network jitter is ensured.

[0054] In a possible implementation, the numerical properties of a plurality of rate adjustment values can also be counted. The plurality of rate adjustment values include the current rate adjustment value and historical rate adjustment values determined in history. If the same numerical property appears continuously for more than a set number of times, the rate absolute value upper limit is increased. In addition, when the numerical properties of the last two rate adjustment values are inconsistent (for example, the last rate adjustment value is a positive value, and the current rate adjustment value is a negative value), the rate absolute value upper limit is restored to the initial value.

[0055] In a possible implementation, the rate control state and / or bandwidth usage state can also be determined according to the quality data in the last two historical downlink transmission processes. Then, the current rate adjustment value is corrected according to the rate control state and / or bandwidth usage state. The rate control state includes one of increase, hold, or decrease. The bandwidth usage state includes one of normal use, underuse, or overuse. In one application example, in a case where the current rate adjustment value is a positive value, if the bandwidth usage states determined by the quality data in the last two historical downlink transmission processes are both underuse, it is indicated that the available bandwidth is relatively idle, and the current rate adjustment value can be determined to be used continuously, or the current rate adjustment value is appropriately corrected to a larger positive value. In another application example, in a case where the current rate adjustment value is a positive value, if the rate control states determined by the quality data in the last two historical downlink transmission processes are increase and hold respectively, it is indicated that the rate control has a decreasing trend, and the current rate adjustment value can be appropriately corrected to a smaller positive value. In addition, the current rate adjustment value can also be corrected in combination with the rate control state and the bandwidth usage state. The specific correction manner or processing logic can be referred to the foregoing description, which is not described herein again.

[0056] In step S205, the target downlink code rate is determined by adjusting the estimated downlink code rate according to the current code rate adjustment value, and the audio and video data is transmitted to the client based on the target downlink code rate.

[0057] That is, the target downlink code rate is determined based on the current code rate adjustment value and the estimated downlink code rate. In an application example, the current code rate adjustment value and the estimated downlink code rate can be added together, and the sum is determined as the target downlink code rate. Compared with the estimated downlink code rate, the target downlink code rate determined based on the current code rate adjustment value and the estimated downlink code rate is closer to the available bandwidth in the downlink transmission process, so that the downlink transmission of audio and video data to the client based on the target downlink code rate can quickly determine the downlink code rate suitable for the current network transmission state when the network transmission state in the downlink transmission process changes, thereby maximizing the transmission efficiency of the audio data, improving the smoothness of the client playing the audio and video, and avoiding packet loss, delay and other problems caused by network congestion.

[0058] In a possible implementation, before the application embodiment is formally started to determine the target downlink code rate, the downlink code rate used for the current downlink transmission of audio and video data can also be assigned in the initial stage, and the downlink code rate is controlled to increase gradually in the assignment process. In actual application, the assignment of the downlink code rate can be gradually increased from 0 or from a pre-set initial assignment. In this process, the transmission efficiency evaluation value estimated by the downlink transmission of audio and video data corresponding to each assignment of the downlink code rate can be determined. When the transmission efficiency evaluation value decreases relative to the last transmission efficiency evaluation value, the step of obtaining the quality data of the historical downlink transmission feedback of the client for the audio and video data is performed, that is, the method provided by the application embodiment is formally started and used.

[0059] Since the transmission efficiency evaluation value can be used to evaluate the transmission quality in the downlink transmission process, by continuously increasing the assignment of the downlink code rate and obtaining the corresponding transmission efficiency evaluation value, the downlink code rate can be determined to have reached the maximum value that can be matured by the network transmission state in the current downlink transmission process when the transmission efficiency evaluation value decreases for the first time. In this way, the initial target downlink code rate of the client can be quickly determined in the initial stage to avoid prematurely starting the step of adjusting the estimated downlink code rate based on the current code rate adjustment value and then determining the target downlink code rate, thereby avoiding the situation that the user experience is not good due to the client using a lower downlink code rate to transmit the audio and video data in the initial stage.

[0060] Corresponding to the application scenario and method of the method provided by the application embodiment, the application embodiment also provides a code rate determination device applied to audio and video transmission. As Figure 3Fig. 3 is a structural block diagram of a code rate determination device 300 for audio and video transmission according to an embodiment of the present application. The device 300 can include:

[0061] A data acquisition module 301 is configured to acquire quality data fed back by a client for a historical downlink transmission process of audio and video data, the quality data including at least one of network delay, packet loss rate and response rate in at least two historical downlink transmission processes;

[0062] A predicted code rate determination module 302 is configured to determine a predicted downlink code rate for downlink transmission of audio and video data in the current downlink transmission process according to the quality data;

[0063] An efficiency evaluation value determination module 303 is configured to determine a transmission efficiency evaluation value for evaluating transmission quality of the downlink transmission process based on the quality data in at least two historical downlink transmission processes, respectively;

[0064] A code rate adjustment value determination module 304 is configured to determine a current code rate adjustment value for the predicted downlink code rate according to the determined transmission efficiency evaluation value;

[0065] A target code rate determination module 305 is configured to adjust the predicted downlink code rate to determine a target downlink code rate according to the current code rate adjustment value, and to downlink transmit audio and video data to the client based on the target downlink code rate.

[0066] In a possible implementation, the efficiency evaluation value determination module 303 can include:

[0067] A code rate input sub-module is configured to input the quality data in at least two historical downlink transmission processes and the predicted downlink code rate into a set code rate adjustment strategy function, respectively, wherein the code rate adjustment strategy function is configured with corresponding transmission performance control coefficients for the quality data, respectively;

[0068] An efficiency evaluation value output sub-module is configured to output a transmission efficiency evaluation value for evaluating transmission quality of the downlink transmission process for the code rate adjustment strategy function.

[0069] In a possible implementation, the code rate adjustment value determination module includes:

[0070] A gradient change data determination sub-module is configured to determine gradient change data of the transmission efficiency evaluation value;

[0071] A code rate adjustment value determination sub-module is configured to determine the current code rate adjustment value according to the gradient change data of the transmission efficiency evaluation value.

[0072] In a possible implementation, the device 300 further includes:

[0073] The first attribute statistics module is configured to count a numerical attribute of a plurality of rate adjustment values, the plurality of rate adjustment values including the current rate adjustment value and historical rate adjustment values determined in the past, and the numerical attribute including a positive value or a negative value.

[0074] The adjustment value up-regulation module is configured to up-regulate an absolute value of the current rate adjustment value when a number of consecutive occurrences of the same numerical attribute exceeds a set number threshold.

[0075] In a possible implementation, the apparatus 300 further includes:

[0076] The rate comparison module is configured to compare the absolute value of the current rate adjustment value with a rate absolute value upper limit.

[0077] The rate replacement module is configured to replace the absolute value of the current rate adjustment value with the rate absolute value upper limit when the absolute value of the current rate adjustment value exceeds the rate absolute value upper limit.

[0078] In a possible implementation, the apparatus 300 further includes:

[0079] The second attribute statistics module is configured to count a numerical attribute of a plurality of rate adjustment values, the plurality of rate adjustment values including the current rate adjustment value and historical rate adjustment values determined in the past, and the numerical attribute including a positive value or a negative value.

[0080] The upper limit up-regulation module is configured to up-regulate the rate absolute value upper limit when a number of consecutive occurrences of the same numerical attribute exceeds a set number threshold.

[0081] In a possible implementation, the apparatus 300 further includes:

[0082] The attribute comparison module is configured to compare a numerical attribute of the current rate adjustment value and a last rate adjustment value, the plurality of rate adjustment values including the current rate adjustment value and historical rate adjustment values determined in the past, and the numerical attribute including a positive value or a negative value.

[0083] The upper limit recovery module is configured to recover the rate absolute value upper limit to an initial value when the numerical attributes of the last two rate adjustment values are inconsistent.

[0084] In a possible implementation, the apparatus 300 further includes:

[0085] The state determination module is configured to determine a rate control state and / or a bandwidth usage state according to quality data in the last two historical downlink transmission processes, wherein the rate control state includes one of increasing, maintaining, or decreasing, and the bandwidth usage state includes one of normal usage, insufficient usage, or overload usage.

[0086] The adjustment value correction module is used to correct the current bitrate adjustment value according to the rate control status and / or the bandwidth usage status.

[0087] In one possible implementation, the device 300 further includes:

[0088] The algorithm invocation module is used to invoke low-pass filtering algorithms and / or linear regression algorithms to perform error correction on the quality data.

[0089] In one possible implementation, the device 300 further includes:

[0090] The bitrate assignment module is used to assign a downlink bitrate to the audio and video data transmitted in this downlink transmission; during the assignment process, the downlink bitrate is controlled to increase sequentially.

[0091] The performance evaluation value determination module is used to determine the estimated transmission performance evaluation value of the downlink audio and video data transmitted in this downlink transmission corresponding to the downlink bit rate assigned in each assignment.

[0092] The step execution module is used to execute the step of obtaining the quality data of the client's historical downlink transmission feedback for audio and video data when the transmission performance evaluation value of a certain transmission performance decreases compared with the previous transmission performance evaluation value.

[0093] In one possible implementation, the data acquisition module 301 can be specifically used to: extract real-time transmission control data packets from the client during the historical uplink transmission process corresponding to the historical downlink transmission process at set time intervals, and extract the quality data obtained by the client analysis from the real-time transmission control data packets.

[0094] The functions of each module in each device in the embodiments of this application can be found in the corresponding description in the above method, and they have corresponding beneficial effects, which will not be repeated here.

[0095] Figure 4 This is a block diagram of an electronic device used to implement embodiments of this application. For example... Figure 4 As shown, the electronic device includes a memory 401 and a processor 402. The memory 401 stores a computer program that can run on the processor 402. When the processor 402 executes the computer program, it implements the method described in the above embodiments. The number of memories 401 and processors 402 can be one or more.

[0096] The electronic device also includes:

[0097] Communication interface 403 is used to communicate with external devices and perform data exchange and transmission.

[0098] If the memory 401, the processor 402 and the communication interface 403 are implemented independently, the memory 401, the processor 402 and the communication interface 403 can be connected to each other through a bus and complete communication between each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For convenience of representation, Figure 4 Only one thick line is used to represent the bus in the figure, but it does not mean that there is only one bus or only one type of bus.

[0099] Optionally, in a specific implementation, if the memory 401, the processor 402 and the communication interface 403 are integrated on a chip, the memory 401, the processor 402 and the communication interface 403 can complete communication between each other through an internal interface.

[0100] The embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to implement the method provided in the embodiment of the present application.

[0101] The embodiment of the present application further provides a chip, which includes a processor, is used for calling and running instructions stored in a memory, and makes a communication device installed with the chip execute the method provided in the embodiment of the present application.

[0102] The embodiment of the present application further provides a chip, which includes an input interface, an output interface, a processor and a memory, the input interface, the output interface, the processor and the memory are connected through an internal connection path, and the processor is used for executing code in the memory, and when the code is executed, the processor is used for executing the method provided in the embodiment of the present application.

[0103] It is to be understood that the above-mentioned processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc. It is to be noted that the processor can be a processor supporting an advanced RISC machine (ARM) architecture.

[0104] Further, the memory can include a read-only memory and a random access memory, optionally. The memory can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. The non-volatile memory can include a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically EPROM (EEPROM), or a flash memory, for example. The volatile memory can include a random access memory (RAM) used as an external cache. Many forms of RAM are available by way of example but not limitation. For example, a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous DRAM (SDRAM), a double data rate SDRAM (DDR SDRAM), an enhanced SDRAM (ESDRAM), a SyncLink DRAM (SLDRAM), and a direct Rambus RAM (DR RAM) can be used.

[0105] In the above-described embodiments, all or part of the embodiments can be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded on a computer, all or part of the processes or functions according to the present application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another computer-readable storage medium.

[0106] In the description of the present specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. Also, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. In addition, a person skilled in the art can combine and combine the different embodiments or examples described in the present specification and the features of the different embodiments or examples without contradiction.

[0107] In addition, the terms "first", "second", etc. are used only for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the technical features indicated. Therefore, the features defined with "first", "second" can explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "a plurality of" is two or more, unless otherwise explicitly specified.

[0108] Any process or method described in the flowchart or otherwise described herein can be understood as a representation of code, including one or more executable instructions for implementing specific logical functions or steps, modules, segments or portions. And the scope of the preferred embodiments of the present application includes additional implementations, in which the functions can be performed in an order other than that shown or discussed, including in a substantially simultaneous manner according to the functions involved or in reverse order.

[0109] The logic and / or steps described in the flowchart or otherwise described herein, for example, can be considered as a list of executable instructions for implementing logical functions, which can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, device or apparatus, such as a computer-based system, a system including a processor, or other system that can take instructions from an instruction execution system, device or apparatus and execute them, or in conjunction with these instruction execution systems, devices or apparatus.

[0110] It should be understood that each part of the present application can be realized by hardware, software, firmware or a combination thereof. In the above embodiments, a plurality of steps or methods can be realized by software or firmware stored in a memory and executed by a suitable instruction execution system. All or part of the steps of the above-mentioned embodiment methods can be completed by a program instructing the relevant hardware, which can be stored in a computer readable storage medium and includes one or a combination of the steps of the embodiment methods when executed.

[0111] In addition, each functional unit in each embodiment of the present application can be integrated in one processing module, or each unit can be physically present separately, or two or more units can be integrated in one module. The above-mentioned integrated module can be realized in the form of hardware or in the form of a software functional module. The above-mentioned integrated module, if realized in the form of a software functional module and sold or used as an independent product, can also be stored in a computer readable storage medium. The storage medium can be a read-only memory, a magnetic disk or an optical disk, etc.

[0112] The above is only an exemplary embodiment of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of various changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for determining bitrate applied to audio and video transmission, wherein, include: Obtain quality data fed back by the client for the historical downlink transmission process of audio and video data. The quality data includes at least one of network latency, packet loss rate and response rate in at least two historical downlink transmission processes. The estimated downlink bitrate used for this downlink audio and video data transmission is determined based on the quality data. Based on quality data from at least two historical downlink transmission processes, a transmission performance evaluation value for evaluating the transmission quality of the downlink transmission process is determined; wherein, for each historical downlink transmission process in the at least two historical downlink transmission processes, the quality data from the historical downlink transmission process and the estimated downlink code rate are input into a set code rate adjustment strategy function to obtain a transmission performance evaluation value determined based on the quality data from the historical downlink transmission process and the estimated downlink code rate. The current bit rate adjustment value for the estimated downlink bit rate is determined based on the determined transmission performance evaluation values. The estimated downlink bitrate is adjusted according to the current bitrate adjustment value to determine the target downlink bitrate, and audio and video data are transmitted downlink to the client based on the target downlink bitrate.

2. The method according to claim 1, wherein, The transmission performance evaluation values ​​for evaluating the transmission quality of the downlink transmission process, determined based on quality data from at least two historical downlink transmission processes, include: The quality data from at least two historical downlink transmission processes are respectively compared with the estimated downlink bit rate input to the bit rate adjustment strategy function, wherein the bit rate adjustment strategy function is configured with corresponding transmission performance control coefficients for the quality data. The bitrate adjustment strategy function outputs a transmission performance evaluation value used to assess the transmission quality of the downlink transmission process.

3. The method according to claim 2, wherein, The step of determining the current bitrate adjustment value for the estimated downlink bitrate based on the determined transmission performance evaluation values ​​includes: Determine the gradient change data of the transmission performance evaluation value; The current bitrate adjustment value is determined based on the gradient change data of the transmission performance evaluation value.

4. The method according to claim 1, wherein, The method further includes: The numerical attributes of multiple bitrate adjustment values ​​are statistically analyzed. The multiple bitrate adjustment values ​​include the current bitrate adjustment value and historically determined bitrate adjustment values. The numerical attributes include positive or negative values. If the number of consecutive occurrences of the same numerical attribute exceeds the set threshold, then the absolute value of the current bitrate adjustment value will be increased.

5. The method according to claim 1, wherein, The method further includes: Compare the absolute value of the current bitrate adjustment with the upper limit of the absolute bitrate value; If the absolute value of the current bitrate adjustment exceeds the upper limit of the absolute bitrate value, then the absolute value of the current bitrate adjustment is replaced with the upper limit of the absolute bitrate value.

6. The method according to claim 5, wherein, The method further includes: The numerical attributes of multiple bitrate adjustment values ​​are statistically analyzed. The multiple bitrate adjustment values ​​include the current bitrate adjustment value and historically determined bitrate adjustment values. The numerical attributes include positive or negative values. If the number of consecutive occurrences of the same numerical attribute exceeds the set threshold, the upper limit of the absolute value of the bitrate will be increased.

7. The method according to claim 5, wherein, The method further includes: Compare the numerical attributes of the current bitrate adjustment value and the previous bitrate adjustment value, where the numerical attributes include positive or negative values; If the numerical attributes of the two most recent bitrate adjustments are inconsistent, the upper limit of the absolute bitrate value will be restored to the initial value.

8. The method according to claim 1, wherein, The method further includes: The rate control status and / or bandwidth usage status are determined based on the quality data from the two most recent historical downlink transmissions; wherein, the rate control status includes one of increasing, maintaining, or decreasing; and the bandwidth usage status includes one of normal usage, insufficient usage, or overloaded usage. The current bitrate adjustment value is corrected based on the rate control status and / or the bandwidth usage status.

9. The method according to claim 1, wherein, The method further includes: The quality data is corrected by invoking a low-pass filtering algorithm and / or a linear regression algorithm.

10. The method according to claim 1, wherein, The method further includes: The downlink bitrate used for this downlink audio and video data transmission is assigned a value; during the assignment process, the downlink bitrate is controlled to increase sequentially. Determine the estimated transmission performance evaluation value of the downlink audio and video data transmitted in this downlink transmission corresponding to the downlink bit rate of each assignment; When a transmission performance evaluation value decreases compared to the previous transmission performance evaluation value, the step of obtaining the historical downlink transmission quality data of the client for audio and video data is executed.

11. The method according to claim 1, wherein, The quality data obtained from the client's historical downlink transmission process for audio and video data includes: The client extracts real-time transmission control data packets from the historical uplink transmission process corresponding to the historical downlink transmission process at set time intervals, and extracts the quality data obtained by the client analysis from the real-time transmission control data packets.

12. A bitrate determination device for audio and video transmission, wherein, include: The data acquisition module is used to acquire quality data fed back by the client for the historical downlink transmission process of audio and video data. The quality data includes at least one of network latency, packet loss rate and response rate in at least two historical downlink transmission processes. The estimated bitrate determination module is used to determine the estimated downlink bitrate used for this downlink transmission of audio and video data based on the quality data. The performance evaluation value determination module is used to determine the transmission performance evaluation value for evaluating the transmission quality of the downlink transmission process based on the quality data of at least two historical downlink transmission processes; wherein, for each historical downlink transmission process in the at least two historical downlink transmission processes, the quality data of the historical downlink transmission process and the estimated downlink code rate are input into a set code rate adjustment strategy function to obtain the transmission performance evaluation value determined based on the quality data of the historical downlink transmission process and the estimated downlink code rate; The bitrate adjustment value determination module is used to determine the current bitrate adjustment value for the estimated downlink bitrate based on the determined transmission performance evaluation values; The target bitrate determination module is used to adjust the estimated downlink bitrate according to the current bitrate adjustment value to determine the target downlink bitrate, and transmit audio and video data to the client downlink based on the target downlink bitrate.

13. An electronic device comprising a memory, a processor, and a computer program stored in the memory, wherein the processor, when executing the computer program, implements the method of any one of claims 1-11.

14. A computer-readable storage medium storing a computer program that, when executed by a processor, implements the method of any one of claims 1-11.

Images