Multimedia data transmission rate control method, device, apparatus and storage medium

Abstract

The application discloses a multimedia data sending rate control method and device, equipment and a storage medium. The live broadcast client determines whether the communication network between the live broadcast client and the live broadcast server is congested and the congestion degree based on the sending total time length and the receiving total time length of at least two groups of multimedia data received from the live broadcast server, avoids determining whether the congestion occurs by determining whether the communication network is packet loss, and informs the live broadcast server of the congestion degree of the communication network, so that the live broadcast server can adjust the multimedia data sending rate according to the congestion degree, avoids the problem that the multimedia data sending rate is reduced in the case that the network bandwidth is sufficient but the packet loss rate is high, thereby stably sending the multimedia data in the case that the network bandwidth is sufficient but the packet loss rate is high, and the data sending rate can be adjusted in time when the video, audio or other data transmission scene is determined to be congested, so that the data transmission is more stable and safe.

Description

Technical Field

[0001] This invention relates to the field of communications, and more specifically to a multimedia data transmission rate control method, apparatus, device, and storage medium. Background Technology

[0002] In a multimedia system consisting of a live streaming server and a live streaming client, multimedia data is transmitted between the server and client via a communication network. During data transmission, network congestion may occur due to the large amount of multimedia data being transmitted and the limited network resources of the communication network. To improve network resource utilization and optimize network transmission, congestion control is necessary. For example, congestion control can be achieved using algorithms such as Cubic, BBR, SRT, and / or GCC in WebRTC. During research and practice with existing technologies, the inventors of this invention discovered that existing algorithms such as Cubic, BBR, SRT, and GCC determine network congestion based on packet loss during multimedia data transmission, i.e., they determine the network state based on packet loss. This can lead to a decrease in the multimedia data transmission rate when network bandwidth is sufficient but the packet loss rate is high. Summary of the Invention

[0003] This invention provides a multimedia data transmission rate control method, apparatus, device, and storage medium, which can avoid reducing the multimedia data transmission rate when the network bandwidth is sufficient but the packet loss rate is high.

[0004] This invention provides a multimedia data transmission rate control method, applied to a multimedia system, the multimedia system including a live streaming server and a live streaming client, comprising:

[0005] The live streaming client receives at least two sets of multimedia data sent by the live streaming server, and obtains the total transmission time spent when the multimedia data is sent by the live streaming server, and the total reception time spent when the live streaming client receives the multimedia data.

[0006] The live streaming client determines the latency gradient generated when each set of multimedia data is transmitted in the communication network between the live streaming server and the live streaming client based on the total transmission and reception duration of each set of multimedia data.

[0007] The live streaming client smooths the latency gradient to obtain several processed latency gradients.

[0008] The live streaming client determines the congestion level information of the communication network based on the changes between the processed delay gradients, and sends the congestion level information to the live streaming server.

[0009] The live streaming server adjusts the multimedia data transmission rate for the live streaming client based on the congestion information.

[0010] Accordingly, embodiments of the present invention also provide a multimedia data transmission rate control device, applied to a multimedia system, the multimedia system including a live streaming server and a live streaming client, comprising:

[0011] The duration determination unit is used to receive at least two sets of multimedia data sent by the live streaming server through the live streaming client, and to obtain the total transmission time spent when the multimedia data is sent by the live streaming server and the total reception time spent when the data is received by the live streaming client.

[0012] The delay determination unit is used to determine the delay gradient generated when each set of multimedia data is transmitted in the communication network between the live streaming server and the live streaming client, based on the total transmission duration and total reception duration of each set of multimedia data by the live streaming client.

[0013] A smoothing unit is used to smooth the latency gradient through the live streaming client to obtain several processed latency gradients.

[0014] A congestion determination unit is used to determine the congestion level information of the communication network based on the changes between the processed delay gradients by the live streaming client, and to send the congestion level information to the live streaming server.

[0015] A rate adjustment unit is used to adjust the multimedia data transmission rate for the live streaming client based on the congestion information provided by the live streaming server.

[0016] Optionally, the smoothing unit is further configured to:

[0017] The live streaming client determines several sets of delay gradients to be processed from multiple delay gradients, wherein a set of delay gradients to be processed includes at least two delay gradients;

[0018] The live streaming client obtains the weights corresponding to each delay gradient included in each group of delay gradients to be processed;

[0019] The live streaming client forms a weighted latency gradient corresponding to each set of latency gradients to be processed based on the latency gradients included in each set of latency gradients to be processed and the weights corresponding to each latency gradient.

[0020] The live streaming client obtains several processed delay gradients based on the weighted delay gradients.

[0021] Optionally, the congestion determination unit is further configured to:

[0022] The live streaming client determines the first arrangement order of each delay gradient based on the receiving order of each group of multimedia data;

[0023] The live streaming client obtains the correspondence between each processed latency gradient and each latency gradient;

[0024] The live streaming client determines the second arrangement order of the processed latency gradient based on the first arrangement order and the correspondence;

[0025] The live streaming client determines the changes in each of the processed latency gradients under the second arrangement order;

[0026] The live streaming client determines the congestion level information of the communication network based on the changes, and sends the congestion level information to the live streaming server.

[0027] Optionally, the congestion determination unit is further configured to:

[0028] The live streaming client performs linear regression on each of the processed time delay gradients based on the second arrangement order to obtain a linear regression equation;

[0029] The live streaming client determines the trend and rate of change of each processed latency gradient based on the slope of the linear regression equation.

[0030] The live streaming client uses the changing trend and the changing rate as the changes in the post-processed delay gradients under the second arrangement order.

[0031] Optionally, the congestion determination unit is further configured to:

[0032] If the change trend is an increasing trend, and the post-processing delay gradient at the end of the second arrangement order is greater than the delay gradient threshold, the live streaming client determines that the communication network is in a congested state, and determines the degree of congestion of the communication network when it is in the congested state based on the change rate.

[0033] The live streaming client uses the congestion status and the degree of congestion as congestion information of the communication network;

[0034] If the trend of change is a decreasing trend and / or the post-processing delay gradient at the end of the second arrangement order is not greater than the delay gradient threshold, the live streaming client determines that the communication network is in a non-congested state, and determines the degree of non-congestion of the communication network when it is in the non-congested state based on the rate of change.

[0035] The live streaming client uses the non-congestion state and the degree of non-congestion as congestion information of the communication network.

[0036] Optionally, the rate adjustment unit is further configured to:

[0037] If the communication network is determined to be in a congested state based on the congestion level information, the live streaming server reduces the multimedia data transmission rate based on the congestion level.

[0038] If the communication network is determined to be in a non-congested state based on the congestion level information, the live streaming server increases the multimedia data transmission rate based on the non-congestion level.

[0039] Optionally, the rate adjustment unit is further configured to:

[0040] The live streaming server obtains the inverse relationship between the rate of change corresponding to the congestion level and the first coefficient, and determines the first coefficient based on the inverse relationship and the rate of change corresponding to the congestion level.

[0041] The live streaming server reduces the multimedia data transmission rate based on the first coefficient.

[0042] The live streaming server obtains the proportional relationship between the rate of change corresponding to the degree of non-congestion and the second coefficient, and determines the second coefficient based on the proportional relationship and the rate of change corresponding to the degree of non-congestion;

[0043] The live streaming server increases the multimedia data transmission rate based on the second coefficient.

[0044] Optionally, the rate adjustment unit is further configured to:

[0045] The live streaming server obtains the input bitrate of the multimedia data that the current application layer sends to the live streaming client.

[0046] The live streaming server determines the retransmission rate when retransmitting the multimedia data to the live streaming client.

[0047] The live streaming server determines the current multimedia data transmission rate for the live streaming client based on the input bitrate and the retransmission rate.

[0048] Optionally, the duration determination unit is further configured to:

[0049] The live streaming client calculates the difference between the total reception time and the total transmission time for each set of multimedia data, which is used as the latency gradient generated when each set of multimedia data is transmitted in the communication network.

[0050] Similarly, embodiments of this application also provide a computer device, including:

[0051] Memory, used to store computer programs;

[0052] A processor configured to implement the steps of any one of the multimedia data transmission rate control methods when executing the computer program.

[0053] Furthermore, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of any one of the multimedia data transmission rate control methods.

[0054] This invention provides a multimedia data transmission rate control method, apparatus, device, and storage medium. The live streaming client can determine whether the communication network between the live streaming client and the live streaming server is congested, and the degree of congestion, based on the total transmission and reception durations of at least two sets of multimedia data received from the live streaming server. This avoids determining congestion based on packet loss. The live streaming client then informs the live streaming server of the network congestion level, allowing the live streaming server to adjust the multimedia data transmission rate accordingly. This avoids reducing the multimedia data transmission rate when network bandwidth is sufficient but the packet loss rate is high. It ensures stable multimedia data transmission even with sufficient bandwidth and a high packet loss rate, and allows for timely adjustment of the multimedia data transmission rate when congestion is detected, preventing severe congestion. Attached Figure Description

[0055] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0056] Figure 1 This is a schematic diagram of a scenario for the multimedia data transmission rate control method provided in an embodiment of the present invention;

[0057] Figure 2 This is a flowchart of the multimedia data transmission rate control method provided in the embodiments of the present invention;

[0058] Figure 3 This is a schematic diagram of a multimedia data transmission scenario provided in an embodiment of the present invention;

[0059] Figure 4 This is a schematic diagram of linear regression provided in an embodiment of the present invention;

[0060] Figure 5This is another flowchart of the multimedia data transmission rate control method provided in the embodiments of the present invention;

[0061] Figure 6 This is an architecture diagram of the multimedia data transmission rate control method provided in an embodiment of the present invention;

[0062] Figure 7 This is a schematic diagram of the structure of the multimedia data transmission rate control device provided in an embodiment of the present invention;

[0063] Figure 8 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0064] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0065] This invention provides a multimedia data transmission rate control method, apparatus, device, and storage medium. Specifically, the multimedia data transmission rate control method of this invention can be applied to live streaming scenarios, i.e., to a multimedia system including a live streaming server and a live streaming client. The live streaming server sends multimedia data (e.g., audio data and / or video data from the live stream) to the live streaming client, enabling the live streaming client to parse the multimedia data and present the live stream to the user. This invention, applied to live streaming platforms, achieves global consistency of video slices through an algorithmic mechanism to reduce downlink buffering and provides higher system disaster recovery capabilities and robustness, thereby optimizing the user's live streaming experience.

[0066] When the multimedia data transmission rate control method is implemented on a terminal, the user terminal includes, but is not limited to, mobile phones, computers, smart voice interaction devices, smart home appliances, vehicle terminals, and aircraft. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud multimedia databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and large-scale multimedia data and artificial intelligence platforms. User terminals and servers can be directly or indirectly connected via wired or wireless communication; this application does not impose any restrictions on this connection.

[0067] For example, a terminal device may store a live streaming application and use it to display scenes from a live stream. The terminal device is used to interact with the user through a live streaming interface, such as by downloading, installing, and running the live streaming application. The terminal device can provide the live streaming interface to the user in various ways, such as rendering it on the terminal device's screen or presenting it via holographic projection. For instance, the terminal device may include a touchscreen display and a processor. The touchscreen display is used to present the live streaming interface and receive user input commands to the live streaming interface, which includes the live stream footage. The processor is used to run the live streaming application, generate the live streaming interface, respond to input commands, and control the display of the live streaming interface on the touchscreen display.

[0068] Furthermore, multimedia data transmission rate control methods can be applied to cloud live streaming scenarios. Cloud live streaming refers to a live streaming method based on cloud computing. In the cloud live streaming operating mode, the main body running the live streaming application and the main body presenting the live stream are separate. The presentation of the live stream is completed on the cloud live streaming client. The cloud live streaming client is mainly used for receiving and sending multimedia data and presenting the live stream. For example, the cloud live streaming client can be a display device with multimedia data transmission capabilities close to the user, such as a mobile terminal, television, computer, PDA, personal digital assistant, etc. However, the terminal device for processing live multimedia data is the cloud live streaming server in the cloud. During live streaming, the user operates the cloud live streaming client to send operation commands to the cloud live streaming server. The cloud live streaming server runs the live streaming program according to the operation commands, encodes and compresses the multimedia data such as the live stream, returns it to the cloud live streaming client through the network, and finally, the cloud live streaming client decodes and outputs the live stream.

[0069] The multimedia data transmission rate control method provided by this invention can be executed on a terminal, on a server, or jointly by both the terminal and the server. The above examples should not be construed as limiting the invention.

[0070] Please see Figure 1 , Figure 1This is a schematic diagram illustrating a scenario for a multimedia data transmission rate control method provided in an embodiment of the present invention. The multimedia system includes a live streaming client 101 and a live streaming server 102. The live streaming client 101 receives at least two sets of multimedia data sent by the live streaming server and obtains the total transmission time spent by the live streaming server and the total reception time spent by the live streaming client for each set of multimedia data. Based on the total transmission and reception times of each set of multimedia data, the live streaming client 101 determines the latency gradient generated during transmission of each set of multimedia data in the communication network between the live streaming server 102 and the live streaming client 101. The live streaming client 101 smooths the latency gradients to obtain several processed latency gradients. Based on the changes between the processed latency gradients, the live streaming client 101 determines the congestion level information of the communication network and sends the congestion level information to the live streaming server 102. The live streaming server 102 adjusts the multimedia data transmission rate for the live streaming client 101 according to the congestion level information.

[0071] The above Figure 1 The example given is merely a system architecture instance for implementing an embodiment of the present invention, and the embodiments of the present invention are not limited to the above. Figure 1 The system structure shown is based on Figure 1 Based on the system architecture, various embodiments of the present invention are proposed.

[0072] The following sections provide detailed descriptions of each example. It should be noted that the order in which the embodiments are described is not intended to limit the preferred order of the embodiments.

[0073] This invention provides a multimedia data transmission rate control method, applied to a multimedia system including a live streaming server and a live streaming client. The method can be executed by the processor of the live streaming client and the processor of the live streaming server, such as... Figure 2 As shown, the specific process of this multimedia data transmission rate control method can be as follows:

[0074] Step 201: The live streaming client receives at least two sets of multimedia data sent by the live streaming server, and obtains the total sending time spent by the live streaming server when sending each set of multimedia data, and the total receiving time spent by the live streaming client when receiving it.

[0075] In this embodiment of the invention, in a live streaming scenario, multimedia data can be live streaming data, such as audio data, video data, live event data, etc., that form the live streaming screen. When a user broadcasts live on a live streaming client, the live streaming server sends the corresponding multimedia data to the live streaming client, which then receives and parses the multimedia data to form the live streaming screen.

[0076] In some embodiments of the present invention, the multimedia data received by the live streaming client may be multimedia data that has already been divided into groups before reception. The live streaming client directly receives multiple groups of multimedia data for further processing. Alternatively, the multimedia data received by the live streaming client may be a single multimedia data set that has not yet formed a multimedia data group. After receiving the multimedia data, the live streaming client divides the received multimedia data into multiple groups. The rules by which the live streaming client divides the multimedia data into multiple groups are not limited and can be flexibly set according to actual conditions. For example, the multimedia data groups can be divided according to the order in which the multimedia data is received, or according to the type of multimedia data, etc.

[0077] In one embodiment of the present invention, the amount of multimedia data contained in each group of multimedia data may be the same, partially the same, or different from each other. Furthermore, the multimedia data can be multimedia data transmitted in any scenario; for example, multimedia data can be video stream data, audio stream data, etc.

[0078] In some embodiments of the present invention, if the multimedia data contained in each group of multimedia data are sent and received sequentially in chronological order, the method for obtaining the total transmission duration of each group of multimedia data on the live streaming server may be: determining the initial transmission time of the first multimedia data sent on the live streaming server in each group of multimedia data, and determining the final transmission time of the last multimedia data sent on the live streaming server, calculating a first difference between the final transmission time and the initial transmission time, and using the difference as the total transmission duration corresponding to each group of multimedia data. Furthermore, the method for obtaining the total reception duration of each group of multimedia data on the live streaming client may be: determining the initial reception time of the first multimedia data received on the live streaming client in each group of multimedia data, and determining the final reception time of the last multimedia data received on the live streaming client, calculating a second difference between the final reception time and the initial reception time, and using the second difference as the total reception duration corresponding to each group of multimedia data.

[0079] For example, such as Figure 3 In the schematic diagram of a multimedia data transmission scenario shown, the live streaming server is in TS i At any given moment, a set of multimedia data is sent to the live streaming client. The live streaming server is in TS. i+k Once the multimedia data set has been sent, the live streaming client will complete the process in TR. i The live streaming client begins receiving this set of multimedia data at a specific time in TR. i+k If the reception of this group of multimedia data is completed at a certain time, then the total transmission time of this group of multimedia data is TS. i+k —TS i The total reception time of this set of multimedia data is TR.i+k —TR i .

[0080] Step 202: The live streaming client determines the latency gradient generated during the transmission of each set of multimedia data in the communication network between the live streaming server and the live streaming client, based on the total transmission and reception duration of each set of multimedia data.

[0081] In one embodiment of the present invention, network congestion refers to a situation in a packet-switched network where the number of packets transmitted is too large, resulting in a decline in network transmission performance due to the limited resources of store-and-forward nodes. Network congestion is a state of continuous network overload, in which user demand for network resources (including link bandwidth, storage space, and processor processing power) exceeds the inherent processing capacity and capacity.

[0082] In some embodiments of the present invention, if the transmission time of each multimedia data is the same during the transmission of a set of multimedia data, it indicates that the communication network is not congested during the transmission of multimedia data. If the transmission time of each multimedia data is different and some multimedia data is delayed during transmission, it indicates that the communication network may be congested and the delay is serious, indicating that the degree of congestion is serious.

[0083] In this embodiment of the invention, the latency gradient is used as a criterion to indicate whether network congestion occurs in the communication network between the live streaming server and the live streaming client. Step 202 above, where "the live streaming client determines the latency gradient generated during the transmission of each set of multimedia data in the communication network between the live streaming server and the live streaming client based on the total transmission and reception time of each set of multimedia data," can be:

[0084] The live streaming client calculates the difference between the total reception time and the total transmission time for each set of multimedia data, which is used as the latency gradient generated when each set of multimedia data is transmitted in the communication network.

[0085] The delay gradient generated during the transmission of each set of multimedia data can represent the delay caused by network congestion beyond the normal transmission time. Therefore, the delay gradient can be used to determine the degree of congestion. For example, ... Figure 3 As shown, the formula for calculating the time delay gradient can be ΔT. i+k =(TR) i+k -R i )-(TS i+k -S i ).

[0086] Step 203: The live streaming client smooths the latency gradient to obtain several processed latency gradients.

[0087] In embodiments of the present invention, since multimedia data may experience jitter during transmission over a communication network, after arranging the delay gradients generated by each group of multimedia data based on their reception time, a very small number of delay gradients may differ significantly from their adjacent delay gradients. These significantly different delay gradients can be considered abnormal multimedia data caused by jitter. To more accurately determine the changes in delay gradients and avoid interference from abnormal multimedia data, a denoising filter can be used to smooth each delay gradient. A low-pass filter is typically used for this smoothing process. The denoising filter is a filter circuit composed of capacitors, inductors, and resistors. The denoising filter can effectively filter out specific frequencies or frequencies outside of those frequencies in the power line, obtaining a power signal of a specific frequency, or eliminating a power signal after a specific frequency has been eliminated. Specifically, the step "the live streaming client smooths the delay gradients to obtain several processed delay gradients" can be:

[0088] The live streaming client determines several sets of delay gradients to be processed from multiple delay gradients, wherein each set of delay gradients to be processed includes at least two delay gradients.

[0089] The live streaming client obtains the weights corresponding to each delay gradient included in each group of delay gradients to be processed;

[0090] The live streaming client generates a weighted latency gradient for each set of latency gradients to be processed, based on the latency gradients included in each set of latency gradients and the weights corresponding to each latency gradient.

[0091] The live streaming client obtains several processed delay gradients based on each weighted delay gradient.

[0092] In this embodiment of the invention, each weighted delay gradient can be used as the processed delay gradient, or each weighted delay gradient and a portion of the delay gradient can be used together as the processed delay gradient. Alternatively, a delay gradient that is not significantly different from the weighted delay gradient can be selected as the processed delay gradient.

[0093] In this embodiment of the invention, the number of delay gradients included in a set of delay gradients to be processed is not limited. Furthermore, in order to ensure that each weighted delay gradient generated after smoothing can more accurately represent the unprocessed delay gradient, after the unprocessed delay gradients are arranged based on the reception time of each set of multimedia data, the delay gradients included in a set of delay gradients to be processed can be several adjacent delay gradients.

[0094] For example, the delay gradients formed by the sequential arrangement of the received multimedia data are t1, t2, t3, t4, t5, and t6. We set up a group of adjacent delay gradients to be processed, dividing t1 and t2 into one group, t2 and t3 into another, t3 and t4 into another, t4 and t5 into another, and t5 and t6 into another. We then determine the weights of the delay gradients included in each group to form a weighted delay gradient, resulting in the weighted delay gradients: T1 = a*t1 + (1-a)*t2, T2 = b*t2 + (1-b)*t3, T3 = c*t3 + (1-c)*t4, T4 = d*t4 + (1-d)*t5, and T5 = e*t5 + (1-e)*t6.

[0095] Step 204: The live streaming client determines the congestion level information of the communication network based on the changes between the processed delay gradients and sends the congestion level information to the live streaming server.

[0096] In one embodiment of the present invention, the congestion level of the communication network can be determined based on the changes between multiple processed delay gradients. Specifically, step 204, "the live streaming client determines the congestion level information of the communication network based on the changes between processed delay gradients and sends the congestion level information to the live streaming server," can be:

[0097] The live streaming client determines the first order of each latency gradient based on the receiving order of each group of multimedia data;

[0098] The live streaming client obtains the correspondence between each processed latency gradient and each latency gradient;

[0099] The live streaming client determines the second arrangement order of the processed latency gradient based on the first arrangement order and the corresponding relationship.

[0100] The live streaming client determines the changes in the post-processing delay gradients under the second permutation order;

[0101] The live streaming client determines the congestion level of the communication network based on the changing situation and sends the congestion level information to the live streaming server.

[0102] For example, the live streaming client receives each set of multimedia data in the order of data1, data2, data3, and data4. The delay gradient formed based on data1 is m1, the delay gradient formed based on data2 is m2, the delay gradient formed based on data3 is m3, and the delay gradient formed based on data4 is m4. The first arrangement order of the delay gradients is m1, m2, m3, and m4. The processed delay gradient n1 is formed based on the delay gradients m1 and m2, the processed delay gradient n2 is formed based on the delay gradients m2 and m3, and the processed delay gradient n3 is formed based on the delay gradients m3 and m4. The second arrangement order of the processed delay gradients is determined based on the receiving order of the multimedia data, which is n1, n2, and n3. The congestion level information of the communication network is determined based on the change from n1 to n2 and then to n3.

[0103] In this embodiment of the invention, to more accurately determine the congestion level of the communication network based on the changes in the post-processed delay gradients, the changes in the post-processed delay gradients can be quantified. Since the post-processed delay gradients are individual discrete values, a linear regression method can be used to fit all post-processed delay gradients to form a straight line, minimizing the sum of the Euclidean distances of all post-processed delay gradients to the straight line. Specifically, the above step "the live streaming client determines the changes in each post-processed delay gradient under the second arrangement order" can be:

[0104] The live streaming client performs linear regression on each processed latency gradient based on the second permutation order to obtain the linear regression equation;

[0105] The live streaming client determines the trend and rate of change of the latency gradient after each processing step based on the slope of the linear regression equation.

[0106] The live streaming client uses the trend and rate of change as the variation of the post-processed delay gradients in the second arrangement.

[0107] In one embodiment of the present invention, the change in the processed time delay gradient—whether it continuously increases or decreases—can be determined based on the sign of the slope of the linear regression equation. If the slope is greater than 0, it indicates that the processed time delay gradient is on an increasing trend; if the slope is less than 0, it indicates that the processed time delay gradient is on a decreasing trend. The rate of change refers to the rate of increase or decrease of the processed time delay gradient. If the slope is greater than 0, and the larger the slope value, the faster the processed time delay gradient increases; if the slope is less than 0, and the larger the absolute value of the slope value, the faster the processed time delay gradient decreases. For example, as... Figure 4 In the linear regression diagram shown, the processed time delay gradient ΔT is formed based on the reception time t of each group of multimedia data. t Based on the post-processing time delay gradient ΔTt 'Forming as Figure 4 The linear regression line shown is shown in the figure.

[0108] In this embodiment of the invention, step 204, "the live streaming client determines the congestion level information of the communication network based on the changes between the processed delay gradients," can be:

[0109] If the trend of change is an increasing trend, and the post-processed delay gradient at the end of the second arrangement is greater than the delay gradient threshold, the live streaming client determines that the communication network is in a congested state, and determines the degree of congestion of the communication network when it is in a congested state based on the rate of change.

[0110] The live streaming client uses the congestion status and congestion level as information about the congestion level of the communication network;

[0111] If the trend of change is a decreasing trend and / or the post-processed delay gradient at the end of the second arrangement is not greater than the delay gradient threshold, the live streaming client determines that the communication network is in a non-congested state, and determines the degree of non-congestion of the communication network when it is in a non-congested state based on the rate of change.

[0112] The live streaming client uses the non-congestion status and non-congestion level as information about the congestion level of the communication network.

[0113] In one embodiment of the present invention, the delay gradient threshold is a post-processing delay gradient indicating that the communication network is in a congested state. If the change is an increasing trend, and the final post-processing delay gradient is also greater than the delay gradient threshold, then the post-processing delay gradient is getting larger and larger, that is, the delay added in addition to the normal transmission delay during multimedia data transmission is increasing, which can indicate that the communication network is in a congested state. If the post-processing delay gradient increases faster and faster, it can indicate that the degree of congestion in the communication network is greater.

[0114] In one embodiment of the present invention, as long as the change is a decreasing trend, it can be indicated that the post-processing delay gradient is getting smaller and smaller, that is, the delay added in the multimedia data transmission process in addition to the normal transmission delay is getting smaller and smaller, which can be indicated that the multimedia data transmission is getting faster and faster. At this time, regardless of whether the final post-processing delay gradient is greater than the delay gradient threshold, it can be determined that the communication network is not in a congested state.

[0115] In some embodiments of the present invention, if the change is not a decreasing trend, although the latency increases more and more in addition to the normal transmission latency during multimedia data transmission, the final post-processing latency gradient is not greater than the latency gradient threshold, indicating that the increase in the post-processing latency gradient is still within the normal range. Although the bandwidth utilization of the communication network is increasing, it can be determined that the communication network is not yet in a congested state.

[0116] Furthermore, if the change shows a decreasing trend and the final processed delay gradient is not greater than the delay gradient threshold, then the additional delay during multimedia data transmission, in addition to the normal transmission delay, is becoming smaller and smaller, indicating that the communication network is not in a congested state.

[0117] In this embodiment of the invention, the congestion level information may only include the result of the live streaming client's judgment on whether the communication network is congested, so that the live streaming server can determine whether to reduce or increase the current multimedia data transmission rate based on the result of whether the communication network is congested. Furthermore, in order to enable the live streaming server to update the current multimedia data transmission rate more accurately based on the congestion level of the communication network, the congestion level information may also include the slope of the linear regression equation, so that the live streaming server can update the current multimedia data transmission rate based on the magnitude of the slope.

[0118] In this embodiment of the invention, after obtaining the time delay gradients of each multimedia data group, the congestion level information can be directly determined based on the changes in the time delay gradients under the first arrangement order. Specifically, the time delay gradients can be linearly regressed to generate corresponding linear regression equations, and the congestion level information can be determined based on the slope of the linear regression equations corresponding to the time delay gradients.

[0119] Step 205: The live streaming server adjusts the multimedia data transmission rate for the live streaming client based on the congestion information.

[0120] In one embodiment of the present invention, after the live streaming client generates congestion information about the communication network, it feeds this information back to the live streaming server. The method by which the live streaming client feeds back the congestion information to the live streaming server is not limited and can be flexibly determined according to the actual situation. For example, the live streaming client can also send congestion information to the live streaming server through the communication network, or it can feed back congestion information to the live streaming server through other networks.

[0121] In this embodiment of the invention, after the live streaming client receives congestion information, it needs to obtain the result of whether the network status is congested, and then adjust the multimedia data transmission rate. If the communication network is congested, the live streaming client can reduce the current multimedia data transmission rate. Specifically, step 205, "the live streaming server adjusts the multimedia data transmission rate for the live streaming client according to the congestion information," can be:

[0122] If the communication network is determined to be congested based on the congestion level information, the live streaming server reduces the multimedia data transmission rate based on the congestion level.

[0123] If the congestion level information determines that the communication network is in a non-congested state, the live streaming server increases the multimedia data transmission rate based on the non-congestion level.

[0124] To ensure that the updated multimedia data transmission rate can alleviate network congestion to a greater extent, the live streaming server needs to determine the degree of congestion. This can be done, for example, by analyzing the rate of change in congestion information. If the communication network is not congested, the live streaming client can directly increase the current multimedia data transmission rate. However, to ensure that the updated multimedia data transmission rate makes better use of the communication network's bandwidth, the live streaming server needs to determine the degree of non-congestion (i.e., the degree of bandwidth underutilization). This can also be done, for example, by analyzing the rate of change in congestion information.

[0125] Specifically, the congestion level information includes the rate of change used to determine the congestion level. In this case, the rate of change can be determined based on the slope of the linear regression equation formed by the time delay gradient after linear regression processing. Specifically, the above step "the live streaming server reduces the multimedia data transmission rate based on the congestion level" can be: the live streaming server obtains the inverse relationship between the rate of change corresponding to the congestion level and the first coefficient, and determines the first coefficient based on the inverse relationship and the rate of change corresponding to the congestion level; the live streaming server reduces the multimedia data transmission rate based on the first coefficient.

[0126] Similarly, the above step "the live streaming server increases the multimedia data transmission rate based on the degree of non-congestion" can be: the live streaming server obtains the proportional relationship between the rate of change corresponding to the degree of non-congestion and the second coefficient, and determines the second coefficient based on the proportional relationship and the rate of change corresponding to the degree of non-congestion; the live streaming server increases the multimedia data transmission rate based on the second coefficient.

[0127] In one embodiment of the present invention, if the communication network experiences congestion, it indicates that the slope of the linear regression equation is greater than 0. A larger slope indicates a higher degree of congestion, and a greater reduction in the current multimedia data transmission rate. If the communication network is not congested and the slope of the linear regression equation is greater than 0, it indicates that the bandwidth utilization of the communication network is also increasing. In this case, to avoid causing congestion after updating the current multimedia data transmission rate, the current multimedia data transmission rate can be increased by a small margin based on the slope. If the communication network is not congested and the slope of the linear regression equation is less than 0, it indicates that the network utilization of the communication network is decreasing. A larger absolute value of the slope indicates a greater degree of non-congestion in the communication network, and a greater increase in the current multimedia data transmission rate.

[0128] In one embodiment of the present invention, before updating the multimedia data transmission rate, it is necessary to first obtain the current multimedia data transmission rate, that is, the multimedia data transmission rate includes the bitrate for transmitting multimedia data that has not been transmitted before, and the bitrate for retransmitting multimedia data packets. Specifically, before step 205 "the live streaming server adjusts the size of the multimedia data transmission rate for the live streaming client according to the congestion information", the method further includes:

[0129] The live streaming server obtains the input bitrate of the multimedia data that the current application layer is sending to the live streaming client.

[0130] The live streaming server determines the retransmission rate when retransmitting multimedia data to the live streaming client.

[0131] The live streaming server determines the current multimedia data transmission rate for the live streaming client based on the input bitrate and retransmission rate.

[0132] Here, bitrate refers to the number of bits of multimedia data transmitted per unit time. For example, if the input bitrate is input_rate and the retransmission rate is retrans_ratio, then the multimedia data transmission rate is send_rate = input_rate * (1 + retrans_ratio).

[0133] In one embodiment of the present invention, the specific process of updating the multimedia data transmission rate based on congestion level can be as follows: When it is determined that the communication network is not congested, the current send_rate value is increased by a coefficient α (i.e., send_rate = send_rate × α). Here, the coefficient α is directly proportional to the absolute value of the slope k of the linear regression equation included in the congestion level information, i.e., α = η × k. When it is determined that the communication network is congested, send_rate is decreased by a coefficient β (send_rate = send_rate × β) until no more congestion is detected. Here, the coefficient β is inversely proportional to the absolute value of k, i.e., β = λ / k. The larger the absolute value of k, the more severe the network congestion or redundancy, and the faster the rate control can converge. The values ​​of η and λ can be set in practice.

[0134] This invention provides a multimedia data transmission rate control method, apparatus, device, and storage medium. The live streaming client can determine whether the communication network between the live streaming client and the live streaming server is congested, and the degree of congestion, based on the total transmission and reception durations of at least two sets of multimedia data received from the live streaming server. This avoids determining congestion based on packet loss. The live streaming client then informs the live streaming server of the network congestion level, allowing the live streaming server to adjust the multimedia data transmission rate accordingly. This avoids reducing the multimedia data transmission rate when network bandwidth is sufficient but the packet loss rate is high. It ensures stable multimedia data transmission even with sufficient bandwidth and a high packet loss rate, and allows for timely adjustment of the multimedia data transmission rate when congestion is detected, preventing severe congestion.

[0135] Based on the method described in the preceding embodiments, another multimedia data transmission rate control method will be further described in detail below.

[0136] This application provides a multimedia data transmission rate control method, such as... Figure 5 As shown, the specific process of this multimedia data transmission rate control method can be as follows:

[0137] Step 501: The live streaming client determines at least two sets of multimedia data received from the live streaming server, wherein each set of multimedia data contains at least two sets of multimedia data.

[0138] In this embodiment of the invention, the multimedia data received by the live streaming client can be multimedia data that has already been divided into groups before reception. The live streaming client directly receives multiple groups of multimedia data for further processing. Alternatively, the multimedia data received by the live streaming client can be a single multimedia data that has not formed a multimedia data group. After receiving the multimedia data, the live streaming client divides the received multimedia data into multiple groups.

[0139] Step 502: The live streaming server obtains the total transmission time of each group of multimedia data when it is sent by the live streaming server, and the total reception time when it is received by the live streaming client.

[0140] In this embodiment of the invention, the method by which the live streaming server obtains the total transmission time of each group of multimedia data on the live streaming server and the total reception time on the live streaming client is the same as the method by which the live streaming client obtains the data in the above embodiments. It is also determined based on a first difference between the end transmission time and the first transmission time of each group of multimedia data, and a second difference between the end reception time and the first reception time of each group of multimedia data. Further details will not be elaborated here.

[0141] Step 503: The live streaming server determines the latency gradient generated during the transmission of each set of multimedia data in the communication network between the live streaming server and the live streaming client, based on the total transmission and reception duration of each set of multimedia data.

[0142] In one embodiment of the present invention, the live streaming client also calculates the difference between the total reception time and the total transmission time for each set of multimedia data as the latency gradient generated when each set of multimedia data is transmitted in the communication network.

[0143] Step 504: The live streaming server smooths the latency gradient to obtain several processed latency gradients.

[0144] Then, based on the reception time of each set of multimedia data, the change of the post-processing delay gradient related to each set of multimedia data is determined. If the change is an increasing trend and the post-processing delay gradient generated by the latest received set of multimedia data is greater than the delay gradient threshold, it is determined that the communication network between the live streaming server and the live streaming client is in a congested state. If the change is a decreasing trend and / or the post-processing delay gradient generated by the latest received set of multimedia data is not greater than the delay gradient threshold, it is determined that the communication network between the live streaming server and the live streaming client is not in a congested state.

[0145] Step 505: The live streaming server determines the congestion level information of the communication network based on the changes between the processed latency gradients.

[0146] In this embodiment of the invention, if the communication network is not in a congested state, the larger the absolute value of the rate of change, the higher the redundancy of the communication network and the lower the network resource utilization. If the communication network is in a congested state, the larger the absolute value of the rate of change, the higher the degree of congestion of the communication network.

[0147] Step 506: The live streaming server adjusts the multimedia data transmission rate for the live streaming client based on the congestion information.

[0148] In this embodiment of the invention, after the live streaming client receives congestion level information, it needs to determine whether the network is congested. If the communication network is congested, the live streaming client can reduce the current multimedia data transmission rate. However, in order for the updated multimedia data transmission rate to alleviate network congestion to a greater extent, the live streaming server needs to determine the degree of congestion, for example, based on the slope in the congestion level information. If the communication network is not congested, the live streaming client can directly increase the current multimedia data transmission rate. However, in order for the adjusted multimedia data transmission rate to make better use of the communication network bandwidth, the live streaming server needs to determine the degree of non-congestion in the communication network (i.e., the degree of bandwidth underutilization), for example, based on the slope in the congestion level information.

[0149] like Figure 6 In the structural diagram of the multimedia data transmission rate control method shown, the live streaming client has a latency calculator, a denoising filter, and a congestion checker. After receiving multiple sets of multimedia data, the latency calculator calculates the total transmission time of all multimedia data in each set when it is sent by the live streaming server, and uses this as the total transmission time. The latency calculator also calculates the total reception time of all multimedia data in each set when it is received by the live streaming client, and uses this as the total reception time. The latency calculator subtracts the calculated total transmission time and total reception time of each set of multimedia data to obtain the latency gradient generated during transmission in the communication network. Then, the latency calculator sends the obtained latency gradient to the denoising filter. The denoising filter removes abnormal multimedia data caused by jitter from the received latency gradient, obtaining a weighted latency gradient. The denoising filter then sends the weighted latency gradient to the congestion checker. The congestion checker determines the degree of congestion in the communication network based on the changes in the weighted latency gradient. Finally, the live streaming client sends the congestion level determined by the congestion checker to the live streaming server, which then performs rate control on the multimedia data transmission rate based on the received congestion level.

[0150] As can be seen from the above, the multimedia data transmission rate control method provided in this embodiment of the invention allows the live streaming client to determine whether the communication network between the live streaming client and the live streaming server is congested and the degree of congestion based on the total transmission time and total reception time of at least two sets of multimedia data received from the live streaming server. This avoids determining whether congestion has occurred based on whether the communication network is losing packets. The live streaming client then informs the live streaming server of the degree of congestion in the communication network, allowing the live streaming server to adjust the multimedia data transmission rate according to the degree of congestion. This avoids the problem of reducing the multimedia data transmission rate when the network bandwidth is sufficient but the packet loss rate is high. Thus, it enables stable transmission of multimedia data even when the network bandwidth is sufficient but the packet loss rate is high, and it can also adjust the multimedia data transmission rate in a timely manner when congestion is detected, avoiding severe congestion.

[0151] To better implement the above methods, this application also provides a multimedia data transmission rate control device, applied to a multimedia system, which includes a live streaming server and a live streaming client, such as... Figure 7 The schematic diagram shown is of a multimedia data transmission rate control device. This device includes a duration determination unit 701, a delay determination unit 702, a smoothing processing unit 703, a congestion determination unit 704, and a rate adjustment unit 705, as follows:

[0152] The duration determination unit 701 is used to receive at least two sets of multimedia data sent by the live streaming server through the live streaming client, and to obtain the total transmission time spent when the multimedia data is sent by the live streaming server and the total reception time spent when the multimedia data is received by the live streaming client.

[0153] The delay determination unit 702 is used to determine the delay gradient generated when each set of multimedia data is transmitted in the communication network between the live streaming server and the live streaming client based on the total transmission duration and total reception duration of each set of multimedia data by the live streaming client.

[0154] The smoothing unit 703 is used to smooth the latency gradient through the live streaming client to obtain several processed latency gradients.

[0155] The congestion determination unit 704 is used to determine the congestion level information of the communication network based on the changes between the processed delay gradients by the live streaming client, and send the congestion level information to the live streaming server.

[0156] The rate adjustment unit 705 is used to adjust the multimedia data transmission rate for the live streaming client based on congestion information from the live streaming server.

[0157] Optionally, the smoothing unit 703 is also used for:

[0158] The live streaming client determines several sets of delay gradients to be processed from multiple delay gradients, wherein each set of delay gradients to be processed includes at least two delay gradients.

[0159] The live streaming client obtains the weights corresponding to each delay gradient included in each group of delay gradients to be processed;

[0160] The live streaming client generates a weighted latency gradient for each set of latency gradients to be processed, based on the latency gradients included in each set of latency gradients and the weights corresponding to each latency gradient.

[0161] The live streaming client obtains several processed delay gradients based on each weighted delay gradient.

[0162] Optionally, the congestion determination unit 704 is also used for:

[0163] The live streaming client determines the first order of each latency gradient based on the receiving order of each group of multimedia data;

[0164] The live streaming client obtains the correspondence between each processed latency gradient and each latency gradient;

[0165] The live streaming client determines the second arrangement order of the processed latency gradient based on the first arrangement order and the corresponding relationship.

[0166] The live streaming client determines the changes in the post-processing delay gradients under the second permutation order;

[0167] The live streaming client determines the congestion level of the communication network based on the changing situation and sends the congestion level information to the live streaming server.

[0168] Optionally, the congestion determination unit 704 is also used for:

[0169] The live streaming client performs linear regression on each processed latency gradient based on the second permutation order to obtain the linear regression equation;

[0170] The live streaming client determines the trend and rate of change of the latency gradient after each processing step based on the slope of the linear regression equation.

[0171] The live streaming client uses the trend and rate of change as the variation of the post-processed delay gradients in the second arrangement.

[0172] Optionally, the congestion determination unit 704 is also used for:

[0173] If the trend of change is an increasing trend, and the post-processed delay gradient at the end of the second arrangement is greater than the delay gradient threshold, the live streaming client determines that the communication network is in a congested state, and determines the degree of congestion of the communication network when it is in a congested state based on the rate of change.

[0174] The live streaming client uses the congestion status and congestion level as information about the congestion level of the communication network;

[0175] If the trend of change is a decreasing trend and / or the post-processed delay gradient at the end of the second arrangement is not greater than the delay gradient threshold, the live streaming client determines that the communication network is in a non-congested state, and determines the degree of non-congestion of the communication network when it is in a non-congested state based on the rate of change.

[0176] The live streaming client uses the non-congestion status and non-congestion level as information about the congestion level of the communication network.

[0177] Optionally, the rate adjustment unit 705 is also used for:

[0178] If the communication network is determined to be congested based on the congestion level information, the live streaming server reduces the multimedia data transmission rate based on the congestion level.

[0179] If the congestion level information determines that the communication network is in a non-congested state, the live streaming server increases the multimedia data transmission rate based on the non-congestion level.

[0180] Optionally, the rate adjustment unit 705 is also used for:

[0181] The live streaming server obtains the inverse relationship between the rate of change corresponding to the congestion level and the first coefficient, and determines the first coefficient based on the inverse relationship and the rate of change corresponding to the congestion level.

[0182] The live streaming server reduces the multimedia data transmission rate based on a first coefficient.

[0183] The live streaming server obtains the proportional relationship between the rate of change corresponding to the degree of non-congestion and the second coefficient, and determines the second coefficient based on the proportional relationship and the rate of change corresponding to the degree of non-congestion;

[0184] The live streaming server increases the multimedia data transmission rate based on a second coefficient.

[0185] Optionally, the rate adjustment unit 705 is also used for:

[0186] The live streaming server obtains the input bitrate of the multimedia data that the current application layer is sending to the live streaming client.

[0187] The live streaming server determines the retransmission rate when retransmitting multimedia data to the live streaming client.

[0188] The live streaming server determines the current multimedia data transmission rate for the live streaming client based on the input bitrate and retransmission rate.

[0189] Optionally, the duration determination unit 701 is also used for:

[0190] The live streaming client calculates the difference between the total reception time and the total transmission time for each set of multimedia data, which is used as the latency gradient generated when each set of multimedia data is transmitted in the communication network.

[0191] As can be seen from the above, the multimedia data transmission rate control device provided in this embodiment of the invention allows the live streaming client to determine whether the communication network between the live streaming client and the live streaming server is congested and the degree of congestion based on the total transmission time and total reception time of at least two sets of multimedia data received from the live streaming server. This avoids determining whether congestion has occurred by checking for packet loss in the communication network. The live streaming client then informs the live streaming server of the degree of congestion in the communication network, allowing the live streaming server to adjust the multimedia data transmission rate according to the degree of congestion. This avoids the problem of reducing the multimedia data transmission rate when the network bandwidth is sufficient but the packet loss rate is high. Thus, it enables stable transmission of multimedia data even when the network bandwidth is sufficient but the packet loss rate is high, and it can also adjust the multimedia data transmission rate in a timely manner when congestion is detected, thus avoiding severe congestion.

[0192] Furthermore, embodiments of the present invention also provide a computer device, which may be a terminal or a server, such as... Figure 8As shown, it illustrates a structural schematic diagram of a computer device involved in an embodiment of the present invention, specifically:

[0193] The computer device may include components such as a processor 901 with one or more processing cores, a memory 902 with one or more computer-readable storage media, a power supply 903, and an input unit 904. Those skilled in the art will understand that... Figure 8 The computer device structure shown does not constitute a limitation on the computer device and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein:

[0194] The processor 901 is the control center of the computer device. It connects various parts of the computer device via various interfaces and lines. By running or executing software programs and / or modules stored in the memory 902, and by calling multimedia data stored in the memory 902, it performs various functions of the computer device and processes multimedia data, thereby performing overall detection of the computer device. Optionally, the processor 901 may include one or more processing cores; preferably, the processor 901 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 901.

[0195] The memory 902 can be used to store software programs and modules. The processor 901 executes various functional applications and multimedia data processing by running the software programs and modules stored in the memory 902. The memory 902 may mainly include a program storage area and a multimedia data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the multimedia data storage area may store multimedia data created according to the use of the computer device, etc. In addition, the memory 902 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 902 may also include a memory controller to provide the processor 901 with access to the memory 902.

[0196] The computer device also includes a power supply 903 that supplies power to the various components. Preferably, the power supply 903 can be logically connected to the processor 901 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 903 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0197] The computer device may also include radio frequency (RF) circuitry, which can be used to transmit and receive RF signals to establish wireless communication with network devices or other computer devices, and to transmit and receive signals with network devices or other computer devices.

[0198] The computer device may also include an input unit 904, which can be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0199] Although not shown, the computer device may also include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 901 in the computer device loads the executable files corresponding to the processes of one or more application programs into the memory 902 according to the following instructions, and the processor 901 runs the application programs stored in the memory 902 to realize various functions, as follows:

[0200] The live streaming client receives at least two sets of multimedia data sent by the live streaming server, and obtains the total transmission time spent when the live streaming server sends each set of multimedia data, as well as the total reception time spent when the live streaming client receives it.

[0201] The live streaming client determines the latency gradient generated during the transmission of each set of multimedia data in the communication network between the live streaming server and the live streaming client based on the total transmission and reception duration of each set of multimedia data.

[0202] The live streaming client smooths the latency gradient, resulting in several processed latency gradients.

[0203] The live streaming client determines the congestion level of the communication network based on the changes in the processed latency gradients and sends the congestion level information to the live streaming server.

[0204] The live streaming server adjusts the rate at which it sends multimedia data to the live streaming client based on congestion information.

[0205] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0206] As can be seen from the above, in this embodiment, the live streaming client can determine whether the communication network between the live streaming client and the live streaming server is congested and the degree of congestion based on the total transmission time and total reception time of at least two sets of multimedia data received from the live streaming server. This avoids determining whether congestion has occurred by checking for packet loss in the communication network. The live streaming client then informs the live streaming server of the degree of congestion in the communication network, allowing the live streaming server to adjust the multimedia data transmission rate according to the degree of congestion. This avoids the problem of reducing the multimedia data transmission rate when the network bandwidth is sufficient but the packet loss rate is high. Thus, it ensures stable transmission of multimedia data even when the network bandwidth is sufficient but the packet loss rate is high. It also allows for timely adjustment of the multimedia data transmission rate when congestion is detected, thus avoiding severe congestion.

[0207] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0208] Therefore, embodiments of this application provide a storage medium storing a plurality of instructions that can be loaded by a processor to execute steps in any of the multimedia data transmission rate control methods provided in embodiments of this application. For example, the instructions can execute the following steps:

[0209] The live streaming client receives at least two sets of multimedia data sent by the live streaming server, and obtains the total transmission time spent when the live streaming server sends each set of multimedia data, as well as the total reception time spent when the live streaming client receives it.

[0210] The live streaming client determines the latency gradient generated during the transmission of each set of multimedia data in the communication network between the live streaming server and the live streaming client based on the total transmission and reception duration of each set of multimedia data.

[0211] The live streaming client smooths the latency gradient, resulting in several processed latency gradients.

[0212] The live streaming client determines the congestion level of the communication network based on the changes in the processed latency gradients and sends the congestion level information to the live streaming server.

[0213] The live streaming server adjusts the rate at which it sends multimedia data to the live streaming client based on congestion information.

[0214] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0215] The storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0216] Since the instructions stored in the storage medium can execute the steps of any of the multimedia data transmission rate control methods provided in the embodiments of this application, the beneficial effects that any of the multimedia data transmission rate control methods provided in the embodiments of this application can achieve can be realized. For details, please refer to the previous embodiments, which will not be repeated here.

[0217] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the various alternative implementations of the data download aspect described above.

[0218] The above provides a detailed description of a multimedia data transmission rate control method, apparatus, device, and storage medium provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A multimedia data transmission rate control method, applied to a multimedia system, the multimedia system including a live streaming server and a live streaming client, characterized in that, include: The live streaming client receives at least two sets of multimedia data sent by the live streaming server, and obtains the total transmission time spent when the multimedia data is sent by the live streaming server, and the total reception time spent when the live streaming client receives the multimedia data. The live streaming client determines the latency gradient generated when each set of multimedia data is transmitted in the communication network between the live streaming server and the live streaming client based on the total transmission and reception duration of each set of multimedia data. The live streaming client smooths the latency gradient to obtain several processed latency gradients. The live streaming client determines the congestion level information of the communication network based on the changes between the processed delay gradients, and sends the congestion level information to the live streaming server. The live streaming server adjusts the multimedia data transmission rate for the live streaming client according to the congestion information, including: if the communication network is determined to be in a congested state according to the congestion information, the live streaming server obtains the inverse relationship between the rate of change corresponding to the congestion and a first coefficient, and determines the first coefficient based on the inverse relationship and the rate of change corresponding to the congestion. The live streaming server reduces the multimedia data transmission rate based on the first coefficient.

2. The method according to claim 1, characterized in that, The live streaming client smooths the latency gradient to obtain several processed latency gradients, including: The live streaming client determines several sets of delay gradients to be processed from multiple delay gradients, wherein a set of delay gradients to be processed includes at least two delay gradients; The live streaming client obtains the weights corresponding to each delay gradient included in each group of delay gradients to be processed; The live streaming client forms a weighted latency gradient corresponding to each set of latency gradients to be processed based on the latency gradients included in each set of latency gradients to be processed and the weights corresponding to each latency gradient. The live streaming client obtains several processed delay gradients based on the weighted delay gradients.

3. The method according to claim 1, characterized in that, The live streaming client determines the congestion level information of the communication network based on the changes between the processed delay gradients, and sends the congestion level information to the live streaming server, including: The live streaming client determines the first arrangement order of each delay gradient based on the receiving order of each group of multimedia data; The live streaming client obtains the correspondence between each processed latency gradient and each latency gradient; The live streaming client determines the second arrangement order of the processed latency gradient based on the first arrangement order and the correspondence; The live streaming client determines the changes in each of the processed latency gradients under the second arrangement order; The live streaming client determines the congestion level information of the communication network based on the changes, and sends the congestion level information to the live streaming server.

4. The method according to claim 3, characterized in that, The live streaming client determines the changes in the post-processed latency gradients under the second arrangement order, including: The live streaming client performs linear regression on each of the processed latency gradients based on the second arrangement order to obtain a linear regression equation. The live streaming client determines the trend and rate of change of each processed latency gradient based on the slope of the linear regression equation. The live streaming client uses the changing trend and the changing rate as the changes in the post-processed delay gradients under the second arrangement order.

5. The method according to claim 4, characterized in that, The live streaming client determines the congestion level information of the communication network based on the changes between the processed latency gradients, including: If the change trend is an increasing trend, and the post-processing delay gradient at the end of the second arrangement order is greater than the delay gradient threshold, the live streaming client determines that the communication network is in a congested state, and determines the degree of congestion of the communication network when it is in the congested state based on the change rate. The live streaming client uses the congestion status and the degree of congestion as congestion information of the communication network. If the trend of change is a decreasing trend and / or the post-processing delay gradient at the end of the second arrangement order is not greater than the delay gradient threshold, the live streaming client determines that the communication network is in a non-congested state, and determines the degree of non-congestion of the communication network when it is in the non-congested state based on the rate of change. The live streaming client uses the non-congestion state and the degree of non-congestion as congestion information of the communication network.

6. The method according to claim 5, characterized in that, The method further includes: If the communication network is determined to be in a non-congested state based on the congestion level information, the live streaming server increases the multimedia data transmission rate based on the non-congestion level.

7. The method according to claim 6, characterized in that, The live streaming server increases the multimedia data transmission rate based on the level of non-congestion, including: The live streaming server obtains the proportional relationship between the rate of change corresponding to the degree of non-congestion and the second coefficient, and determines the second coefficient based on the proportional relationship and the rate of change corresponding to the degree of non-congestion; The live streaming server increases the multimedia data transmission rate based on the second coefficient.

8. The method according to claim 1, characterized in that, Before the live streaming server adjusts the multimedia data transmission rate for the live streaming client based on the congestion information, it further includes: The live streaming server obtains the input bitrate of the multimedia data that the current application layer sends to the live streaming client. The live streaming server determines the retransmission rate when retransmitting the multimedia data to the live streaming client. The live streaming server determines the current multimedia data transmission rate for the live streaming client based on the input bitrate and the retransmission rate.

9. The method according to claim 1, characterized in that, The live streaming client determines the latency gradient generated during the transmission of each set of multimedia data in the communication network between the live streaming server and the live streaming client, based on the total transmission and reception duration of each set of multimedia data, including: The live streaming client calculates the difference between the total reception time and the total transmission time for each set of multimedia data, which is used as the latency gradient generated when each set of multimedia data is transmitted in the communication network.

10. A multimedia data transmission rate control device, applied to a multimedia system, the multimedia system including a live streaming server and a live streaming client, characterized in that, include: The duration determination unit is used to receive at least two sets of multimedia data sent by the live streaming server through the live streaming client, and to obtain the total transmission time spent when the multimedia data is sent by the live streaming server and the total reception time spent when the data is received by the live streaming client. The delay determination unit is used to determine the delay gradient generated when each set of multimedia data is transmitted in the communication network between the live streaming server and the live streaming client, based on the total transmission duration and total reception duration of each set of multimedia data by the live streaming client. A smoothing unit is used to smooth the latency gradient through the live streaming client to obtain several processed latency gradients. A congestion determination unit is used to determine the congestion level information of the communication network based on the changes between the processed delay gradients by the live streaming client, and to send the congestion level information to the live streaming server. A rate adjustment unit is used to adjust the multimedia data transmission rate for the live streaming client according to the congestion information via the live streaming server, including: if it is determined that the communication network is in a congested state according to the congestion information, the live streaming server obtains the inverse relationship between the rate of change corresponding to the congestion level and a first coefficient, and determines the first coefficient based on the inverse relationship and the rate of change corresponding to the congestion level. The live streaming server reduces the multimedia data transmission rate based on the first coefficient.

11. The apparatus according to claim 10, characterized in that, The smoothing unit is also used for: The live streaming client determines several sets of delay gradients to be processed from multiple delay gradients, wherein a set of delay gradients to be processed includes at least two delay gradients; The live streaming client obtains the weights corresponding to each delay gradient included in each group of delay gradients to be processed; The live streaming client forms a weighted latency gradient corresponding to each set of latency gradients to be processed based on the latency gradients included in each set of latency gradients to be processed and the weights corresponding to each latency gradient. The live streaming client obtains several processed delay gradients based on the weighted delay gradients.

12. The apparatus according to claim 10, characterized in that, The congestion determination unit is also used for: The live streaming client determines the first arrangement order of each delay gradient based on the receiving order of each group of multimedia data; The live streaming client obtains the correspondence between each processed latency gradient and each latency gradient; The live streaming client determines the second arrangement order of the processed latency gradient based on the first arrangement order and the correspondence; The live streaming client determines the changes in each of the processed latency gradients under the second arrangement order; The live streaming client determines the congestion level information of the communication network based on the changes, and sends the congestion level information to the live streaming server.

13. The apparatus according to claim 12, characterized in that, The congestion determination unit is also used for: The live streaming client performs linear regression on each of the processed latency gradients based on the second arrangement order to obtain a linear regression equation. The live streaming client determines the trend and rate of change of each processed latency gradient based on the slope of the linear regression equation. The live streaming client uses the changing trend and the changing rate as the changes in the post-processed delay gradients under the second arrangement order.

14. The apparatus according to claim 13, characterized in that, The congestion determination unit is also used for: If the change trend is an increasing trend, and the post-processing delay gradient at the end of the second arrangement order is greater than the delay gradient threshold, the live streaming client determines that the communication network is in a congested state, and determines the degree of congestion of the communication network when it is in the congested state based on the change rate. The live streaming client uses the congestion status and the degree of congestion as congestion information of the communication network; If the trend of change is a decreasing trend and / or the post-processing delay gradient at the end of the second arrangement order is not greater than the delay gradient threshold, the live streaming client determines that the communication network is in a non-congested state, and determines the degree of non-congestion of the communication network when it is in the non-congested state based on the rate of change. The live streaming client uses the non-congestion state and the degree of non-congestion as congestion information of the communication network.

15. The apparatus according to claim 14, characterized in that, The rate adjustment unit is also used for: If the communication network is determined to be in a non-congested state based on the congestion level information, the live streaming server increases the multimedia data transmission rate based on the non-congestion level.

16. The apparatus according to claim 15, characterized in that, The rate adjustment unit is also used for: The live streaming server obtains the proportional relationship between the rate of change corresponding to the degree of non-congestion and the second coefficient, and determines the second coefficient based on the proportional relationship and the rate of change corresponding to the degree of non-congestion; The live streaming server increases the multimedia data transmission rate based on the second coefficient.

17. The apparatus according to claim 10, characterized in that, The rate adjustment unit is also used for: The live streaming server obtains the input bitrate of the multimedia data that the current application layer sends to the live streaming client. The live streaming server determines the retransmission rate when retransmitting the multimedia data to the live streaming client. The live streaming server determines the current multimedia data transmission rate for the live streaming client based on the input bitrate and the retransmission rate.

18. The apparatus according to claim 10, characterized in that, The duration determination unit is also used for: The live streaming client calculates the difference between the total reception time and the total transmission time for each set of multimedia data, which is used as the latency gradient generated when each set of multimedia data is transmitted in the communication network.

19. A computer device, characterized in that, include: Memory, used to store computer programs; A processor configured to implement the steps of the multimedia data transmission rate control method as described in any one of claims 1 to 9 when executing the computer program.

20. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the multimedia data transmission rate control method as described in any one of claims 1 to 9.

21. A computer program product, characterized in that, The computer program product includes computer instructions stored in a computer-readable storage medium; the processor of the computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps in the multimedia data transmission rate control method as described in any one of claims 1 to 9.

Images