Method for rate control, video playing method, device and electronic equipment

Abstract

The application discloses a code rate control method, a video playing method, a device, an electronic device and a computer readable storage medium. The code rate control method comprises the following steps: determining a target video frame to be played by a client; predicting a network transmission state when the target video frame is sent to the client; determining an estimated encoding code rate of the target video frame; determining a user experience value of the target video frame according to the network transmission state and the estimated encoding code rate of the target video frame; adjusting the estimated encoding code rate of the target video frame based on the user experience value of the target video frame, so as to obtain a target encoding code rate corresponding to the target video frame, and sending the target video frame to the client after the target video frame is encoded by using the target encoding code rate. Through the scheme provided in the application, the phenomenon that the user is caused to have bad influences such as lag and delay or waste of network resources can be better avoided, so that the video watching experience of the user is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a bitrate control method, a video playing method and device, an electronic device and a computer readable storage medium. BACKGROUND

[0002] With the rapid development of computer technology, audio and video technology is increasingly applied in live streaming field, e-commerce field and other industries, for example, through cloud rendering technology to provide a display video of goods for the client, to realize the immersive 3D shopping experience of the user. In such applications, the bitrate control method can determine the encoding bitrate of each frame of video to adapt to the change of network transmission state, which is a key technology to realize low delay and high user experience.

[0003] In related technologies, when performing bitrate control, the current network state (such as available bandwidth, bandwidth utilization, sending queue length, etc.) is usually observed in real time, and after detecting network congestion, the bitrate of each frame is reduced to alleviate the congestion phenomenon, and when detecting network idle, the bitrate of each frame is increased to improve the quality of the user watching the video.

[0004] The way of adjusting the bitrate according to the current network state in related technologies has already caused the user to have bad effects such as lag and delay or has wasted network resources, thus reducing the user's video watching experience as a whole. SUMMARY

[0005] The present application provides a bitrate control method, a video playing method and device, an electronic device and a computer readable storage medium, which can more timely and accurately adjust the bitrate of the video, to better avoid causing the user to have bad effects such as lag and delay or wasting network resources, thus improving the user's video watching experience. The specific scheme is as follows:

[0006] In a first aspect, the present application provides a bitrate control method, the method comprising:

[0007] determining a target video frame to be played by a client;

[0008] predicting a network transmission state when the target video frame is sent to the client;

[0009] determining an estimated encoding bitrate of the target video frame;

[0010] determining a user experience value of the target video frame according to the network transmission state and the estimated encoding code rate of the target video frame, the user experience value being an experience value brought to a user after sending the encoded target video frame to the client, the encoded target video frame being a video frame obtained by encoding the target video frame at the estimated encoding code rate;

[0011] adjusting the estimated encoding code rate of the target video frame based on the user experience value of the target video frame to obtain a target encoding code rate corresponding to the target video frame, so as to send the target video frame to the client after encoding the target video frame at the target encoding code rate.

[0012] Optionally, before the determining the user experience value of the target video frame according to the network transmission state and the estimated encoding code rate of the target video frame, the method further comprises:

[0013] predicting a motion state of the client to the target video frame, the motion state including having motion or no motion to a video frame;

[0014] The determining the user experience value of the target video frame according to the network transmission state and the estimated encoding code rate of the target video frame comprises:

[0015] determining the user experience value of the target video frame according to the network transmission state, the motion state and the estimated encoding code rate of the target video frame.

[0016] Optionally, the predicting the motion state of the client to the target video frame comprises:

[0017] predicting a motion state of the client to N frames of video frames to be determined target encoding code rate, a first frame of the N frames of video frames being the target video frame, N being an integer greater than 1;

[0018] The predicting the network transmission state when sending the target video frame to the client comprises:

[0019] predicting the network transmission state when sending each video frame in the N frames of video frames to the client;

[0020] The determining the user experience value of the target video frame according to the network transmission state, the motion state and the estimated encoding code rate of the target video frame comprises:

[0021] determining a user experience value of the i-th video frame according to a network transmission state, an action state and a predicted encoding rate corresponding to the i-th video frame in the N video frames, the user experience value of the i-th video frame being an experience value brought to a user after the i-th encoded video frame is sent to the client, the i-th encoded video frame being a video frame obtained by encoding the i-th video frame according to the corresponding predicted encoding rate, i being 1 to N;

[0022] adjusting an encoding rate of the target video frame based on the user experience value of the target video frame to obtain a target encoding rate corresponding to the target video frame, the target video frame being a next video frame to be played by the client.

[0023] adjusting predicted encoding rates of the first to N-th video frames based on a total user experience value of the first to N-th video frames to obtain target encoding rates corresponding to the first to N-th video frames respectively.

[0024] Optionally, the target encoding rate corresponding to the first video frame in the first to N-th video frames is used to encode the target video frame and send the target video frame to the client, the target video frame being the next video frame to be played by the client.

[0025] Alternatively, the target encoding rates corresponding to the first to N-th video frames are used to encode the first to N-th video frames respectively and send the first to N-th video frames to the client, the target video frame being an earliest undetermined rate video frame, the earliest undetermined rate video frame being a video frame in the video frames to be played by the client, for which a corresponding target encoding rate is not determined and which has the earliest playing time.

[0026] Optionally, the target video frame is the next video frame to be played by the client.

[0027] The prediction of the action state of the target video frame by the client includes:

[0028] obtaining an action sequence corresponding to M adjacent video frames before the target video frame, wherein M is an integer greater than 1, and the action sequence includes action states corresponding to the M video frames.

[0029] predicting the action state of the target video frame by the client according to the action sequence corresponding to the M video frames.

[0030] Optionally, the prediction of the action state of the target video frame by the client according to the action sequence corresponding to the M video frames includes:

[0031] finding, from a pre-set probability and statistics table, an action probability / actionless probability corresponding to a next frame of the action sequence corresponding to the M frame video frame, wherein the probability and statistics table statistically records probabilities of a next frame corresponding to an action state / actionless state for various action sequences;

[0032] randomly sampling according to the action probability / actionless state to determine the action state of the target video frame by the client.

[0033] Optionally, the probability of a next frame of the first action sequence corresponding to an action state / actionless state in the probability and statistics table is determined by:

[0034] determining the probability of a next frame of the first action sequence corresponding to an action state / actionless state according to a number of times that a second action sequence appears in the user historical behavior data and a number of times that the first action sequence appears in the user historical behavior data, wherein the second action sequence is an action sequence obtained by adding an action state / actionless state at the end of the first action sequence.

[0035] Optionally, the network transmission state includes at least one of a network transmission rate, a network one-way transmission delay, and a data amount in a sending queue.

[0036] Optionally, the determining of the user experience value of the target video frame according to the network transmission state, the action state, and a predicted encoding code rate of the target video frame includes:

[0037] determining a video quality of the target video frame according to the predicted encoding code rate of the target video frame;

[0038] predicting a total delay of the target video frame from starting to be sent to being received by the client according to the network transmission state;

[0039] determining an experience loss of the target video frame due to the delay according to the total delay and the action state, wherein the experience loss due to the delay is greater for an action state with action than for an action state without action;

[0040] determining a user experience value of the target video frame according to the video quality and the experience loss.

[0041] Optionally, the predicting of the total delay of the target video frame from starting to be sent to being received by the client according to the network transmission state includes:

[0042] predicting a waiting duration of the target video frame in a sending queue according to a data amount in the sending queue;

[0043] The sum of the waiting time length and the network one-way transmission delay is determined as the total delay from the start of sending the target video frame to the reception of the target video frame by the client.

[0044] Optionally, the prediction of the waiting time length of the target video frame in the sending queue according to the amount of data in the sending queue comprises:

[0045] The amount of data in the sending queue is determined according to the estimated encoding code rate corresponding to the target video frame.

[0046] The waiting time length of the target video frame in the sending queue is predicted according to the amount of data in the sending queue, the amount of data of the target video frame after encoding, and the network transmission rate.

[0047] Optionally, the prediction of the waiting time length of the target video frame in the sending queue according to the amount of data in the sending queue, the amount of data of the target video frame after encoding, and the network transmission rate comprises:

[0048] The waiting time length of the target video frame in the sending queue is determined as the ratio of the amount of data in the sending queue to the first amount of data of the target video frame after encoding and the network transmission rate.

[0049] Optionally, the amount of data in the sending queue is predicted by:

[0050] The second amount sum of the amount of data in the sending queue when sending the previous video frame and the amount of data of the previous video frame after encoding is determined.

[0051] The amount of data in the sending queue when sending the target video frame is determined according to the second amount sum, the interval between the time when the target video frame starts to be sent and the time when the previous video frame starts to be sent, and the network transmission rate.

[0052] Optionally, the determination of the video quality of the target video frame according to the encoding code rate of the target video frame comprises:

[0053] The video quality of the target video frame is determined based on the principle that the video quality is in a non-linear positive proportion to the encoding code rate, and the marginal utility of the improvement of the video quality due to the increase of the encoding code rate decreases.

[0054] Optionally, the determination of the experience loss of the target video frame due to the delay according to the total delay and the action state comprises:

[0055] The action state weight coefficient is determined according to the action state, wherein the action state weight coefficient corresponding to the action state with action is greater than the action state weight coefficient corresponding to the action state without action.

[0056] determine a delay loss caused by delay according to the total delay;

[0057] determine an experience loss caused by delay of the target video frame according to the action state weight coefficient and the delay loss.

[0058] Optionally, the determining the delay loss caused by delay according to the total delay comprises:

[0059] determine the delay loss caused by delay according to the principle that the delay loss increases as a quadratic function of the increase of the total delay.

[0060] Optionally, a network transmission rate when the target video frame is sent to the client is: a current network transmission rate between the client and the server.

[0061] The network one-way transmission delay is predicted by:

[0062] determine a predicted round-trip delay between the client and the server, and determine half of the predicted round-trip delay as the one-way transmission delay of the target video frame in the process of being sent to the client.

[0063] Optionally, the predicted round-trip delay is determined by:

[0064] obtain a latest round-trip delay between the client and the server;

[0065] obtain a last smoothed round-trip delay;

[0066] determine a current smoothed round-trip delay according to the latest round-trip delay and the last smoothed round-trip delay, and determine the current smoothed round-trip delay as the predicted round-trip delay.

[0067] In a second aspect, the present application provides a video playing method applied to a server, the method comprising:

[0068] determine a target video frame to be played by a client;

[0069] predict a network transmission state when the target video frame is sent to the client;

[0070] determine an estimated encoding code rate of the target video frame;

[0071] determine a user experience value of the target video frame according to the network transmission state and the estimated encoding code rate of the target video frame, the user experience value being an experience value brought to a user after a coded video frame is sent to the client, the coded video frame being a video frame obtained by encoding the target video frame according to the estimated encoding code rate;

[0072] adjust the estimated encoding code rate of the target video frame based on the user experience value of the target video frame, to obtain a target encoding code rate corresponding to the target video frame;

[0073] encode the target video frame through the target encoding code rate and send the encoded target video frame to the client, so that the client decodes and plays the decoded target video frame.

[0074] In a third aspect, the present application provides a code rate control device, the device comprising:

[0075] a video determination unit configured to determine a target video frame to be played by a client;

[0076] a network prediction unit configured to predict a network transmission state when the target video frame is sent to the client;

[0077] a code rate determination unit configured to determine an estimated encoding code rate of the target video frame;

[0078] an experience determination unit configured to determine a user experience value of the target video frame according to the network transmission state and the estimated encoding code rate of the target video frame, the user experience value being an experience value brought to a user after a client receives an encoded video frame, the encoded video frame being a video frame obtained by encoding the target video frame through the estimated encoding code rate;

[0079] a code rate adjustment unit configured to adjust the estimated encoding code rate of the target video frame based on the user experience value of the target video frame, to obtain a target encoding code rate corresponding to the target video frame, and to encode the target video frame through the target encoding code rate and send the encoded target video frame to the client.

[0080] In a fourth aspect, the present application further provides an electronic device comprising a processor, a memory, and computer program instructions stored in the memory and executable on the processor; the processor implements the method of any one of the first aspect to the second aspect when executing the computer program instructions.

[0081] In a fifth aspect, the present application provides a computer readable storage medium, the computer readable storage medium storing computer execution instructions, the computer execution instructions being executed by a processor to implement the method of any one of the first aspect to the second aspect.

[0082] In a sixth aspect, the present application provides a computer program product comprising a computer program, the computer program being executed by a processor to implement the method of any one of the first aspect to the second aspect.

[0083] Compared with the prior art, the present application has the following advantages:

[0084] The method provided by the embodiment of the present application comprises the following steps: determining a target video frame to be played by a client; predicting a network transmission state of the target video frame in a process of sending the target video frame to the client, the network transmission state reflecting a good or bad network transmission state in the process of transmitting the target video frame, which will affect video sending delay, freezing and other phenomena; determining a user experience value of the target video frame according to the predicted network transmission state and a predicted encoding code rate of the target video frame, the user experience value being an experience value brought to a user after sending an encoded video frame to the client, the encoded video frame being a video frame obtained by encoding the target video frame according to the predicted encoding code rate; adjusting the predicted encoding code rate of the target video frame based on the user experience value of the target video frame, taking improving the user experience value of the target video frame as a code rate adjustment principle, to obtain a target encoding code rate corresponding to the target video frame; and sending the target video frame to the client after encoding the target video frame according to the target encoding code rate.

[0085] The method provided by the embodiment of the present application determines the target encoding code rate of the target video frame based on the user experience value of the target video frame, so that the determined target encoding code rate can better improve the video watching experience of the user. In addition, the method provided by the embodiment of the present application predicts the network transmission state in the process of sending the target video frame to the client in advance for each target video frame to be played, that is, the method provided by the embodiment of the present application makes a forward-looking prediction on the network condition for the target video frame to be played, and makes a code rate prediction in advance based on the predicted network state and taking improving the user experience as a principle. Compared with the related art which adjusts the code rate when detecting network congestion or idling, the method provided by the embodiment of the present application can actively and timely adjust the code rate to be consistent with the predicted network state, can actively avoid delay accumulation and data congestion problems in advance, so as to more timely and accurately adjust the code rate of the video, to better avoid causing bad effects such as freezing and delay to the user or wasting network resources, thereby improving the video watching experience of the user, and significantly reducing the delay and freezing phenomenon compared with the traditional responsive method.

[0086] Further, the embodiment of the present application also predicts the action state of the client to the target video frame, the action state includes action or no action to the video frame, that is, the present application predicts whether the user has action to the target video frame on the client, because whether the user continues to operate the target video frame will affect the user's video watching experience, for example, when the user performs zooming, rotating and other actions on the video, the user is more likely to pay attention to the video stall in the action process, thereby affecting the user's watching experience, the user's sensitivity to video delay in the interactive real-time audio / video communication application changes dynamically, especially in the frequent interaction stage, the sensitivity to delay is higher, the present application also considers the action state of the user to the video frame when estimating the code rate, determines the user experience value according to the estimated user action state to estimate the target coding code rate, which can effectively reduce the delay in the dynamic interaction stage of the user in the interactive real-time audio / video communication, and significantly reduce the stall phenomenon in the user interaction stage, improve the fluency and satisfaction of the user in the user interaction, and accurately adapt to the differentiated needs of the user experience in different interaction stages. BRIEF DESCRIPTION OF DRAWINGS

[0087] Figure 1 FIG. 1 is a schematic diagram of an application scenario of the code rate control scheme provided by the present application;

[0088] Figure 2 FIG. 1 is a schematic diagram of an application scenario of the code rate control scheme provided by the present application;

[0089] Figure 3 FIG. 1 is a schematic diagram of an application scenario of the code rate control scheme provided by the present application;

[0090] Figure 4 FIG. 1 is a schematic diagram of an application scenario of the code rate control scheme provided by the present application;

[0091] Figure 5 FIG. 1 is a schematic diagram of an application scenario of the code rate control scheme provided by the present application; DETAILED DESCRIPTION

[0092] In order for those skilled in the art to better understand the technical solutions of the present application, the present application will be described in detail below in conjunction with the drawings in the embodiments of the present application. However, the present application can be implemented in many other ways different from the description below, therefore, based on the embodiments provided by the present application, all other embodiments obtained by those skilled in the art without creative labor shall belong to the scope of protection of the present application.

[0093] It should be noted that the terms "first", "source domain", "third", and the like in the claims, the specification, and the drawings of the present application are used to distinguish similar objects, and are not intended to describe a specific order or sequence. The data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include", "have" and their variants are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0094] In order to facilitate the understanding of the embodiments of the present application, the application background of the embodiments is described.

[0095] With the rapid development of computer technology, audio and video technology is increasingly applied in live streaming, e-commerce and other industries, for example, cloud rendering technology is used to provide a display video of a commodity for a client, so as to realize an immersive 3D shopping experience of a user. In such applications, a bitrate control method can determine the encoding bitrate of each frame of video to adapt to the changes in network transmission state, which is a key technology to realize low latency and high user experience.

[0096] In the related art, when performing bitrate control, the current network state (such as available bandwidth, bandwidth utilization, sending queue length, etc.) is usually observed in real time, and after detecting network congestion, the bitrate of each frame is reduced to alleviate the congestion phenomenon, and when detecting network idle, the bitrate of each frame is increased to improve the quality of the video watched by the user.

[0097] The way of adjusting the bitrate according to the current network state in the related art has already caused the user to have a bad effect such as lag, delay, or has wasted network resources, thereby reducing the user's video watching experience as a whole.

[0098] To solve the above problems, the embodiments of the present application provide a bitrate control method, a video playing method, a device, an electronic device and a computer readable storage medium. The purpose is to better avoid providing inaccurate answers for the user, to provide more accurate answers for the user, so as to reduce the situation that the user is misled due to providing inaccurate answers.

[0099] The bitrate control method provided by the present application can be applied to bitrate control in various professional fields, specifically, it can be applied to live streaming, 3D cloud rendering video playing, recorded video playing, stored video playing, etc., but is not limited thereto.

[0100] In order to facilitate the understanding of the method embodiments of the present application, the application scenarios are introduced. Please refer to Figure 1 , Figure 1 The application scenario of the scheme provided by the embodiments of the present application is shown in the figure. The application scenario is a schematic example and does not constitute a specific description of the application scenario. As Figure 1 shown, the application scenario is provided with a server 102 and a client 101. In this embodiment, the client 101 and the server 102 establish a connection through network communication to perform data transmission.

[0101] The client 101 can be a mobile phone, a tablet computer (pad), a smart watch, a desktop computer, a smart television, a VR device, a vehicle-mounted device, a wearable device, a notebook computer, etc. The client 101 is used to receive an operation instruction of a video input by a user, the operation instruction can include rotating, zooming, local magnifying, etc. of the video, and send the operation instruction to the server 102, so that the server 102 obtains a motion state corresponding to a video frame played by the client according to the operation instruction of the user, estimates a motion state of a next frame of the video played according to the motion state corresponding to the video frame played, and determines a coding code rate of the next frame of the video played, encodes the video frame and sends it to the client. The client 101 is also used to obtain the encoded video frame from the server 102 and play it after decoding. The client 101 is also used to send a video playing request to the server, so that the server 102 sends the corresponding video frame after encoding to the client 101 for display. The client 101 can also be used to send an access request, interactive information, etc. to the server 102, so that the server 102 sends the corresponding request data to the client 101 for display.

[0102] The server 102 has high computing power. The server 102 can be a server, and the server 102 has high central processing unit (CPU) computing power, long-time reliable operation, powerful input / output (I / O) external data throughput capacity and better scalability. The server 102 can be a single server or a server cluster. The server 102 is used to encode the video frame to be played by the client 101 and send it to the client 101. The server 102 can also provide other specific services for the client 101, such as user information access, website access, application program access, etc. The present application does not make specific limitations.

[0103] The client 101 and the server 102 can communicate with each other by using various communication systems, for example, can be by using a wired communication system or a wireless communication system. The wireless communication system can be, for example, a global system for mobile communications (GSM) system, a code division multiple access (CDMA) system, a wideband code division multiple access (WCDMA) system, a general packet radio service (GPRS), a long term evolution (LTE) system, an LTE frequency division duplex (FDD) system, an LTE time division duplex (TDD), a universal mobile telecommunication system (UMTS), a worldwide interoperability for microwave access (WiMAX) communication system, a 5th generation (5G) system or a new radio (NR), a satellite communication system, etc.

[0104] Embodiment one

[0105] The first embodiment of the present application provides a code rate control method, which can be applied to an electronic device. The electronic device can be a server, i.e., the code rate of a video frame to be played is controlled by the server. The electronic device can be, for example, a server, a desktop computer, a notebook computer, a mobile phone, a tablet computer (pad), a smart watch, a smart television, a VR device, a vehicle-mounted device, a wearable device, etc.

[0106] As shown in Figure 2 The code rate control method provided by the first embodiment of the present application includes the following steps S110-S150.

[0107] Step S110: Determine the target video frame to be played by the client.

[0108] The target video frame can be a next frame of video frame to be played by the client. In this case, for each frame of video frame to be played by the client, i.e., for each next frame of video frame to be played by the client, the target encoding code rate of the frame is determined according to the code rate control method provided in the application before playing. In this way, the code rate control can respond to the dynamic viewing experience demand changes of the user in real time, and the code rate of each next frame can be adjusted in time according to the dynamically changing network environment, thereby improving the overall user experience.

[0109] The target video frame can also be a video frame with the earliest playing time among the video frames to be played by the client and for which the corresponding target encoding code rate has not been determined, i.e., the target video frame is the earliest video frame with undetermined code rate. In this case, the target encoding code rates of multiple video frames to be played next are determined at one time during the last determination of the encoding code rate, and the target encoding code rates of the multiple video frames are used for encoding the respective video frames. Therefore, the target encoding code rates of the multiple video frames can be determined through one encoding determination process, and the specific determination method will be described in detail below. In this case, the process of determining the target encoding code rate can be performed every several frames, i.e., steps S110 to S140 are performed every several frames, thereby improving the code rate control efficiency and reducing the calculation resources consumed for code rate control.

[0110] Step S120: predicting a network transmission state when the target video frame is sent to the client.

[0111] The network transmission state when the target video frame is sent to the client reflects the network environment of sending the target video frame to the client as a whole, and the network transmission state affects the delay of sending the target video frame to the client. The network transmission state can include at least one of a network transmission rate, a network one-way transmission delay, and a data amount in a sending queue, but is not limited thereto.

[0112] The network one-way transmission delay refers to a time length consumed from the start of sending a data packet to the reception of the data packet by the client in a short future period. The short future period can be, for example, 2 seconds, 1 second, or another short period in the future. The network one-way transmission delay can specifically refer to a time length consumed from the start of sending the target video frame to the reception of the target video frame by the client.

[0113] The data amount in the sending queue refers to the accumulated data amount in the sending queue when the target video frame is sent.

[0114] The network transmission rate refers to the data amount transmitted per unit time.

[0115] The prediction method is usually different for different network transmission state parameters. The prediction methods for different network transmission state parameters are described below.

[0116] In an embodiment, the network transmission rate when the target video frame is sent to the client is the current network transmission rate between the server and the client. That is, the network transmission rate between the server and the client at the current time is determined as the network transmission rate when the target video frame is sent to the client. The current network transmission rate between the server and the client can be understood as the current sending rate. The server can conveniently obtain the current sending rate. For example, a rate monitoring module can be set on the server to monitor the current sending rate in real time. Alternatively, the current sending rate can be determined according to feedback information of the client. The feedback information includes the receiving time of the data received by the client. The server can calculate the current sending rate according to the sending time and the receiving time. Since the transmission rate remains almost stable in a short time (such as 300 ms), the current sending rate can be estimated as the network transmission rate of the next frame or several frames to be played in the future, so that the rate determination is simple and convenient, and the accuracy can be better guaranteed. Alternatively, the average sending rate in a fixed-size time window can be calculated and used as the estimated value of the future rate. The window size can be adjusted to balance the trade-off between response speed and stability, that is, the network transmission rate when the target video frame is sent to the client is estimated by means of sliding window average.

[0117] In an embodiment, the one-way transmission delay of the network can be predicted by determining a predicted round-trip delay between the server and the client, and determining half of the predicted round-trip delay as the one-way transmission delay of the target video frame in the process of being sent to the client.

[0118] The predicted round-trip delay mentioned above refers to the predicted round-trip delay when the target video frame is sent to the client. The round-trip delay refers to the time length between sending data from the server to the client and receiving feedback data of the client. Since the round-trip delay is usually easier to obtain, the one-way transmission delay is estimated by predicting the round-trip delay in this embodiment.

[0119] Specifically, the predicted round-trip delay can be determined by the following steps A to C.

[0120] Step A: Obtain the latest round-trip delay between the server and the client.

[0121] The latest round-trip delay mentioned above can be the round-trip delay corresponding to the latest video frame.

[0122] Step B: Obtain the last smoothed round-trip delay.

[0123] Step C: Determine the current smoothed round-trip delay according to the latest round-trip delay and the smoothed round-trip delay, and determine the current smoothed round-trip delay as the predicted round-trip delay.

[0124] Specifically, an attenuation coefficient can be set, the attenuation coefficient being used to indicate the weight of the latest observed round-trip delay, so that step C can determine the current smoothed round-trip delay according to the latest round-trip delay, the attenuation coefficient and the smoothed round-trip delay.

[0125] For example, if the attenuation coefficient = 0.6, indicating that the "newly observed round-trip delay" accounts for 60% of the weight, then the last smoothed round-trip delay accounts for 40% of the weight. If the last smoothed round-trip delay is 38 ms and the newly measured latest round-trip delay is 42 ms, then the new current smoothed round-trip delay may be calculated as formula (1) as follows:

[0126]

[0127] (1)

[0128] The predicted round-trip delay is 40.4 ms, and then the one-way transmission delay of the target video frame in the process of sending to the client is 20.2 ms.

[0129] The embodiment can accurately and conveniently predict the predicted round-trip delay by the smoothing window method, so as to obtain the network one-way transmission delay.

[0130] Alternatively, the predicted round-trip delay can also be determined in other ways, for example, the predicted round-trip delay can be determined by a pre-trained round-trip prediction model, the average value of the round-trip delays in the last time period is determined as the predicted round-trip delay, and the like, which are not specifically limited by the present application.

[0131] The network one-way transmission delay can also be predicted in other ways, for example, the network one-way transmission delay when the target video frame is sent to the client can be predicted by a pre-trained one-way delay prediction model, the average value of the network one-way transmission delays in the last time period is determined as the network one-way transmission delay when the target video frame is sent to the client, and the like, which are not specifically limited by the present application.

[0132] The amount of data in the sending queue can be queried by using an existing network query interface, or the total amount of data can be recorded each time data is written into the sending queue, and the corresponding value is subtracted from the count after the data is sent out, so as to obtain the amount of data in the sending queue at different times. Those skilled in the art can flexibly select the method of obtaining the amount of data in the sending queue, which is not specifically limited by the present application.

[0133] Step S130: determining the estimated encoding code rate of the target video frame.

[0134] The estimated encoding rate of the target video frame can be the encoding rate of the last video frame sent to the client, can be a preset encoding rate, or can be the average encoding rate of the video frames that have been sent to the client, and the application does not make a specific limitation. The estimated encoding rate is a predetermined estimated value.

[0135] Step S140: determining a user experience value of the target video frame according to the network transmission state and the estimated encoding rate of the target video frame.

[0136] The user experience value is the experience value brought to the user after the encoded video frame is sent to the client, and the encoded video frame is a video frame obtained by encoding the target video frame according to the estimated encoding rate.

[0137] Specifically, the total time length (i.e., total delay) spent in sending the target video frame to the client can be estimated according to the network transmission state and the estimated encoding rate of the target video frame, the video quality of the target video frame after being sent to the client can be determined according to the estimated encoding rate of the target video frame, and the user experience value of the target video frame can be determined according to the video quality and the total time length. For example, the difference between the video quality and the total time length can be determined as the user experience value of the target video frame; or a relationship function among the video quality, the total time length, and the user experience value can be determined, and the user experience value can be determined according to the relationship function, wherein the user experience value is positively correlated with the video quality and is negatively correlated with the total delay in the relationship function, and the specific function formula of the relationship function is not specifically limited by the application.

[0138] In an embodiment, before step S140, the following step S140a can also be included.

[0139] Step S140a: predicting the action state of the client to the target video frame, wherein the action state includes having an action or having no action to the video frame.

[0140] The action state of the video frame having an action indicates that the user will operate the frame video frame on the client, such as scaling, rotating, and the like, and the action state of the video frame having no action indicates that the user will not operate the frame video frame on the client.

[0141] Optionally, the action state of the target video frame can be predicted by a pre-trained action prediction model, the action prediction model being configured to predict the action state of the target video frame according to an action sequence composed of action states of a plurality of adjacent video frames before the target video frame. Specifically, the action sequence composed of the action states of the plurality of adjacent video frames before the target video frame can be input into the action prediction model to obtain the action state corresponding to the target video frame. When the target video frame is the next video frame to be played by the client, the plurality of adjacent video frames before the target video frame are the video frames that have been played by the client. The training of the action prediction model can use a training algorithm such as supervised, unsupervised or semi-supervised training algorithm in the related art, which will not be described in detail herein.

[0142] In one embodiment, step S140a can predict the action state of the target video frame by steps S140a1-S140a2.

[0143] Step S140a1: Obtain an action sequence corresponding to M adjacent video frames before the target video frame, wherein M is an integer greater than 1, and the action sequence includes action states corresponding to the M video frames.

[0144] When the target video frame is the next video frame to be played by the client, the M adjacent video frames before the target video frame are the M video frames that have been played by the client. When the target video frame is the earliest video frame for which the target encoding rate has not been determined, the M adjacent video frames before the target video frame are the M video frames for which the corresponding target encoding rate has been determined. Hereinafter, the target video frame will be taken as the next video frame to be played by the client as an example.

[0145] The value of M can be any number between 5 and 20, or other more or less numbers, which will not be specifically limited herein.

[0146] When the M video frames are the M video frames that have been played by the client, the action states corresponding to the M video frames can be easily obtained since the M video frames have been played by the client. When the M video frames are the M video frames for which the corresponding target encoding rate has been determined, the action states corresponding to the M video frames can be determined at the same time as the action state is determined last time, thereby obtaining the action states corresponding to the M video frames.

[0147] Step S140a2: Predict the action state of the target video frame according to the action sequence corresponding to the M video frames.

[0148] Specifically, the motion probability / motionless probability of the next frame corresponding to the motion sequence of the M-frame video frame can be found from a pre-set probability statistical table, where probabilities of the next frame corresponding to various motion sequences being in the motion state / motionless state are statistically recorded in the probability statistical table; and the motion state of the target video frame is determined by random sampling according to the motion probability / motionless state.

[0149] For example, the motion probability of the next frame corresponding to the motion sequence of the M-frame video frame being in the motion state can be found from a pre-set probability statistical table, where probabilities of the next frame corresponding to various motion sequences being in the motion state are statistically recorded in the probability statistical table; and the motion state of the target video frame is determined by random sampling according to the motion probability; or the motionless probability of the next frame corresponding to the motion sequence of the M-frame video frame being in the motionless state can be found from a pre-set probability statistical table, where probabilities of the next frame corresponding to various motion sequences being in the motionless state are statistically recorded in the probability statistical table; and the motion state of the target video frame is determined by random sampling according to the motionless probability.

[0150] By setting the probability statistical table, the embodiment can conveniently and efficiently find the probability of the next frame being in the motion state or the motionless state, and the motion state of the target video frame can be efficiently and accurately estimated through random experiments.

[0151] The probability of the next frame corresponding to the first motion sequence being in the motion state / motionless state in the probability statistical table is determined according to the number of times that the second motion sequence appears in the user historical behavior data and the number of times that the first motion sequence appears in the user historical behavior data, where the second motion sequence is obtained by adding a motion state / motionless state at the end of the first motion sequence. The embodiment can quickly obtain the probability statistical table, thereby improving the determination efficiency of the statistical table. For example, the probability of the next frame corresponding to the first motion sequence being in the motion state / motionless state can be determined by the following formula (2).

[0152] (2)

[0153] wherein, represents the probability of the motion state of the t-th frame being in the motion state, the t-th frame being the next frame of the first motion sequence, represents the motion state of the t-th frame (1 represents a motion frame and 0 represents a non-motion frame), represents the number of times that the corresponding state sequence appears in the user historical behavior data.

[0154] Alternatively, the action sequence corresponding to the M video frames can be input into the action prediction model to obtain the action state of the target video frame of the client through the action prediction model.

[0155] Correspondingly, step S140 can be implemented in the following step S141.

[0156] Step S141: determining the user experience value of the target video frame according to the network transmission state, the action state, and the estimated encoding code rate of the target video frame.

[0157] Specifically, the total time length (i.e., total delay) taken to send the target video frame to the client can be estimated according to the network transmission state and the estimated encoding code rate of the target video frame, the experience loss caused by the delay to the user can be determined according to the action state of the target video frame and the total time length taken to send the target video frame, for example, different loss weights can be assigned to different action states, the loss weight of the action state with motion is greater than the loss weight of the action state without motion, that is, when there is motion, the delay will bring worse viewing experience to the user, the experience loss caused by the delay to the user is determined according to the loss weight corresponding to the action state and the total delay, for example, the product of the loss weight corresponding to the action state and the total delay can be determined as the experience loss caused by the delay to the user, or the sum of the loss weight corresponding to the action state and the total delay can be determined as the experience loss caused by the delay to the user. The video quality of the target video frame after being sent to the client is determined according to the estimated encoding code rate of the target video frame, the user experience value of the target video frame is determined according to the video quality and the experience loss. For example, the difference between the video quality and the experience loss can be determined as the user experience value of the target video frame; or a relationship function between the video quality, the experience loss, and the user experience value can be determined, and the user experience value is determined according to the relationship function, wherein the user experience value is positively correlated with the video quality and negatively correlated with the experience loss in the relationship function, and the specific function of the relationship function is not limited in the present application.

[0158] Step S150: adjusting the estimated encoding code rate of the target video frame based on the user experience value of the target video frame to obtain a target encoding code rate corresponding to the target video frame, so as to encode the target video frame through the target encoding code rate and send it to the client.

[0159] Specifically, the pre-estimated encoding code rate of the target video frame can be adjusted according to a code rate adjustment principle of improving the user experience value of the target video frame, for example, the pre-estimated encoding code rate of the target video frame can be adjusted according to a code rate adjustment principle of maximizing the user experience value of the target video frame, or the pre-estimated encoding code rate of the target video frame can be adjusted according to a principle that the user experience value of the target video frame is greater than a preset experience threshold. In the adjustment process, the pre-estimated encoding code rate of the target video frame can be adjusted multiple times in a manner of multiple rounds of iterative adjustment until the user experience value of the target video frame is improved to meet the requirements.

[0160] The code rate control method provided in the embodiments of the present application determines a target video frame to be played by a client, predicts a network transmission state of the target video frame in a process of sending the target video frame to the client, determines a user experience value of the target video frame according to the predicted network transmission state and a pre-estimated encoding code rate of the target video frame, and adjusts the pre-estimated encoding code rate of the target video frame according to a code rate adjustment principle of improving the user experience value of the target video frame to obtain a target encoding code rate corresponding to the target video frame, so that the target video frame is encoded according to the target encoding code rate and then sent to the client.

[0161] The code rate control method provided in the embodiments of the present application determines a target video frame to be played by a client, predicts a network transmission state of the target video frame in a process of sending the target video frame to the client, determines a user experience value of the target video frame according to the predicted network transmission state and a pre-estimated encoding code rate of the target video frame, and adjusts the pre-estimated encoding code rate of the target video frame according to a code rate adjustment principle of improving the user experience value of the target video frame to obtain a target encoding code rate corresponding to the target video frame, so that the target video frame is encoded according to the target encoding code rate and then sent to the client.

[0162] Further, the embodiment of the present application also predicts the action state of the client on the target video frame, the action state including having action or no action on the video frame, that is, the present application predicts whether the user has action on the target video frame on the client, because whether the user continues to operate on the target video frame will affect the video watching experience of the user, for example, when the user performs zooming, rotating and other actions on the video, the user is more likely to pay attention to the video lag in the action process, thereby affecting the user watching experience, the user's sensitivity to video delay in the interactive real-time audio / video communication application changes dynamically, especially in the frequent interaction stage, the sensitivity to delay is higher, the present application also considers the action state of the user on the video frame when estimating the code rate, determines the user experience value according to the estimated user action state to estimate the target encoding code rate, which can effectively reduce the delay in the dynamic interaction stage of the user in the interactive real-time audio / video communication, and significantly reduce the lag phenomenon in the user interaction stage, improve the fluency and satisfaction of the user in the interaction, and accurately adapt to the differentiated needs of the user experience in different interaction stages.

[0163] In an embodiment, as shown in FIG. 14A, step S140a can be implemented in the following step S140a3. Figure 3

[0164] Step S140a3: predicting the action state of the client on N frames of video frames for which the target encoding code rate is to be determined, the first frame of the N frames of video frames being the target video frame, and N being an integer greater than 1.

[0165] The N frames of video frames can be specifically N frames of video frames to be played next by the client, and when the target encoding code rates corresponding to the first to N frames of video frames are determined, the N frames of video frames can also be specifically N frames of video frames for which the encoding rate is not determined recently.

[0166] The determination manner of the action state of each frame in the N frames of video frames is similar to the determination manner of the action state of the target video frame in step S140a, which will not be described in detail here.

[0167] The embodiment simultaneously predicts N frames of video frames to be played next by the client, and the N frames of video frames can be the first N frames of video frames to be played by the client, or the first N frames of video frames for which the target encoding code rate is not determined. In the embodiment, when predicting the action state of a frame, if the M frames before the frame are played video frames, the actual action state corresponding to the M frames of video frames can be obtained according to the actual operation of the user, and if there are unplayed video frames in the M frames before the frame, the action state of the current predicted video frame can be predicted according to the predicted action state corresponding to the unplayed video frame.

[0168] ​Step S120 can predict the network transmission state when the target video frame is sent to the client in the following step S121.

[0169] Step S121: predict the network transmission state when each video frame in the N video frames is sent to the client.

[0170] The prediction method of the network transmission state of other video frames in the N video frames can refer to the prediction of the network transmission state of the target video frame in step S120 above, which will not be described in detail here. The network transmission rate corresponding to the N video frames can be the same, and the one-way transmission delay corresponding to the N video frames can also be the same.

[0171] Step S141 can be implemented in the following step S141a.

[0172] Step S141a: according to the network transmission state, motion state and estimated encoding code rate corresponding to the i-th video frame in the N video frames, determine the user experience value of the i-th video frame, i traverses 1-N.

[0173] The user experience value of the i-th video frame is the experience value brought to the user after sending the i-th encoded video frame to the client, and the i-th encoded video frame is the video frame obtained by encoding the i-th video frame according to the corresponding estimated encoding code rate.

[0174] The determination method of the estimated encoding code rate corresponding to each frame in the N video frames can refer to the method of determining the estimated encoding code rate of the target video frame in step S130 above, and the calculation method of the user experience value corresponding to each frame can refer to the determination method of the user experience value corresponding to the target video frame in step S141 above, which will not be described in detail here.

[0175] Step S150 can adjust the encoding code rate of the target video frame in the following step S151 to obtain the target encoding code rate corresponding to the target video frame.

[0176] Step S151: based on the total user experience value of the first N video frames, adjust the estimated encoding code rate of the first N video frames to obtain the target encoding code rate corresponding to the first N video frames respectively.

[0177] Specifically, the user experience total value of the first to Nth video frames can be used as a code rate adjustment principle to improve the user experience total value of the first to Nth video frames, for example, the user experience total value of the first to Nth video frames can be used as a code rate adjustment principle to maximize the user experience total value of the first to Nth video frames, or the user experience total value of the first to Nth video frames can be used as a principle to adjust the estimated encoding code rate of the target video frame to be greater than a preset experience total threshold. The step S151 can also be adjusted in multiple rounds of iteration, and specific details can be referred to the description of the step S150, which will not be described here.

[0178] The embodiment is based on the user experience total value of the first to Nth video frames, that is, the user experience value of the whole multiple continuous frames is comprehensively considered to estimate the target encoding code rate of the target video frame, which can make the determined target encoding code rate more stably and reliably meet the user experience, and can also determine the target encoding code rates of the next multiple video frames, thereby improving the code rate determination efficiency.

[0179] The target encoding code rate corresponding to the first video frame in the first to Nth video frames is used to encode the target video frame and then sent to the client, and the target video frame is the next video frame to be played by the client. That is, although the target encoding code rates corresponding to the first to Nth video frames are determined in the step S151, only the target encoding code rate corresponding to the first video frame (i.e., the target video frame) is used to encode the target video frame and then sent to the client. In this case, the target video frame is the next video frame to be played by the client, that is, the process of determining the code rate in the steps S110 to S150 is performed once before each video frame is played by the client, so that a more accurate encoding code rate can be determined in real time and dynamically, and the determined encoding code rate is more consistent with the current real-time network status and user operation.

[0180] Alternatively, the target encoding code rates corresponding to the first to Nth video frames are respectively used to encode the first to Nth video frames and then sent to the client, and the target video frame is the earliest video frame whose code rate is not determined, and the earliest video frame whose code rate is not determined is the video frame whose corresponding target encoding code rate is not determined and whose playing time is the earliest among the video frames to be played by the client. That is, the target encoding code rates corresponding to the first to Nth video frames determined in the step S151 are all used to encode the corresponding video frames and then sent to the client. In this case, since the target encoding code rates of the N video frames have been determined, the target video frame determined in the next time of determining the encoding code rate is the video frame after the N video frames, that is, the earliest video frame whose code rate is not determined. This way can reduce the number of times of executing the process of determining the target encoding code rate, thereby better saving the computing resources.

[0181] In an embodiment, step S141 can determine the user experience value of the target video frame according to the following steps S141b-S141e.

[0182] Step S141b: determining the video quality of the target video frame according to the estimated encoding code rate of the target video frame.

[0183] Generally, the higher the encoding code rate, the higher the video quality, thus, the video quality of the target video frame can be determined based on the principle that the video quality is positively correlated with the estimated encoding code rate, for example, the estimated encoding code rate can be directly determined as the video quality of the target video frame. In practical applications, when the encoding code rate increases to a certain threshold, the display effect of a higher encoding code rate can be similar to that of a lower encoding code rate, that is, the marginal utility of the increase of the encoding code rate on the improvement of the video quality is diminishing, and is nonlinearly proportional, thus, the video quality of the target video frame can be determined based on the principle that the video quality is nonlinearly proportional to the encoding code rate, and the marginal utility of the increase of the encoding code rate on the improvement of the video quality is diminishing, so that the determined video quality is more accurate and more in line with the actual situation.

[0184] Step S141c: predicting the total delay of the target video frame from being sent to being received by the client according to the network transmission state.

[0185] The total delay from being sent to being received by the client refers to the total time length consumed by the target video frame from being sent to being received by the client.

[0186] In a specific embodiment, the total delay of the target video frame from being sent to being received by the client can be simply and accurately predicted according to the following steps 1-2.

[0187] Step 1: predicting the waiting time length of the target video frame in the sending queue according to the data amount in the sending queue.

[0188] The data amount in the sending queue refers to the data amount in the sending queue when the target video frame is sent, specifically, the data amount of the target video frame after encoding can be determined according to the estimated encoding code rate corresponding to the target video frame; the waiting time length of the target video frame in the sending queue can be predicted according to the data amount in the sending queue, the data amount of the target video frame after encoding, and the network transmission rate. Specifically, the waiting time length of the target video frame in the sending queue can be obtained by dividing the sum of the data amount in the sending queue and the data amount of the target video frame after encoding by the network transmission rate, that is, the ratio of the first amount of the data amount in the sending queue and the data amount of the target video frame after encoding to the network transmission rate can be determined as the waiting time length of the target video frame in the sending queue.

[0189] Step 2: summing the waiting time length and the network one-way transmission delay to determine the total delay from the start of sending the target video frame to the reception of the target video frame by the client.

[0190] The data amount in the sending queue can be predicted by determining a second sum of the data amount in the sending queue when the previous video frame is sent and the data amount of the encoded previous video frame, and determining the data amount in the sending queue when the target video frame is sent according to the second sum, the interval between the time when the target video frame starts to be sent and the time when the previous video frame starts to be sent, and the network transmission rate.

[0191] The previous video frame refers to the previous video frame of the target video frame, and the second sum can be understood as the total data amount in the sending queue after the previous video frame is encoded and added to the sending queue. Specifically, the product of the interval between the time when the target video frame starts to be sent and the time when the previous video frame starts to be sent and the network transmission rate corresponding to the target video frame can be determined as the data amount sent out from the sending queue within the interval between the time when the target video frame starts to be sent and the time when the previous video frame starts to be sent, and the difference between the second sum and the data amount sent out from the sending queue within the interval can be determined as the data amount in the sending queue when the target video frame is sent.

[0192] In the embodiments of the present application, the interval between the time when the target video frame starts to be sent and the time when the previous video frame starts to be sent can be the waiting time length of the previous video frame in the sending queue.

[0193] Step S141d: determining the experience loss of the target video frame caused by the delay according to the total delay and the action state, wherein the experience loss caused by the delay is greater for the action state with action than for the action state without action.

[0194] Specifically, the experience loss of the target video frame caused by the delay can be determined according to the following steps S141d1-S141d3.

[0195] Step S141d1: determining an action state weight coefficient according to the action state, wherein the action state weight coefficient corresponding to the action state with action is greater than the action state weight coefficient corresponding to the action state without action.

[0196] The greater the action state weight coefficient is, the worse the delay will bring to the user. Specifically, the action state weight coefficient corresponding to the action state with action can be a number greater than 1, for example, can be any value in 1.1-1.6, or can be other numerical values greater than 1. The action state weight coefficient corresponding to the action state without action can be 1. Those skilled in the art can flexibly set other ranges of weight coefficients, for example, the action state weight coefficients corresponding to the action state with action and the action state without action can be a number less than 1, or can be a number greater than 2, etc. The present application does not specifically limit that the action state weight coefficient corresponding to the action state with action is greater than the action state weight coefficient corresponding to the action state without action.

[0197] Step S141d2: determining the delay loss caused by delay according to the total delay from the start of sending to the reception of the target video frame by the client.

[0198] Generally, the longer the total delay is, the greater the experience loss brought to the user, that is, the greater the delay loss is. Therefore, the delay loss caused by delay can be determined according to the principle that the total delay is positively correlated with the delay loss caused by delay, for example, the delay loss can be proportional to the total delay. In actual application, the longer the delay is, the faster the experience loss brought to the user will significantly increase, therefore, the delay loss caused by delay can be determined according to the principle that the delay loss increases as a quadratic function of the increase of the total delay, so that the experience loss caused by delay to the user in actual application can be more accurately reflected.

[0199] Step S141d3: determining the experience loss caused by delay of the target video frame according to the action state weight coefficient and the delay loss.

[0200] Specifically, the product of the action state weight coefficient and the delay loss can be determined as the experience loss caused by delay of the target video frame, or the sum of the action state weight coefficient and the delay loss can also be determined as the experience loss caused by delay of the target video frame, which can be flexibly selected by those skilled in the art. Among them, determining the product of the action state weight coefficient and the delay loss as the experience loss caused by delay of the target video frame can more accurately reflect the actual loss.

[0201] Step S141e: determining the user experience value of the target video frame according to the video quality and the experience loss.

[0202] The difference between the video quality and the experience loss can be determined as the user experience value of the target video frame, or an experience coefficient can be set for the video quality and an experience coefficient can be set for the experience loss, and the user experience value of the target video frame can be determined according to the video quality, the corresponding experience coefficient, and the experience loss corresponding to the experience coefficient. For example, the difference between the product of the video quality and the corresponding experience coefficient and the product of the experience loss and the corresponding experience coefficient can be determined as the user experience value of the target video frame.

[0203] The embodiment can determine the final user experience value according to the predicted video quality of the target video frame encoded at the estimated encoding code rate and the experience loss of the target video frame caused by the delay, and can comprehensively consider the comprehensive situation of the video quality of the target video frame, the user action state, and the network state to predict a more accurate user experience value.

[0204] When the application adjusts the estimated encoding code rate of the first N video frames based on the total user experience value of the first N video frames according to step S151 in the above embodiment, the action state, the network transmission state, the user experience value, the video quality, the total delay, the experience loss, and the user experience value corresponding to each frame in the first N video frames are similar to the determination method of the target video frame, and will not be described in detail here. The determination of each parameter in the code rate control process will be introduced in detail through the following formula examples.

[0205] In the embodiment of the application, the video quality of each frame in the N video frames can be determined by the following formula (3).

[0206] (3)

[0207] wherein, Q i represents the video quality of the i th video frame, R i represents the estimated encoding code rate of the i th video frame, R max represents the maximum threshold of the code rate, a is a quality control constant, and specifically, a can be taken as , and the specific values of a can be flexibly selected.

[0208] When i is 1, the first video frame is the target video frame.

[0209] The total delay of the target video frame from the start of sending to being received by the client can be determined by the following formula (4).

[0210] (4)

[0211] wherein, T i represents the total delay of the i th video frame from the start of sending to being received by the client, denotes the one-way transmission delay of the network in the process of transmitting the i-th video frame to the client, denotes the waiting time of the i-th video frame in the sending queue.

[0212] the waiting time of the i-th video frame in the sending queue It can be determined by the following formula (5).

[0213] (5)

[0214] wherein, denotes the data volume of the i-th video frame after encoding according to the corresponding estimated encoding rate, denotes the data volume in the sending queue when the i-th video frame is sent, denotes the network transmission rate corresponding to the i-th video frame.

[0215] The data volume in the sending queue when each video frame is sent can be updated according to the following formula (6).

[0216]

[0217] (6)

[0218] wherein, denotes the data volume in the sending queue when the i+1-th video frame is sent, denotes the time when the i+1-th video frame is sent, denotes the time when the i-th video frame is sent.

[0219] The delay loss of each frame due to delay can be predicted according to the following formula (7).

[0220] (7)

[0221] wherein, denotes the delay loss of the i-th video frame due to delay, denotes the total delay from the start of sending to the reception of the i-th video frame by the client, denotes the maximum delay threshold, It can be any value between 100ms and 200ms, or other more or less values, which can be flexibly set by those skilled in the art.

[0222] In step S151, the estimated encoding rate of the 1st to Nth video frames can be adjusted based on the following formula (8).

[0223] (8)

[0224] wherein, wherein, represents the motion state weight coefficient corresponding to the i-th video frame, and other parameters in formula (7) refer to the above description and will not be repeated here.

[0225] Formula (8) is to adjust the estimated encoding code rate of the 1st to Nth video frames according to the principle of maximizing the total user experience value of the 1st to Nth video frames.

[0226] Embodiment Two

[0227] The second embodiment of the present application also provides a video playing method, which is applied to a server and includes the following steps S210 to S260.

[0228] Step S210: determining a target video frame to be played by a client;

[0229] Step S220: predicting a network transmission state when the target video frame is sent to the client;

[0230] Step S230: determining an estimated encoding code rate of the target video frame;

[0231] Step S240: determining a user experience value of the target video frame according to the network transmission state and the estimated encoding code rate of the target video frame, wherein the user experience value is an experience value brought to a user after the encoded video frame is sent to the client, and the encoded video frame is a video frame obtained by encoding the target video frame according to the estimated encoding code rate;

[0232] Step S250: adjusting the estimated encoding code rate of the target video frame based on the user experience value of the target video frame to obtain a target encoding code rate corresponding to the target video frame;

[0233] Step S260: encoding the target video frame according to the target encoding code rate and sending the encoded target video frame to the client, so that the client decodes and plays the decoded target video frame.

[0234] This embodiment applies the code rate control method from the perspective of a specific application scenario of video playing. The execution processes of this embodiment are similar to those of the first embodiment, and the details of the related technical features and the effects achieved can be found in the corresponding description of the code rate control method embodiment provided in the first embodiment above.

[0235] Embodiment Three

[0236] The third embodiment of the present application also provides a code rate control device corresponding to the code rate control method embodiment provided in the first embodiment. Since the device embodiment is basically similar to the method embodiment, it is described more simply, and the details of the related technical features and the effects achieved can be found in the corresponding description of the code rate control method embodiment provided above. For example,Figure 4 The code rate control device provided in the embodiment includes:

[0237] The video determining unit 310 is configured to determine a target video frame to be played by a client;

[0238] The network predicting unit 320 is configured to predict a network transmission state when the target video frame is transmitted to the client;

[0239] The code rate determining unit 330 is configured to determine an estimated encoding code rate of the target video frame;

[0240] The experience determining unit 340 is configured to determine a user experience value of the target video frame according to the network transmission state and the estimated encoding code rate of the target video frame, the user experience value being an experience value brought to a user after a coded video frame is transmitted to the client, the coded video frame being a video frame obtained by encoding the target video frame according to the estimated encoding code rate;

[0241] The code rate adjusting unit 350 is configured to adjust the estimated encoding code rate of the target video frame based on the user experience value of the target video frame, to obtain a target encoding code rate corresponding to the target video frame, so that the target video frame is encoded according to the target encoding code rate and then transmitted to the client.

[0242] Embodiment Four

[0243] The fourth embodiment of the present application further provides an electronic device embodiment corresponding to the code rate control method provided in the first embodiment. The following description of the electronic device embodiment is merely illustrative. The electronic device embodiment includes the following:

[0244] Please refer to Figure 5 The above electronic device is understood as Figure 5 The electronic device provided in the embodiment includes a processor 1001, a memory 1002, a communication bus 1003, and a communication interface 1004;

[0245] The memory 1002 is configured to store computer instructions for data processing. When the computer instructions are read and executed by the processor 1001, the following steps are performed:

[0246] determine a target video frame to be played by a client;

[0247] predict a network transmission state when the target video frame is transmitted to the client;

[0248] determine an estimated encoding code rate of the target video frame;

[0249] determine a user experience value of the target video frame according to the network transmission state and the estimated encoding code rate of the target video frame, the user experience value being an experience value brought to a user after a coded video frame is sent to the client, the coded video frame being a video frame obtained after the target video frame is encoded at the estimated encoding code rate;

[0250] adjust the estimated encoding code rate of the target video frame based on the user experience value of the target video frame, to obtain a target encoding code rate corresponding to the target video frame, so that the target video frame is encoded at the target encoding code rate and then sent to the client.

[0251] The fifth embodiment of the application also provides an electronic device corresponding to the video playing method provided by the second embodiment. The following description of the electronic device embodiment is merely illustrative. The electronic device embodiment is as follows:

[0252] The electronic device provided in this embodiment includes a processor, a memory, a communication bus, and a communication interface.

[0253] The memory is configured to store computer instructions for data processing. When the computer instructions are read and executed by the processor, the following steps are performed:

[0254] determine a target video frame to be played by the client;

[0255] predict a network transmission state when the target video frame is sent to the client;

[0256] determine an estimated encoding code rate of the target video frame;

[0257] determine a user experience value of the target video frame according to the network transmission state and the estimated encoding code rate of the target video frame, the user experience value being an experience value brought to a user after a coded video frame is sent to the client, the coded video frame being a video frame obtained after the target video frame is encoded at the estimated encoding code rate;

[0258] adjust the estimated encoding code rate of the target video frame based on the user experience value of the target video frame, to obtain a target encoding code rate corresponding to the target video frame;

[0259] encode the target video frame at the target encoding code rate and then send the target video frame to the client, so that the client decodes and plays the decoded target video frame.

[0260] The sixth embodiment of the present application further provides a computer readable storage medium for implementing the method of the first embodiment. The computer readable storage medium embodiment provided by the present application is relatively simple, and the relevant parts refer to the corresponding description of the above method embodiment. The following described embodiments are only illustrative.

[0261] The computer readable storage medium provided by the present embodiment stores computer instructions, which are executed by a processor to implement the following steps:

[0262] Determining a target video frame to be played by the client;

[0263] Predicting a network transmission state when the target video frame is sent to the client;

[0264] Determining an estimated encoding code rate of the target video frame;

[0265] According to the network transmission state and the estimated encoding code rate of the target video frame, determining a user experience value of the target video frame, the user experience value being an experience value brought to a user after the encoded video frame is sent to the client, the encoded video frame being a video frame obtained by encoding the target video frame according to the estimated encoding code rate;

[0266] Based on the user experience value of the target video frame, adjusting the estimated encoding code rate of the target video frame to obtain a target encoding code rate corresponding to the target video frame, so as to send the target video frame to the client after encoding the target video frame according to the target encoding code rate.

[0267] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0268] The memory can include non-persistent memory in computer readable media, random access memory (RAM), and / or non-volatile memory, such as read-only memory (ROM) or flash memory (flash RAM). The memory is an example of computer readable media.

[0269] 1. Computer-readable media includes permanent and non-permanent, removable and non-removable media can be implemented by any method or technology to store information. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape disk storage or other magnetic storage devices or any other non-transmission medium that can be used to store information that can be accessed by a computing device. According to the definition herein, computer-readable media does not include non-transitory computer-readable media (transitory media), such as modulated data signals and carriers.

[0270] 2. Those skilled in the art should understand that the embodiments of the present application can be provided as a method, system or computer program product. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0271] Although the present application is disclosed with reference to the preferred embodiments above, it is not intended to limit the present application, and any person skilled in the art can make possible changes and modifications without departing from the spirit and scope of the present application. Therefore, the scope of protection of the present application should be defined by the scope defined by the claims of the present application.

Claims

1. A rate control method, characterized in that: The method comprises: Determine the target video frame to be played by the client; Predicting a network transmission state when sending the target video frame to the client; Determine an estimated encoding bit rate of the target video frame, where the estimated encoding bit rate is a predetermined estimated value; Predicting an action state of the client on the target video frame, where the action state includes whether there is an action or no action on the video frame; Determining a user experience value of the target video frame according to the network transmission state, the action state, and the estimated encoding bit rate of the target video frame, including: determining the video quality of the target video frame according to the estimated encoding bit rate of the target video frame; predicting the total delay of the target video frame from the start of sending to the time of being received by the client according to the network transmission state; determining the experience loss of the target video frame due to the delay according to the total delay and the action state, wherein the experience loss due to the delay of the action state with action is greater than that of the action state without action; determining the user experience value of the target video frame according to the video quality and the experience loss; the user experience value is the experience value brought to the user after the encoded video frame is sent to the client, and the encoded video frame is a video frame obtained after encoding the target video frame according to the estimated encoding bit rate; Based on the user experience value of the target video frame, the estimated encoding bit rate of the target video frame is adjusted by adopting multiple rounds of iterative adjustment until the user experience value of the target video frame is improved to meet the requirements, and the target encoding bit rate corresponding to the target video frame is obtained. The target video frame is encoded at the target encoding bit rate and then sent to the client.

2. The rate control method according to claim 1, wherein: The predicting the action state of the client on the target video frame includes: Predicting an action state of the client with respect to N video frames for which a target encoding bit rate is to be determined, where a first frame of the N video frames is the target video frame, and N is an integer greater than 1; The predicting of the network transmission state when sending the target video frame to the client includes: Predicting a network transmission state when sending each of the N video frames to the client; The determining, according to the network transmission state, the action state, and the estimated encoding bit rate of the target video frame, a user experience value of the target video frame includes: determining a user experience value of the i-th video frame according to a network transmission state, an action state, and an estimated encoding bit rate corresponding to the i-th video frame among the N video frames, where the user experience value of the i-th video frame is an experience value brought to the user after the i-th encoded video frame is sent to the client, the i-th encoded video frame being a video frame obtained by encoding the i-th video frame at the corresponding estimated encoding bit rate, where i traverses 1 to N; The adjusting the encoding bit rate of the target video frame based on the user experience value of the target video frame to obtain the target encoding bit rate corresponding to the target video frame includes: Based on the total user experience value of the 1st to Nth video frames, the estimated encoding bit rates of the 1st to Nth video frames are adjusted to obtain the target encoding bit rates corresponding to the 1st to Nth video frames respectively.

3. The rate control method according to claim 2, wherein: The target encoding bit rate corresponding to the first frame of the first to Nth video frames is used to encode the target video frame and then send it to the client, where the target video frame is the next video frame to be played by the client; Alternatively, the target encoding bit rates corresponding to the 1st to Nth video frames are used to encode the 1st to Nth video frames and then send them to the client, the target video frame is the earliest video frame with undetermined bit rate, and the earliest video frame with undetermined bit rate is the video frame for which the corresponding target encoding bit rate has not been determined among the video frames to be played by the client and the video frame with the earliest playback time.

4. The rate control method according to claim 1, wherein: The target video frame is the next video frame to be played by the client; The predicting the action state of the client on the target video frame includes: Obtaining an action sequence corresponding to M adjacent video frames preceding the target video frame, where M is an integer greater than 1, and the action sequence includes action states corresponding to the M adjacent video frames; The action state of the client for the target video frame is predicted according to the action sequence corresponding to the M video frames.

5. The rate control method according to claim 4, wherein: The predicting, based on the action sequence corresponding to the M video frames, the action state of the client for the target video frame includes: Searching a pre-set probability statistics table for the action probability / no-action probability of the next frame of the action sequence corresponding to the M frames of video frames being in the action state / no-action state, wherein the probability statistics table counts the probabilities of the next frame of various action sequences being in the action state / no-action state; Random sampling is performed based on the action probability / no-action state to determine the action state of the client for the target video frame.

6. The rate control method according to claim 5, wherein: The probability that the next frame of the first action sequence in the probability statistics table corresponds to the action state / no action state is determined by the following method: Based on the number of times the second action sequence appears in the user's historical behavior data and the number of times the first action sequence appears in the user's historical behavior data, the probability that the next frame of the first action sequence corresponds to an action state / no action state is determined, where the second action sequence is the action sequence obtained by adding an action state / no action state to the end of the first action sequence.

7. The rate control method according to claim 1, wherein: The network transmission status includes at least one of a network transmission rate, a network one-way transmission delay, and a data volume in a sending queue.

8. The rate control method according to claim 7, wherein: The predicting, based on the network transmission status, the total delay from the start of sending the target video frame to the time of being received by the client comprises: Predicting the waiting time of the target video frame in the sending queue according to the amount of data in the sending queue; The sum of the waiting time and the one-way transmission delay of the network is determined as the total delay from the start of sending the target video frame to being received by the client.

9. The rate control method according to claim 8, wherein: The predicting the waiting time of the target video frame in the sending queue according to the amount of data in the sending queue includes: Determining the amount of data after encoding the target video frame according to the estimated encoding bit rate corresponding to the target video frame; The waiting time of the target video frame in the sending queue is predicted according to the amount of data in the sending queue, the amount of data after the target video frame is encoded, and the network transmission rate.

10. The rate control method according to claim 9, wherein: The predicting, based on the amount of data in the sending queue, the amount of data after encoding the target video frame, and the network transmission rate, a waiting time of the target video frame in the sending queue includes: The waiting time of the target video frame in the sending queue is determined as the ratio of the first amount of the data volume in the sending queue to the encoded data volume of the target video frame and the network transmission rate.

11. The rate control method according to claim 10, wherein: The amount of data in the sending queue is predicted by: Determine a second sum of the amount of data in the sending queue when sending the previous video frame and the amount of data after encoding the previous video frame; The amount of data in the sending queue when the target video frame is sent is determined according to the second number, the time interval between the start time of sending the target video frame and the start time of sending the previous video frame, and the network transmission rate.

12. The rate control method according to claim 1, wherein: The determining the video quality of the target video frame according to the encoding bit rate of the target video frame includes: The video quality of the target video frame is determined based on the principle that video quality is nonlinearly proportional to encoding bit rate and that an increase in encoding bit rate has a decreasing marginal effect on improving video quality.

13. The rate control method according to claim 1, wherein: The determining, according to the total delay and the action state, an experience loss of the target video frame caused by the delay includes: Determining an action state weight coefficient according to the action state, wherein the action state weight coefficient corresponding to the action state with action is greater than the action state weight coefficient corresponding to the action state without action; determining a delay loss caused by the delay based on the total delay; The experience loss of the target video frame caused by the delay is determined according to the action state weight coefficient and the delay loss.

14. The rate control method according to claim 13, wherein: The determining of the delay loss caused by the delay according to the total delay includes: The delay loss caused by the delay is determined based on the principle that the delay loss increases quadratically with the increase of the total delay.

15. The rate control method according to claim 7, wherein: The network transmission rate when sending the target video frame to the client is: the current network transmission rate between the client and the network; The network one-way transmission delay is predicted by: A predicted round-trip delay between the target video frame and the client is determined, and half of the predicted round-trip delay is determined as a one-way transmission delay of the target video frame in a process of sending the target video frame to the client.

16. The rate control method according to claim 15, characterized in that: The predicted round trip delay is determined as follows: Get the latest round-trip latency with the client; Get the last smoothed round trip delay; A current smoothed round-trip delay is determined according to the latest round-trip delay and the previous smoothed round-trip delay, and the current smoothed round-trip delay is determined as the predicted round-trip delay.

17. A video playback method, characterized in that: Applied to the server, the method includes: Determine the target video frame to be played by the client; Predicting a network transmission state when sending the target video frame to the client; Determine an estimated encoding bit rate of the target video frame, where the estimated encoding bit rate is a predetermined estimated value; Predicting an action state of the client on the target video frame, where the action state includes whether there is an action or no action on the video frame; Determining a user experience value of the target video frame according to the network transmission state, the action state, and the estimated encoding bit rate of the target video frame, including: determining the video quality of the target video frame according to the estimated encoding bit rate of the target video frame; predicting the total delay of the target video frame from the start of sending to the time of being received by the client according to the network transmission state; determining the experience loss of the target video frame due to the delay according to the total delay and the action state, wherein the experience loss due to the delay of the action state with action is greater than that of the action state without action; determining the user experience value of the target video frame according to the video quality and the experience loss; the user experience value is the experience value brought to the user after the encoded video frame is sent to the client, and the encoded video frame is a video frame obtained after encoding the target video frame according to the estimated encoding bit rate; Based on the user experience value of the target video frame, adjusting the estimated encoding bit rate of the target video frame by using multiple rounds of iterative adjustment until the user experience value of the target video frame is improved to meet the requirements, thereby obtaining a target encoding bit rate corresponding to the target video frame; The target video frame is encoded at the target encoding rate and then sent to the client, so that the client decodes and plays the decoded target video frame.

18. A rate control device, characterized in that: The device comprises: A video determination unit, configured to determine a target video frame to be played by the client; a network prediction unit, configured to predict a network transmission state when sending the target video frame to the client; A bit rate determination unit, configured to determine an estimated encoding bit rate of the target video frame, wherein the estimated encoding bit rate is a predetermined estimated value; An experience determination unit is configured to predict an action state of the client with respect to the target video frame, where the action state includes whether there is action or no action on the video frame; and determine a user experience value of the target video frame according to the network transmission state, the action state, and the estimated encoding bit rate of the target video frame, including: determining a video quality of the target video frame according to the estimated encoding bit rate of the target video frame, predicting a total delay from the start of sending the target video frame to being received by the client according to the network transmission state, and determining an experience loss of the target video frame due to the delay according to the total delay and the action state, wherein an action state with action causes a greater experience loss due to the delay than an action state without action, and determining the user experience value of the target video frame according to the video quality and the experience loss; the user experience value is an experience value brought to the user after the encoded video frame is sent to the client, where the encoded video frame is a video frame obtained after encoding the target video frame according to the estimated encoding bit rate; A bit rate adjustment unit is used to adjust the estimated encoding bit rate of the target video frame based on the user experience value of the target video frame by adopting a multi-round iterative adjustment method until the user experience value of the target video frame is improved to meet the requirements, thereby obtaining a target encoding bit rate corresponding to the target video frame, and encoding the target video frame at the target encoding bit rate and sending it to the client.

19. An electronic device, characterized in that: include: a processor, a memory, and computer program instructions stored on the memory and executable on the processor; When the processor executes the computer program instructions, the method according to any one of claims 1 to 17 is implemented.

20. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method according to any one of claims 1 to 17.

Images