Data transmission method, apparatus, device, medium, and program product

Abstract

The application relates to a data transmission method, device, equipment, medium and program product. The method comprises the following steps: first, receiving history decoding state information and network state information sent by a target terminal, the history decoding state information being used for indicating a state parameter of the target terminal for decoding a second video frame before a first video frame to be sent; then, acquiring first decoding complexity information of the first video frame and second decoding complexity information of the second video frame; determining a decoding timeliness of the first video frame according to the history decoding state information, the network state information, the first decoding complexity information and the second decoding complexity information; finally, determining a target encoding mode according to the decoding timeliness, encoding the first video frame based on the target encoding mode, and sending the encoded first video frame to the target terminal. The method can improve the efficiency of data transmission.

Description

Technical Field

[0001] This application relates to the field of data transmission technology, and in particular to a data transmission method, apparatus, device, medium, and program product. Background Technology

[0002] With the widespread adoption of cloud terminals, users can access various cloud services such as cloud office, cloud gaming, and cloud live streaming. Among these, cloud gaming and cloud live streaming are low-latency cloud services. When network transmission jitter, packet loss, or delays occur, it can affect the smooth transmission of video streams in these cloud services, causing video stuttering.

[0003] Currently, when packet loss or delay occurs, the data packets need to be retransmitted to the cloud terminal for processing, which is inefficient. Summary of the Invention

[0004] Therefore, it is necessary to provide a data transmission method, apparatus, device, medium, and program product with higher transmission efficiency to address the aforementioned technical problems.

[0005] Firstly, this application provides a data transmission method. The method includes: receiving historical decoding status information and network status information sent by a target terminal, wherein the historical decoding status information is used to indicate status parameters for the target terminal to decode a second video frame preceding a first video frame to be transmitted; acquiring first decoding complexity information of the first video frame and second decoding complexity information of the second video frame; determining the decoding timeliness of the first video frame based on the historical decoding status information, network status information, first decoding complexity information, and second decoding complexity information; determining a target encoding method based on the decoding timeliness, encoding the first video frame based on the target encoding method, and sending the encoded first video frame to the target terminal.

[0006] In one embodiment, the historical decoding status information includes frame decoding time, and the network status information includes packet loss rate and round-trip latency. Determining the decoding timeliness of the first video frame based on the historical decoding status information, network status information, first decoding complexity information, and second decoding complexity information includes: calculating the ratio of the decoding complexity value indicated by the first decoding complexity information and the decoding complexity value indicated by the second decoding complexity information, and calculating a first product of the frame decoding time and the ratio; calculating the sum of the packet loss rate of the second video frame and a preset constant, and calculating a second product of the round-trip latency and the sum; and determining the decoding timeliness based on the sum of the first product and the second product.

[0007] In one embodiment, the first decoding complexity information is determined based on the total number of macroblock groups in the first video frame and the sum of squared residual errors of the P-frames; the second decoding complexity information is determined based on the total number of macroblock groups in the second video frame and the sum of squared residual errors of the P-frames.

[0008] In one embodiment, determining the target encoding method based on decoding timeliness includes: if the decoding timeliness is less than or equal to a preset frame rendering threshold, then the target encoding method is determined to be a video encoding method; if the decoding timeliness is greater than the preset frame rendering threshold, then the target encoding method is determined to be an IDR encoding method.

[0009] In one embodiment, the preset frame rendering threshold is determined based on the difference between the presentation timestamp of the first video frame and the presentation timestamp of the second video frame and a preset time base.

[0010] In one embodiment, receiving historical decoding status information and network status information sent by the target terminal includes: receiving RR messages, wherein the RR messages carry historical decoding status information and network status information.

[0011] In one embodiment, the RR message also carries a preset frame rendering threshold.

[0012] In one embodiment, the method further includes: determining the value of the encoding flag bit in the RTP packet according to the target encoding method; and sending the RTP packet to the target terminal.

[0013] Secondly, this application provides a data transmission method. The method includes: sending historical decoding status information and network status information to a cloud platform server, wherein the historical decoding status information is used to indicate the status parameters for a target terminal to decode a second video frame preceding a first video frame to be sent by the cloud platform server; receiving encoding method indication information sent by the cloud platform server based on the historical decoding status information and network status information; determining the encoding method of the first video frame according to the encoding method indication information; and decoding the first video frame sent by the cloud platform server based on the encoding method.

[0014] In one embodiment, receiving encoding method indication information sent by the cloud platform server based on historical decoding status information and network status information includes: receiving an RTP packet sent by the cloud platform server based on historical decoding status information and network status information, wherein the RTP packet includes an encoding flag bit; and using the value of the encoding flag bit as the encoding method indication information.

[0015] In one embodiment, the encoding method of the first video frame is determined according to the encoding method indication information, and the first video frame sent by the cloud platform server is decoded based on the encoding method, including: if the encoding method indication information determines that the encoding method of the first video frame is a video encoding method, then the first video frame is video decoded; if the encoding method indication information determines that the encoding method of the first video frame is an IDR encoding method, then the first video frame is IDR decoded.

[0016] In one embodiment, the method further includes: if packet loss exists, detecting whether the time the target video frame to be decoded waits for decoding in the decoding buffer is less than the packet loss retransmission time; if the time the target video frame waits for decoding in the decoding buffer is less than the packet loss retransmission time, determining a third video frame whose nearest encoding method is IDR encoding; and replacing the target video frame with the third video frame for decoding.

[0017] In one embodiment, the method further includes: if the time the target video frame waits for decoding in the decoding buffer is greater than or equal to the time for packet loss retransmission, performing packet loss retransmission processing.

[0018] Thirdly, this application also provides a data transmission apparatus. The apparatus includes:

[0019] The status information receiving module is used to receive historical decoding status information and network status information sent by the target terminal. The historical decoding status information is used to indicate the status parameters of the target terminal for decoding the second video frame before the first video frame to be sent.

[0020] The acquisition module is used to acquire the first decoding complexity information of the first video frame and the second decoding complexity information of the second video frame;

[0021] The decoding timeliness determination module is used to determine the decoding timeliness of the first video frame based on historical decoding status information, network status information, first decoding complexity information, and second decoding complexity information.

[0022] The data transmission module is used to determine the target encoding method based on the decoding timeliness, encode the first video frame based on the target encoding method, and send the encoded first video frame to the target terminal.

[0023] In one embodiment, the historical decoding status information includes frame decoding time, and the network status information includes packet loss rate and round-trip time. The decoding timeliness determination module is specifically used to calculate the ratio of the decoding complexity value indicated by the first decoding complexity information and the decoding complexity value indicated by the second decoding complexity information, and to calculate a first product of the frame decoding time and the ratio; calculate the sum of the packet loss rate of the second video frame and a preset constant, and to calculate a second product of the round-trip time and the sum; and determine the decoding timeliness based on the sum of the first product and the second product.

[0024] In one embodiment, the first decoding complexity information is determined based on the total number of macroblock groups in the first video frame and the sum of squared residual errors of the P-frames; the second decoding complexity information is determined based on the total number of macroblock groups in the second video frame and the sum of squared residual errors of the P-frames.

[0025] In one embodiment, the data sending module is specifically used to determine the target encoding method as a video encoding method if the decoding timeliness is less than or equal to a preset frame rendering threshold; and to determine the target encoding method as an IDR encoding method if the decoding timeliness is greater than the preset frame rendering threshold.

[0026] In one embodiment, the preset frame rendering threshold is determined based on the difference between the presentation timestamp of the first video frame and the presentation timestamp of the second video frame and a preset time base.

[0027] In one embodiment, the status information receiving module is specifically used to receive RR messages, which carry historical decoding status information and network status information.

[0028] In one embodiment, the RR message also carries a preset frame rendering threshold.

[0029] In one embodiment, the system further includes an encoding flag determination module, which determines the value of the encoding flag in the RTP packet according to the target encoding method and sends the RTP packet to the target terminal.

[0030] Fourthly, this application also provides a data transmission apparatus. The apparatus includes:

[0031] The status information sending module is used to send historical decoding status information and network status information to the cloud platform server. The historical decoding status information is used to indicate the status parameters of the target terminal for decoding the second video frame before the first video frame to be sent by the cloud platform server.

[0032] The instruction information receiving module is used to receive encoding method instruction information sent by the cloud platform server based on historical decoding status information and network status information;

[0033] The encoding method determination module is used to determine the encoding method of the first video frame according to the encoding method indication information, and to decode the first video frame sent by the cloud platform server based on the encoding method.

[0034] In one embodiment, the instruction information receiving module is specifically used to receive RTP packets sent by the cloud platform server based on historical decoding status information and network status information. The RTP packets include encoding flag bits; the value of the encoding flag bits is used as encoding mode indication information.

[0035] In one embodiment, the encoding method determination module is specifically used to perform video decoding on the first video frame if the encoding method indication information determines that the encoding method of the first video frame is video encoding; and to perform IDR decoding on the first video frame if the encoding method indication information determines that the encoding method of the first video frame is IDR encoding.

[0036] In one embodiment, a detection module is further included, which is used to detect whether the time the target video frame to be decoded waits for decoding in the decoding buffer is less than the time for packet loss retransmission if packet loss exists; if the time the target video frame waits for decoding in the decoding buffer is less than the time for packet loss retransmission, a third video frame whose nearest encoding method is determined to be IDR encoding is used; and the target video frame is replaced by the third video frame for decoding.

[0037] In one embodiment, a retransmission module is also included, which is used to perform packet loss retransmission processing if the time for the target video frame to wait for decoding in the decoding buffer is greater than or equal to the packet loss retransmission time.

[0038] Fifthly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the data transmission method described in either the first or second aspect above.

[0039] Sixthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, implements the data transmission method described in either the first or second aspect above.

[0040] In a seventh aspect, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the data transmission method described in either the first or second aspect above.

[0041] The aforementioned data transmission method, apparatus, device, medium, and program product first receive historical decoding status information and network status information sent by the target terminal, which instruct the target terminal to decode the second video frame preceding the first video frame to be transmitted. Then, it acquires the first decoding complexity information of the first video frame and the second decoding complexity information of the second video frame. Based on the historical decoding status information, network status information, first decoding complexity information, and second decoding complexity information, it determines the decoding timeliness of the first video frame. Finally, it determines the target encoding method based on the decoding timeliness, encodes the first video frame based on the target encoding method, and sends the encoded first video frame to the target terminal. In this way, before sending the first video frame, the decoding timeliness of the first video frame is calculated, and then the target encoding method of the first video frame is determined based on the decoding timeliness. The first video frame is then encoded according to the target encoding method and sent, thereby ensuring that the first video frame can be decoded in a timely manner, without causing video stuttering or delay, resulting in higher transmission efficiency. Attached Figure Description

[0042] Figure 1 This is an application environment diagram of a data transmission method in one embodiment;

[0043] Figure 2 This is a flowchart of a conventional data transmission method in one embodiment;

[0044] Figure 3 This is a flowchart illustrating the data transmission method in another embodiment;

[0045] Figure 4 This is a flowchart illustrating the data transmission method in another embodiment;

[0046] Figure 5 This is a flowchart illustrating the data transmission method in another embodiment;

[0047] Figure 6 This is a flowchart illustrating the data transmission method in another embodiment;

[0048] Figure 7 This is a flowchart illustrating the data transmission method in another embodiment;

[0049] Figure 8 This is a flowchart illustrating the data transmission method in another embodiment;

[0050] Figure 9 This is a flowchart illustrating the data transmission method in another embodiment;

[0051] Figure 10 A flowchart of a data transmission method in another embodiment;

[0052] Figure 11 An interaction diagram of the data transmission method in another embodiment;

[0053] Figure 12 This is a structural block diagram of a data transmission device in one embodiment;

[0054] Figure 13 This is a structural block diagram of a data transmission device in another embodiment;

[0055] Figure 14 This is an internal structural diagram of a computer device in one embodiment;

[0056] Figure 15 This is a diagram of the internal structure of a computer device in another embodiment. Detailed Implementation

[0057] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0058] The data transmission method provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, the target terminal 101 is communicatively connected to the cloud platform server 102. The target terminal 101 can be a cloud terminal, such as a cloud phone or cloud computer. Figure 1 As shown, the cloud platform server 102 includes the following components:

[0059] (1) Virtual machine CPU / GPU: Virtual cloud phones or cloud computers simulated by x86 / ARM servers to provide cloud services to users.

[0060] (2) Screen capture: After rendering the screen, the screen capture encoder is responsible for capturing the audio and video screen of the cloud service.

[0061] (3) IDR encoder: Real-time encoding of IDR frames in cloud service video streams, which is triggered when IDR encoding is required.

[0062] (4) Video encoder: Real-time encoding of cloud service video streams, which continues to encode after the cloud service is started.

[0063] (5) Decoding analysis and early warning: Receive the RR (Receiver Report) information from the target terminal, calculate the decoding timeliness, and determine whether to enable the IDR encoder.

[0064] (6) Network transmission: responsible for transmitting video streams to target terminals and notifying decoders and analysts of RR information.

[0065] Target terminal 101 includes the following parts:

[0066] (1) Network transmission: Receive video data sent by the cloud platform server and send RR information to the cloud platform server.

[0067] (2) Decoding load calculator: Statistics on video decoder decoding latency, GPU utilization and other information.

[0068] (3) Decoding buffer controller: controls the video buffer received by the network packet and hands the data to the video decoder frame by frame for decoding.

[0069] (4) Video decoder: responsible for decoding video streams for cloud services.

[0070] (5) Rendering output: Render the screen and output it to the user.

[0071] In traditional technologies, such as Figure 2 As shown, for a single frame of data, the cloud platform server encodes it and sends the data stream to the target terminal. Upon receiving the data stream, the target terminal checks for packet loss. If no packet loss occurs, the target terminal decodes the data and determines if decoding was successful. If successful, the data is rendered and the next frame is transmitted. If packet loss occurs, retransmission is performed. If decoding fails, an IDR frame is requested for decoding. However, this may require waiting for retransmitted data packets or IDR frames to arrive before decoding can begin. When the video frame data in the decoding buffer is insufficient, it may cause stuttering or delays in the video.

[0072] Based on the aforementioned traditional technologies, the data transmission method provided in this application, applied to a cloud platform server, firstly receives historical decoding status information and network status information sent by a target terminal, indicating the target terminal to decode a second video frame preceding the first video frame to be transmitted. Then, it acquires the first decoding complexity information of the first video frame and the second decoding complexity information of the second video frame. Based on the historical decoding status information, network status information, first decoding complexity information, and second decoding complexity information, it determines the decoding timeliness of the first video frame. Finally, it determines the target encoding method based on the decoding timeliness, encodes the first video frame based on the target encoding method, and sends the encoded first video frame to the target terminal. In this way, before sending the first video frame, the decoding timeliness of the first video frame is calculated, and then the target encoding method of the first video frame is determined based on the decoding timeliness. After encoding the first video frame according to the target encoding method, it is sent, thereby ensuring that the first video frame can be decoded in a timely manner, without causing video stuttering or delay, resulting in higher transmission efficiency.

[0073] It should be noted that the beneficial effects or technical problems solved by the embodiments of this application are not limited to this one, but may also be other implicit or related problems. For details, please refer to the description of the embodiments below.

[0074] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0075] In one embodiment, such as Figure 3 As shown, a data transmission method is provided, which is applied to... Figure 1 Taking a cloud platform server as an example, the explanation includes the following steps:

[0076] Step 301: Receive historical decoding status information and network status information sent by the target terminal.

[0077] The historical decoding status information is used to indicate the status parameters of the target terminal during the decoding of the second video frame preceding the first video frame to be sent. This historical decoding status information can be the frame decoding time of the second video frame, which is the time from when the target terminal reads a frame of data from the buffer to when the decoder completes decoding; this time can be obtained from the decoder. Network status information can include the packet loss rate, which is the packet loss rate over a certain period of time reported by the target terminal via RTP RR feedback; this rate can be statistically analyzed by the network card. The network status information can also include the round-trip latency of network packets interacting between the target terminal and the cloud platform service.

[0078] Step 302: Obtain the first decoding complexity information of the first video frame and the second decoding complexity information of the second video frame.

[0079] The first decoding complexity information is used to represent the decoding complexity of the first video frame, and the second decoding complexity information is used to represent the decoding complexity of the second video frame. The decoding complexity is estimated and determined by the encoder during video frame encoding.

[0080] Step 303: Determine the decoding timeliness of the first video frame based on historical decoding status information, network status information, first decoding complexity information, and second decoding complexity information.

[0081] Based on the historical decoding status information, network status information, first decoding complexity information, and second decoding complexity information obtained above, the decoding timeliness of the first video frame can be calculated.

[0082] Step 304: Determine the target encoding method based on the decoding timeliness, encode the first video frame based on the target encoding method, and send the encoded first video frame to the target terminal.

[0083] Optionally, the cloud platform server can determine the decoding timeliness of video frames at preset intervals or for each frame. Based on the decoding timeliness, it determines the target encoding method, which can be IDR (Instantaneous Decoder Refresh) encoding or video encoding. If the target encoding method is IDR encoding, the first video frame is encoded according to the IDR encoding method; if the target encoding method is video encoding, the first video frame is encoded according to the video encoding method. The encoded first video frame is then sent to the target terminal. The IDR frame generated by IDR encoding is a type of I-frame, which includes complete data for one frame; decoding only requires this frame's data. The video encoding generates p-frames, which require overlaying previously cached frames with the differences defined in this frame during decoding to generate the final image.

[0084] In the above embodiment, firstly, historical decoding status information and network status information sent by the target terminal are received, which instruct the target terminal to decode the second video frame preceding the first video frame to be transmitted. Then, the first decoding complexity information of the first video frame and the second decoding complexity information of the second video frame are obtained. Based on the historical decoding status information, network status information, first decoding complexity information, and second decoding complexity information, the decoding timeliness of the first video frame is determined. Finally, the target encoding method is determined based on the decoding timeliness, and the first video frame is encoded based on the target encoding method. The encoded first video frame is then sent to the target terminal. In this way, before sending the first video frame, the decoding timeliness of the first video frame is calculated, and then the target encoding method of the first video frame is determined based on the decoding timeliness. The first video frame is then encoded according to the target encoding method and sent, thereby ensuring that the first video frame can be decoded in a timely manner, without causing video stuttering or delay, resulting in higher transmission efficiency.

[0085] In one embodiment, historical decoding status information includes frame decoding time, and network status information includes packet loss rate and round-trip latency. The steps for determining the decoding time of the first video frame based on historical decoding status information, network status information, first decoding complexity information, and second decoding complexity information are as follows: Figure 4 As shown, it includes:

[0086] Step 401: Calculate the ratio of the decoding complexity value indicated by the first decoding complexity information to the decoding complexity value indicated by the second decoding complexity information, and calculate the first product of the frame decoding time and the ratio.

[0087] Optionally, the first decoding complexity information is determined based on the total number of macroblock groups in the first video frame and the sum of squared residual errors of the P-frames; the second decoding complexity information is determined based on the total number of macroblock groups in the second video frame and the sum of squared residual errors of the P-frames.

[0088] Taking the first decoding complexity information as an example, the decoding complexity value is estimated by the encoder during frame encoding, using the computational cost generated during macroblock prediction, and calculated based on the macroblock group allocation, i.e., the number of macroblock groups. The calculation formula for the decoding complexity value indicated by the first decoding complexity information is as follows:

[0089]

[0090] Where DC is the decoding complexity value, SSD (SUM of Squared Difference) is the sum of squared residual errors of P-frames, and the number of macroblock groups is the total number of macroblock groups in the first video frame.

[0091] After calculating the decoding complexity value indicated by the first decoding complexity information and the decoding complexity value indicated by the second decoding complexity information, the ratio between the two is calculated, and then the ratio is multiplied by the frame decoding time to obtain the first product.

[0092] Step 402: Calculate the sum of the packet loss rate of the second video frame and the preset constant, and calculate the second product of the round-trip delay and the sum.

[0093] The preset constant can be 1. Calculate the sum of the packet loss rate of the second video frame and the preset constant, and then multiply it by the round-trip delay of the second video frame to obtain the second product.

[0094] Step 403: Determine the decoding time based on the sum of the first product and the second product.

[0095] When the preset constant is 1, the decoding and timing calculation methods are as follows:

[0096]

[0097] Where DT is decoding timeliness, FDT is frame decoding time, DC is decoding complexity, RTT is round-trip time, which is the round-trip time of network packets between the target terminal and the cloud platform server, LR is packet loss rate, i represents the first video frame, and i-1 represents the second video frame.

[0098] In one embodiment, after determining the decoding timeliness of the first video frame, the target encoding method is determined based on the decoding timeliness, including the following two cases:

[0099] In the first case, if the decoding timeliness is less than or equal to the preset frame rendering threshold, then the target encoding method is determined to be the video encoding method.

[0100] The preset frame rendering threshold is determined by the difference between the presentation timestamps of the first and second video frames and a preset time base. The frame rendering threshold is obtained by subtracting the playback times of the two frames. Optionally, the formula for calculating the preset frame rendering threshold is as follows:

[0101] FRT i =(PTS) i -PTS i-1 )×TimeBase (3)

[0102] Where FRT is the preset frame rendering threshold, PTS (Presentation Time Stamp) is the presentation timestamp of the video frame, TimeBase is the preset time base, i is the first video frame, and i-1 is the second video frame.

[0103] When the decoding timeliness is less than or equal to the preset frame rendering threshold, that is, the first video frame can be decoded in time, it can be encoded using video encoding methods.

[0104] In the second case, if the decoding timeliness is greater than the preset frame rendering threshold, then the target encoding method is determined to be IDR encoding method.

[0105] If the decoding timeliness is greater than the preset frame rendering threshold, it means that the current first video frame is not decoded in time. Therefore, the target encoding method is determined to be IDR encoding method, and encoding is performed using IDR encoding method.

[0106] In the above embodiments, the encoding method of the video frame is determined by the decoding timeliness and the preset frame rendering threshold. By predicting whether the video frame can be decoded in time, if it cannot be decoded in time, the IDR frame is sent by the IDR encoding method. This avoids the delay that may be caused by requesting to send the IDR frame when the video frame cannot be decoded in time, thus ensuring the smoothness of the video picture and avoiding stuttering.

[0107] Furthermore, when the predicted video frames can be decoded in time, they are encoded using video encoding methods, and the IDR encoder is turned off to save resources.

[0108] Optionally, after determining the target encoding method, the cloud platform server sends IDR video frames to the target terminal in the following steps: Figure 5 As shown, it also includes:

[0109] Step 501: Determine the value of the encoding flag bit in the RTP packet according to the target encoding method.

[0110] The cloud platform server inserts encoding flag information into the PT bit of the sent RTP data. If the target encoding method is IDR encoding, the cloud platform server sets the encoding flag in the RTP packet to 1. If the target encoding method is video encoding, the cloud platform server sets the encoding flag in the RTP packet to 0.

[0111] Step 502: Send the RTP packet to the target terminal.

[0112] Then, the cloud platform server sends the RTP packet to the target terminal, which can determine the target encoding method by checking the value of the encoding flag bit of the PT bit in the RTP data.

[0113] In the above embodiments, by setting an encoding flag in the RTP packet, the target terminal can determine the target encoding method according to the encoding flag after receiving the RTP packet, thereby achieving synchronization of the encoding and decoding methods between the cloud platform server and the target terminal.

[0114] In one embodiment, the cloud platform server receives historical decoding status information and network status information sent by the target terminal, including: receiving RR messages, wherein the RR messages carry historical decoding status information and network status information.

[0115] The cloud platform server receives RR messages sent by the target terminal. The RR messages carry historical decoding status information and network status information. Optionally, the RR messages also carry a preset frame rendering threshold.

[0116] In one embodiment, such as Figure 6 As shown, a data transmission method is provided, which is applied to... Figure 1 Taking the target terminal as an example, the explanation includes the following steps:

[0117] Step 601: Send historical decoding status information and network status information to the cloud platform server.

[0118] Historical decoding status information is used to indicate the status parameters of the target terminal during the decoding of the second video frame preceding the first video frame to be sent by the cloud platform server. This historical decoding status information may include frame decoding time (the frame decoding time of the second video frame by the target terminal) and network status information between the target terminal and the cloud platform server, such as packet loss rate and round-trip latency.

[0119] Step 602: Receive encoding method indication information sent by the cloud platform server based on historical decoding status information and network status information.

[0120] The cloud platform server determines the target encoding method based on the historical decoding status information and network status information sent by the target terminal using the method described in the above embodiments, and then determines encoding method indication information based on the target encoding method and sends it to the target terminal. Optionally, such as Figure 7 As shown, the steps for the target terminal to receive the encoding method indication information include:

[0121] Step 701: Receive RTP packets sent by the cloud platform server based on historical decoding status information and network status information.

[0122] RTP packets include encoding flags. The values ​​of the encoding flags are determined based on the target encoding method.

[0123] Step 702: Use the value of the encoding flag bit as the encoding method indication information.

[0124] Step 603: Determine the encoding method of the first video frame according to the encoding method indication information, and decode the first video frame sent by the cloud platform server based on the encoding method.

[0125] The target terminal determines the encoding method of the first video frame based on the encoding method indication information, and decodes the first video frame sent by the cloud platform server based on the encoding method, including:

[0126] If the encoding method indication information determines that the encoding method of the first video frame is video encoding, then the first video frame is video decoded.

[0127] If the target terminal detects that the value of the encoding flag is 0, that is, the encoding method of the first video frame is video encoding, then the target terminal performs video decoding on the first video frame.

[0128] If the encoding method indication information determines that the encoding method of the first video frame is IDR encoding, then IDR decoding is performed on the first video frame.

[0129] If the target terminal detects that the value of the encoding flag bit is 1, that is, the encoding method of the first video frame is IDR encoding, then IDR decoding is performed on the first video frame.

[0130] In the embodiments of this application, after the target terminal receives video frame data, it selects different processing methods according to different situations, such as... Figure 8 As shown, it includes:

[0131] Step 801: If packet loss exists, check whether the time the target video frame to be decoded waits for decoding in the decoding buffer is less than the time for packet loss retransmission.

[0132] The process involves the target terminal receiving video frame data and checking for packet loss. If no packet loss occurs, the video frame data is placed in the decoding buffer for decoding. If packet loss does occur, and the target video frame is the one with packet loss, the terminal checks whether the time the target video frame waits for decoding in the decoding buffer is less than the packet retransmission time.

[0133] Step 802: If the time the target video frame waits for decoding in the decoding buffer is less than the time for packet loss retransmission, determine the third video frame whose nearest encoding method is IDR encoding.

[0134] If the time the target video frame waits for decoding in the decoding buffer is less than the time for packet loss retransmission, that is, the target video frame does not have enough time to be retransmitted, the video frame with the nearest encoding method of IDR encoding is determined as the third video frame based on the encoding flag bit of the video frame in the decoding buffer.

[0135] Step 803: Decode the target video frame by replacing it with the third video frame.

[0136] The third video frame replaces the target video frame that experienced packet loss for decoding. Since the third video frame is an IDR frame, which contains complete data of a single frame, all frame data preceding the third video frame in the decoding buffer are deleted after decoding using the third video frame.

[0137] Optionally, if the time the target video frame waits for decoding in the decoding buffer is greater than or equal to the time for packet loss retransmission, packet loss retransmission processing is performed.

[0138] If the time the target video frame spends waiting for decoding in the decoding buffer is greater than or equal to the time for packet loss retransmission, then the target video frame has enough time to be retransmitted, i.e., the target video frame is resent.

[0139] In the above embodiments, the processing method of video frames is determined based on whether packets are lost and the time for packet loss retransmission. This can provide early warning. When the time for the target video frame to wait for decoding in the decoding buffer is less than the time for packet loss retransmission, a suitable IDR frame is selected for decoding to avoid stuttering and delay in the picture.

[0140] In the embodiments of this application, please refer to Figure 9 The diagram illustrates a flowchart of a data transmission method provided in an embodiment of this application, which includes the following steps:

[0141] Step 901: The target terminal sends historical decoding status information and network status information to the cloud platform server.

[0142] Step 902: The cloud platform server receives the historical decoding status information and network status information sent by the target terminal.

[0143] Step 903: The cloud platform server obtains the first decoding complexity information of the first video frame and the second decoding complexity information of the second video frame.

[0144] Step 904: The cloud platform server determines the decoding timeliness of the first video frame based on historical decoding status information, network status information, first decoding complexity information, and second decoding complexity information.

[0145] Step 905: The cloud platform server determines the target encoding method based on the decoding timeliness, encodes the first video frame based on the target encoding method, and sends the encoded first video frame to the target terminal.

[0146] Determine the value of the encoding flag in the RTP packet based on the target encoding method. Send the RTP packet to the target terminal.

[0147] Step 906: The target terminal receives encoding method indication information sent by the cloud platform server based on historical decoding status information and network status information.

[0148] Step 907: The target terminal determines the encoding method of the first video frame according to the encoding method instruction information, and decodes the first video frame sent by the cloud platform server based on the encoding method.

[0149] In one embodiment, the data transmission method of this application flows as follows: Figure 10 As shown, the screen captures and encodes the i-th frame of data, calculates the decoding timeliness of the i-th frame, and determines whether the decoding timeliness exceeds a preset frame rendering threshold. If it does, the IDR encoder is started, IDR encoding is performed, and then the IDR frame is sent. If the decoding timeliness does not exceed the preset frame rendering threshold, the IDR encoder is stopped, video encoding is performed, and then the video frame is sent.

[0150] The target terminal receives data, determines if packet loss exists, and if not, decodes it according to the encoding method. It then checks if the decoding is normal. If decoding is normal, the target terminal sends status information. If decoding is abnormal, it determines the nearest IDR frame for decoding and then sends status information. If packet loss exists and the waiting time for decoding is greater than or equal to the packet loss retransmission time, packet retransmission is performed. If the waiting time for decoding is less than the packet loss retransmission time, the nearest IDR frame is determined for decoding.

[0151] In one embodiment, to facilitate the reader's understanding of the technical solution provided in the embodiments of this application, the data transmission method of this application is applied to... Figure 1 For example, please refer to the following application environment. Figure 11 .

[0152] 1. For the decoding and transmission of the i-th frame of data, the target terminal uses normal video decoding of either the i-th or p-th frame, with a decoding complexity of 1.295017495.

[0153] 1.1 The cloud platform server video encoder performs video encoding normally and sends the encoded video stream to the target terminal. The IDR encoding flag bit in the PT bit of the RTP packet is 0.

[0154] 1.2 After the target terminal receives the RTP packet of the video stream, it checks that the encoding flag is 0 and performs normal decoding using video decoding. The video frame can be an I-frame or a P-frame.

[0155] 2. Data transmission process of the (i+1)th frame.

[0156] 2.1 The target terminal decoder sends the frame decoding time (12ms), packet loss rate (0%), RTT (10ms), and preset frame rendering threshold of the i-th frame to the cloud platform server.

[0157] The calculation of the preset frame rendering threshold is shown below, with the video having a fixed frame rate of 25fps.

[0158]

[0159] 2.2 The decoding complexity of the decoder for the cloud platform server's decoding analysis and early warning receiver is:

[0160]

[0161] 2.3. Combining the frame decoding time, packet loss rate, and RTT of the i-th frame, the predicted decoding time of the (i+1)-th frame is:

[0162]

[0163] As can be seen, the decoding timeliness of the (i+1)th frame is less than the preset frame rendering threshold of 40ms, therefore the prediction decoding is timely, and the (i+1)th frame uses normal video encoding.

[0164] 2.4. Use a video encoder to encode the video and send the (i+1)th frame to the target terminal for decoding.

[0165] 2.5 The target terminal receives video frame data, checks the encoding flag bit, and determines that the (i+1)th frame uses video decoding.

[0166] 3. Data transmission process of the (i+2)th frame.

[0167] 3.1 The target terminal decoder sends the decoding time (12ms), packet loss rate (6%), RTT (15ms), and preset frame rendering threshold of frame i+1 to the cloud platform server. The preset frame rendering threshold is shown below.

[0168]

[0169] 3.2 The decoding complexity of the cloud platform server decoding analysis early warning receiver decoder is shown below.

[0170]

[0171] 3.3 Combining the frame decoding time, packet loss rate, and RTT of the (i+1)th frame, the decoding time of the (i+2)th frame is predicted as follows.

[0172]

[0173] It can be seen that the decoding time of the (i+2)th frame is greater than the preset frame rendering threshold of 40ms. Therefore, the (i+2)th frame is encoded using IDR encoding.

[0174] 3.4 Start the IDR encoder. The i+2th frame is encoded using the IDR encoding method.

[0175] 3.5 Set the encoding flag of the (i+2)th frame of video stream data to 1 and send it to the target terminal.

[0176] 3.6 The target terminal receives video frame data, checks the encoding flag bit, and determines that the (i+2)th frame is decoded using IDR.

[0177] 4. Data transmission process of the (i+3)th frame.

[0178] 4.1 The target terminal decoder sends the decoding time (20ms), packet loss rate (2%), RTT (12ms), and preset frame rendering threshold of the (i+2)th frame to the cloud platform server. The preset frame rendering threshold is shown below.

[0179]

[0180] 4.2 The decoding complexity of the cloud platform decoding analysis early warning receiver decoder is shown below.

[0181]

[0182] 4.3. Combining the frame decoding time, packet loss rate, and RTT of the (i+2)th frame, the decoding time of the (i+3)th frame is predicted as follows.

[0183]

[0184] As can be seen, the decoding time of the (i+3)th frame is less than the preset frame rendering threshold of 40ms, so video encoding is used for encoding.

[0185] 4.4. Turn off the IDR encoder, and encode the (i+3)th frame using video encoding.

[0186] 4.5 The encoding flag of the (i+3)th video frame data is set to 0 and sent to the target terminal.

[0187] 4.6 The cloud terminal receives the (i+3)th video frame data, checks the encoding flag, and determines that normal video decoding is used.

[0188] 5. Continue to perform data transmission and encoding / decoding for cloud services according to the above process.

[0189] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0190] Based on the same inventive concept, this application also provides a data transmission apparatus for implementing the data transmission method described above. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, specific limitations in one or more data transmission apparatus embodiments provided below can be found in the limitations of the data transmission method described above, and will not be repeated here.

[0191] In one embodiment, such as Figure 12 As shown, a data transmission device 1200 is provided, including: a status information receiving module 1201, an acquisition module 1202, a decoding and timeliness determination module 1203, and a data sending module 1204, wherein:

[0192] The status information receiving module 1201 is used to receive historical decoding status information and network status information sent by the target terminal. The historical decoding status information is used to indicate the status parameters of the target terminal for decoding the second video frame before the first video frame to be sent.

[0193] The acquisition module 1202 is used to acquire the first decoding complexity information of the first video frame and the second decoding complexity information of the second video frame;

[0194] The decoding timeliness determination module 1203 is used to determine the decoding timeliness of the first video frame based on historical decoding status information, network status information, first decoding complexity information, and second decoding complexity information.

[0195] The data transmission module 1204 is used to determine the target encoding method based on the decoding timeliness, encode the first video frame based on the target encoding method, and send the encoded first video frame to the target terminal.

[0196] In one embodiment, the historical decoding status information includes frame decoding time, and the network status information includes packet loss rate and round-trip delay. The decoding timeliness determination module 1203 is specifically used to calculate the ratio of the decoding complexity value indicated by the first decoding complexity information and the decoding complexity value indicated by the second decoding complexity information, and to calculate the first product of the frame decoding time and the ratio; calculate the sum of the packet loss rate of the second video frame and a preset constant, and to calculate the second product of the round-trip delay and the sum; and determine the decoding timeliness based on the sum of the first product and the second product.

[0197] In one embodiment, the first decoding complexity information is determined based on the total number of macroblock groups in the first video frame and the sum of squared residual errors of the P-frames; the second decoding complexity information is determined based on the total number of macroblock groups in the second video frame and the sum of squared residual errors of the P-frames.

[0198] In one embodiment, the data sending module 1204 is specifically used to determine the target encoding method as a video encoding method if the decoding timeliness is less than or equal to a preset frame rendering threshold; and to determine the target encoding method as an IDR encoding method if the decoding timeliness is greater than the preset frame rendering threshold.

[0199] In one embodiment, the preset frame rendering threshold is determined based on the difference between the presentation timestamp of the first video frame and the presentation timestamp of the second video frame and a preset time base.

[0200] In one embodiment, the status information receiving module 1201 is specifically used to receive RR messages, which carry historical decoding status information and network status information.

[0201] In one embodiment, the RR message also carries a preset frame rendering threshold.

[0202] In one embodiment, the system further includes an encoding flag determination module, which determines the value of the encoding flag in the RTP packet according to the target encoding method and sends the RTP packet to the target terminal.

[0203] Each module in the aforementioned data transmission device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0204] In one embodiment, such as Figure 13 As shown, a data transmission device 1300 is also provided, including: a status information sending module 1301, an indication information receiving module 1302, and an encoding method determining module 1303, wherein:

[0205] The status information sending module 1301 is used to send historical decoding status information and network status information to the cloud platform server. The historical decoding status information is used to indicate the status parameters of the target terminal for decoding the second video frame before the first video frame to be sent by the cloud platform server.

[0206] The instruction information receiving module 1302 is used to receive encoding method instruction information sent by the cloud platform server based on historical decoding status information and network status information;

[0207] The encoding method determination module 1303 is used to determine the encoding method of the first video frame according to the encoding method indication information, and to decode the first video frame sent by the cloud platform server based on the encoding method.

[0208] In one embodiment, the indication information receiving module 1302 is specifically used to receive RTP packets sent by the cloud platform server based on historical decoding status information and network status information. The RTP packets include encoding flag bits; the value of the encoding flag bits is used as encoding mode indication information.

[0209] In one embodiment, the encoding method determination module 1303 is specifically used to perform video decoding on the first video frame if the encoding method indication information determines that the encoding method of the first video frame is video encoding; and to perform IDR decoding on the first video frame if the encoding method indication information determines that the encoding method of the first video frame is IDR encoding.

[0210] In one embodiment, a detection module is further included, which is used to detect whether the time the target video frame to be decoded waits for decoding in the decoding buffer is less than the time for packet loss retransmission if packet loss exists; if the time the target video frame waits for decoding in the decoding buffer is less than the time for packet loss retransmission, a third video frame whose nearest encoding method is determined to be IDR encoding is used; and the target video frame is replaced by the third video frame for decoding.

[0211] In one embodiment, a retransmission module is also included, which is used to perform packet loss retransmission processing if the time for the target video frame to wait for decoding in the decoding buffer is greater than or equal to the packet loss retransmission time.

[0212] Each module in the aforementioned data transmission device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0213] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 14 As shown, this computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores decoding status information and network status information. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements a data transmission method.

[0214] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 15As shown, the computer device includes a processor, memory, input / output interface, communication interface, display unit, and input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interface. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The input / output interface is used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a data transmission method. The display unit is used to form a visually visible image and can be a display screen, projection device, or virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0215] Those skilled in the art will understand that Figure 15 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0216] In one embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to perform the following steps: receiving historical decoding status information and network status information sent by a target terminal, wherein the historical decoding status information is used to indicate the status parameters of the target terminal for decoding a second video frame preceding the first video frame to be sent; acquiring first decoding complexity information of the first video frame and second decoding complexity information of the second video frame; determining the decoding timeliness of the first video frame based on the historical decoding status information, network status information, first decoding complexity information, and second decoding complexity information; determining a target encoding method based on the decoding timeliness, encoding the first video frame based on the target encoding method, and sending the encoded first video frame to the target terminal.

[0217] In one embodiment, the historical decoding status information includes frame decoding time, and the network status information includes packet loss rate and round-trip time. When the processor executes the computer program, it further implements the following steps: calculating the ratio of the decoding complexity value indicated by the first decoding complexity information and the decoding complexity value indicated by the second decoding complexity information, and calculating a first product of the frame decoding time and the ratio; calculating the sum of the packet loss rate of the second video frame and a preset constant, and calculating a second product of the round-trip time and the sum; and determining the decoding time based on the sum of the first product and the second product.

[0218] In one embodiment, the first decoding complexity information is determined based on the total number of macroblock groups in the first video frame and the sum of squared residual errors of the P-frames; the second decoding complexity information is determined based on the total number of macroblock groups in the second video frame and the sum of squared residual errors of the P-frames.

[0219] In one embodiment, when the processor executes the computer program, it further performs the following steps: if the decoding timeliness is less than or equal to a preset frame rendering threshold, the target encoding method is determined to be a video encoding method; if the decoding timeliness is greater than the preset frame rendering threshold, the target encoding method is determined to be an IDR encoding method.

[0220] In one embodiment, the preset frame rendering threshold is determined based on the difference between the presentation timestamp of the first video frame and the presentation timestamp of the second video frame and a preset time base.

[0221] In one embodiment, when the processor executes the computer program, it also performs the following steps: receiving RR messages, which carry historical decoding status information and network status information.

[0222] In one embodiment, the RR message also carries a preset frame rendering threshold.

[0223] In one embodiment, when the processor executes the computer program, it further performs the following steps: determining the value of the encoding flag bit in the RTP packet according to the target encoding method; and sending the RTP packet to the target terminal.

[0224] In one embodiment, when the processor executes the computer program, it further performs the following steps: sending historical decoding status information and network status information to the cloud platform server, wherein the historical decoding status information is used to indicate the status parameters of the target terminal for decoding the second video frame before the first video frame to be sent by the cloud platform server; receiving encoding method indication information sent by the cloud platform server based on the historical decoding status information and network status information; determining the encoding method of the first video frame according to the encoding method indication information, and decoding the first video frame sent by the cloud platform server based on the encoding method.

[0225] In one embodiment, when the processor executes the computer program, it further performs the following steps: receiving an RTP packet sent by the cloud platform server based on historical decoding status information and network status information, wherein the RTP packet includes an encoding flag; and using the value of the encoding flag as encoding mode indication information.

[0226] In one embodiment, when the processor executes the computer program, it further performs the following steps: if the encoding mode indication information determines that the encoding mode of the first video frame is a video encoding mode, then the first video frame is video decoded; if the encoding mode indication information determines that the encoding mode of the first video frame is an IDR encoding mode, then the first video frame is IDR decoded.

[0227] In one embodiment, when the processor executes the computer program, it further implements the following steps: if packet loss exists, it detects whether the time the target video frame to be decoded waits for decoding in the decoding buffer is less than the time for packet loss retransmission; if the time the target video frame waits for decoding in the decoding buffer is less than the time for packet loss retransmission, it determines the third video frame whose nearest encoding method is IDR encoding; and it replaces the target video frame with the third video frame for decoding.

[0228] In one embodiment, when the processor executes the computer program, it further performs the following steps: if the time for the target video frame to wait for decoding in the decoding buffer is greater than or equal to the time for packet loss retransmission, packet loss retransmission processing is performed.

[0229] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, it performs the following steps: receiving historical decoding status information and network status information sent by a target terminal, wherein the historical decoding status information is used to indicate the status parameters of the target terminal for decoding a second video frame preceding the first video frame to be sent; obtaining first decoding complexity information of the first video frame and second decoding complexity information of the second video frame; determining the decoding timeliness of the first video frame based on the historical decoding status information, network status information, first decoding complexity information, and second decoding complexity information; determining a target encoding method based on the decoding timeliness, encoding the first video frame based on the target encoding method, and sending the encoded first video frame to the target terminal.

[0230] In one embodiment, the historical decoding status information includes frame decoding time, and the network status information includes packet loss rate and round-trip time. When the computer program is executed by the processor, it further implements the following steps: calculating the ratio of the decoding complexity value indicated by the first decoding complexity information and the decoding complexity value indicated by the second decoding complexity information, and calculating a first product of the frame decoding time and the ratio; calculating the sum of the packet loss rate of the second video frame and a preset constant, and calculating a second product of the round-trip time and the sum; and determining the decoding time based on the sum of the first product and the second product.

[0231] In one embodiment, the first decoding complexity information is determined based on the total number of macroblock groups in the first video frame and the sum of squared residual errors of the P-frames; the second decoding complexity information is determined based on the total number of macroblock groups in the second video frame and the sum of squared residual errors of the P-frames.

[0232] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: if the decoding timeliness is less than or equal to a preset frame rendering threshold, the target encoding method is determined to be a video encoding method; if the decoding timeliness is greater than the preset frame rendering threshold, the target encoding method is determined to be an IDR encoding method.

[0233] In one embodiment, the preset frame rendering threshold is determined based on the difference between the presentation timestamp of the first video frame and the presentation timestamp of the second video frame and a preset time base.

[0234] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: receiving RR messages, which carry historical decoding status information and network status information.

[0235] In one embodiment, the RR message also carries a preset frame rendering threshold.

[0236] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: determining the value of the encoding flag bit in the RTP packet according to the target encoding method; and sending the RTP packet to the target terminal.

[0237] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: sending historical decoding status information and network status information to the cloud platform server, wherein the historical decoding status information is used to indicate the status parameters of the target terminal for decoding the second video frame before the first video frame to be sent by the cloud platform server; receiving encoding method indication information sent by the cloud platform server based on the historical decoding status information and network status information; determining the encoding method of the first video frame according to the encoding method indication information, and decoding the first video frame sent by the cloud platform server based on the encoding method.

[0238] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: receiving an RTP packet sent by the cloud platform server based on historical decoding status information and network status information, wherein the RTP packet includes an encoding flag; and using the value of the encoding flag as encoding mode indication information.

[0239] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: if the encoding mode indication information determines that the encoding mode of the first video frame is a video encoding mode, then the first video frame is video decoded; if the encoding mode indication information determines that the encoding mode of the first video frame is an IDR encoding mode, then the first video frame is IDR decoded.

[0240] In one embodiment, when the computer program is executed by the processor, it further implements the following steps: if packet loss exists, it detects whether the time the target video frame to be decoded waits for decoding in the decoding buffer is less than the time for packet loss retransmission; if the time the target video frame waits for decoding in the decoding buffer is less than the time for packet loss retransmission, it determines the third video frame whose nearest encoding method is IDR encoding; and it replaces the target video frame with the third video frame for decoding.

[0241] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: if the time for the target video frame to wait for decoding in the decoding buffer is greater than or equal to the time for packet loss retransmission, packet loss retransmission processing is performed.

[0242] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, performs the following steps: receiving historical decoding status information and network status information sent by a target terminal, wherein the historical decoding status information is used to indicate the status parameters of the target terminal for decoding a second video frame preceding the first video frame to be sent; acquiring first decoding complexity information of the first video frame and second decoding complexity information of the second video frame; determining the decoding timeliness of the first video frame based on the historical decoding status information, network status information, first decoding complexity information, and second decoding complexity information; determining a target encoding method based on the decoding timeliness, encoding the first video frame based on the target encoding method, and sending the encoded first video frame to the target terminal.

[0243] In one embodiment, the historical decoding status information includes frame decoding time, and the network status information includes packet loss rate and round-trip time. When the computer program is executed by the processor, it further implements the following steps: calculating the ratio of the decoding complexity value indicated by the first decoding complexity information and the decoding complexity value indicated by the second decoding complexity information, and calculating a first product of the frame decoding time and the ratio; calculating the sum of the packet loss rate of the second video frame and a preset constant, and calculating a second product of the round-trip time and the sum; and determining the decoding time based on the sum of the first product and the second product.

[0244] In one embodiment, the first decoding complexity information is determined based on the total number of macroblock groups in the first video frame and the sum of squared residual errors of the P-frames; the second decoding complexity information is determined based on the total number of macroblock groups in the second video frame and the sum of squared residual errors of the P-frames.

[0245] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: if the decoding timeliness is less than or equal to a preset frame rendering threshold, the target encoding method is determined to be a video encoding method; if the decoding timeliness is greater than the preset frame rendering threshold, the target encoding method is determined to be an IDR encoding method.

[0246] In one embodiment, the preset frame rendering threshold is determined based on the difference between the presentation timestamp of the first video frame and the presentation timestamp of the second video frame and a preset time base.

[0247] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: receiving RR messages, which carry historical decoding status information and network status information.

[0248] In one embodiment, the RR message also carries a preset frame rendering threshold.

[0249] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: determining the value of the encoding flag bit in the RTP packet according to the target encoding method; and sending the RTP packet to the target terminal.

[0250] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: sending historical decoding status information and network status information to the cloud platform server, wherein the historical decoding status information is used to indicate the status parameters of the target terminal for decoding the second video frame before the first video frame to be sent by the cloud platform server; receiving encoding method indication information sent by the cloud platform server based on the historical decoding status information and network status information; determining the encoding method of the first video frame according to the encoding method indication information, and decoding the first video frame sent by the cloud platform server based on the encoding method.

[0251] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: receiving an RTP packet sent by the cloud platform server based on historical decoding status information and network status information, wherein the RTP packet includes an encoding flag; and using the value of the encoding flag as encoding mode indication information.

[0252] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: if the encoding mode indication information determines that the encoding mode of the first video frame is a video encoding mode, then the first video frame is video decoded; if the encoding mode indication information determines that the encoding mode of the first video frame is an IDR encoding mode, then the first video frame is IDR decoded.

[0253] In one embodiment, when the computer program is executed by the processor, it further implements the following steps: if packet loss exists, it detects whether the time the target video frame to be decoded waits for decoding in the decoding buffer is less than the time for packet loss retransmission; if the time the target video frame waits for decoding in the decoding buffer is less than the time for packet loss retransmission, it determines the third video frame whose nearest encoding method is IDR encoding; and it replaces the target video frame with the third video frame for decoding.

[0254] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: if the time for the target video frame to wait for decoding in the decoding buffer is greater than or equal to the time for packet loss retransmission, packet loss retransmission processing is performed.

[0255] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data shall comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0256] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0257] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0258] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A data transmission method, characterized in that, The method includes: The system receives historical decoding status information and network status information sent by the target terminal. The historical decoding status information is used to indicate the status parameters of the target terminal for decoding the second video frame before the first video frame to be sent. Obtain the first decoding complexity information of the first video frame and the second decoding complexity information of the second video frame; The decoding timeliness of the first video frame is determined based on the historical decoding status information, the network status information, the first decoding complexity information, and the second decoding complexity information. The target encoding method is determined based on the decoding timeliness, and the first video frame is encoded based on the target encoding method. The encoded first video frame is then sent to the target terminal. The historical decoding status information includes frame decoding time, and the network status information includes packet loss rate and round-trip latency. Determining the decoding timeliness of the first video frame based on the historical decoding status information, the network status information, the first decoding complexity information, and the second decoding complexity information includes: Calculate the ratio of the decoding complexity value indicated by the first decoding complexity information to the decoding complexity value indicated by the second decoding complexity information, and calculate the first product of the frame decoding time and the ratio; Calculate the sum of the packet loss rate of the second video frame and a preset constant, and calculate the second product of the round-trip delay and the sum; The decoding timeliness is determined based on the sum of the first product and the second product.

2. The method according to claim 1, characterized in that, The first decoding complexity information is determined based on the total number of macroblock groups in the first video frame and the sum of squared residual errors of P frames; The second decoding complexity information is determined based on the total number of macroblock groups in the second video frame and the sum of squared residual errors of the P-frames.

3. The method according to any one of claims 1 to 2, characterized in that, The step of determining the target encoding method based on the decoding timeliness includes: If the decoding timeliness is less than or equal to the preset frame rendering threshold, then the target encoding method is determined to be the P-frame encoding method; If the decoding timeliness is greater than the preset frame rendering threshold, then the target encoding method is determined to be IDR encoding method.

4. The method according to claim 3, characterized in that, The preset frame rendering threshold is determined based on the difference between the demonstration timestamp of the first video frame and the demonstration timestamp of the second video frame and a preset time base.

5. The method according to claim 3, characterized in that, The historical decoding status information and network status information received from the target terminal include: Receive RR messages, which carry the historical decoding status information and the network status information.

6. The method according to claim 5, characterized in that, The RR message also carries the preset frame rendering threshold.

7. The method according to any one of claims 1 to 2, characterized in that, The method further includes: Determine the value of the encoding flag bit in the RTP packet according to the target encoding method; The RTP packet is sent to the target terminal.

8. A data transmission method, characterized in that, The method includes: Send historical decoding status information and network status information to the cloud platform server. The historical decoding status information is used to indicate the status parameters of the target terminal for decoding the second video frame before the first video frame to be sent by the cloud platform server. The system receives encoding method indication information sent by the cloud platform server based on the historical decoding status information and the network status information. The encoding method indication information is determined according to a target encoding method, which is determined by the cloud platform server based on the decoding timeliness of the first video frame. The decoding timeliness is determined by the cloud platform server based on the historical decoding status information, the network status information, the first decoding complexity information of the first video frame, and the second decoding complexity information of the second video frame. The historical decoding status information includes frame decoding time, and the network status information includes packet loss rate and round-trip latency. The cloud platform server calculates the ratio of the decoding complexity value indicated by the first decoding complexity information to the decoding complexity value indicated by the second decoding complexity information, and calculates a first product of the frame decoding time and the ratio. It also calculates the sum of the packet loss rate of the second video frame and a preset constant, and calculates a second product of the round-trip latency and the sum. The decoding timeliness is determined based on the sum of the first product and the second product. The encoding method of the first video frame is determined according to the encoding method indication information, and the first video frame sent by the cloud platform server is decoded based on the encoding method.

9. The method according to claim 8, characterized in that, The receipt of encoding method indication information sent by the cloud platform server based on the historical decoding status information and the network status information includes: Receives an RTP packet sent by the cloud platform server based on the historical decoding status information and the network status information, wherein the RTP packet includes an encoding flag bit; The value of the encoding flag is used as the encoding method indication information.

10. The method according to claim 8, characterized in that, The step of determining the encoding method of the first video frame according to the encoding method indication information, and decoding the first video frame sent by the cloud platform server based on the encoding method, includes: If the encoding method indication information determines that the encoding method of the first video frame is P-frame encoding, then the first video frame is P-frame decoded. If the encoding method indication information determines that the encoding method of the first video frame is IDR encoding, then IDR decoding is performed on the first video frame.

11. The method according to claim 10, characterized in that, The method further includes: If packet loss exists, check whether the time the target video frame to be decoded waits for decoding in the decoding buffer is less than the time for packet retransmission. If the time the target video frame waits for decoding in the decoding buffer is less than the time for packet loss retransmission, then the third video frame whose nearest encoding method is the IDR encoding method is determined to be the target video frame. Decoding is performed by replacing the target video frame with the third video frame.

12. The method according to claim 11, characterized in that, The method further includes: If the time the target video frame spends waiting for decoding in the decoding buffer is greater than or equal to the time for packet loss retransmission, packet loss retransmission processing is performed.

13. A data transmission device, characterized in that, The device includes: The status information receiving module is used to receive historical decoding status information and network status information sent by the target terminal. The historical decoding status information is used to indicate the status parameters of the target terminal for decoding the second video frame before the first video frame to be sent. The acquisition module is used to acquire the first decoding complexity information of the first video frame and the second decoding complexity information of the second video frame; The decoding timeliness determination module is used to determine the decoding timeliness of the first video frame based on the historical decoding status information, the network status information, the first decoding complexity information, and the second decoding complexity information. The data transmission module is used to determine the target encoding method according to the decoding timeliness, encode the first video frame based on the target encoding method, and send the encoded first video frame to the target terminal; The historical decoding status information includes frame decoding time, and the network status information includes packet loss rate and round-trip time. The decoding timeliness determination module is specifically used to calculate the ratio of the decoding complexity value indicated by the first decoding complexity information to the decoding complexity value indicated by the second decoding complexity information, and to calculate the first product of the frame decoding time and the ratio; calculate the sum of the packet loss rate of the second video frame and a preset constant, and to calculate the second product of the round-trip time and the sum; and determine the decoding timeliness based on the sum of the first product and the second product.

14. A data transmission device, characterized in that, The device includes: The status information sending module is used to send historical decoding status information and network status information to the cloud platform server. The historical decoding status information is used to indicate the status parameters of the target terminal for decoding the second video frame before the first video frame to be sent by the cloud platform server. An instruction information receiving module is used to receive encoding method instruction information sent by the cloud platform server based on the historical decoding status information and the network status information. The encoding method instruction information is determined according to a target encoding method, which is determined by the cloud platform server based on the decoding timeliness of the first video frame. The decoding timeliness is determined by the cloud platform server based on the historical decoding status information, the network status information, the first decoding complexity information of the first video frame, and the second decoding complexity information of the second video frame. The historical decoding status information includes frame decoding time, and the network status information includes packet loss rate and round-trip time. The cloud platform server calculates the ratio of the decoding complexity value indicated by the first decoding complexity information to the decoding complexity value indicated by the second decoding complexity information, and calculates a first product of the frame decoding time and the ratio. It also calculates the sum of the packet loss rate of the second video frame and a preset constant, and calculates a second product of the round-trip time and the sum. The decoding timeliness is determined based on the sum of the first product and the second product. The encoding method determination module is used to determine the encoding method of the first video frame according to the encoding method indication information, and to decode the first video frame sent by the cloud platform server based on the encoding method.

15. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7 or 8 to 12.

16. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7 or 8 to 12.

17. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 7 or 8 to 12.

Images