Data transmission method and apparatus, and device, medium and program product

Abstract

The present application relates to a data transmission method and apparatus, and a device, a medium and a program product. The method comprises: first, receiving historical decoding state information and network state information which are sent by a target terminal, wherein the historical decoding state information is used for indicating a state parameter for the target terminal to decode a second video frame before a first video frame to be sent; then, acquiring first decoding complexity information of the first video frame and second decoding complexity information of the second video frame (S302); determining a decoding timeliness of the first video frame on the basis of the historical decoding state information, the network state information, the first decoding complexity information and the second decoding complexity information (S303); and finally, determining a target encoding mode on the basis of the decoding timeliness, encoding the first video frame on the basis of the target encoding mode, and sending the encoded first video frame to the target terminal (S304). By means of using the method, the data transmission efficiency can be improved.

Description

Data transmission method, device, equipment, medium and program product

[0001] Related applications

[0002] This application claims priority to Chinese patent application number 202310985259.5, filed on August 7, 2023, entitled “Data Transmission Method, Apparatus, Equipment, Medium and Program Product,” the entire text of which is hereby incorporated by reference. Technical Field

[0003] The present application relates to the field of data transmission technology, and in particular to a data transmission method, apparatus, device, medium and program product. Background Art

[0004] With the widespread use of cloud terminals, users can now access a variety of cloud services, including cloud office, cloud gaming, and cloud live streaming. These services are low-latency, and network jitter, packet loss, or delays can affect the smooth transmission of cloud service video streams, causing video freezes.

[0005] Currently, when packet loss or delay occurs, it is necessary to wait for the retransmitted data packet to be sent to the cloud terminal and then processed. This transmission method is not efficient.

[0006] Summary of the Invention

[0007] Based on this, it is necessary to provide a data transmission method, device, equipment, medium and program product with higher transmission efficiency to address the above technical problems.

[0008] In a first aspect, the present application provides a data transmission method. The method includes: receiving historical decoding state information and network state information sent by a target terminal, the historical decoding state information being used to indicate state parameters for the target terminal to decode a second video frame preceding a first video frame to be sent; obtaining first decoding complexity information of the first video frame and second decoding complexity information of the second video frame; determining decoding timeliness of the first video frame based on the historical decoding state information, the network state information, the first decoding complexity information, and the second decoding complexity information; determining a target encoding mode based on the decoding timeliness, encoding the first video frame based on the target encoding mode, and transmitting the encoded first video frame to the target terminal.

[0009] In one embodiment, the historical decoding status information includes the frame decoding time of the second video frame, the network status information includes the packet loss rate and the round-trip delay, and the decoding timeliness of the first video frame is determined based on the historical decoding status information, the network status information, the first decoding complexity information, and the second decoding complexity information, including: calculating the ratio of the decoding complexity value indicated by the first decoding complexity information to the decoding complexity value indicated by the second decoding complexity information, and calculating the first product of the frame decoding time of the second video frame and the ratio; calculating the sum of the packet loss rate of the second video frame and a preset constant, and calculating the second product of the round-trip delay of the second video frame and the sum; determining the decoding timeliness of the first video frame based on the sum of the first product and the second product.

[0010] In one embodiment, the first decoding complexity information is determined based on the total number of macroblock groups in the first video frame and the sum of residual squared errors of the P frames; the second decoding complexity information is determined based on the total number of macroblock groups in the second video frame and the sum of residual squared errors of the P frames.

[0011] In one embodiment, the target encoding mode is determined based on the decoding timeliness, including: if the decoding timeliness is less than or equal to a preset frame rendering threshold, the target encoding mode is determined to be a video encoding mode; if the decoding timeliness is greater than the preset frame rendering threshold, the target encoding mode is determined to be an IDR encoding mode.

[0012] In one embodiment, the preset frame rendering threshold is determined based on a difference between a presentation timestamp of the first video frame and a presentation timestamp of the second video frame and a preset time base.

[0013] In one embodiment, receiving historical decoding status information and network status information sent by the target terminal includes: receiving an RR message, where the RR message carries the historical decoding status information and network status information.

[0014] In one of the embodiments, the RR message also carries a preset frame rendering threshold.

[0015] In one embodiment, the method further includes: determining a value of an encoding flag in the RTP packet according to a target encoding mode; and sending the RTP packet to a target terminal.

[0016] In a second aspect, the present application provides a data transmission method. The method includes: sending historical decoding state information and network state information to a cloud platform server, where the historical decoding state information is used to indicate state parameters for a target terminal to decode a second video frame preceding a first video frame sent by the cloud platform server; receiving encoding mode indication information sent by the cloud platform server based on the historical decoding state information and the network state information; determining an encoding mode for the first video frame according to the encoding mode indication information, and decoding the first video frame sent by the cloud platform server based on the encoding mode.

[0017] In one embodiment, receiving encoding mode indication information sent by the cloud platform server based on historical decoding status information and network status information includes: receiving an RTP packet sent by the cloud platform server based on historical decoding status information and network status information, the RTP packet including an encoding flag; and using the value of the encoding flag as encoding mode indication information.

[0018] In one embodiment, the encoding mode of the first video frame is determined according to the encoding mode indication information, and the first video frame sent by the cloud platform server is decoded based on the encoding mode, including: if the encoding mode indication information determines that the encoding mode of the first video frame is a video encoding mode, then the first video frame is video decoded; if the encoding mode indication information determines that the encoding mode of the first video frame is an IDR encoding mode, then the first video frame is IDR decoded.

[0019] In one embodiment, the method further includes: if packet loss currently occurs, detecting whether the time the target video frame to be decoded waits for decoding in the decoding buffer is less than the time for packet loss retransmission; if the time the target video frame waits for decoding in the decoding buffer is less than the time for packet loss retransmission, determining that the video frame closest to the target video frame and encoded in an IDR encoding mode is a third video frame; and using the third video frame to replace the target video frame for decoding.

[0020] In one embodiment, the method further includes: if the time the target video frame waits for decoding in the decoding buffer is greater than or equal to the packet loss retransmission time, performing packet loss retransmission processing.

[0021] In a third aspect, the present application further provides a data transmission device. The device includes:

[0022] a status information receiving module, configured to receive historical decoding status information and network status information sent by a target terminal, wherein the historical decoding status information is used to indicate a status parameter of the target terminal for decoding a second video frame preceding a first video frame to be sent;

[0023] An acquisition module, configured to acquire first decoding complexity information of a first video frame and second decoding complexity information of a second video frame;

[0024] A decoding timeliness determination module, configured to determine the decoding timeliness of the first video frame based on historical decoding state information, network state information, first decoding complexity information, and second decoding complexity information;

[0025] The data sending module is used to determine a target encoding method according to decoding timeliness, encode the first video frame based on the target encoding method, and send the encoded first video frame to a target terminal.

[0026] In one embodiment, the historical decoding status information includes the frame decoding time of the second video frame, the network status information includes the packet loss rate and the round-trip delay, and the decoding timeliness determination module is specifically used to calculate the ratio of the decoding complexity value indicated by the first decoding complexity information and the decoding complexity value indicated by the second decoding complexity information, and calculate the first product of the frame decoding time of the second video frame and the ratio; calculate the sum of the packet loss rate of the second video frame and a preset constant, and calculate the second product of the round-trip delay of the second video frame and the sum; determine the decoding timeliness of the first video frame based on the sum of the first product and the second product.

[0027] In one embodiment, the first decoding complexity information is determined based on the total number of macroblock groups in the first video frame and the sum of residual squared errors of the P frames; the second decoding complexity information is determined based on the total number of macroblock groups in the second video frame and the sum of residual squared errors of the P frames.

[0028] In one embodiment, the data sending module is specifically used to determine that the target encoding mode is a video encoding mode if the decoding timeliness is less than or equal to a preset frame rendering threshold; if the decoding timeliness is greater than the preset frame rendering threshold, determine that the target encoding mode is an IDR encoding mode.

[0029] In one embodiment, the preset frame rendering threshold is determined based on a difference between a presentation timestamp of the first video frame and a presentation timestamp of the second video frame and a preset time base.

[0030] In one embodiment, the status information receiving module is specifically configured to receive RR messages, where the RR messages carry historical decoding status information and network status information.

[0031] In one of the embodiments, the RR message also carries a preset frame rendering threshold.

[0032] In one embodiment, the method further includes a coding flag determination module for determining the value of the coding flag in the RTP packet according to the target coding mode; and sending the RTP packet to the target terminal.

[0033] In a fourth aspect, the present application further provides a data transmission device. The device includes:

[0034] a status information sending module, configured to send historical decoding status information and network status information to the cloud platform server, wherein the historical decoding status information is used to indicate status parameters for decoding a second video frame preceding a first video frame sent by the cloud platform server by the target terminal;

[0035] An indication information receiving module is used to receive encoding mode indication information sent by the cloud platform server based on historical decoding status information and network status information;

[0036] The encoding mode determination module is used to determine the encoding mode of the first video frame according to the encoding mode indication information, and decode the first video frame sent by the cloud platform server based on the encoding mode.

[0037] In one embodiment, the indication information receiving module is specifically used to receive an RTP packet sent by a cloud platform server based on historical decoding status information and network status information, where the RTP packet includes an encoding flag; and the value of the encoding flag is used as encoding mode indication information.

[0038] In one embodiment, the encoding mode determination module is specifically used to perform video decoding on the first video frame if the encoding mode indication information determines that the encoding mode of the first video frame is a video encoding mode; if the encoding mode indication information determines that the encoding mode of the first video frame is an IDR encoding mode, perform IDR decoding on the first video frame.

[0039] In one of the embodiments, a detection module is further included for detecting whether the time the target video frame to be decoded waits for decoding in the decoding buffer is less than the time for packet loss retransmission if packet loss currently exists; if the time the target video frame waits for decoding in the decoding buffer is less than the time for packet loss retransmission, determining that the video frame closest to the target video frame and coded in the IDR coding mode is the third video frame; and using the third video frame to replace the target video frame for decoding.

[0040] In one embodiment, a retransmission module is further included, which is used to perform packet loss retransmission processing if the time the target video frame waits for decoding in the decoding buffer is greater than or equal to the packet loss retransmission time.

[0041] In a fifth aspect, the present application further provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the data transmission method described in any embodiment of the first or second aspect when executing the computer program.

[0042] In a sixth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the data transmission method described in any embodiment of the first or second aspect.

[0043] In a seventh aspect, the present application further provides a computer program product, comprising a computer program that, when executed by a processor, implements the data transmission method described in any one of the first and second aspects.

[0044] The above-mentioned data transmission method, apparatus, device, medium and program product first receives historical decoding status information and network status information sent by the target terminal for indicating the status parameters of the target terminal to decode the second video frame before the first video frame to be sent, then obtains the first decoding complexity information of the first video frame and the second decoding complexity information of the second video frame, determines the decoding timeliness of the first video frame based on the historical decoding status information, the network status information, the first decoding complexity information and the second decoding complexity information, and finally determines the target encoding method based on the decoding timeliness, encodes the first video frame based on the target encoding method, and sends the encoded first video frame to the target terminal. In this way, before sending the first video frame, the decoding timeliness of the first video frame is calculated, and then the target encoding method of the first video frame is determined based on the decoding timeliness of the first video frame, and the first video frame is encoded according to the target encoding method and sent, thereby ensuring that the first video frame can be decoded in time, without causing video screen freezes and delays, and the transmission efficiency is higher. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technology descriptions. Obviously, the drawings described below are only some embodiments of this application. For those of ordinary skill in the art, other drawings can be derived from these drawings without inventive effort.

[0046] FIG1 is a diagram showing an application environment of a data transmission method according to an embodiment of the present application.

[0047] FIG2 is a flow chart of conventional data transmission.

[0048] FIG3 is a flow chart of a data transmission method according to another embodiment of the present application.

[0049] FIG4 is a flow chart of a data transmission method according to another embodiment of the present application.

[0050] FIG5 is a flow chart of a data transmission method according to another embodiment of the present application.

[0051] FIG6 is a flow chart of a data transmission method according to another embodiment of the present application.

[0052] FIG7 is a flow chart of a data transmission method according to another embodiment of the present application.

[0053] FIG8 is a flow chart of a data transmission method according to another embodiment of the present application.

[0054] FIG9 is a flow chart of a data transmission method according to another embodiment of the present application.

[0055] FIG10 is a flowchart of a data transmission method in another embodiment of the present application.

[0056] FIG11 is an interaction diagram of a data transmission method in another embodiment of the present application.

[0057] FIG12 is a structural block diagram of a data transmission device according to an embodiment of the present application.

[0058] FIG13 is a structural block diagram of a data transmission device in another embodiment of the present application.

[0059] FIG14 is a diagram showing the internal structure of a computer device according to an embodiment of the present application.

[0060] FIG15 is a diagram showing the internal structure of a computer device in another embodiment of the present application. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0062] The data transmission method provided in the embodiment of the present application can be applied in the application environment shown in Figure 1. In which, the target terminal 101 is connected to the cloud platform server 102 for communication. The target terminal 101 can be a cloud terminal, such as a cloud phone, a cloud computer, etc. As shown in Figure 1, the cloud platform server 102 includes the following parts:

[0063] (1) Virtual machine CPU / GPU: A virtual cloud phone or cloud computer simulated by an x86 / ARM server provides cloud business services to users.

[0064] (2) Screen capture: After rendering the screen, the screen capture encoder is responsible for capturing the audio and video images of the cloud business.

[0065] (3) Instantaneous Decoder Refresh (IDR) encoder: Real-time encoding of IDR frames in cloud service video streams, triggered when IDR encoding is required.

[0066] (4) Video encoder: Real-time encoding of cloud service video streams, which is performed continuously after the cloud service is started.

[0067] (5) Decoding analysis and warning: Receive the receiver report (RR) information from the target terminal, calculate the decoding timeliness, and determine whether to enable the IDR encoder.

[0068] (6) Network transmission: responsible for transmitting the video stream to the target terminal and notifying the decoding analysis and early warning of RR information.

[0069] The target terminal 101 includes the following components:

[0070] (1) Network transmission: Receive video data sent by the cloud platform server and send RR information to the cloud platform server.

[0071] (2) Decoding Load Calculator: collects statistics on the decoding delay, GPU utilization, and other information of the video decoder.

[0072] (3) Decoding Cache Controller: Controls the video cache of network packets received and passes the data frame by frame to the video decoder for decoding.

[0073] (4) Video decoder: responsible for decoding the video stream of cloud business images.

[0074] (5) Rendering output: Render the image and output it to the user.

[0075] In traditional technology, as shown in Figure 2, for a frame of data, the cloud platform server encodes and sends the data stream to the target terminal. After receiving the data stream, the target terminal determines whether there is packet loss. If there is no packet loss, the target terminal decodes the data stream and determines whether the decoding is successful. If successful, the output is rendered and the next frame of data is transmitted. If packet loss occurs, the packet is retransmitted. If decoding fails, an IDR frame is requested for decoding. However, this may require waiting for the retransmitted data packet or IDR frame to arrive before decoding can proceed. If the decoding buffer is insufficient for video frame data, it may cause image freezes or delays.

[0076] Based on the above-mentioned traditional technology, the data transmission method provided by the embodiment of the present application can be applied to a cloud platform server. First, the historical decoding status information and network status information of the status parameters sent by the target terminal for indicating that the target terminal decodes the second video frame before the first video frame to be sent are received; then, the first decoding complexity information of the first video frame and the second decoding complexity information of the second video frame are obtained, and the decoding timeliness of the first video frame is determined according to the historical decoding status information, the network status information, the first decoding complexity information and the second decoding complexity information; finally, the target encoding method is determined according to the decoding timeliness, and the first video frame is encoded based on the target encoding method, and the encoded first video frame is sent to the target terminal. In this way, before sending the first video frame, the decoding timeliness of the first video frame is calculated, and then the target encoding method of the first video frame is determined according to the decoding timeliness of the first video frame, and the first video frame is encoded according to the target encoding method and then sent, thereby ensuring that the first video frame can be decoded in time, without causing video screen freezes and delays, and the transmission efficiency is higher.

[0077] It should be noted that the beneficial effects or technical problems solved by the embodiments of the present application are not limited to this one, but may also include other implicit or related problems. For details, please refer to the description of the following embodiments.

[0078] The following specific embodiments describe in detail the technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0079] In one embodiment, as shown in FIG3 , a data transmission method is provided, which is described by taking the method applied to the cloud platform server in FIG1 as an example, and includes the following steps S301 to S304 .

[0080] Step S301: Receive historical decoding status information and network status information sent by a target terminal.

[0081] Among them, the historical decoding state information is used to indicate the state parameters of the target terminal for decoding the second video frame before the first video frame to be sent. The historical decoding state information can be the frame decoding time of the second video frame. The frame decoding time is the time taken by the target terminal to read a frame of data from the buffer and the decoder to complete the decoding, which can be obtained by the decoder. The network status information may include the packet loss rate. The packet loss rate is the packet loss rate within a period of time fed back by the RR transmitted by the target terminal through RTP, which can be counted by the network card. The network status information can also include the round-trip delay of the network packet interacting with the target terminal and the cloud platform server.

[0082] Step S302: Obtain first decoding complexity information of the first video frame and second decoding complexity information of the second video frame.

[0083] Specifically, the first decoding complexity information is used to represent the decoding complexity of the first video frame, and the second decoding complexity information is used to represent the decoding complexity of the second video frame. The decoding complexity is estimated and determined by the encoder when encoding the video frame.

[0084] Step S303: Determine the decoding timeliness of the first video frame according to the historical decoding state information, the network state information, the first decoding complexity information, and the second decoding complexity information.

[0085] Specifically, the decoding timeliness of the first video frame can be obtained by performing calculations based on the historical decoding state information, network state information, first decoding complexity information, and second decoding complexity information obtained above.

[0086] Step S304: determining a target encoding mode according to the decoding timeliness, encoding the first video frame based on the target encoding mode, and sending the encoded first video frame to the target terminal.

[0087] Optionally, the cloud platform server can judge the decoding timeliness of the video frame for each frame or at a preset time interval, and determine the target encoding method based on the decoding timeliness. The target encoding method can be instantaneous decoder refresh (IDR) encoding, or video encoding. If the target encoding method is IDR encoding, the first video frame is encoded according to the IDR encoding method; if the target encoding method is video encoding, the first video frame is encoded according to the video encoding method. The encoded first video frame is then sent to the target terminal. Among them, the IDR frame generated by IDR encoding is a type of I frame. The I frame includes the complete data of a frame, and only the data of this frame is needed to complete the decoding. Video encoding generates a p frame. When decoding, it is necessary to superimpose the difference defined by this frame with the previously cached picture to generate the final picture.

[0088] In the above embodiment, first, the target terminal receives historical decoding status information and network status information sent by the target terminal for indicating the status parameters for decoding the second video frame before the first video frame to be sent; then, the first decoding complexity information of the first video frame and the second decoding complexity information of the second video frame are obtained, and the decoding timeliness of the first video frame is determined based on the historical decoding status information, the network status information, the first decoding complexity information, and the second decoding complexity information; finally, the target encoding method is determined based on the decoding timeliness, and the first video frame is encoded based on the target encoding method, and the encoded first video frame is sent to the target terminal. In this way, before sending the first video frame, the decoding timeliness of the first video frame is calculated, and then the target encoding method of the first video frame is determined based on the decoding timeliness of the first video frame, and the first video frame is encoded according to the target encoding method and sent, thereby ensuring that the first video frame can be decoded in time, without causing video screen freezes and delays, and the transmission efficiency is higher.

[0089] In one embodiment, the historical decoding status information includes the frame decoding time, and the network status information includes the packet loss rate and the round-trip delay. The steps of determining the decoding timeliness of the first video frame based on the historical decoding status information, the network status information, the first decoding complexity information and the second decoding complexity information are shown in Figure 4, including steps S401 to S403.

[0090] Step S401: Calculate a ratio of a decoding complexity value indicated by the first decoding complexity information to a decoding complexity value indicated by the second decoding complexity information, and calculate a first product of a frame decoding time of a second video frame and the ratio.

[0091] Optionally, the first decoding complexity information is determined based on the total number of macroblock groups in the first video frame and the sum of residual squared errors of the P frames; the second decoding complexity information is determined based on the total number of macroblock groups in the second video frame and the sum of residual squared errors of the P frames.

[0092] Specifically, taking the first decoding complexity information as an example, the decoding complexity value is estimated by the encoder during frame encoding, using the computational amount generated during macroblock prediction, and is calculated based on the macroblock group allocation amount, that is, the number of macroblock groups. The calculation formula for the decoding complexity value indicated by the first decoding complexity information is shown in the following formula (1):

[0093] Wherein, DC is the decoding complexity value, SSD (SUM of Squared Difference) is the sum of squared errors of the P frame residual, and the number of macroblock groups is the total number of macroblock groups in the first video frame.

[0094] After calculating and obtaining the decoding complexity value indicated by the first decoding complexity information and the decoding complexity value indicated by the second decoding complexity information, a ratio between the two is calculated, and then the ratio is multiplied by the frame decoding time of the second video frame to obtain a first product.

[0095] Step S402: Calculate the sum of the packet loss rate of the second video frame and a preset constant, and calculate a second product of the round-trip delay of the second video frame and the sum.

[0096] Specifically, the preset constant may be 1, and the sum of the packet loss rate of the second video frame and the preset constant is calculated and then multiplied by the round-trip delay of the second video frame to obtain the second product.

[0097] Step S403: Determine the decoding timeliness of the first video frame according to the sum of the first product and the second product.

[0098] Specifically, when the preset constant is 1, the decoding timeliness is calculated as shown in the following formula (2):

[0099] Among them, DT is decoding timeliness, FDT is frame decoding time, DC is decoding complexity, RTT is round-trip delay, that is, the round-trip delay of network packets interacting between the target terminal and the cloud platform server, LR is the packet loss rate, i represents the first video frame, and i-1 represents the second video frame.

[0100] In one embodiment, after the decoding timeliness of the first video frame is determined, a target encoding mode is determined according to the decoding timeliness, including the following two cases.

[0101] In the first case, if the decoding timeliness is less than or equal to a preset frame rendering threshold, the target encoding mode is determined to be a video encoding mode.

[0102] The preset frame rendering threshold is determined based on the difference between the presentation timestamp of the first video frame and the presentation timestamp of the second video frame and the preset time base. The frame rendering threshold can be obtained by subtracting the playback time of the two frames. Optionally, the calculation formula of the preset frame rendering threshold is shown in the following formula (3): FRT i =(PTS i -PTS i-1 )×TimeBase (3)

[0103] Among them, FRT is the preset frame rendering threshold, PTS (Presentation Time Stamp) is the presentation timestamp of the video frame, TimeBase is the preset time base, i is the first video frame, and i-1 is the second video frame.

[0104] When the decoding timeliness is less than or equal to the preset frame rendering threshold, that is, the first video frame can be decoded in time, and thus can be encoded using a video encoding method.

[0105] In the second case, if the decoding timeliness is greater than the preset frame rendering threshold, the target encoding mode is determined to be the IDR encoding mode.

[0106] If the decoding timeliness is greater than the preset frame rendering threshold, it means that the current first video frame is not decoded in time, so the target encoding mode is determined to be the IDR encoding mode, and encoding is performed using the IDR encoding mode.

[0107] In the above embodiment, the encoding method of the video frame is determined by decoding timeliness and a preset frame rendering threshold, and it is predicted whether the video frame can be decoded in time. If it cannot be decoded in time, the IDR frame is sent through the IDR encoding method. This can avoid the delay that may be caused by requesting to send the IDR frame when the video frame cannot be decoded in time, thereby ensuring the smoothness of the video picture and avoiding freezes.

[0108] Furthermore, when the predicted video frame can be decoded in time, it is encoded using a video encoding method and the IDR encoder is turned off to save resources.

[0109] Optionally, after determining the target encoding mode, the step of the cloud platform server sending the IDR video frame to the target terminal is shown in FIG5 , and also includes steps S501 to S502 .

[0110] Step S501: Determine the value of the encoding flag in the RTP packet according to the target encoding mode.

[0111] The cloud platform server inserts the encoding flag information into the PT bit of the RTP data sent. If the target encoding method is IDR encoding, the cloud platform server sets the encoding flag in the RTP packet to 1; if the target encoding method is video encoding, the cloud platform server sets the encoding flag in the RTP packet to 0.

[0112] Step S502: Send the RTP packet to the target terminal.

[0113] Specifically, the cloud platform server sends the RTP packet to the target terminal, and the target terminal can determine the target encoding mode by checking the value of the encoding flag bit of the PT bit of the RTP data.

[0114] In the above embodiment, by setting the encoding flag in the RTP packet, the target terminal can determine the target encoding method according to the value of the encoding flag after receiving the RTP packet, thereby achieving synchronization of the encoding and decoding methods of the cloud platform server and the target terminal.

[0115] In one embodiment, the step of the cloud platform server receiving the historical decoding status information and the network status information sent by the target terminal may further include: receiving an RR message, wherein the RR message carries the historical decoding status information and the network status information.

[0116] Specifically, the cloud platform server receives an RR message sent by the target terminal, wherein the RR message carries historical decoding status information and network status information. Optionally, the RR message also carries a preset frame rendering threshold.

[0117] In one embodiment, as shown in FIG6 , a data transmission method is provided, which is described by taking the method applied to the target terminal in FIG1 as an example, and includes the following steps S601 to S603 .

[0118] Step S601: Send historical decoding status information and network status information to the cloud platform server.

[0119] Specifically, the historical decoding state information is used to indicate the state parameters of the target terminal decoding the second video frame preceding the first video frame to be sent by the cloud platform server. The historical decoding state information may include the frame decoding time of the second video frame. The target terminal sends the frame decoding time of the second video frame and network status information between the target terminal and the cloud platform server, such as packet loss rate and round-trip delay, to the cloud platform server.

[0120] Step S602: Receive encoding mode indication information sent by the cloud platform server based on historical decoding status information and network status information.

[0121] Specifically, the cloud platform server determines the target encoding mode based on the historical decoding state information and network state information sent by the target terminal using the method in the above embodiment, and then determines encoding mode indication information based on the target encoding mode and sends it to the target terminal. Optionally, as shown in Figure 7, the step of the target terminal receiving the encoding mode indication information may include steps S701 to S702.

[0122] Step S701: Receive RTP packets sent by the cloud platform server based on historical decoding status information and network status information.

[0123] In one embodiment, the RTP packet includes an encoding flag, the value of which is determined according to a target encoding mode.

[0124] Step S702: Use the value of the encoding flag as encoding mode indication information.

[0125] In one embodiment, the target terminal determines the encoding mode of the first video frame according to the encoding mode indication information, and the step of decoding the first video frame sent by the cloud platform server based on the encoding mode may include:

[0126] If the encoding mode indication information determines that the encoding mode of the first video frame is a video encoding mode, performing video decoding on the first video frame;

[0127] Specifically, if the target terminal detects that the value of the encoding flag is 0, that is, the encoding mode of the first video frame is the video encoding mode, the target terminal performs video decoding on the first video frame;

[0128] If the encoding mode indication information determines that the encoding mode of the first video frame is the IDR encoding mode, performing IDR decoding on the first video frame;

[0129] Specifically, if the target terminal detects that the value of the encoding flag is 1, that is, the encoding mode of the first video frame is the IDR encoding mode, then IDR decoding is performed on the first video frame.

[0130] In an embodiment of the present application, after receiving the video frame data, the target terminal selects different processing methods according to different situations. As shown in FIG8 , the above data transmission method may further include the following steps S801 to S803 .

[0131] Step S801: If there is currently packet loss, it is detected whether the time for the target video frame to be decoded to wait for decoding in the decoding buffer is less than the time for packet loss retransmission.

[0132] Specifically, after receiving the video frame data, the target terminal detects whether packet loss has occurred. If not, the video frame data is placed in the decoding buffer for decoding. If packet loss has occurred, and the target video frame is the one with packet loss, the target terminal detects whether the time the target video frame has been waiting for decoding in the decoding buffer is less than the packet loss retransmission time.

[0133] Step S802: If the time the target video frame waits for decoding in the decoding buffer is less than the packet loss retransmission time, the video frame closest to the target video frame and coded in the IDR coding mode is determined to be the third video frame.

[0134] Specifically, if the time the target video frame waits for decoding in the decoding buffer is less than the time for packet loss retransmission, that is, the target video frame does not have time for packet loss retransmission, according to the encoding flag of the video frame in the decoding buffer, it is determined that the video frame closest to the target video frame and encoded in the IDR encoding mode is the third video frame.

[0135] Step S803: Use the third video frame to replace the target video frame for decoding.

[0136] Specifically, the third video frame is used to replace the target video frame with packet loss for decoding. Since the third video frame is an IDR frame, which includes complete data of a frame, after decoding using the third video frame, all frame data before the third video frame in the decoding buffer is deleted.

[0137] Optionally, if the time the target video frame waits for decoding in the decoding buffer is greater than or equal to the packet loss retransmission time, packet loss retransmission processing is performed.

[0138] Specifically, if the time the target video frame waits for decoding in the decoding buffer is greater than or equal to the packet loss retransmission time, the target video frame has time to be retransmitted, so the packet loss retransmission processing can be performed, that is, the target video frame can be resent.

[0139] In the above embodiment, the processing method of the video frame is determined according to whether there is packet loss and the time for packet loss retransmission, which can provide early warning. When the time the target video frame waits for decoding in the decoding buffer is less than the time for packet loss retransmission, the appropriate IDR frame is selected for decoding to avoid image freeze and delay.

[0140] In an embodiment of the present application, please refer to FIG9 , which shows a flow chart of a data transmission method provided in an embodiment of the present application. The data transmission method includes the following steps S901 to S907 .

[0141] Step S901: The target terminal sends historical decoding status information and network status information to the cloud platform server.

[0142] Step S902: The cloud platform server receives historical decoding status information and network status information sent by the target terminal.

[0143] Step S903: The cloud platform server obtains first decoding complexity information of the first video frame and second decoding complexity information of the second video frame.

[0144] Step S904: The cloud platform server determines the decoding timeliness of the first video frame according to the historical decoding state information, the network state information, the first decoding complexity information, and the second decoding complexity information.

[0145] Step S905: The cloud platform server determines a target encoding method according to the decoding timeliness, encodes the first video frame based on the target encoding method, and sends the encoded first video frame to the target terminal.

[0146] In one embodiment, the cloud platform server determines the value of the encoding flag in the RTP packet according to the target encoding method, and sends the RTP packet to the target terminal.

[0147] Step S906: The target terminal receives the encoding mode indication information sent by the cloud platform server based on the historical decoding status information and the network status information.

[0148] Step S907: The target terminal determines the encoding mode of the first video frame according to the encoding mode indication information, and decodes the first video frame sent by the cloud platform server based on the encoding mode.

[0149] In one embodiment, the data transmission method of the present application proceeds as shown in FIG10 . The i-th frame of data is captured and encoded, the decoding timeliness of the i-th frame is calculated, and a determination is made as to whether the decoding timeliness exceeds a preset frame rendering threshold. If so, the IDR encoder is activated, IDR encoding is performed, and the IDR frame is transmitted. If the decoding timeliness does not exceed the preset frame rendering threshold, the IDR encoder is deactivated, video encoding is performed, and the video frame is transmitted.

[0150] The target terminal receives data and determines whether there is packet loss. If there is no packet loss, it decodes the data according to the encoding method and then determines whether the decoding is normal. If the decoding is normal, the target terminal sends a status message. If the decoding is abnormal, the most recent IDR frame is determined for decoding and the target terminal sends a status message. If there is packet loss and the waiting time for decoding is greater than or equal to the packet loss retransmission time, the packet loss retransmission is performed. If the waiting time for decoding is less than the packet loss retransmission time, the most recent IDR frame is determined for decoding.

[0151] In one embodiment, in order to facilitate readers to understand the technical solution provided by the embodiment of the present application, an example of applying the data transmission method of the present application to the application environment of Figure 1 is given, please refer to Figure 11.

[0152] 1. During the decoding and transmission of the i-th frame data, the target terminal uses normal video decoding of the I-frame or P-frame, with a decoding complexity of 1.295017495.

[0153] 1.1 The cloud platform server video encoder performs video encoding normally and sends the encoded video stream to the target terminal. The IDR encoding flag bit in the PT bit of the RTP packet is 0.

[0154] 1.2 After receiving the RTP packet of the video stream, the target terminal checks that the encoding flag is 0 and uses video decoding to perform normal decoding. The video frame can be an I frame or a P frame.

[0155] 2. Data transmission process of the i+1th frame.

[0156] 2.1. The target terminal decoder sends the frame decoding time (12ms), packet loss rate (0%), RTT (10ms) and preset frame rendering threshold of the i-th frame to the cloud platform server.

[0157] The calculation of the preset frame rendering threshold is as follows. The video has a fixed frame rate of 25fps.

[0158] 2.2. Decoding analysis of the cloud platform server The decoding complexity of the early warning receiver decoder is:

[0159] 2.3. Combining the frame decoding time, packet loss rate, and RTT of the i-th frame, the decoding timeliness of the i+1-th frame is predicted to be:

[0160] It can be seen that the decoding timeliness of the i+1th frame is less than the preset frame rendering threshold of 40ms, so the decoding is predicted to be timely and the i+1th frame adopts normal video encoding.

[0161] 2.4. Use a video encoder to encode the video and send the i+1th frame to the target terminal for decoding.

[0162] 2.5. The target terminal receives the video frame data, checks the encoding flag, and determines that the i+1th frame adopts video decoding.

[0163] 3. Data transmission process of the i+2th frame.

[0164] 3.1. The target terminal decoder sends the i+1 frame decoding time (12ms), packet loss rate (6%), RTT (15ms), and the preset frame rendering threshold to the cloud platform server. The preset frame rendering threshold is as follows:

[0165] 3.2. Cloud Platform Server Decoding Analysis Warning The decoding complexity of the receiving decoder is as follows:

[0166] 3.3. Combining the frame decoding time, packet loss rate, and RTT of the i+1th frame, the decoding timeliness of the i+2th frame is predicted as follows:

[0167] It can be seen that the decoding timeliness of the i+2th frame is greater than the preset frame rendering threshold of 40ms, so the i+2th frame is encoded using IDR encoding.

[0168] 3.4. Start the IDR encoder and encode the i+2th frame using the IDR encoding method.

[0169] 3.5. Set the encoding flag of the i+2th frame video stream data to 1 and send it to the target terminal.

[0170] 3.6. The target terminal receives the video frame data, checks the encoding flag, and determines that the i+2th frame adopts IDR decoding.

[0171] 4. Data transmission process of the i+3th frame.

[0172] 4.1. The target terminal decoder sends the i+2 frame decoding time (20ms), packet loss rate (2%), RTT (12ms), and the preset frame rendering threshold to the cloud platform server. The preset frame rendering threshold is as follows:

[0173] 4.2. Cloud Platform Decoding Analysis Warning The decoding complexity of the receiving decoder is as follows:

[0174] 4.3. Combining the frame decoding time, packet loss rate, and RTT of the i+2th frame, the decoding timeliness of the i+3th frame is predicted as follows:

[0175] It can be seen that the decoding timeliness of the i+3th frame is less than the preset frame rendering threshold of 40ms, so video coding is used for encoding.

[0176] 4.4. Turn off the IDR encoder and encode the i+3th frame using video encoding.

[0177] 4.5. The encoding flag position of the i+3th frame video frame data is 0 and is sent to the target terminal.

[0178] 4.6. The cloud terminal receives the i+3th frame of video data, checks the encoding flag, and determines to use the normal video decoding method for decoding.

[0179] 5. Continue to perform data transmission and encoding and decoding of cloud services according to the above process.

[0180] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0181] Based on the same inventive concept, embodiments of the present application further provide a data transmission device for implementing the aforementioned data transmission method. The implementation solution provided by this device is similar to the implementation solution described in the aforementioned method. Therefore, the specific limitations of one or more data transmission device embodiments provided below can be found in the above-mentioned limitations of the data transmission method and will not be further elaborated here.

[0182] In one embodiment, as shown in FIG12 , a data transmission device 1200 is provided, including: a status information receiving module 1201 , an acquisition module 1202 , a decoding timeliness determination module 1203 and a data sending module 1204 .

[0183] The status information receiving module 1201 is used to receive historical decoding status information and network status information sent by the target terminal. The historical decoding status information is used to indicate the status parameters of the target terminal decoding the second video frame before the first video frame to be sent;

[0184] The acquisition module 1202 is configured to acquire first decoding complexity information of a first video frame and second decoding complexity information of a second video frame.

[0185] The decoding timeliness determination module 1203 is configured to determine the decoding timeliness of the first video frame according to historical decoding state information, network state information, first decoding complexity information, and second decoding complexity information.

[0186] The data sending module 1204 is configured to determine a target encoding method according to the decoding timeliness of the first video frame, encode the first video frame based on the target encoding method, and send the encoded first video frame to a target terminal.

[0187] In one embodiment, the historical decoding status information includes the frame decoding time, and the network status information includes the packet loss rate and the round-trip delay. The decoding timeliness determination module 1203 is specifically used to calculate the ratio of the decoding complexity value indicated by the first decoding complexity information to the decoding complexity value indicated by the second decoding complexity information, and calculate the first product of the frame decoding time of the second video frame and the ratio; calculate the sum of the packet loss rate of the second video frame and a preset constant, and calculate the second product of the round-trip delay of the second video frame and the sum; determine the decoding timeliness of the first video frame based on the sum of the first product and the second product.

[0188] In one embodiment, the first decoding complexity information is determined based on the total number of macroblock groups in the first video frame and the sum of residual squared errors of the P frames; the second decoding complexity information is determined based on the total number of macroblock groups in the second video frame and the sum of residual squared errors of the P frames.

[0189] In one embodiment, the data sending module 1204 is specifically used to: if the decoding timeliness is less than or equal to the preset frame rendering threshold, determine the target encoding mode as the video encoding mode; if the decoding timeliness is greater than the preset frame rendering threshold, determine the target encoding mode as the IDR encoding mode.

[0190] In one embodiment, the preset frame rendering threshold is determined according to a difference between a presentation timestamp of the first video frame and a presentation timestamp of the second video frame and a preset time base.

[0191] In one embodiment, the status information receiving module 1201 is specifically configured to receive RR messages, wherein the RR messages carry historical decoding status information and network status information.

[0192] In one embodiment, the RR message also carries a preset frame rendering threshold.

[0193] In one embodiment, the data transmission device further includes a coding flag determining module, configured to determine a value of a coding flag in an RTP packet according to a target coding method, and send the RTP packet to a target terminal.

[0194] Each module in the above-mentioned data transmission device can be implemented in whole or in part through software, hardware, or a combination thereof. Each module can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the corresponding operations of each module.

[0195] In one embodiment, as shown in FIG13 , a data transmission device 1300 is further provided, including: a status information sending module 1301 , an indication information receiving module 1302 , and a coding mode determining module 1303 .

[0196] The status information sending module 1301 is used to send historical decoding status information and network status information to the cloud platform server. The historical decoding status information is used to indicate the status parameters of the target terminal for decoding the second video frame before the first video frame sent by the cloud platform server.

[0197] The indication information receiving module 1302 is used to receive encoding mode indication information sent by the cloud platform server based on historical decoding status information and network status information.

[0198] The encoding mode determination module 1303 is used to determine the encoding mode of the first video frame according to the encoding mode indication information, and decode the first video frame sent by the cloud platform server based on the encoding mode.

[0199] In one embodiment, the indication information receiving module 1302 is specifically used to receive the RTP packet sent by the cloud platform server based on historical decoding status information and network status information, wherein the RTP packet includes an encoding flag; and the value of the encoding flag is used as encoding mode indication information.

[0200] In one embodiment, the encoding mode determination module 1303 is specifically used to perform video decoding on the first video frame if the encoding mode indication information determines that the encoding mode of the first video frame is a video encoding mode; if the encoding mode indication information determines that the encoding mode of the first video frame is an IDR encoding mode, then perform IDR decoding on the first video frame.

[0201] In one embodiment, the above-mentioned data transmission device also includes a detection module for detecting whether the time the target video frame to be decoded waits for decoding in the decoding buffer is less than the time for packet loss retransmission if packet loss currently exists; if the time the target video frame waits for decoding in the decoding buffer is less than the time for packet loss retransmission, determining that the video frame closest to the target video frame and encoded in the IDR encoding mode is the third video frame; and using the third video frame to replace the target video frame for decoding.

[0202] In one embodiment, the data transmission device further includes a retransmission module configured to perform packet loss retransmission processing if the time the target video frame waits for decoding in the decoding buffer is greater than or equal to the packet loss retransmission time.

[0203] Each module in the above-mentioned data transmission device can be implemented in whole or in part through software, hardware, or a combination thereof. Each module can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the corresponding operations of each module.

[0204] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be shown in Figure 14. The computer device includes a processor, a memory, an input / output interface (I / O), and a communication interface. The processor, memory, and I / O interface are connected via a system bus, and the communication interface is connected to the system bus via the I / O interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the computer device is used to store decoding status information and network status information data. The I / O interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, a data transmission method is implemented.

[0205] In one embodiment, a computer device is provided, which may be a terminal. A diagram of its internal structure may be shown in FIG15 . The computer device includes a processor, memory, an input / output interface, a communication interface, a display unit, and an input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are connected to the system bus via the input / output interface. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The input / output interface of the computer device is configured to exchange information between the processor and an external device. The communication interface of the computer device is configured to communicate with an external terminal via wired or wireless communication, where the wireless communication may be achieved via Wi-Fi, a mobile cellular network, near-field communication (NFC), or other technologies. When executed by the processor, the computer program implements a data transmission method. The display unit of the computer device is configured to produce a visually visible image and may be a display screen, a projection device, or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device casing, or an external keyboard, touchpad or mouse.

[0206] Those skilled in the art will understand that the structure shown in FIG15 is merely a block diagram of a portion of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0207] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the following steps when executing the computer program: receiving historical decoding status information and network status information sent by a target terminal, the historical decoding status information being used to indicate status parameters for the target terminal to decode a second video frame before a first video frame to be sent; obtaining first decoding complexity information of the first video frame and second decoding complexity information of the second video frame; determining decoding timeliness of the first video frame based on the historical decoding status information, the network status information, the first decoding complexity information, and the second decoding complexity information; determining a target encoding method based on the decoding timeliness, encoding the first video frame based on the target encoding method, and sending the encoded first video frame to the target terminal.

[0208] In one embodiment, the historical decoding status information includes the frame decoding time of the second video frame, and the network status information includes the packet loss rate and the round-trip delay. When the processor executes the computer program, the following steps are also implemented: calculating the ratio of the decoding complexity value indicated by the first decoding complexity information to the decoding complexity value indicated by the second decoding complexity information, and calculating the first product of the frame decoding time of the second video frame and the ratio; calculating the sum of the packet loss rate of the second video frame and a preset constant, and calculating the second product of the round-trip delay of the second video frame and the sum; determining the decoding timeliness of the first video frame based on the sum of the first product and the second product.

[0209] In one embodiment, the first decoding complexity information is determined based on the total number of macroblock groups in the first video frame and the sum of residual squared errors of the P frames; the second decoding complexity information is determined based on the total number of macroblock groups in the second video frame and the sum of residual squared errors of the P frames.

[0210] In one embodiment, when the processor executes the computer program, the following steps are also implemented: if the decoding timeliness is less than or equal to the preset frame rendering threshold, the target encoding mode is determined to be the video encoding mode; if the decoding timeliness is greater than the preset frame rendering threshold, the target encoding mode is determined to be the IDR encoding mode.

[0211] In one embodiment, the preset frame rendering threshold is determined according to a difference between a presentation timestamp of the first video frame and a presentation timestamp of the second video frame and a preset time base.

[0212] In one embodiment, when the processor executes the computer program, the processor further implements the following steps: receiving an RR message, where the RR message carries historical decoding status information and network status information.

[0213] In one embodiment, the RR message also carries a preset frame rendering threshold.

[0214] In one embodiment, when the processor executes the computer program, the processor further implements the following steps: determining the value of the encoding flag in the RTP packet according to the target encoding mode; and sending the RTP packet to the target terminal.

[0215] In one embodiment, when the processor executes the computer program, it also implements the following steps: sending historical decoding status information and network status information to the cloud platform server, the historical decoding status information is used to indicate the status parameters of the target terminal for decoding the second video frame before the first video frame sent by the cloud platform server; receiving the encoding method indication information sent by the cloud platform server based on the historical decoding status information and the network status information; determining the encoding method of the first video frame according to the encoding method indication information, and decoding the first video frame sent by the cloud platform server based on the encoding method.

[0216] In one embodiment, when the processor executes the computer program, it also implements the following steps: receiving an RTP packet sent by the cloud platform server based on historical decoding status information and network status information, the RTP packet including an encoding flag; and using the value of the encoding flag as encoding mode indication information.

[0217] In one embodiment, when the processor executes the computer program, the following steps are also implemented: if the encoding mode indication information determines that the encoding mode of the first video frame is a video encoding mode, video decoding is performed on the first video frame; if the encoding mode indication information determines that the encoding mode of the first video frame is an IDR encoding mode, IDR decoding is performed on the first video frame.

[0218] In one embodiment, when executing the computer program, the processor further implements the following steps: if there is currently packet loss, detecting whether the time the target video frame to be decoded waits for decoding in the decoding buffer is less than the time for packet loss retransmission; if the time the target video frame waits for decoding in the decoding buffer is less than the time for packet loss retransmission, determining that the video frame closest to the target video frame and encoded in the IDR encoding mode is the third video frame; and using the third video frame to replace the target video frame for decoding.

[0219] In one embodiment, when the processor executes the computer program, the following steps are further implemented: if the time the target video frame waits for decoding in the decoding buffer is greater than or equal to the packet loss retransmission time, performing packet loss retransmission processing.

[0220] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented: receiving historical decoding status information and network status information sent by a target terminal, the historical decoding status information is used to indicate the status parameters of the target terminal for decoding a second video frame before a first video frame to be sent; obtaining first decoding complexity information of the first video frame and second decoding complexity information of the second video frame; determining the decoding timeliness of the first video frame based on the historical decoding status information, the network status information, the first decoding complexity information and the second decoding complexity information; determining a target encoding method based on the decoding timeliness, encoding the first video frame based on the target encoding method, and sending the encoded first video frame to the target terminal.

[0221] In one embodiment, the historical decoding status information includes the frame decoding time of the second video frame, the network status information includes the packet loss rate and the round-trip delay, and the computer program, when executed by the processor, further implements the following steps: calculating the ratio of the decoding complexity value indicated by the first decoding complexity information to the decoding complexity value indicated by the second decoding complexity information, and calculating the first product of the frame decoding time of the second video frame and the ratio; calculating the sum of the packet loss rate of the second video frame and a preset constant, and calculating the second product of the round-trip delay of the second video frame and the sum; determining the decoding timeliness of the first video frame based on the sum of the first product and the second product.

[0222] In one embodiment, the first decoding complexity information is determined based on the total number of macroblock groups in the first video frame and the sum of residual squared errors of the P frames; the second decoding complexity information is determined based on the total number of macroblock groups in the second video frame and the sum of residual squared errors of the P frames.

[0223] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: if the decoding timeliness is less than or equal to the preset frame rendering threshold, the target encoding mode is determined to be the video encoding mode; if the decoding timeliness is greater than the preset frame rendering threshold, the target encoding mode is determined to be the IDR encoding mode.

[0224] In one embodiment, the preset frame rendering threshold is determined according to a difference between a presentation timestamp of the first video frame and a presentation timestamp of the second video frame and a preset time base.

[0225] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: receiving an RR message, where the RR message carries historical decoding status information and network status information.

[0226] In one embodiment, the RR message also carries a preset frame rendering threshold.

[0227] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented: determining the value of the encoding flag in the RTP packet according to the target encoding mode; and sending the RTP packet to the target terminal.

[0228] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: sending historical decoding status information and network status information to the cloud platform server, the historical decoding status information is used to indicate the status parameters of the target terminal for decoding the second video frame before the first video frame sent by the cloud platform server; receiving the encoding method indication information sent by the cloud platform server based on the historical decoding status information and the network status information; determining the encoding method of the first video frame according to the encoding method indication information, and decoding the first video frame sent by the cloud platform server based on the encoding method.

[0229] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: receiving an RTP packet sent by the cloud platform server based on historical decoding status information and network status information, the RTP packet including an encoding flag; and using the value of the encoding flag as encoding mode indication information.

[0230] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: if the encoding mode indication information determines that the encoding mode of the first video frame is a video encoding mode, video decoding is performed on the first video frame; if the encoding mode indication information determines that the encoding mode of the first video frame is an IDR encoding mode, IDR decoding is performed on the first video frame.

[0231] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: if there is currently packet loss, it is detected whether the time the target video frame to be decoded waits for decoding in the decoding buffer is less than the time for packet loss retransmission; if the time the target video frame waits for decoding in the decoding buffer is less than the time for packet loss retransmission, it is determined that the video frame closest to the target video frame and encoded in the IDR encoding mode is the third video frame; and the third video frame is used to replace the target video frame for decoding.

[0232] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: if the time the target video frame waits for decoding in the decoding buffer is greater than or equal to the packet loss retransmission time, performing packet loss retransmission processing.

[0233] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps: receiving historical decoding status information and network status information sent by a target terminal, the historical decoding status information being used to indicate status parameters for the target terminal to decode a second video frame preceding a first video frame to be sent; obtaining first decoding complexity information of the first video frame and second decoding complexity information of the second video frame; determining decoding timeliness of the first video frame based on the historical decoding status information, the network status information, the first decoding complexity information, and the second decoding complexity information; determining a target encoding method based on the decoding timeliness, encoding the first video frame based on the target encoding method, and sending the encoded first video frame to the target terminal.

[0234] In one embodiment, the historical decoding status information includes the frame decoding time of the second video frame, the network status information includes the packet loss rate and the round-trip delay, and the computer program, when executed by the processor, further implements the following steps: calculating the ratio of the decoding complexity value indicated by the first decoding complexity information to the decoding complexity value indicated by the second decoding complexity information, and calculating the first product of the frame decoding time of the second video frame and the ratio; calculating the sum of the packet loss rate of the second video frame and a preset constant, and calculating the second product of the round-trip delay of the second video frame and the sum; determining the decoding timeliness of the first video frame based on the sum of the first product and the second product.

[0235] In one embodiment, the first decoding complexity information is determined based on the total number of macroblock groups in the first video frame and the sum of residual squared errors of the P frames; the second decoding complexity information is determined based on the total number of macroblock groups in the second video frame and the sum of residual squared errors of the P frames.

[0236] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: if the decoding timeliness is less than or equal to the preset frame rendering threshold, the target encoding mode is determined to be the video encoding mode; if the decoding timeliness is greater than the preset frame rendering threshold, the target encoding mode is determined to be the IDR encoding mode.

[0237] In one embodiment, the preset frame rendering threshold is determined according to a difference between a presentation timestamp of the first video frame and a presentation timestamp of the second video frame and a preset time base.

[0238] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: receiving an RR message, where the RR message carries historical decoding status information and network status information.

[0239] In one embodiment, the RR message also carries a preset frame rendering threshold.

[0240] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented: determining the value of the encoding flag in the RTP packet according to the target encoding mode; and sending the RTP packet to the target terminal.

[0241] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: sending historical decoding status information and network status information to the cloud platform server, the historical decoding status information is used to indicate the status parameters of the target terminal for decoding the second video frame before the first video frame sent by the cloud platform server; receiving the encoding method indication information sent by the cloud platform server based on the historical decoding status information and the network status information; determining the encoding method of the first video frame according to the encoding method indication information, and decoding the first video frame sent by the cloud platform server based on the encoding method.

[0242] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: receiving an RTP packet sent by the cloud platform server based on historical decoding status information and network status information, the RTP packet including an encoding flag; and using the value of the encoding flag as encoding mode indication information.

[0243] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: if the encoding mode indication information determines that the encoding mode of the first video frame is a video encoding mode, video decoding is performed on the first video frame; if the encoding mode indication information determines that the encoding mode of the first video frame is an IDR encoding mode, IDR decoding is performed on the first video frame.

[0244] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: if there is currently packet loss, it is detected whether the time the target video frame to be decoded waits for decoding in the decoding buffer is less than the time for packet loss retransmission; if the time the target video frame waits for decoding in the decoding buffer is less than the time for packet loss retransmission, it is determined that the video frame closest to the target video frame and encoded in the IDR encoding mode is the third video frame; and the third video frame is used to replace the target video frame for decoding.

[0245] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: if the time the target video frame waits for decoding in the decoding buffer is greater than or equal to the packet loss retransmission time, performing packet loss retransmission processing.

[0246] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions.

[0247] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processor involved in the various embodiments provided herein may be, but are not limited to, a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, and the like.

[0248] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0249] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A data transmission method, comprising: Receiving historical decoding state information and network state information sent by a target terminal, wherein the historical decoding state information is used to indicate a state parameter of decoding a second video frame before a first video frame to be sent by the target terminal; Obtaining first decoding complexity information of the first video frame and second decoding complexity information of the second video frame; Determining the decoding timeliness of the first video frame according to the historical decoding state information, the network state information, the first decoding complexity information, and the second decoding complexity information; A target encoding method is determined according to the decoding timeliness, the first video frame is encoded based on the target encoding method, and the encoded first video frame is sent to the target terminal.

2. The method according to claim 1, wherein the historical decoding state information includes a frame decoding time of the second video frame, the network state information includes a packet loss rate and a round-trip delay, and determining the decoding timeliness of the first video frame according to the historical decoding state information, the network state information, the first decoding complexity information, and the second decoding complexity information comprises: Calculating a ratio of a decoding complexity value indicated by the first decoding complexity information to a decoding complexity value indicated by the second decoding complexity information, and calculating a first product of a frame decoding time of the second video frame and the ratio; Calculate a sum of a packet loss rate of the second video frame and a preset constant, and calculate a second product of a round-trip delay of the second video frame and the sum; The decoding timeliness of the first video frame is determined according to the sum of the first product and the second product.

3. The method according to claim 1, wherein the first decoding complexity information is determined according to the total number of macroblock groups of the first video frame and a P-frame residual square error; The second decoding complexity information is determined according to the total number of macroblock groups of the second video frame and the P frame residual square error sum.

4. The method according to any one of claims 1 to 3, wherein determining the target encoding method according to the decoding timeliness comprises: If the decoding timeliness is less than or equal to a preset frame rendering threshold, determining that the target encoding mode is a video encoding mode; If the decoding timeliness is greater than the preset frame rendering threshold, the target encoding mode is determined to be an IDR encoding mode. 5 . The method according to claim 4 , wherein the preset frame rendering threshold is determined based on a difference between a presentation timestamp of the first video frame and a presentation timestamp of the second video frame and a preset time base.

6. The method according to claim 4, wherein the receiving the historical decoding status information and the network status information sent by the target terminal comprises: An RR message is received, where the RR message carries the historical decoding state information and the network state information.

7. The method according to claim 4, wherein the RR message also carries the preset frame rendering threshold.

8. The method according to any one of claims 1 to 3, further comprising: Determine the value of the encoding flag bit in the RTP packet according to the target encoding mode; The RTP packet is sent to the target terminal.

9. A data transmission method, comprising: Sending historical decoding state information and network state information to the cloud platform server, wherein the historical decoding state information is used to indicate state parameters for the target terminal to decode a second video frame preceding a first video frame sent by the cloud platform server; Receive the code sent by the cloud platform server based on the historical decoding state information and the network state information Code mode indication information; The encoding method of the first video frame is determined according to the encoding method indication information, and the first video frame sent by the cloud platform server is decoded based on the encoding method.

10. The method according to claim 9, wherein the receiving the encoding mode indication information sent by the cloud platform server based on the historical decoding state information and the network state information comprises: Receiving an RTP packet sent by the cloud platform server based on the historical decoding state information and the network state information, wherein the RTP packet includes a coding flag; The value of the encoding flag bit is used as the encoding mode indication information.

11. The method according to claim 9, wherein determining the encoding mode of the first video frame according to the encoding mode indication information, and decoding the first video frame sent by the cloud platform server based on the encoding mode, comprises: If the encoding mode indication information determines that the encoding mode of the first video frame is a video encoding mode, performing video decoding on the first video frame; If the encoding mode indication information determines that the encoding mode of the first video frame is an IDR encoding mode, IDR decoding is performed on the first video frame.

12. The method according to claim 11, further comprising: If there is currently packet loss, check whether the time the target video frame to be decoded waits for decoding in the decoding buffer is less than the time for packet loss retransmission; If the time the target video frame waits for decoding in the decoding buffer is less than the time for packet loss retransmission, determining that the video frame closest to the target video frame and encoded in the IDR encoding mode is the third video frame; The target video frame is replaced by the third video frame for decoding.

13. The method according to claim 12, further comprising: If the time the target video frame waits for decoding in the decoding buffer is greater than or equal to the packet loss retransmission time, packet loss retransmission processing is performed.

14. A data transmission device, comprising: A state information receiving module, used to receive historical decoding state information and network state information sent by a target terminal, wherein the historical decoding state information is used to indicate a state parameter for the target terminal to decode a second video frame before a first video frame to be sent; An acquisition module, used to acquire first decoding complexity information of the first video frame and second decoding complexity information of the second video frame; A decoding timeliness determination module, configured to determine the decoding timeliness of the first video frame according to the historical decoding state information, the network state information, the first decoding complexity information, and the second decoding complexity information; A data sending module is used to determine a target encoding method according to the decoding timeliness, encode the first video frame based on the target encoding method, and send the encoded first video frame to the target terminal.

15. A data transmission device, comprising: A status information sending module, used to send historical decoding status information and network status information to the cloud platform server, wherein the historical decoding status information is used to indicate a status parameter for a target terminal to decode a second video frame preceding a first video frame sent by the cloud platform server; An indication information receiving module, used to receive encoding mode indication information sent by the cloud platform server based on the historical decoding state information and the network state information; The coding mode determination module is used to determine the coding mode of the first video frame according to the coding mode indication information, and decode the first video frame sent by the cloud platform server based on the coding mode.

16. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein the processor implements the steps of the method according to any one of claims 1 to 8 or 9 to 13 when executing the computer program.

17. A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 or 9 to 13 are implemented.

18. A computer program product, comprising a computer program, wherein the computer program, when executed by a processor, implements the steps of the method according to any one of claims 1 to 8 or 9 to 13.

Images