A data transmission method and related device

Abstract

The present application discloses a data transmission method and related devices. When a packet loss of the i-th forward reference frame is detected, during the video decoding process, if the (i + 1)-th forward reference frame is decoded, the (i + 1)-th forward reference frame is directly decoded by using the corresponding second reference frame combination. If the (i + T + 1)-th forward reference frame is decoded, since the video frames in the first reference frame combination of the (i + T + 1)-th forward reference frame are before the (i + T + 1)-th forward reference frame, and there are T forward reference frames between the i-th forward reference frame and the (i + T + 1)-th forward reference frame, where T is a positive integer determined according to the retransmission delay, so as to complete the retransmission of the i-th forward reference frame before decoding the (i + T + 1)-th forward reference frame. Therefore, the (i + T + 1)-th forward reference frame is decoded by using the first reference frame combination including the retransmitted i-th forward reference frame. This reduces the delay of video playback at the receiving end and improves the smoothness of video playback.

Description

Technical Field

[0001] This application relates to the field of communication technologies, and in particular, to a data transmission method and related devices. Background Art

[0002] With the development of communication technologies, multimedia communication mainly based on video and audio has gradually become the mainstream service of communication technologies, providing people with multimedia communication services anytime and anywhere.

[0003] Currently, in the process of video communication services, the encoding end sends the encoded video frames to the receiving end. During the transmission process, there may be packet loss for some video frames (such as forward reference frames). To reduce the impact caused by packet loss of forward reference frames, in the traditional method, mainly reference frame selection technology is adopted, that is, through the acknowledgment information fed back by the receiving end, the encoding end selects the nearest correctly decoded frame as the reference frame for subsequent inter-frame encoding, thereby reducing the impact caused by packet loss of forward reference frames.

[0004] However, in this way, the encoding end needs to wait for the acknowledgment information fed back by the receiving end before starting the encoding of the next forward reference frame, which leads to an increase in the latency of the encoding end, and further leads to the latency and jitter of video playback at the receiving end. Summary of the Invention

[0005] To solve the above technical problems, this application provides a data transmission method and related devices, which ensure that the retransmission of the forward reference frame with packet loss in the reference frame combination is completed before decoding a certain forward reference frame, thereby reducing the impact of packet loss of the forward reference frame on video playback at the receiving end. Based on this, the encoding end does not need to wait for the acknowledgment information fed back by the receiving end and can directly complete the encoding of the next forward reference frame, reducing the latency of the encoding end, further reducing the latency of video playback at the receiving end, and improving the smoothness of video playback.

[0006] The embodiments of this application disclose the following technical solutions:

[0007] In a first aspect, an embodiment of this application provides a data transmission method, and the method includes:

[0008] Detect the reception situation of forward reference frames;

[0009] It is detected that the i-th forward reference frame in the frame group has a packet loss during the transmission of the forward reference frame. The video frame corresponding to the i-th forward reference frame is located in the first reference frame combination of the (i + T + 1)-th forward reference frame. The video frames in the first reference frame combination are located before the (i + T + 1)-th forward reference frame. The i-th forward reference frame is the video frame closest to the (i + T + 1)-th forward reference frame in the first reference frame combination, and there are T forward reference frames between them. T is a positive integer determined according to the retransmission delay, so as to complete the retransmission of the i-th forward reference frame before decoding the (i + T + 1)-th forward reference frame; i is a positive integer greater than 0;

[0010] During the video decoding process, when decoding the (i + 1)-th forward reference frame, the second reference frame combination of the (i + 1)-th forward reference frame is used to decode the (i + 1)-th forward reference frame. The video frames in the second reference frame combination are located before the (i + 1)-th forward reference frame and are at least T forward reference frames apart from the (i + 1)-th forward reference frame;

[0011] When decoding the (i + T + 1)-th forward reference frame, the first reference frame combination including the retransmitted i-th forward reference frame is used to decode the (i + T + 1)-th forward reference frame.

[0012] In a second aspect, an embodiment of the present application provides a data transmission method, and the method includes:

[0013] Determine the reference frame combination corresponding to each video frame to be encoded in the frame group. The video frames in the reference frame combination are located before the corresponding video frame to be encoded and are at least T video frames apart from the corresponding video frame to be encoded. T is a positive integer determined according to the retransmission delay, so as to complete the retransmission of the forward reference frame with a packet loss in the reference frame combination before decoding the forward reference frame;

[0014] Perform inter-frame encoding on each video frame in sequence through the corresponding reference frame combination to obtain the corresponding forward reference frame, and transmit the obtained forward reference frame to the receiving end;

[0015] If the acknowledgment information from the receiving end is received and the acknowledgment information indicates that the i-th forward reference frame has a packet loss during the transmission of the forward reference frame, retransmit the i-th forward reference frame to the receiving end, where i is a positive integer greater than 0.

[0016] In a third aspect, an embodiment of the present application provides a data transmission device, and the device includes a detection unit and a decoding unit:

[0017] The detection unit is used to detect the reception situation of the forward reference frame;

[0018] The detection unit is configured to detect that the i-th forward reference frame in the frame group is lost during the transmission of the forward reference frames. The i-th forward reference frame is in the first reference frame combination of the (i + T + 1)-th forward reference frame. The video frames in the first reference frame combination are before the (i + T + 1)-th forward reference frame. The i-th forward reference frame is the video frame closest to the (i + T + 1)-th forward reference frame in the first reference frame combination, and there are T forward reference frames between the i-th forward reference frame and the (i + T + 1)-th forward reference frame. T is a positive integer determined according to the retransmission delay, so as to complete the retransmission of the i-th forward reference frame before decoding the (i + T + 1)-th forward reference frame. The i is a positive integer greater than 0;

[0019] The decoding unit is configured to, during video decoding, when decoding the (i + 1)-th forward reference frame, decode the (i + 1)-th forward reference frame by using the second reference frame combination of the (i + 1)-th forward reference frame. The video frames in the second reference frame combination are before the (i + 1)-th forward reference frame and there are at least T forward reference frames between the video frames in the second reference frame combination and the (i + 1)-th forward reference frame;

[0020] The decoding unit is further configured to, when decoding the (i + T + 1)-th forward reference frame, decode the (i + T + 1)-th forward reference frame by using the first reference frame combination including the retransmitted i-th forward reference frame.

[0021] In a fourth aspect, an embodiment of the present application provides a data transmission device, which includes a determination unit, an encoding unit, a transmission unit, and a retransmission unit:

[0022] The determination unit is configured to determine a reference frame combination corresponding to each video frame to be encoded in the frame group. The video frames in the reference frame combination are before the corresponding video frame to be encoded and there are at least T video frames between the video frames in the reference frame combination and the corresponding video frame to be encoded. T is a positive integer determined according to the retransmission delay, so as to complete the retransmission of the lost forward reference frame in the reference frame combination before decoding the forward reference frame;

[0023] The encoding unit is configured to perform inter-frame encoding on each video frame in sequence through the corresponding reference frame combination to obtain the corresponding forward reference frame;

[0024] The transmission unit is configured to transmit the obtained forward reference frame to the receiving end;

[0025] The retransmission unit is configured to, if receiving the response information from the receiving end and the response information indicates that the i-th forward reference frame is lost during the transmission of the forward reference frames, retransmit the i-th forward reference frame to the receiving end. The i is a positive integer greater than 0.

[0026] Fifth aspect, an embodiment of the present application provides a device for data transmission, the device includes a processor and a memory:

[0027] The memory is used to store program code and transmit the program code to the processor;

[0028] The processor is used to execute the method described in the foregoing aspect according to the instructions in the program code.

[0029] Sixth aspect, an embodiment of the present application provides a computer-readable storage medium, the computer-readable storage medium is used to store program code, and the program code is used to execute the method described in the foregoing aspect.

[0030] It can be seen from the above technical solutions that the encoding end can encode video frames and transmit the obtained forward reference frames to the receiving end. The receiving end can detect whether packet loss occurs during the transmission of the forward reference frames. If it is determined that the i-th forward reference frame in the frame group has packet loss during the transmission of the forward reference frames, during the video decoding process, if the (i + 1)-th forward reference frame is decoded, since the (i + 1)-th forward reference frame is not encoded based on the video frame corresponding to the i-th forward reference frame, there is no need to wait for the retransmission of the i-th forward reference frame received, and directly use the corresponding second reference frame combination to decode the (i + 1)-th forward reference frame. The video frames in the second reference frame combination are before the (i + 1)-th forward reference frame and are at least separated from the (i + 1)-th forward reference frame by T forward reference frames. If the (i + T + 1)-th forward reference frame is decoded, since the video frames in the first reference frame combination of the (i + T + 1)-th forward reference frame are before the (i + T + 1)-th forward reference frame, the i-th forward reference frame is the video frame closest to the (i + T + 1)-th forward reference frame in the first reference frame combination and is separated from the (i + T + 1)-th forward reference frame by T forward reference frames, and T is a positive integer determined according to the retransmission delay, so as to complete the retransmission of the i-th forward reference frame before decoding the (i + T + 1)-th forward reference frame. Therefore, the receiving end does not need to wait for the retransmission of the i-th forward reference frame and can directly use the first reference frame combination including the retransmitted i-th forward reference frame to decode the (i + T + 1)-th forward reference frame. This method determines a reasonable cross-frame reference interval according to the retransmission delay, so as to ensure that the retransmission of the forward reference frame with packet loss in its reference frame combination is completed before decoding a certain forward reference frame, thereby reducing the impact of forward reference frame packet loss on video playback at the receiving end. Based on this, the encoding end does not need to wait for the response information feedback from the receiving end and can directly encode the next forward reference frame, reducing the delay of the encoding end and further reducing the delay of video playback at the receiving end, improving the smoothness of video playback. Description of the Drawings

[0031] To more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the accompanying drawings required for the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0032] Figure 1 Schematic diagram of a frame group provided by an embodiment of the present application;

[0033] Figure 2 Schematic diagram of a data transmission method provided by the related art;

[0034] Figure 3 Schematic diagram of the system architecture of a data transmission method provided by an embodiment of the present application;

[0035] Figure 4 Signaling interaction diagram of a data transmission method provided by an embodiment of the present application;

[0036] Figure 5 Schematic diagram of the reference relationship between video frames in a frame group in a single-frame reference scenario provided by an embodiment of the present application;

[0037] Figure 6 Schematic diagram of the processing flow of a receiving end provided by an embodiment of the present application;

[0038] Figure 7 Structure diagram of a data transmission device provided by an embodiment of the present application;

[0039] Figure 8 Structure diagram of a data transmission device provided by an embodiment of the present application;

[0040] Figure 9 Structure diagram of a terminal provided by an embodiment of the present application;

[0041] Figure 10 Structure diagram of a server provided by an embodiment of the present application. Detailed implementation manners

[0042] The following describes the embodiments of the present application in conjunction with the accompanying drawings.

[0043] First, the terms related to the present application are explained:

[0044] Inter-frame reference coding is a commonly used coding technique and is adopted in specifications such as H.264 / H.265. The prerequisite for decoding a video frame with inter-frame reference coding is to correctly receive all the reference frames used for reference coding. Currently, in order to improve the coding effect, inter-frame coding with multiple reference frames can be adopted. In low-latency transmission, forward reference coding is generally used, that is, when encoding the current video frame, only the previous video frames are referred to.

[0045] Forward reference frame: In video coding, it is a frame that refers to the previous video frames and is encoded using the inter-frame coding technique. Its coding size is relatively small. When decoding, it depends on the video frames referred to forward during encoding, which is different from the frames encoded using the intra-frame coding technique.

[0046] Group Of Pictures (GoP): A group of consecutive pictures in a video, serving as a group of frames in video coding. In low-latency video transmission, usually in a GoP, the first encoded video frame is an I-frame, and the subsequent video frames are forward reference frames (P-frames).

[0047] See Figure 1 as shown Figure 1 shows a GOP (i.e., a group of frames), which includes 8 encoded video frames. Among them, the first video frame is an I-frame. Starting from the second video frame until the last video frame, they need to refer to the previous video frames and are encoded using the inter-frame coding technique, so they are forward reference frames (P-frames). There are multiple P-frames in this group of frames. Then, the second video frame is the first P-frame (i.e., P1), the third video frame is the second P-frame (i.e., P2), the fourth video frame is the third P-frame (i.e., P3), the fifth video frame is the fourth P-frame (i.e., P4), the sixth video frame is the fifth P-frame (i.e., P5), the seventh video frame is the sixth P-frame (i.e., P6), and the eighth video frame is the seventh P-frame (i.e., P7).

[0048] Packet loss may occur during the transmission of forward reference frames. In order to reduce the impact caused by the packet loss of forward reference frames, in the traditional method, the reference frame selection technique is mainly adopted, that is, through the acknowledgment information such as ACK information or NACK information feedback by the receiving end, the encoding end selects the nearest correctly decoded frame as the reference frame for subsequent inter-frame coding, so as to reduce the impact caused by the packet loss of forward reference frames. See Figure 2As shown, an error occurs in P3, and the receiving end feeds back a NACK message to the encoding end. Since P4 refers to P3 and cannot be correctly decoded either, it will also transmit a NACK message indicating the error in P3. Since P5 refers to P3 and cannot be correctly decoded either, it will also transmit a NACK message indicating the error in P3. When the encoding end receives the NACK signal of the error in P3 during the encoding of P6, it adjusts the frame buffer and uses the correctly decoded P2 frame as the reference frame. The receiving end returns to normal after correctly receiving the encoded data of P6.

[0049] However, in this way, the encoding end needs to wait for the receiving end to feedback the received response message before starting the encoding of the next forward reference frame, which leads to an increase in the latency of the encoding end, and further leads to the latency and jitter of the video playback at the receiving end.

[0050] To solve the above technical problems, the embodiments of the present application provide a data transmission method, which can be applied to various low-latency transmission scenarios, such as scenarios with low packet loss rate and extremely strict latency requirements, such as the remote control of the 5th Generation Mobile Communication Technology (5G) private network. Remote control of the 5G private network can include, for example, remotely controlling rovers on the moon and Mars, unmanned ships, and remotely driven vehicles (such as assisted driving, autonomous driving, remote control driving, etc.).

[0051] This scenario has a high frame rate. To reduce latency, a high frame rate, such as 60 FPS (Frames Per Second), is introduced; a low packet loss rate, because the 5G air interface has built-in retransmission, and the number of routers in the private network is extremely small, and packet loss rarely occurs. The latency from the receiving end detecting and sending a NACK message or an ACK message to notifying the encoding end is about 10 - 15 ms. If packet loss occurs and the encoding end sends the data packet to the receiving end again for reception, it also takes about 10 - 15 ms. For a 60 FPS frame rate, if the reference frame selection technology is used and the encoding end waits for the NACK message or the ACK message to perform encoding, it will add at least 1 frame of latency.

[0052] The method provided by the embodiments of the present application determines a reasonable cross-frame reference interval according to the retransmission latency, so as to ensure that the retransmission of the forward reference frame with packet loss in the reference frame combination is completed before decoding a certain forward reference frame, thereby reducing the impact of the forward reference frame packet loss on the video playback at the receiving end. Based on this, the encoding end does not need to wait for the response message feedback by the receiving end and can directly perform the encoding of the next forward reference frame, reducing the latency of the encoding end, further reducing the latency of the video playback at the receiving end, and improving the smoothness of the video playback.

[0053] The method provided by the embodiments of the present application relates to the field of cloud technology, for example, the field of cloud computing. Cloud computing is a computing model that distributes computing tasks across a resource pool composed of a large number of computing devices, enabling various application systems to obtain computing power, storage space, and information services as needed. The network that provides resources is called the "cloud". The resources in the "cloud" seem to users to be infinitely scalable, and can be obtained at any time, used on demand, expanded at any time, and paid according to usage. As a basic capability provider of cloud computing, a cloud computing resource pool (abbreviated as a cloud platform, generally called an IaaS (Infrastructure as a Service) platform) will be established, and various types of virtual resources will be deployed in the resource pool for external customers to select and use. The cloud computing resource pool mainly includes: computing devices (virtualized machines, including operating systems), storage devices, and network devices. According to logical function division, a PaaS (Platform as a Service) layer can be deployed on the IaaS (Infrastructure as a Service) layer, and a SaaS (Software as a Service) layer can be deployed on top of the PaaS layer. The SaaS can also be directly deployed on the IaaS. PaaS is a platform for software operation, such as databases, web containers, etc. SaaS is various business software, such as web portals, SMS mass senders, etc. Generally speaking, SaaS and PaaS are upper layers relative to IaaS.

[0054] Next, the system architecture of data transmission will be introduced. Refer to Figure 3 , Figure 3 which is a schematic diagram of the system architecture of the data transmission method provided by the embodiments of the present application. The system architecture includes a terminal 301 and a server 302. Among them, the terminal 301 can be used as a receiving end, and the server 302 can be used as an encoding end. The server 302 encodes the video frames in the video frame sequence to obtain the encoded video frames, and transmits the encoded video frames to the terminal 301 for rendering and playing on the terminal 301. Among them, a group of consecutive pictures in the video is used as a group of frames in video encoding. Therefore, the first encoded video frame can be called an I-frame, and the subsequent video frames can be called forward reference frames.

[0055] In the same group of frames, the forward reference frames need to refer to the previous video frames and use the inter-frame encoding technology for encoding. Therefore, if a forward reference frame is lost during transmission, it may affect the decoding of subsequent forward reference frames and may also affect the encoding of the encoding end, resulting in latency.

[0056] Therefore, in the embodiments of the present application, the server 302 encodes video frames in a cross-frame reference manner, that is, when performing inter-frame encoding on a video frame to be encoded, it encodes with reference to a combination of reference frames before the T-th frame forward. That is, the video frames in the reference frame combination corresponding to the video frame to be encoded are located before the corresponding video frame to be encoded, and there are at least T video frames between them and the corresponding video frame to be encoded. T is a positive integer determined according to the retransmission delay, so as to complete the retransmission of the forward reference frame with packet loss in the reference frame combination before decoding the forward reference frame. Among them, the reference frame combination may include one video frame (i.e., single-frame reference), or may include multiple video frames (i.e., multi-frame reference). Figure 3 Taking the single-frame reference as an example, T = 1, and the finally determined reference frame combination is: P2 refers to the I frame, P3 refers to P1, P4 refers to P2, P5 refers to P3, P6 refers to P4, and P7 refers to P5.

[0057] In this way, the encoding end can perform inter-frame encoding on each video frame in turn through the corresponding reference frame combination to obtain the corresponding forward reference frame, and transmit the obtained forward reference frame to the receiving end. Since T is a positive integer determined according to the retransmission delay, so as to complete the retransmission of the forward reference frame with packet loss in the reference frame combination before decoding the forward reference frame, the server 302 does not need to worry about the packet loss of the referenced video frame during transmission, which affects the decoding of the decoding end during encoding. Therefore, the server 302 can encode the next forward reference frame without waiting for the response information of the receiving end (terminal 301), thereby reducing the delay of the encoding end (server 302).

[0058] The terminal 301 can determine whether a forward reference frame has packet loss according to the reception situation. If it detects that the i-th forward reference frame in the frame group has packet loss during the transmission of the forward reference frame, it can send a response information to the server 302. If the server 302 receives the response information of the terminal 301, and the response information indicates that the i-th forward reference frame has packet loss during the transmission of the forward reference frame, it retransmits the i-th forward reference frame to the terminal 301, where i is a positive integer greater than 0.

[0059] The terminal 301 can decode the received forward reference frame. During the video decoding process, if it decodes the (i + 1)-th forward reference frame, since it does not encode with reference to the video frame corresponding to the i-th forward reference frame, it can directly use the second reference frame combination of the (i + 1)-th forward reference frame to decode the (i + 1)-th forward reference frame. The video frames in the second reference frame combination are located before the (i + 1)-th forward reference frame, and there are at least T forward reference frames between them and the (i + 1)-th forward reference frame.

[0060] If the terminal 301 decodes the (i + T + 1)-th forward reference frame, since the retransmission of the i-th forward reference frame has been completed at this time, the (i + T + 1)-th forward reference frame can be directly decoded by using the first reference frame combination including the retransmitted i-th forward reference frame.

[0061] It should be noted that the server 302 can be an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. The terminal 301 can be a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, a vehicle-mounted terminal, a smart TV, etc., but is not limited thereto. The terminal 301 and the server 302 can be directly or indirectly connected through wired or wireless communication means, which is not limited in this application.

[0062] Next, from the perspective of the interaction between the encoding end and the receiving end, where the encoding end can be a server and the receiving end can be a terminal, the data transmission method provided by the embodiments of the present application will be introduced in detail with reference to the accompanying drawings.

[0063] See Figure 4 , Figure 4 which shows a signaling interaction diagram of a data transmission method, and the method includes:

[0064] S401. The encoding end determines a reference frame combination corresponding to each video frame to be encoded in the frame group.

[0065] It can be understood that the present application mainly reduces the delay impact of forward reference frame loss on the encoding end and the receiving end. The reason why forward reference frame loss has a delay impact on the encoding end and the receiving end is mainly that the forward reference frame encoded depending on the lost forward reference frame also needs to depend on it for decoding. However, due to the loss, it takes a certain retransmission time to retransmit the lost forward reference frame, resulting in the inability to complete the retransmission before decoding and thus unable to decode. Therefore, the encoding end needs to wait for the response information from the receiving end to determine whether the previous forward reference frame is lost before encoding the next forward reference frame.

[0066] Therefore, in the embodiments of the present application, the encoding end performs inter-frame encoding in a cross-frame reference manner, that is, when performing inter-frame encoding on a video frame to be encoded, it encodes with reference to a combination of reference frames before the T-th frame forward. That is, the video frames in the reference frame combination corresponding to the video frame to be encoded are located before the corresponding video frame to be encoded, and there are at least T video frames between them and the corresponding video frame to be encoded. T is a positive integer determined according to the retransmission delay, so as to complete the retransmission of the lost forward reference frames in the reference frame combination before decoding the forward reference frames. In this way, the encoding end does not need to wait for the acknowledgment information fed back by the receiving end, and does not need to determine whether the transmitted forward reference frames are lost, and can encode the next forward reference frame. It will neither cause the receiving end to be unable to decode in time due to the loss of forward reference frames, resulting in the delay of the receiving end, nor cause the delay of the encoding end due to the encoding end waiting for the acknowledgment information.

[0067] It should be noted that in this embodiment, the reference frame combination may include one video frame (i.e., single-frame reference), or may include multiple video frames (i.e., multi-frame reference). Refer to Figure 5 , Figure 5 , taking the single-frame reference as an example, T = 1, which reflects the reference relationship between the video frames in the group of frames. Then the finally determined reference frame combination is: P2 refers to the I frame (the reference frame combination corresponding to the video frame to be encoded by P2 includes the I frame), P3 refers to P1 (the reference frame combination corresponding to the video frame to be encoded by P3 includes the P1 frame), P4 refers to P2 (the reference frame combination corresponding to the video frame to be encoded by P4 includes the P2 frame), P5 refers to P3 (the reference frame combination corresponding to the video frame to be encoded by P5 includes the P3 frame), P6 refers to P4 (the reference frame combination corresponding to the video frame to be encoded by P6 includes the P4 frame), and P7 refers to P5 (the reference frame combination corresponding to the video frame to be encoded by P7 includes the P5 frame).

[0068] It should be understood that in this embodiment, T is a positive integer determined according to the retransmission delay, so as to complete the retransmission of the lost forward reference frames in the reference frame combination before decoding the forward reference frames. The retransmission delay is the time required to retransmit the lost forward reference frames. In order to ensure that the retransmission of the lost forward reference frames in the reference frame combination is completed before decoding a certain forward reference frame, the product of the frame duration of the forward reference frame and T + 1 should be greater than the retransmission delay.

[0069] Among them, the retransmission delay is determined according to the duration of detecting the loss of the forward reference frame, the network retransmission duration, the decoding duration of the forward reference frame, and the catch-up duration of the forward reference frame delay. Generally, the retransmission delay can be equal to the duration of the forward reference frame loss (including the network notification of NACK information or ACK information) + the network retransmission duration + the decoding duration of the forward reference frame - the catch-up duration (the reception delay tolerated for normal playback). That is to say, when determining T, the frame duration * (T + 1) needs to be greater than the duration of detecting the loss of the forward reference frame (including the network notification of NACK information or ACK information) + the network retransmission duration + the decoding duration of the forward reference frame - the catch-up duration (the reception delay tolerated for normal playback).

[0070] It can be understood that since the network retransmission duration is determined according to the network quality, and the network quality may change. Sometimes the network quality is good, then the network retransmission duration may be shorter; sometimes the network quality is poor, then the network retransmission duration may be longer. Therefore, in the embodiments of the present application, the interval T of cross-frame reference can be adjusted according to the network quality.

[0071] However, the timing of adjusting T may be different. In order to avoid adjusting T too frequently and reducing the processing efficiency, and at the same time be able to select a suitable T as much as possible according to the network quality, in the embodiments of the present application, T is mainly adjusted at the following two timings. Since video frames may be in units of a group of frames during transmission, in a possible implementation manner, T can be adjusted according to the network quality after the transmission of a group of frames is completed.

[0072] In some cases, consecutive loss of forward reference frames may occur. For example, P2, P3, and P4 all experience packet loss. Then it may be considered that the current network quality is poor, and thus a group of frames is refreshed to re-determine the reference relationship between the video frames in the group of frames and the refreshed group of frames. Therefore, in a possible implementation manner, T can be adjusted according to the network quality when a group of frames is refreshed.

[0073] If a group of frames is referred to Figure 5 As shown, P2, P3, and P4 all experience packet loss. Then a refresh is performed at the receiving end at P4. At this time, P5 - P7 and other supplementary video frames form a new group of frames. The number of video frames and the composition of the video frames included in the refreshed group of frames and the group of frames before refreshing can be the same. Thus, T is determined according to the current network quality, and T before refreshing is adjusted.

[0074] S402. The encoding end performs inter-frame encoding on each video frame in turn through the corresponding reference frame combination to obtain the corresponding forward reference frame.

[0075] S403. The encoding end transmits the obtained forward reference frame to the receiving end.

[0076] For each video frame, perform inter-frame coding in sequence according to the determined reference frame combination to obtain the corresponding forward reference frame, transmit the obtained forward reference frame to the receiving end, and leave a cache locally.

[0077] S404. The receiving end detects the reception situation of the forward reference frame.

[0078] The receiving end sequentially receives the forward reference frames transmitted by the encoding end, detects the reception situation of the forward reference frames, and notifies the encoding end of the response information according to the detection result. The response information can be, for example, NACK information or ACK information, etc.

[0079] S405. The receiving end detects that the i-th forward reference frame in the frame group is lost during the transmission of the forward reference frame.

[0080] S406. The receiving end sends response information to the encoding end.

[0081] If the receiving end detects that the i-th forward reference frame in the frame group is lost during the transmission of the forward reference frame, the response information sent to the encoding end indicates that the i-th forward reference frame is lost during the transmission of the forward reference frame. The video frame corresponding to the i-th forward reference frame is in the first reference frame combination of the (i + T + 1)-th forward reference frame. The video frames in the first reference frame combination are before the (i + T + 1)-th forward reference frame. The i-th forward reference frame is the video frame closest to the (i + T + 1)-th forward reference frame in the first reference frame combination, and there are T forward reference frames between the i-th forward reference frame and the (i + T + 1)-th forward reference frame. T is a positive integer determined according to the retransmission delay, so as to complete the retransmission of the i-th forward reference frame before decoding the (i + T + 1)-th forward reference frame. i is a positive integer greater than 0. Among them, the i-th forward reference frame may be completely lost or partially lost.

[0082] See Figure 5 as shown Figure 5 Taking single-frame reference as an example, T = 1. Then P2 refers to the I frame (the reference frame combination of the video frame to be encoded corresponding to P2 includes the I frame), P3 refers to P1 (the reference frame combination of the video frame to be encoded corresponding to P3 includes the P1 frame), P4 refers to P2 (the reference frame combination of the video frame to be encoded corresponding to P4 includes the P2 frame), P5 refers to P3 (the reference frame combination of the video frame to be encoded corresponding to P5 includes the P3 frame), P6 refers to P4 (the reference frame combination of the video frame to be encoded corresponding to P6 includes the P4 frame), P7 refers to P5 (the reference frame combination of the video frame to be encoded corresponding to P7 includes the P5 frame). If i = 3, that is, the 3rd forward reference frame P3 is lost, then send response information to the encoding end, and this response information indicates that the 3rd forward reference frame is lost during the transmission of the forward reference frame.

[0083] According to the reference relationship, P3 is in the first reference frame combination of the 5th (3 + 1 + 1) forward reference frame P5. P3 is the video frame closest to P5 in the first reference frame combination and is separated from P5 by 1 forward reference frame P4. Only Figure 5 Taking single-frame reference as an example, only P3 is in the first reference frame combination; in the case of multi-frame reference, the first reference frame combination includes multiple video frames before P3.

[0084] S407. The encoding end retransmits the i-th forward reference frame to the receiving end.

[0085] The encoding end retransmits the i-th forward reference frame to the receiving end according to the received acknowledgment information. If all packets of the i-th forward reference frame are lost, the entire i-th forward reference frame is retransmitted; if only some packets of the i-th forward reference frame are lost, only the lost part of the i-th forward reference frame is retransmitted, avoiding wasting resources.

[0086] S408. During video decoding, when decoding the (i + 1)-th forward reference frame, the second reference frame combination of the (i + 1)-th forward reference frame is used to decode the (i + 1)-th forward reference frame.

[0087] It can be understood that in addition to detecting the reception situation of forward reference frames, the receiving end can mainly decode and render the received forward reference frames to play the corresponding pictures at the receiving end.

[0088] When the receiving end normally receives the current forward reference frame, normal decoding and recovery are performed according to the local reference frame buffer. If it is detected that the (i + 1)-th forward reference frame is normally received, when decoding the (i + 1)-th forward reference frame, the second reference frame combination of the (i + 1)-th forward reference frame is used to decode the (i + 1)-th forward reference frame. The video frames in the second reference frame combination are before the (i + 1)-th forward reference frame and are separated from the (i + 1)-th forward reference frame by at least T forward reference frames, that is, the i-th forward reference frame is not included.

[0089] Continuing with the example of P3 packet loss mentioned above, if the receiving end normally receives Figure 5 P4 as shown, since P4 is not encoded or decoded with reference to P3, P4 can be directly decoded based on P2 in the reference frame buffer.

[0090] When the receiving end detects that the current forward reference frame is lost, the image of the previous frame is reused to render the current forward reference frame. That is, if it is detected that the i-th forward reference frame is lost during the transmission of forward reference frames, when decoding the i-th forward reference frame, the image corresponding to the target video frame is reused to render the i-th forward reference frame, and the target video frame is the previous frame of the i-th forward reference frame.

[0091] S409. Decode the (i + T + 1)-th forward reference frame, and decode the (i + T + 1)-th forward reference frame by using a first reference frame combination including the retransmitted i-th forward reference frame.

[0092] Even if the (i + T + 1)-th forward reference frame encoded depending on the i-th forward reference frame is decoded, since the cross-frame reference interval T is determined according to the retransmission delay, the retransmission of the i-th forward reference frame can be completed before decoding the (i + T + 1)-th forward reference frame. When the receiving end receives the retransmitted i-th forward reference frame, the i-th forward reference frame will be put into the local reference frame buffer. Thus, when decoding the (i + T + 1)-th forward reference frame, the (i + T + 1)-th forward reference frame is directly decoded by using the first reference frame combination including the retransmitted i-th forward reference frame.

[0093] Continuing with the previous example of P3 packet loss, when the receiving end decodes the 5th (3 + 1 + 1) forward reference frame P5, the receiving end will receive the retransmitted P3 from the encoding end before receiving P5. Therefore, P5 can be decoded based on the retransmitted P3.

[0094] It should be noted that the video frames received by the receiving end will carry identification information, and the identification information is used to reflect the reference relationship between the video frames in the same frame group, so as to facilitate knowing which video frames a forward reference frame is encoded with reference to, that is, which video frames are included in the reference frame combination corresponding to the forward reference frame. In this way, during the decoding process, it is possible to determine which video frames to decode according to the identification information. That is to say, the first reference frame combination or the second reference frame combination is determined according to the identification information of the video frames.

[0095] If frame group refreshing has been performed, the video frames received by the receiving end will also carry refreshing information, and the refreshing information can reflect from which video frame the refreshing is performed and which video frames are included in the frame group after refreshing. At this time, the first reference frame combination or the second reference frame combination is determined according to the refreshing information of the frame group refreshing and the identification information of the video frames included in the frame group after refreshing.

[0096] As can be seen from the above technical solution, the encoding end can encode video frames and transmit the obtained forward reference frames to the receiving end. The receiving end can detect whether packet loss occurs during the transmission of the forward reference frames. If it is determined that the i-th forward reference frame in the frame group has packet loss during the transmission of the forward reference frames, during the video decoding process, if the (i + 1)-th forward reference frame is decoded, since the (i + 1)-th forward reference frame is not encoded based on the video frame corresponding to the i-th forward reference frame, there is no need to wait for the retransmission of the i-th forward reference frame received, and directly use the corresponding second reference frame combination to decode the (i + 1)-th forward reference frame. The video frames in the second reference frame combination are before the (i + 1)-th forward reference frame and are at least separated by T forward reference frames from the (i + 1)-th forward reference frame. If the (i + T + 1)-th forward reference frame is decoded, since the video frames in the first reference frame combination of the (i + T + 1)-th forward reference frame are before the (i + T + 1)-th forward reference frame, the i-th forward reference frame is the video frame closest to the (i + T + 1)-th forward reference frame in the first reference frame combination and is separated by T forward reference frames from the (i + T + 1)-th forward reference frame, and T is a positive integer determined according to the retransmission delay, so as to complete the retransmission of the i-th forward reference frame before decoding the (i + T + 1)-th forward reference frame. Therefore, the receiving end does not need to wait for the retransmission of the i-th forward reference frame and can directly use the first reference frame combination including the retransmitted i-th forward reference frame to decode the (i + T + 1)-th forward reference frame. This method determines a reasonable cross-frame reference interval according to the retransmission delay, so as to ensure that the retransmission of the forward reference frame with packet loss in its reference frame combination is completed before decoding a certain forward reference frame, thereby reducing the impact of forward reference frame packet loss on video playback at the receiving end. Based on this, the encoding end does not need to wait for the acknowledgment information feedback from the receiving end and can directly encode the next forward reference frame, reducing the delay of the encoding end and further reducing the delay of video playback at the receiving end, improving the smoothness of video playback.

[0097] In addition, although the embodiment of the present application adopts cross-frame reference, it only slightly increases the transmission code rate of the forward reference frame and has little impact on the increase of the transmission code rate of the forward reference frame, and is suitable for low-delay data transmission scenarios.

[0098] Next, the data transmission method provided by the embodiment of the present application will be introduced in combination with an actual application scenario. In a 5G private network remote control scenario, such as a remote driving scenario, a server (encoding end) can transmit the captured high-definition picture to a user's terminal (receiving end) so that the user can perform remote control according to the high-definition picture displayed on the terminal.

[0099] Continuing with the Figure 5 frame group shown as an example, the reference relationship of the video frames in the frame group is also as Figure 5As shown, this reference relationship is pre-determined according to T (assuming T is 1), and the server can encode and transmit video frames according to the reference relationship shown in Figure 5 . The processing flow at the receiving end can be referred to Figure 6 . It includes the receiving stream at the receiving end, the retransmission stream at the receiving end, and the decoding and playback stream at the receiving end. The receiving stream is the data stream transmitted by the encoding end received by the receiving end. The retransmission stream is the data stream of a certain forward reference frame retransmitted by the encoding end received by the receiving end. The decoding and playback stream is the continuous picture rendered by the receiving end after decoding. The frame duration of each video frame is 16.6 ms, and the receiving end sequentially receives I frame, P1, P2, P3, P4, P5, P6, P7. From the start of receiving P3 (indicating the left edge of the rectangular frame of the forward reference frame P3) to a certain moment at the receiving end, for example Figure 6 , it is found that P3 is lost at the dot position shown. 20 ms after the loss is detected, the receiving end receives the retransmitted P3 from the encoding end. Correspondingly, when decoding and playing the stream at the receiving end, for the normally received I, P1, P2, the local reference frame buffer is used for normal decoding and recovery. Since P3 is lost, when decoding P3, the image corresponding to P2 can be reused for rendering to output the image corresponding to P3. At this time, the image corresponding to P3 is the same as the image corresponding to P2. When decoding P4, P4 is decoded and played with reference to P2. When decoding P5, since P3 has been retransmitted before receiving P5, P5 can be decoded based on the retransmitted P3. When decoding P6, P6 is decoded and played with reference to P4. When decoding P7, P7 is decoded and played with reference to P5. Thus, the playback of this frame group is completed.

[0100] Based on Figure 4 the data transmission method provided in the corresponding embodiment, the embodiment of the present application also provides a data transmission device. Refer to Figure 7 . The device 700 includes a detection unit 701 and a decoding unit 702:

[0101] The detection unit 701 is used to detect the reception situation of the forward reference frame;

[0102] The detection unit 701 is configured to detect that a packet loss occurs to the i-th forward reference frame in the frame group during the transmission of the forward reference frames. The i-th forward reference frame is located in the first reference frame combination of the (i + T + 1)-th forward reference frame. The video frames in the first reference frame combination are located before the (i + T + 1)-th forward reference frame. The i-th forward reference frame is the video frame closest to the (i + T + 1)-th forward reference frame in the first reference frame combination, and there are T forward reference frames between the i-th forward reference frame and the (i + T + 1)-th forward reference frame. T is a positive integer determined according to the retransmission delay, so as to complete the retransmission of the i-th forward reference frame before decoding the (i + T + 1)-th forward reference frame; i is a positive integer greater than 0;

[0103] The decoding unit 702 is configured to, during video decoding, when decoding to the (i + 1)-th forward reference frame, decode the (i + 1)-th forward reference frame by using the second reference frame combination of the (i + 1)-th forward reference frame. The video frames in the second reference frame combination are located before the (i + 1)-th forward reference frame, and there are at least T forward reference frames between the second reference frame combination and the (i + 1)-th forward reference frame;

[0104] The decoding unit 702 is further configured to, when decoding to the (i + T + 1)-th forward reference frame, decode the (i + T + 1)-th forward reference frame by using the first reference frame combination including the retransmitted i-th forward reference frame.

[0105] In a possible implementation manner, the decoding unit 702 is further configured to:

[0106] When decoding to the i-th forward reference frame, multiplex the image corresponding to the target video frame to render the i-th forward reference frame, where the target video frame is the previous frame of the i-th forward reference frame.

[0107] In a possible implementation manner, the product of the frame duration of the forward reference frame and T + 1 is greater than the retransmission delay.

[0108] In a possible implementation manner, the retransmission delay is determined according to the duration of detecting packet loss of the forward reference frame, the network retransmission duration, the decoding duration of the forward reference frame, and the catch-up duration of the forward reference frame delay.

[0109] In a possible implementation manner, the first reference frame combination or the second reference frame combination is determined according to the identification information of the video frame, and the identification information is used to reflect the reference relationship between the video frames in the same frame group.

[0110] In a possible implementation manner, the first reference frame combination or the second reference frame combination is determined according to the refresh information of the frame group refresh and the identification information of the video frames included in the refreshed frame group.

[0111] An embodiment of the present application further provides a data transmission device. Refer to Figure 8 , the device 800 includes a determination unit 801, an encoding unit 802, a transmission unit 803, and a retransmission unit 804:

[0112] The determination unit 801 is configured to determine a reference frame combination corresponding to each video frame to be encoded in a frame group. The video frames in the reference frame combination are located before the corresponding video frame to be encoded and are at least separated from the corresponding video frame to be encoded by T video frames. T is a positive integer determined according to the retransmission delay, so as to complete the retransmission of the forward reference frame with packet loss in the reference frame combination before decoding the forward reference frame;

[0113] The encoding unit 802 is configured to perform inter-frame encoding on each video frame in sequence through the corresponding reference frame combination to obtain a corresponding forward reference frame;

[0114] The transmission unit 803 is configured to transmit the obtained forward reference frame to the receiving end;

[0115] The retransmission unit 804 is configured to, if receiving the acknowledgment information from the receiving end and the acknowledgment information indicates that the i-th forward reference frame has packet loss during the forward reference frame transmission, retransmit the i-th forward reference frame to the receiving end, where i is a positive integer greater than 0.

[0116] In a possible implementation manner, the product of the frame duration of the forward reference frame and T + 1 is greater than the retransmission delay.

[0117] In a possible implementation manner, the retransmission delay is determined according to the duration of detecting packet loss of the forward reference frame, the network retransmission duration, the decoding duration of the forward reference frame, and the catch-up duration of the forward reference frame delay.

[0118] In a possible implementation manner, the network retransmission duration is determined according to the network quality, and the device further includes an adjustment unit:

[0119] The adjustment unit is configured to adjust the T according to the network quality after completing the transmission of a frame group;

[0120] Alternatively, when a frame group refresh occurs, adjust the T according to the network quality.

[0121] Based on the above method, an embodiment of the present application further provides a device for data transmission. This device may be a terminal. Taking the terminal as a smart phone as an example:

[0122] Figure 9 Shown is a block diagram of a part of the structure of a smart phone related to the terminal provided by the embodiment of the present application. Refer toFigure 9 , the smart phone includes components such as a Radio Frequency (RF) circuit 910, a memory 920, an input unit 930, a display unit 940, a sensor 950, an audio circuit 960, a wireless fidelity (WiFi) module 970, a processor 980, and a power supply 990. The input unit 930 may include a touch panel 931 and other input devices 932. The display unit 940 may include a display panel 941. The audio circuit 960 may include a speaker 961 and a microphone 962. Those skilled in the art can understand that Figure 9 the structure of the smart phone shown in

[0123] does not limit the smart phone, and it may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0124] The memory 920 can be used to store software programs and modules. The processor 980 executes various functional applications and data processing of the smart phone by running the software programs and modules stored in the memory 920. The memory 920 mainly includes a program storage area and a data storage area. Among them, the program storage area can store an operating system, application programs required for at least one function (such as a sound playback function, an image playback function, etc.); the data storage area can store data created according to the use of the smart phone (such as audio data, phone book, etc.). In addition, the memory 920 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0125] In this embodiment, the processor 980 in the terminal may execute the following steps:

[0126] Detect the reception situation of the forward reference frame;

[0127] It is detected that the i-th forward reference frame in the frame group has a packet loss during the transmission of the forward reference frame. The video frame corresponding to the i-th forward reference frame is in the first reference frame combination of the (i + T + 1)-th forward reference frame. The video frames in the first reference frame combination are before the (i + T + 1)-th forward reference frame. The i-th forward reference frame is the video frame closest to the (i + T + 1)-th forward reference frame in the first reference frame combination, and there are T forward reference frames between the i-th forward reference frame and the (i + T + 1)-th forward reference frame. T is a positive integer determined according to the retransmission delay, so as to complete the retransmission of the i-th forward reference frame before decoding the (i + T + 1)-th forward reference frame; i is a positive integer greater than 0;

[0128] During the video decoding process, when decoding the (i + 1)-th forward reference frame, the second reference frame combination of the (i + 1)-th forward reference frame is used to decode the (i + 1)-th forward reference frame. The video frames in the second reference frame combination are before the (i + 1)-th forward reference frame and are at least T forward reference frames apart from the (i + 1)-th forward reference frame;

[0129] When decoding the (i + T + 1)-th forward reference frame, the first reference frame combination including the retransmitted i-th forward reference frame is used to decode the (i + T + 1)-th forward reference frame.

[0130] Or,

[0131] Determine the reference frame combination corresponding to each video frame to be encoded in the frame group. The video frames in the reference frame combination are before the corresponding video frame to be encoded and are at least T video frames apart from the corresponding video frame to be encoded. T is a positive integer determined according to the retransmission delay, so as to complete the retransmission of the forward reference frame with a packet loss in the reference frame combination before decoding the forward reference frame;

[0132] Perform inter-frame encoding on each video frame in sequence through the corresponding reference frame combination to obtain the corresponding forward reference frame, and transmit the obtained forward reference frame to the receiving end;

[0133] If the acknowledgment information from the receiving end is received and the acknowledgment information indicates that the i-th forward reference frame has a packet loss during the transmission of the forward reference frame, retransmit the i-th forward reference frame to the receiving end. i is a positive integer greater than 0.

[0134] The embodiment of the present application also provides a server. Please refer to Figure 10 as shown Figure 10The structural diagram of server 1000 provided by an embodiment of this application. Server 1000 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPUs) 1022 (for example, one or more processors) and a memory 1032, and one or more storage media 1030 (for example, one or more mass storage devices) storing application programs 1042 or data 1044. Among them, the memory 1032 and the storage media 1030 may be transient storage or persistent storage. The programs stored in the storage media 1030 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the server. Further, the central processing unit 1022 may be configured to communicate with the storage media 1030 and execute a series of instruction operations in the storage media 1030 on the server 1000.

[0135] Server 1000 may further include one or more power supplies 1026, one or more wired or wireless network interfaces 1050, one or more input / output interfaces 1058, and / or one or more operating systems 1041, such as Windows Server TM , Mac OS X TM , Unix TM , Linux TM , FreeBSD TM and so on.

[0136] In this embodiment, the central processing unit 1022 in the server may perform the following steps:

[0137] Determine the reference frame combination corresponding to each video frame to be encoded in the frame group. The video frames in the reference frame combination are located before the corresponding video frame to be encoded and are at least separated by T video frames from the corresponding video frame to be encoded. T is a positive integer determined according to the retransmission delay, so as to complete the retransmission of the forward reference frames with packet loss in the reference frame combination before decoding the forward reference frames;

[0138] Perform inter-frame encoding on each video frame in sequence through the corresponding reference frame combination to obtain the corresponding forward reference frame, and transmit the obtained forward reference frame to the receiving end;

[0139] If the acknowledgment information from the receiving end is received and the acknowledgment information indicates that the i-th forward reference frame has packet loss during the forward reference frame transmission, retransmit the i-th forward reference frame to the receiving end, where i is a positive integer greater than 0.

[0140] According to one aspect of the present application, there is provided a computer-readable storage medium for storing program code for executing the data transmission method described in each of the foregoing embodiments.

[0141] According to one aspect of the present application, there is provided a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the methods provided in various alternative implementations of the foregoing embodiments.

[0142] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not have to be used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order different from those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units does not have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0143] In several embodiments provided by the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection to each other can be through some interfaces, and the indirect coupling or communication connection of the device or unit can be in an electrical, mechanical or other form.

[0144] The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0145] In addition, in each embodiment of the present application, each functional unit may be integrated into a processing unit, may exist physically as individual units, or two or more units may be integrated into one unit. The above integrated unit may be implemented in the form of hardware or in the form of a software functional unit.

[0146] If the above integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it may be stored in a computer-readable storage medium. Based on such understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, may be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in each embodiment of the present application. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM), random access memories (RAM), magnetic disks, or optical discs that can store program codes.

[0147] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of each embodiment of the present application.

Claims

1. A data transmission method, characterized in that, The method includes: Detecting the reception status of forward reference frames; Detecting that a packet loss occurs to the i-th forward reference frame in the frame group during the process of transmitting the forward reference frame at the encoding end, the video frame corresponding to the i-th forward reference frame is located in the first reference frame combination of the (i + T + 1)-th forward reference frame, the video frames in the first reference frame combination are located before the (i + T + 1)-th forward reference frame, the i-th forward reference frame is the video frame closest to the (i + T + 1)-th forward reference frame in the first reference frame combination, and there are T forward reference frames between the i-th forward reference frame and the (i + T + 1)-th forward reference frame, where T is a positive integer determined according to the retransmission delay caused by retransmitting the lost forward reference frame at the encoding end, so as to complete the retransmission of the i-th forward reference frame before decoding the (i + T + 1)-th forward reference frame; i is a positive integer greater than 0; the product of the frame duration of the forward reference frame and T + 1 is greater than the retransmission delay; During the video decoding process, when decoding the (i + 1)-th forward reference frame, decoding the (i + 1)-th forward reference frame by using the second reference frame combination of the (i + 1)-th forward reference frame, the video frames in the second reference frame combination are located before the (i + 1)-th forward reference frame, and there are at least T forward reference frames between the video frames in the second reference frame combination and the (i + 1)-th forward reference frame; When decoding the (i + T + 1)-th forward reference frame, decoding the (i + T + 1)-th forward reference frame by using the first reference frame combination including the retransmitted i-th forward reference frame.

2. The method according to claim 1, wherein The method further includes: When decoding the i-th forward reference frame, multiplexing the image corresponding to the target video frame to render the i-th forward reference frame, and the target video frame is the previous frame of the i-th forward reference frame.

3. The method according to claim 1, wherein The retransmission delay is determined according to the duration of detecting packet loss of the forward reference frame, the network retransmission duration, the decoding duration of the forward reference frame, and the catch-up duration of the forward reference frame delay.

4. The method according to claim 1 or 2, characterized in that, The first reference frame combination or the second reference frame combination is determined according to the identification information of the video frame, and the identification information is used to reflect the reference relationship between video frames in the same frame group.

5. The method according to claim 4, characterized in that, The first reference frame combination or the second reference frame combination is determined according to the refresh information of frame group refresh and the identification information of the video frames included in the refreshed frame group.

6. A data transmission method, characterized in that, The method includes: Determining the reference frame combination corresponding to each video frame to be encoded in the frame group, the video frames in the reference frame combination are located before the corresponding video frame to be encoded, and there are at least T video frames between the video frames in the reference frame combination and the corresponding video frame to be encoded, where T is a positive integer determined according to the retransmission delay caused by retransmitting the lost forward reference frame at the encoding end, so as to complete the retransmission of the forward reference frame with packet loss in the reference frame combination before decoding the forward reference frame; the product of the frame duration of the forward reference frame and T + 1 is greater than the retransmission delay; Performing inter-frame encoding on each video frame in sequence through the corresponding reference frame combination to obtain the corresponding forward reference frame, and transmitting the obtained forward reference frame to the receiving end; If the response information of the receiving end is received, and the response information indicates that the i-th forward reference frame is lost during the transmission of the forward reference frame at the encoding end, the i-th forward reference frame is retransmitted to the receiving end, where i is a positive integer greater than 0.

7. The method according to claim 6, characterized in that, The retransmission delay is determined according to the duration of detecting the loss of the forward reference frame, the network retransmission duration, the decoding duration of the forward reference frame, and the catch-up duration of the forward reference frame delay.

8. The method according to claim 7, characterized in that The network retransmission duration is determined according to the network quality, and the method further includes: After completing the transmission of a group of frames, adjusting the T according to the network quality; Alternatively, when a group of frames is refreshed, adjusting the T according to the network quality.

9. A data transmission device, characterized in that, The device includes a detection unit and a decoding unit: The detection unit is configured to detect the reception situation of the forward reference frame; The detection unit is configured to detect that the i-th forward reference frame in the group of frames is lost during the transmission of the forward reference frame at the encoding end. The i-th forward reference frame is located in the first reference frame combination of the (i + T + 1)-th forward reference frame. The video frames in the first reference frame combination are located before the (i + T + 1)-th forward reference frame. The i-th forward reference frame is the video frame closest to the (i + T + 1)-th forward reference frame in the first reference frame combination, and there are T forward reference frames between the i-th forward reference frame and the (i + T + 1)-th forward reference frame. T is a positive integer determined according to the retransmission delay caused by retransmitting the lost forward reference frame at the encoding end, so as to complete the retransmission of the i-th forward reference frame before decoding the (i + T + 1)-th forward reference frame; i is a positive integer greater than 0; the product of the frame duration of the forward reference frame and T + 1 is greater than the retransmission delay; The decoding unit is configured to, during video decoding, when decoding to the (i + 1)-th forward reference frame, decode the (i + 1)-th forward reference frame by using the second reference frame combination of the (i + 1)-th forward reference frame. The video frames in the second reference frame combination are located before the (i + 1)-th forward reference frame and are at least T forward reference frames away from the (i + 1)-th forward reference frame; The decoding unit is further configured to, when decoding to the (i + T + 1)-th forward reference frame, decode the (i + T + 1)-th forward reference frame by using the first reference frame combination including the retransmitted i-th forward reference frame.

10. The device according to claim 9, characterized in that, The decoding unit is further configured to: When decoding to the i-th forward reference frame, multiplex the image corresponding to the target video frame to render the i-th forward reference frame, where the target video frame is the previous frame of the i-th forward reference frame.

11. The device according to claim 9, characterized in that, The retransmission delay is determined according to the duration of detecting the loss of the forward reference frame, the network retransmission duration, the decoding duration of the forward reference frame, and the catch-up duration of the forward reference frame delay.

12. The device according to claim 9 or 10, characterized in that, The first reference frame combination or the second reference frame combination is determined according to the identification information of the video frame, and the identification information is used to reflect the reference relationship between the video frames in the same group of frames.

13. The device according to claim 12, characterized in that, The first reference frame combination or the second reference frame combination is determined according to the refresh information of the group of frames and the identification information of the video frames included in the refreshed group of frames.

14. A data transmission device, characterized in that, The device includes a determination unit, an encoding unit, a transmission unit, and a retransmission unit: The determination unit is configured to determine a reference frame combination corresponding to each video frame to be encoded in a frame group. The video frames in the reference frame combination are located before the corresponding video frame to be encoded and are at least separated by T video frames from the corresponding video frame to be encoded. T is a positive integer determined according to the retransmission delay caused by the forward reference frame of the retransmitted packet at the encoding end, so as to complete the retransmission of the forward reference frame with a packet loss in the reference frame combination before decoding the forward reference frame; the product of the frame duration of the forward reference frame and T + 1 is greater than the retransmission delay; The encoding unit is configured to perform inter-frame encoding on each video frame in sequence through the corresponding reference frame combination to obtain a corresponding forward reference frame; The transmission unit is configured to transmit the obtained forward reference frame to the receiving end; The retransmission unit is configured to, if receiving the response information from the receiving end and the response information indicates that the i-th forward reference frame has a packet loss during the transmission of the forward reference frame at the encoding end, retransmit the i-th forward reference frame to the receiving end, where i is a positive integer greater than 0.

15. The device according to claim 14, characterized in that, The retransmission delay is determined according to the duration of detecting the packet loss of the forward reference frame, the network retransmission duration, the decoding duration of the forward reference frame, and the catch-up duration of the forward reference frame delay.

16. The device according to claim 15, characterized in that, The network retransmission duration is determined according to the network quality, and the device further includes an adjustment unit; The adjustment unit is configured to, after completing the transmission of a frame group, adjust the T according to the network quality; or, when a frame group refresh occurs, adjust the T according to the network quality.

17. A device for data transmission, characterized in that, The device includes a processor and a memory: The memory is configured to store program code and transmit the program code to the processor; The processor is configured to execute the method according to any one of claims 1-5, or the method according to any one of claims 6-8 according to the instructions in the program code.

18. A computer-readable storage medium, characterized in that, The computer-readable storage medium is configured to store program code, and the program code is used to execute the method according to any one of claims 1-5, or the method according to any one of claims 6-8.

19. A computer program product, characterized in that, The computer program product includes computer instructions, and the computer instructions are stored in a computer-readable storage medium; the processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method according to any one of claims 1-5, or the method according to any one of claims 6-8.

Images