Frame loss recovery method and device, data sending equipment, data receiving equipment and storage medium

Abstract

The invention discloses a frame loss recovery method and device, data sending equipment, data receiving equipment and a storage medium, and belongs to the technical field of data transmission. The method is applied to a data sending device, and comprises the following steps: sending at least one data frame to a data receiving device, the at least one data frame comprising a long reference frame LTR added with a target reference identifier; and when a frame loss recovery request sent by the data receiving equipment is received, selecting a target LTR from the at least one LTR added with the target reference identifier, generating an LTR-P frame according to the target LTR, and sending the LTR-P frame to the data receiving equipment. According to the method and the device, the target reference identifier is added in the at least one sent data frame in advance, and when the frame loss recovery request is subsequently received, one target LTR is selected from the LTRs in the target reference identifier as the reference frame to generate the LTR-P frame, so that the fluency of data transmission during frame loss recovery is improved.

Description

Technical Field

[0001] The present application relates to the technical field of data transmission, and in particular to a frame loss recovery method, apparatus, data sending device, data receiving device and storage medium. Background Art

[0002] With the development of science and technology, various terminal devices appear in people's daily lives. These terminal devices can transmit various audio data and video data by establishing communication connections.

[0003] At present, when a terminal device is transmitting audio data or video data, data loss often occurs on the side of the data receiving device due to various reasons. For example, in most video transmission processes, the data sending device will use I frames or P frames to encode the video frames to be transmitted. In this video transmission scenario, if the data receiving device encounters delays, freezes, or packet loss, some data will be lost. In order to ensure the integrity of data transmission, some terminal devices can initiate a packet retransmission request. The data sending device will reselect the reference frame based on the packet retransmission request and re-encode the I frame to send to the data receiving device. In this process, the algorithm for reselecting the reference frame is usually too complicated, and the amount of data required to be transmitted is large, which affects the video smoothness of the frame loss recovery. Summary of the Invention

[0004] To address the problems of related technologies, improve the efficiency of a data transmission device in selecting reference frames, and enhance the video smoothness of the data transmission device in responding to frame loss recovery, the present invention provides a frame loss recovery method, apparatus, data transmission device, and storage medium. The technical solution is as follows:

[0005] In one aspect, the present application provides a frame loss recovery method, applied to a data sending device, the method comprising:

[0006] Sending at least one data frame to a data receiving device, wherein the at least one data frame includes a long reference frame LTR added with a target reference identifier;

[0007] When receiving the frame loss recovery request sent by the data receiving device, a target LTR is selected from at least one LTR added with the target reference identifier, and an LTR-P frame is generated according to the target LTR and sent to the data receiving device.

[0008] In one aspect, the present application provides a frame loss recovery method, applied to a data receiving device, the method comprising:

[0009] receiving at least one data frame sent by a data sending device, wherein the at least one data frame includes a long reference frame LTR to which a target reference identifier is added;

[0010] In the event of data frame loss, a frame loss recovery request is sent to the data sending device, where the frame loss recovery request is used to instruct the data sending device to select a target LTR from at least one LTR to which the target reference identifier is added, so that the data sending device generates an LTR-P frame based on the target LTR and sends the LTR-P frame to the data receiving device.

[0011] In one aspect, the present application provides a frame loss recovery device, applied to a data sending device, the device comprising:

[0012] A data sending module, configured to send at least one data frame to a data receiving device, wherein the at least one data frame includes a long reference frame LTR to which a target reference identifier is added;

[0013] The reference frame selection module is used to select a target LTR from at least one LTR added with the target reference identifier when receiving a frame loss recovery request sent by the data receiving device, and generate an LTR-P frame according to the target LTR and send the LTR-P frame to the data receiving device.

[0014] In one aspect, the present application provides a frame loss recovery device, applied to a data receiving device, the device comprising:

[0015] A first receiving module is configured to receive at least one data frame sent by a data sending device, wherein the at least one data frame includes a long reference frame LTR to which a target reference identifier is added;

[0016] A request sending module is used to send a frame loss recovery request to a data sending device in the event of data frame loss. The frame loss recovery request is used to instruct the data sending device to select a target LTR from at least one LTR to which the target reference identifier is added, so that the data sending device generates an LTR-P frame according to the target LTR and sends the LTR-P frame to the data receiving device.

[0017] In another aspect, the present application provides a data sending device, which includes a processor and a memory, wherein the memory stores a computer program, and the computer program is executed by the processor to implement the frame loss recovery method as described in one aspect.

[0018] In another aspect, the present application provides a data receiving device, which includes a processor and a memory, wherein the memory stores a computer program, and the computer program is executed by the processor to implement the frame loss recovery method as described in one aspect.

[0019] In another aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program is executed by a processor to implement the frame loss recovery method as described in one aspect.

[0020] On the other hand, an embodiment of the present application provides a computer program product. When the computer program product is run on a computer, the computer is caused to execute the frame loss recovery method as described in one aspect above.

[0021] On the other hand, an embodiment of the present application provides an application publishing platform, which is used to publish a computer program product. When the computer program product runs on a computer, the computer executes the frame loss recovery method described in one aspect above.

[0022] The beneficial effects of the technical solutions provided in the embodiments of the present application include at least:

[0023] A data transmitting device transmits at least one data frame to a data receiving device, wherein the at least one data frame includes a long reference frame (LTR) with a target reference identifier added thereto. Upon receiving a lost frame recovery request from the data receiving device, a target LTR is selected from the at least one LTR with the target reference identifier added thereto, and an LTR-P frame is generated based on the target LTR and transmitted to the data receiving device. The present application pre-adds a target reference identifier to at least one transmitted data frame, and upon subsequent receipt of a lost frame recovery request, a target LTR is selected from the LTRs in the target reference identifier as a reference frame to generate the LTR-P frame. This process is simple and convenient, does not require the design of complex algorithms for calculation, reduces the amount of data that needs to be encoded by the data transmitting device, and improves the smoothness of data transmission when lost frame recovery occurs. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0025] Figure 1 This is a schematic diagram of an example of a terminal device provided in an embodiment of the present application;

[0026] Figure 2 This is a schematic diagram of a data transmission scenario involved in an exemplary embodiment of the present application;

[0027] Figure 3 This is a schematic diagram of a process of requesting I frame recovery according to an exemplary embodiment of the present application;

[0028] Figure 4 This is a flowchart of requesting LTR recovery according to an exemplary embodiment of the present application;

[0029] Figure 5 This is a flow chart of a frame loss recovery method provided by an exemplary embodiment of the present application;

[0030] Figure 6 This is a flow chart of a frame loss recovery method provided by an exemplary embodiment of the present application;

[0031] Figure 7 This is a flow chart of a frame loss recovery method provided by an exemplary embodiment of the present application;

[0032] Figure 8 This is an interactive flow chart of a frame loss recovery method provided by an exemplary embodiment of the present application;

[0033] Figure 9 This is a structural block diagram of a frame loss recovery device provided by an exemplary embodiment of the present application;

[0034] Figure 10 This is a structural block diagram of a frame loss recovery device provided by an exemplary embodiment of the present application;

[0035] Figure 11 This is a structural diagram of another example of a lost frame recovery device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0036] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0037] In this document, "plurality" refers to two or more. "And / or" describes a relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone, A and B exist simultaneously, or B exists alone. The character " / " generally indicates an "or" relationship between the associated objects.

[0038] The solution provided in this application can be used in real-life scenarios where people use terminal devices to transmit data in their daily lives. To facilitate understanding, the following first briefly introduces the application scenarios involved in the embodiments of this application.

[0039] Frame loss recovery: refers to a fallback strategy initiated by the receiving device after it has failed to request a data packet through data retransmission requests.

[0040] With the development of science and technology, various terminal devices have appeared in people's daily lives. People can use terminal devices for work, entertainment, study, etc. Communication connections can be established between terminal devices to realize the transmission of various multimedia data through communication connections.

[0041] Please refer to Figure 1 , which is a schematic diagram of an example of the structure of the terminal device provided in the embodiment of the present application. Figure 1 The terminal device shown includes components such as a processor 110, a memory 120, a transceiver 130, a display unit 140, an input unit 150, a sensor 160, an audio circuit 170, and a battery module 180.

[0042] Processor 110 is the control center of the terminal device. It connects the various components of the entire terminal device using various interfaces and lines. By running or executing software programs and / or modules stored in memory 120 and accessing data stored in memory 120, it performs various terminal device functions and processes data, thereby monitoring the terminal device as a whole. Optionally, processor 110 may include one or more processing units; alternatively, processor 110 may integrate an application processor, which primarily processes operating devices, user interfaces, and application programs. Of course, other processors may also be included, which are not listed here.

[0043] The memory 120 can be used to store software programs and modules. The processor 110 executes various functional applications and data processing of the terminal device by running the software programs and modules stored in the memory 120. The memory 120 may mainly include a program storage area and a data storage area. The program storage area may store operating devices, at least one application required for a function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created based on the use of the terminal device (such as audio data, a phone book, etc.). In addition, the memory 120 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage device.

[0044] The transceiver 130 can provide wireless communication solutions for terminal devices, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc. The transceiver 130 can be one or more devices that integrate at least one communication processing module, for example, an antenna and a baseband processor are integrated into the transceiver 130, or an antenna and a modem processor are integrated into the transceiver 130, etc., and there is no limitation here.

[0045] The display unit 140 can be used to display information input by the user or information provided to the user, as well as various menus of the terminal device. The display unit 140 can be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), etc., without limitation.

[0046] The input unit 150 can be used to receive input digital or character information, and to generate key signal input related to the user settings and function control of the terminal device. Specifically, the input unit 150 can collect the user's operations on or near it, and drive the corresponding connection device according to a pre-set program. In addition, the input unit 150 may include a touch panel, which can be implemented using various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch panel, the input unit 150 may also include other input devices. Specifically, other input devices may include but are not limited to one or more of function keys (such as volume control buttons, switch buttons, etc.), trackballs, joysticks, etc.

[0047] The terminal device may also include at least one sensor 160, such as a gyroscope, motion sensor, or other sensor. The motion sensor may include an accelerometer, which detects the magnitude of acceleration in various directions and the magnitude and direction of gravity when stationary. This can be used for applications that recognize the terminal device's posture, such as landscape or portrait screen switching, related games, and magnetometer posture calibration. Other sensors that may be configured in the terminal device, such as a pressure gauge, barometer, hygrometer, thermometer, and infrared sensor, are not described here.

[0048] Audio circuit 170 may include a speaker and a microphone, providing an audio interface between the user and the terminal device. Audio circuit 170 converts received audio data into electrical signals and transmits them to the speaker, which then converts them into sound signals for output. The microphone, on the other hand, converts collected sound signals into electrical signals, which are then received by audio circuit 170 and converted into audio data. The audio data is then output to processor 110 for processing, then transmitted via the video circuit to, for example, another terminal device, or to memory 120 for further processing.

[0049] The terminal device also includes a battery module 180 for supplying power to various components. Optionally, the battery module 180 can be logically connected to the processor 110 through a power management device, thereby managing functions such as charging, discharging, and power consumption through the power management device.

[0050] Although not shown, the terminal device may further include a camera. Optionally, the camera may be positioned in the front or rear of the terminal device, which is not limited in this embodiment of the present application.

[0051] It is understood that the structures illustrated in the embodiments of the present application do not constitute specific limitations on the terminal device. In other embodiments of the present application, the terminal device may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0052] Optional, above Figure 1 The terminal devices shown can establish communication connections with each other or with a server and transmit multimedia data. Figure 2 , which shows a schematic diagram of a data transmission scenario involved in an exemplary embodiment of the present application. Figure 2 As shown, it includes various terminal devices 210, servers 220, and network providing devices 230.

[0053] The terminal device 210 may be an electronic device with a data transmission function. For example, the terminal device may include but is not limited to a wearable device (such as a wristband, a smart watch, smart glasses, etc.), a mobile phone, a tablet computer, a laptop computer, smart glasses, a smart watch, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 player (Moving Picture Experts Group Audio Layer IV), a desktop computer, a laptop computer, a smart home device, etc. Alternatively, the terminal device may be a front-end video device such as an IP camera (IP Camera, IPC), which can transmit video frames.

[0054] Server 220 can include at least one of a single server, multiple servers, a cloud computing platform, and a virtualization center. Server 220 provides backend services for applications that support virtual environments. Optionally, server 220 can handle primary computing tasks, while terminal device 210 handles secondary computing tasks. Alternatively, server 220 can handle secondary computing tasks, while terminal device 210 handles primary computing tasks. Alternatively, server 220 and terminal device 210 can collaborate on computing tasks using a distributed computing architecture.

[0055] The network providing device 230 may be a WiFi device, a wireless access point (AP) device, a base station, etc. in a home environment.

[0056] Each terminal device 210 can interact with each other or with the server 220 by establishing a wireless communication connection with the network providing device 230. Optionally, a communication connection is established between the terminal device and the server, and then data such as images and videos are transmitted through the communication connection. The communication connection can be referred to as a communication network or a network connection, and the communication connection uses standard communication technology and / or protocols. The network is usually the Internet, but it can also be any network, including but not limited to a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a mobile, wired or wireless network, a private network or any combination of a virtual private network). In some embodiments, technologies and / or formats including Hyper Text Mark-up Language (HTML), Extensible Markup Language (XML), etc. are used to represent data exchanged through the network. In addition, conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), Internet Protocol Security (IPsec), etc. may be used to encrypt all or some links. In other embodiments, customized and / or dedicated data communication technologies may be used instead of or in addition to the above data communication technologies.

[0057] At present, video communication is a common communication method, which can provide live broadcast, video news, video conference and other contents. Figure 2When transmitting video data between various terminal devices that have established communication connections, it is extremely important to ensure that the video stream is transmitted in real time and efficiently. Among them, in order to ensure low latency in real-time playback, there are usually only two types of frames in this type of video: I-frames and P-frames. I-frames refer to the compression of all data, and the amount of data that needs to be encoded is large, while P-frames are differentially encoded based on the previous I-frames, and the amount of data that needs to be encoded is relatively small. For example, in a real-time audio and video communication (RTC) transmission scenario, if the terminal device as the receiving end cannot always receive the Real-time Transport Protocol (RTP) data packet sent by the sending end through packet loss retransmission, data loss will occur at the receiving end. The terminal device as the receiving end can trigger an Instantaneous Decoding Refresh (IDR) request to ensure subsequent correct decoding and playback.

[0058] Please refer to Figure 3 , in fact, it shows a schematic diagram of a process of requesting I frame recovery involved in an exemplary embodiment of the present application. Figure 3 As shown, the sender sends the data of each I frame and P frame to the receiver. In RTC real-time audio and video transmission, if the receiver cannot receive the RTP data packet of the sender through packet loss and retransmission, it will cause data loss at the receiver (for example, Figure 3 The P frame with a cross in it) triggers an IDR refresh request (equivalent to Figure 3 The sender retransmits the frame to ensure correct decoding and playback at the receiver. IDR refresh requests can only use intra-frame prediction, which is inefficient. When retransmission is requested, the network is often already poor, and retransmitting the frame further exacerbates network congestion.

[0059] The purpose of the long reference LTR-P frame refresh strategy is to ensure that the receiver can correctly decode and play the video without re-requesting the I frame in the case of P frame loss. Compared with I frame, it can significantly improve the encoding efficiency, save bandwidth, and improve the video smoothness in weak network scenarios. Figure 4 , which actually shows a flow chart of requesting LTR recovery involved in an exemplary embodiment of the present application. Figure 4 As shown, the transmitter sends the data of each I frame and P frame to the receiver. When the receiver loses data, it requests LTR recovery so that the transmitter can retransmit based on the LTR frame. Among them, the LTR frame is essentially a reference frame in the resident reference frame queue. For hardware encoding and decoding, the definition of the long reference frame LTR is to record the key frames that have been received and decoded by the receiver into the reference frame list. When the encoder receives packet loss feedback or detects a scene switch, the use of the long reference frame LTR is triggered. Among them, Figure 4 The straight arrow in the middle may represent a forward reference frame of a P frame after requesting LTR recovery.

[0060] In the above scheme, since IDR refresh request can only use intra-frame prediction, the coding efficiency is low. When retransmission is requested, the network is usually not good. Retransmitting I frame will aggravate the network congestion. Therefore, some manufacturers have adopted the long reference frame LTR scheme, that is, according to Figure 4 This approach can be used to recover the video. This prevents widespread errors from causing undecodeable content on the decoder side, while also minimizing the significant bit consumption associated with re-encoding I-frames. However, existing algorithms for reselecting reference frames are often overly complex, requiring large amounts of data to be transmitted, and impacting video smoothness during frame recovery.

[0061] In order to solve the problems existing in the above-mentioned related technologies, improve the efficiency of selecting reference frames during data transmission, reduce the amount of transmitted data during frame loss recovery, and speed up the smoothness of data transmission when frame loss recovery occurs, the present application provides a frame loss recovery method, which can add a target reference identifier to the data frame in advance. If the data receiving device initiates a frame loss recovery request, the data sending device selects one from the LTR frames with the target reference identifier as the target LTR, quickly generates an LTR-P frame and sends it to the data receiving device, without the need for excessive calculations and other processes.

[0062] Please refer to Figure 5 , which shows a method flow chart of a frame loss recovery method provided by an exemplary embodiment of the present application. The frame loss recovery method can be applied to a data sending device, which can be the above-mentioned Figure 1 or Figure 2 The terminal device in Figure 5 As shown, the frame loss recovery method may include the following steps:

[0063] Step 501: Send at least one data frame to a data receiving device, wherein the at least one data frame includes a long reference frame LTR with a target reference identifier added thereto.

[0064] The at least one data frame may be a data frame generated during interaction between the data sending device and the data receiving device. For example, if the transmitted multimedia data is video data, the at least one data frame may be a video frame.

[0065] At least one data frame is a video frame. A data transmitting device and a data receiving device communicate via a communication connection. When transmitting video data, I-frames and P-frames may be used for transmission encoding of the video frames. When the video frames are transmitted to the data receiving device, the transmission information of the video I-frames or P-frames includes, but is not limited to, a timestamp of the video frame, a peak traffic volume, and a device identifier of each terminal device.

[0066] Optionally, the data transmitting device may add a target reference identifier to one or more data frames in at least one data frame. The target reference identifier may indicate that the data frame is a long reference frame (LTR). The data transmitting device may add the target reference identifier during or after encoding the video frame, and this application does not limit this.

[0067] Step 502: upon receiving a frame loss recovery request sent by a data receiving device, a target LTR is selected from at least one LTR with a target reference identifier added thereto, and an LTR-P frame is generated according to the target LTR and sent to the data receiving device.

[0068] Optionally, after transmitting at least one data frame, if the data receiving device experiences packet loss or other circumstances, triggering a frame loss recovery request, the data transmitting device will receive the frame loss recovery request sent by the data receiving device and then select a target LTR from at least one LTR with a target reference identifier. For example, if each data frame previously transmitted to the data receiving device includes a first LTR and a second LTR with a target reference identifier, upon receiving the frame loss recovery request from the data receiving device, the data transmitting device may select one of the first and second LTRs as the target LTR, generate an LTR-P frame based on the target LTR, and transmit the LTR-P frame to the data receiving device.

[0069] That is to say, in this solution, the data sending device marks the selectable LTR frames in advance through the target reference identifier. When frame loss recovery is required, one of the LTRs with the target reference identifier is selected as the target LTR, and the target LTR is used to generate an LTR-P frame. The LTR-P frame is sent to the data receiving device to complete the frame loss recovery effect without the need for a large amount of data calculation and other processes.

[0070] In summary, a data sending device sends at least one data frame to a data receiving device, wherein the at least one data frame includes a long reference frame LTR with a target reference identifier added thereto; upon receiving a lost frame recovery request sent by the data receiving device, a target LTR is selected from the at least one LTR with the target reference identifier added thereto, and an LTR-P frame is generated based on the target LTR and sent to the data receiving device. The present application adds a target reference identifier to at least one data frame sent in advance, and upon subsequent receipt of a lost frame recovery request, a target LTR is selected from the LTRs in the target reference identifier as a reference frame to generate an LTR-P frame. This process is simple and convenient, does not require the design of a complex algorithm for calculation, reduces the amount of data that needs to be encoded by the data sending device, and improves the smoothness of data transmission when lost frame recovery occurs.

[0071] In one possible implementation, before sending at least one data frame to the data receiving device, the data sending device needs to add a target reference identifier in advance and receive feedback information from the data receiving device. When selecting the target LTR, it selects from the LTRs successfully transmitted to the data receiving device, thereby ensuring the accuracy of the data.

[0072] Please refer to Figure 6 , which shows a method flow chart of a frame loss recovery method provided by an exemplary embodiment of the present application. The frame loss recovery method can be applied to a data sending device, which can be the above-mentioned Figure 1 or Figure 2 The terminal device in Figure 6 As shown, the frame loss recovery method may include the following steps:

[0073] Step 601: Add a target reference identifier to at least one data frame.

[0074] Optionally, when the data sending device sends a data frame to the data receiving device, it can add a target reference identifier in the data frame to indicate that the data frame is an LTR frame. For example, the target reference identifier can be an index identifier. After encoding the data frame, the data sending device can establish a correspondence between the index of the LTR frame and the encoding output counter (output counter), encapsulate the output counter of the LTR frame into an index identifier, use Sei to represent the index identifier, and use Sei to indicate that the current data frame is an LTR frame. The Sei identifier target reference identifier here is also exemplary. In actual applications, a combination of names such as letters, numbers, and symbols can also be used as the target reference identifier to achieve the effect of adding to the data frame.

[0075] In one possible implementation, the data transmitting device may perform the addition as follows: According to a preset addition period, the target reference identifier is added to at least one data frame that meets the preset addition period. For example, if the preset addition period is 2 seconds, the data transmitting device may mark a data frame every 2 seconds, i.e., add the target reference identifier to the data frame. In other words, the data transmitting device may add the target reference identifier to the data frame at a fixed interval, adding the target reference identifier to the current data frame every time the preset addition period passes, thereby marking the long reference frame (LTR).

[0076] In one possible implementation, the data transmitting device may further perform the addition in the following manner: obtaining a degree of picture change between the current data frame and the previous long reference frame (LTR) to which the target reference identifier is added; and when the degree of picture change is greater than or equal to a preset change threshold, adding the target reference identifier to the current data frame. Specifically, the data transmitting device calculates the picture change between the current data frame and the previous long reference frame (LTR) to which the target reference identifier is added, uses the picture change degree to represent the picture change between two adjacent data frames, and adds the target reference identifier to the current data frame if the degree of picture change is greater than or equal to the preset change threshold.

[0077] That is to say, the data sending device of this scheme can choose two schemes to add a target reference identifier to a certain data frame. The first scheme is to add a target reference identifier to a data frame at a fixed interval of the preset adding period according to the normal preset adding period, and mark it as a long reference frame LTR. The second scheme is based on the degree of picture change between the current data frame and the previous long reference frame LTR to which the target reference identifier is added. If the degree of picture change is greater than or equal to the preset change threshold, the target reference identifier is added to the current data frame and the long reference frame LTR is marked. In addition, these two schemes can be combined. For example, on the basis of marking the long-term reference frame at intervals, the frame with obvious changes in picture content is marked as a long-term reference frame and placed in the long reference frame LTR reference queue.

[0078] Optionally, the data sending device may obtain the degree of picture change between the current data frame and the previous long reference frame LTR to which the target reference identifier is added as follows: obtain histogram information of each of the current data frame and the previous long reference frame LTR to which the target reference identifier is added; compare the histogram information of the current data frame with the previous long reference frame LTR to which the target reference identifier is added, and obtain picture change information of the current data frame relative to the previous long reference frame LTR to which the target reference identifier is added; and obtain the degree of picture change based on the picture change information. This process may be performed before the data sending device completes encoding of the current data frame. For example, for each data frame encoded by the data sending device, the data frame is the current data frame. The data sending device may obtain the histogram information of the current data frame and compare it with the histogram information of the previous long reference frame LTR to which the target reference identifier is added, to obtain picture change information of the current data frame relative to the previous long reference frame LTR to which the target reference identifier is added, and obtain the degree of picture change based on the picture change information. The picture change information may be each pixel point, pixel coordinate, image position, picture content, etc. of the current data frame that has changed relative to the previous long reference frame LTR to which the target reference identifier is added.

[0079] It should be noted that the above histogram information is exemplary. In practical applications, the Peak Signal to Noise Ratio (PSNR) of the data frame can also be used for comparison to obtain the picture change information of the current data frame relative to the previous long reference frame LTR to which the target reference identifier is added; or the structural similarity index (SSIM) of the data frame can be used for comparison to obtain the picture change information of the current data frame relative to the previous long reference frame LTR to which the target reference identifier is added. The specific image parameters used are not limited in this solution.

[0080] Step 602: Send at least one data frame to a data receiving device, wherein the at least one data frame includes a long reference frame LTR with a target reference identifier added thereto.

[0081] Optionally, the data transmitting device transmits the encoded data frame to the data receiving device. If a target reference identifier is added to the data frame, each transmitted data frame includes a long reference frame (LTR) to which the target reference identifier is added. For example, during data transmission between the data transmitting device and the data receiving device, the data transmitting device transmits each encoded data frame to the data receiving device. Therefore, at least one data frame transmitted during the transmission process includes the long reference frame (LTR) to which the target reference identifier is added.

[0082] Optionally, the manner in which the data sending device and the data receiving device transmit data frames may refer to the above-mentioned existing transmission manner, which is not limited here.

[0083] Step 603: Receive feedback information from the data receiving device, where the feedback information is used to indicate that the data receiving device has received the long reference frame LTR with the target reference identifier added thereto.

[0084] Optionally, during the aforementioned process of sending each data frame, if a data frame containing a target reference identifier is sent, the data receiving device will send feedback information to the data sending device, thereby indicating whether it has successfully received the long reference frame (LTR) containing the target reference identifier. Still taking the target reference identifier as an index identifier (Sei) as an example, after encapsulating a data frame with an Sei, the data sending device can insert the Sei into the code stream and package it with the Real-time Transport Protocol (RTP) to send to the data receiving device. If the data receiving device receives the LTR frame, it can reply with feedback information to inform the data sending device that it has successfully received the LTR frame containing the target reference identifier. Optionally, the feedback information can be pre-set in the protocol based on the principle of positive feedback, and the data receiving device can reply with feedback information to the successfully received LTR frame containing the target reference identifier. For example, the feedback information can be provided by the data receiving device via a custom payload-specific feedback packet (PSFB) message of the RTCP protocol.

[0085] For example, in the process of sending edited data frames to the data receiving device in sequence, data frames one to five are sent within a period of time. If the data frames containing the target reference identifier are data frames two and four respectively, then if the data receiving device successfully receives data frames two and four, it will feed back corresponding feedback information upon receipt. If only data frame two is received, the data sending device will only receive the feedback information fed back by the data receiving device for data frame two.

[0086] Step 604: upon receiving the frame loss recovery request sent by the data receiving device, a target LTR is selected from at least one LTR with a target reference identifier added thereto, and an LTR-P frame is generated according to the target LTR and sent to the data receiving device.

[0087] Optionally, when packet loss or other phenomena occur and video frames need to be retransmitted, the data receiving device can send a frame loss recovery request to the data sending device. In response to the frame loss recovery request, the data sending device selects a target LTR from at least one LTR with a target reference identifier added.

[0088] Optionally, a real-time audio and video communication RTC connection is established between the data sending device and the data receiving device, and the frame loss recovery request is a key frame retransmission request (Intra Frame Request, FIR) of the RTP Control Protocol (RTP Control Protocol, RTCP).

[0089] That is, after receiving the lost frame recovery request sent by the data receiving device, the data sending device selects a target LTR from the LTRs to which the target reference identifier was previously added. In one possible implementation, when selecting the target LTR from the LTRs to which the target reference identifier was previously added, the data sending device needs to select from the LTRs successfully received by the data receiving device. As can be seen from the feedback mechanism established above, the data receiving device will provide feedback for each LTR to which the target reference identifier is added. Therefore, the data sending device, based on the received feedback information, selects the target LTR from the LTRs with target reference identifiers corresponding to the received feedback information.

[0090] For example, the data frames sent by the data transmitting device include data frame 1, data frame 2, data frame 3, data frame 4, data frame 5, data frame 6, data frame 7, and data frame 8, wherein data frame 4, data frame 6, and data frame 8 are LTR frames with target reference identifiers added. In this step, after receiving the frame loss recovery request sent by the data receiving device, the target LTR is selected from the LTRs with target reference identifiers added, that is, the target LTR is selected from data frame 4, data frame 6, and data frame 8. When receiving feedback information, if the data transmitting device receives feedback information from the data receiving device for data frame 4, data frame 6, and data frame 8, then the target LTR can be selected from data frame 4, data frame 6, and data frame 8. If the data transmitting device only receives feedback information from the data receiving device for data frame 4 and data frame 8, then the target LTR can be selected from data frame 4 and data frame 8.

[0091] Alternatively, the data transmitting device may select a target LTR from among the LTRs with target reference identifiers in the following manner. For example, the data transmitting device may obtain the current data frame; and from among the LTRs with target reference identifiers for which feedback information has been received, determine the LTR with the highest similarity to the current data frame as the target LTR. In other words, based on the current data frame, the data transmitting device determines the similarity between the LTRs with target reference identifiers for which feedback information has been received and the current data frame, and determines the LTR with the highest similarity as the target LTR.

[0092] For example, the data sending device needs to select a target LTR from data frame four, data frame six and data frame eight. When selecting, the similarity between the current data frame and data frame four, data frame six and data frame eight is calculated respectively. If the similarity between data frame six and the current data frame is the highest in each calculation result, then data frame six is used as the target LTR determined this time.

[0093] Optionally, the similarity can be obtained by using the histogram information of the current data frame and the histogram information of each LTR with the target reference identifier, or by using the PSNR of the current data frame and the PSNR of each LTR with the target reference identifier, or by using the SSIM of the current data frame and the SSIM of each LTR with the target reference identifier. That is, the data transmitting device can traverse and calculate the similarity between the current data frame and each LTR to be selected, thereby determining the target LTR.

[0094] In one possible implementation, the data transmitting device may select a target LTR from among the LTRs with target reference identifiers as follows. For example, the data transmitting device obtains the current data frame; from among the LTRs with target reference identifiers for which feedback information has been received, the LTR with the closest time interval to the current data frame is determined as the target LTR. In other words, based on the current data frame, the data transmitting device selects the LTR frame that is temporally adjacent to the current data frame as the target LTR. For example, if the data transmitting device needs to select a target LTR from data frame four, data frame six, and data frame eight, and if data frame eight is temporally closest to the current data frame, the data transmitting device selects data frame eight as the target LTR.

[0095] In one possible implementation, upon receiving a lost frame recovery request from a data receiving device, if the data transmitting device is unable to select a target LTR from at least one LTR with a target reference identifier, the data transmitting device may also retransmit an I-frame to the data receiving device by feeding back an instantaneous decoding refresh (IDR) request. Specifically, if the data transmitting device fails to select a suitable LTR as the target LTR using the two aforementioned methods, it will revert to the I-frame recovery logic and respond to the lost frame recovery request by retransmitting the I-frame.

[0096] In summary, a data sending device sends at least one data frame to a data receiving device, wherein the at least one data frame includes a long reference frame LTR with a target reference identifier added thereto; upon receiving a lost frame recovery request sent by the data receiving device, a target LTR is selected from the at least one LTR with the target reference identifier added thereto, and an LTR-P frame is generated based on the target LTR and sent to the data receiving device. The present application adds a target reference identifier to at least one data frame sent in advance, and upon subsequent receipt of a lost frame recovery request, a target LTR is selected from the LTRs in the target reference identifier as a reference frame to generate an LTR-P frame. This process is simple and convenient, does not require the design of a complex algorithm for calculation, reduces the amount of data that needs to be encoded by the data sending device, and improves the smoothness of data transmission when lost frame recovery occurs.

[0097] In addition, this solution is equivalent to using an LTR-P refresh request instead of an IDR refresh request, and using a long-term reference frame that has been received before the lost frame as a reference for encoding / decoding display. This significantly improves encoding efficiency, saves transmission bandwidth under weak network conditions, and improves video smoothness in weak network scenarios. Moreover, when marking long-term reference frames, this solution detects the picture changes of the video frames to be encoded, and on the basis of interval marking of long-term reference frames, marks the frames with obvious changes in picture content as long-term reference frames, and puts them into the long reference frame LTR reference queue, ensuring that effective reference information is provided under the condition of a limited number of long reference frames LTR. There is no need to record and store the relevant information of each video frame. Instead, it is only necessary to record and store the relevant information of the LTR frame with the target reference identifier to calculate the degree of picture change, thereby reducing the amount of data recorded.

[0098] In addition, when the receiving end initiates an FIR retransmission request, it uses the custom PSFB information fed back by the receiving end to determine whether the previously marked LTR long reference frame LTR has been correctly received by the receiving end; then it finds the LTR frame with the strongest similarity as the reference frame, bypassing the previously lost frame for a more effective and reasonable reference.

[0099] The following describes the frame loss recovery method provided by this solution from the perspective of the data receiving device. Figure 7 , which shows a method flow chart of a frame loss recovery method provided by an exemplary embodiment of the present application. The frame loss recovery method can be applied to a data receiving device, which can also be the above-mentioned Figure 1 or Figure 2 The terminal device in Figure 7 As shown, the frame loss recovery method may include the following steps:

[0100] Step 701: Receive at least one data frame sent by a data sending device, where the at least one data frame includes a long reference frame LTR with a target reference identifier added thereto.

[0101] The data receiving device receives at least one data frame sent by the data sending device through the communication connection established with the data sending device. The process of the data sending device sending at least one data frame can refer to the above Figure 5 or Figure 6 The description in the illustrated embodiment will not be repeated here.

[0102] Step 702: In the event of data frame loss, a frame loss recovery request is sent to the data sending device. The frame loss recovery request is used to instruct the data sending device to select a target LTR from at least one LTR to which a target reference identifier is added, so that the data sending device generates an LTR-P frame based on the target LTR and sends the LTR-P frame to the data receiving device.

[0103] Optionally, in the process of receiving at least one data frame on the data receiving device side, if any data frame is not received, it can be regarded as a data frame loss. That is, for multiple data frames sent by the data sending device, if a data frame fails to be received, it can be regarded as a data frame loss. The data sending device can trigger a frame loss recovery request to the data sending device, so that the data sending device selects a target LTR from at least one LTR with a target reference identifier added according to the data sending request, and generates an LTR-P frame according to the target LTR and sends the LTR-P frame to the data receiving device. The process of the data sending device selecting the target LTR and generating the LTR-P frame can refer to the above Figure 6 The embodiments in are not described in detail here.

[0104] Optionally, after the data sending device sends the LTR frame, the data receiving device can receive the LTR-P frame sent by the data sending device, thereby completing the re-receiving process after the frame loss recovery request, and improving the smoothness of data transmission.

[0105] In summary, a data receiving device receives at least one data frame sent by a data sending device, wherein the at least one data frame includes a long reference frame LTR with a target reference identifier added thereto; in the event of data frame loss, a frame loss recovery request is sent to the data sending device, so that the data sending device selects a target LTR from the at least one LTR with the target reference identifier added thereto, and generates an LTR-P frame based on the target LTR and sends the LTR-P frame to the data receiving device. The present application adds a target reference identifier in advance to at least one data frame sent, and upon subsequent receipt of a frame loss recovery request, selects a target LTR from the LTRs in the target reference identifier as a reference frame to generate an LTR-P frame. This process is simple and convenient, does not require the design of a complex algorithm for calculation, reduces the amount of data that needs to be encoded by the data sending device, and improves the smoothness of data transmission when frame loss recovery occurs.

[0106] Below, we take the interaction between the data sending device and the data receiving device as an example, and the data transmission between the two is carried out according to RTP and RTCP. Figure 8 , which shows an interactive flow chart of a frame loss recovery method provided by an exemplary embodiment of the present application. The frame loss recovery method can be executed by a data sending device, which can also be regarded as an encoder, and a data receiving device can be regarded as a decoder. Figure 8 As shown, the frame loss recovery method may include the following steps:

[0107] Step 801: The data sending device marks the LTR long-term reference frame.

[0108] Step 802: The data sending device performs encoding.

[0109] Step 803: The data sending device establishes a correspondence between the LTR frame index and the encoding output counter.

[0110] Step 804: The data sending device encapsulates the encoding output counter of the LTR frame into Sei.

[0111] Step 805: The data sending device inserts the Sei corresponding to the LTR into the code stream and packages it into RTP and sends it to the data receiving device.

[0112] Optionally, Sei corresponding to the LTR is equivalent to the target reference identifier added above, marking the current LTR frame.

[0113] The above step 801 is equivalent to the process of starting the marking process, until step 804 completes marking of a data frame, and Sei is equivalent to the result of marking the LTR long-term reference frame.

[0114] Step 806: The data receiving device feeds back whether the marked LTR is received correctly.

[0115] Step 807: The data sending device determines each successfully received LTR frame based on the information fed back by RTCP.

[0116] Optionally, if the data receiving device successfully receives the RTCP-defined PSFB message, it will feedback the information. If it does not receive the information, it will not feedback the information. In addition, after receiving the marked data frame, the data receiving device will also parse the Sei corresponding to the LTR, obtain the encoding output counter of the LTR frame, and enter the decoding process. In other words, the encoding output counter inserted with the LTR frame is mainly used as the data receiving device at the receiving end to feed back the PSFB (payload feedback) message information to the transmitting end after receiving it. The transmitting end determines which LTR frame was successfully received based on the encoding output counter.

[0117] Step 808: Select a target LTR based on the screen similarity.

[0118] Step 809: Determine whether the target LTR is found.

[0119] If the target LTR is found, step 810 is executed; if not, step 812 is executed.

[0120] Step 810: Restore according to the target LTR.

[0121] Step 811: Degenerate to traditional I frame restoration.

[0122] In summary, the solution of the present application is based on the RTC real-time audio and video communication LTR-P long reference frame LTR refresh request technology, and proposes to detect the changes in the picture content of the frame to be encoded according to the picture similarity before encoding, and mark the frame with obvious picture content changes as a long-term reference frame; and when the data receiving device initiates a retransmission request, it is judged according to the RTCP customized PSFB message information whether the previously marked LTR frame is received normally, and then the image similarity between the current encoding frame and the previously marked LRT frame is calculated to select the most appropriate long-term reference frame as the reference encoding, thereby improving the encoding efficiency, saving the transmission bandwidth under weak network conditions, and improving the video fluency in weak network scenarios.

[0123] The following are device embodiments of the present application, which can be used to implement the method embodiments of the present application. For details not disclosed in the device embodiments of the present application, please refer to the method embodiments of the present application.

[0124] Please refer to Figure 9 , which shows a structural block diagram of a frame loss recovery device provided by an exemplary embodiment of the present application. The frame loss recovery device 900 can be used in a data sending device to perform Figure 5 , Figure 6 ,as well as Figure 8 All or part of the steps in the method provided by the embodiment shown are performed by the data sending device. The frame loss recovery device 900 includes:

[0125] The data sending module 901 is configured to send at least one data frame to a data receiving device, wherein the at least one data frame includes a long reference frame LTR to which a target reference identifier is added;

[0126] The reference frame selection module 902 is used to select a target LTR from at least one LTR added with the target reference identifier when receiving a frame loss recovery request sent by the data receiving device, and generate an LTR-P frame according to the target LTR and send the LTR-P frame to the data receiving device.

[0127] In summary, the data sending device sends at least one data frame to the data receiving device, and the at least one data frame includes a long reference frame LTR with a target reference identifier added thereto; upon receiving a frame loss recovery request sent by the data receiving device, the data sending device selects a target LTR from the at least one LTR with a target reference identifier added thereto, and generates an LTR-P frame based on the target LTR and sends the LTR-P frame to the data receiving device. The present application adds a target reference identifier to at least one data frame sent in advance, and upon subsequent receipt of a frame loss recovery request, selects a target LTR from the LTRs in the target reference identifier as a reference frame to generate an LTR-P frame. This process is simple and convenient, does not require the design of a complex algorithm for calculation, reduces the amount of data that needs to be encoded by the data sending device, and improves the video fluency when frame loss recovery occurs.

[0128] Optionally, the device further includes:

[0129] A first receiving module is configured to receive feedback information from the data receiving device after sending at least one data frame to the data receiving device, wherein the feedback information is used to indicate that the data receiving device has received a long reference frame LTR with the target reference identifier added thereto;

[0130] The reference frame selection module 902 is further configured to select a target LTR from the LTRs having the target reference identifier for which the feedback information is received.

[0131] Optionally, the reference frame selection module 902 further includes: a first acquiring unit, a first determining unit;

[0132] The first acquiring unit is configured to acquire a current data frame;

[0133] The first determining unit is configured to determine, from among the LTRs having the target reference identifiers received and corresponding to the feedback information, an LTR having the highest similarity with the current data frame as the target LTR.

[0134] Optionally, the reference frame selection module 902 further includes: a second acquisition unit, a second determination unit;

[0135] The second acquiring unit is configured to acquire a current data frame;

[0136] The second determining unit is configured to determine, from among the LTRs having the target reference identifiers for which the feedback information is received, an LTR with the shortest time interval to the current data frame as the target LTR.

[0137] Optionally, the device further includes:

[0138] The first adding module is configured to add the target reference identifier to the at least one data frame before sending the at least one data frame to the data receiving device.

[0139] Optionally, the first adding module further includes a first adding unit;

[0140] The first adding unit is configured to add the target reference identifier to the data frames meeting the preset adding period in the at least one data frame according to a preset adding period.

[0141] Optionally, the first adding module further includes a first acquiring unit and a second adding unit;

[0142] The first acquisition unit is used to acquire the degree of picture change between the current data frame and the previous long reference frame LTR to which the target reference identifier is added;

[0143] The second adding unit is configured to add the target reference identifier to the current data frame when the degree of change in the picture is greater than or equal to a preset change threshold.

[0144] Optionally, the first acquiring unit is further configured to:

[0145] Respectively obtaining histogram information of the current data frame and the previous long reference frame LTR to which the target reference identifier is added;

[0146] Comparing the histogram information of the current data frame with the previous long reference frame LTR to which the target reference identifier is added, to obtain picture change information of the current data frame relative to the previous long reference frame LTR to which the target reference identifier is added;

[0147] The picture change degree is acquired according to the picture change information.

[0148] Optionally, the device further includes:

[0149] The first retransmission module is used to retransmit the I frame to the data receiving device in a manner of feedback instant decoding refresh (IDR) request when receiving a frame loss recovery request sent by the data receiving device and if the data sending device cannot select the target LTR from at least one LTR to which the target reference identifier is added.

[0150] Please refer to Figure 10 , which shows a structural block diagram of a frame loss recovery device provided by an exemplary embodiment of the present application. The frame loss recovery device 1000 can be used in a data sending device to perform Figure 6 , Figure 7 ,as well as Figure 8 All or part of the steps in the method provided by the embodiment shown are performed by the data receiving device. The frame loss recovery device 1000 includes:

[0151] The data receiving module 1001 is configured to receive at least one data frame sent by a data sending device, wherein the at least one data frame includes a long reference frame LTR to which a target reference identifier is added;

[0152] The request sending module 1002 is used to send a frame loss recovery request to the data sending device in the event of data frame loss. The frame loss recovery request is used to instruct the data sending device to select a target LTR from at least one LTR to which the target reference identifier is added, so that the data sending device generates an LTR-P frame according to the target LTR and sends the LTR-P frame to the data receiving device.

[0153] In summary, a data receiving device receives at least one data frame sent by a data sending device, wherein the at least one data frame includes a long reference frame LTR with a target reference identifier added thereto; in the event of data frame loss, a frame loss recovery request is sent to the data sending device, so that the data sending device selects a target LTR from the at least one LTR with the target reference identifier added thereto, and generates an LTR-P frame based on the target LTR and sends the LTR-P frame to the data receiving device. The present application adds a target reference identifier in advance to at least one data frame sent, and upon subsequent receipt of a frame loss recovery request, selects a target LTR from the LTRs in the target reference identifier as a reference frame to generate an LTR-P frame. This process is simple and convenient, does not require the design of a complex algorithm for calculation, reduces the amount of data that needs to be encoded by the data sending device, and improves the smoothness of data transmission when frame loss recovery occurs.

[0154] Please refer to Figure 11 , which is a schematic diagram of another exemplary structure of a frame loss recovery device provided in an embodiment of the present application. The frame loss recovery device 1100 can be a data transmitting device or a data receiving device, capable of implementing the functions of the method provided in an embodiment of the present application. The frame loss recovery device 1100 can be a chip system. In the embodiment of the present application, the chip system can be composed of a chip or include a chip and other discrete components.

[0155] In hardware implementation, the communication module may be a transceiver, which is integrated into the lost frame recovery device 1100 to form the communication interface 1103 .

[0156] The lost frame recovery device 1100 includes at least one processor 1101, which is used to implement or support the lost frame recovery device 1100 in implementing the functions of the data sending device in the method provided in the embodiment of the present application. For example, the processor 1101 can send at least one data frame from the data sending device to the data receiving device, and the at least one data frame includes a long reference frame LTR with a target reference identifier added thereto; upon receiving a lost frame recovery request sent by the data receiving device, the processor 1101 selects a target LTR from the at least one LTR with a target reference identifier added thereto, generates an LTR-P frame based on the target LTR, and sends the LTR-P frame to the data receiving device, etc. For details, please refer to the detailed description in the method example, which will not be repeated here.

[0157] The frame loss recovery device 1100 may further include at least one memory 1102 for storing program instructions and / or data. The memory 1102 is coupled to the processor 1101. Coupling in the embodiments of the present application is an indirect coupling or communication connection between devices, units, or modules, and may be electrical, mechanical, or other forms, for information exchange between the devices, units, or modules. The processor 1101 may operate in conjunction with the memory 1102. The processor 1101 may execute program instructions stored in the memory 1102. At least one of the at least one memory may be included in the processor.

[0158] Lost frame recovery apparatus 1100 may also include a communication interface 1103 for communicating with other devices via a transmission medium, thereby enabling the apparatus in lost frame recovery apparatus 1100 to communicate with the other devices. Exemplarily, the other devices may be network-side devices. Processor 1101 may utilize communication interface 1103 to transmit and receive data. Communication interface 1103 may specifically be a transceiver.

[0159] The specific connection medium between the communication interface 1103, the processor 1101 and the memory 1102 is not limited in the embodiment of the present application. Figure 11 The memory 1102, the processor 1101 and the communication interface 1103 are connected via a bus 1104. Figure 11 The connections between the other components are shown in bold lines, which are only for illustration and not intended to be limiting. The bus can be divided into address bus, data bus, control bus, etc. Figure 11 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0160] In the embodiments of the present application, the processor 1101 may be a general-purpose processor, a digital signal processor, an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component, and may implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of the present application may be directly implemented as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0161] In an embodiment of the present application, the memory 1102 may be a non-volatile memory, such as a hard disk drive (HDD) or a solid-state drive (SSD), or may be a volatile memory (volatile memory), such as a random-access memory (RAM). A memory is any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory in the embodiment of the present application may also be a circuit or any other device that can implement a storage function, for storing program instructions and / or data.

[0162] Optionally, an embodiment of the present application further provides a data sending device, which includes a processor and a memory, wherein the memory stores a computer program, and the computer program is executed by the processor to implement all or part of the steps performed by the data sending device in the frame loss recovery method of each of the above embodiments.

[0163] Optionally, an embodiment of the present application further provides a data receiving device, which includes a processor and a memory, wherein the memory stores a computer program, and the computer program is executed by the processor to implement all or part of the steps performed by the data receiving device in the frame loss recovery method of each of the above embodiments.

[0164] Optionally, an embodiment of the present application further provides a computer-readable medium, which stores a computer program, and the computer program is executed by a processor to implement all or part of the steps performed by the data sending device or the data receiving device in the frame loss recovery method of each of the above embodiments.

[0165] Optionally, an embodiment of the present application further provides a computer program product. When the computer program product runs on a computer, it enables the computer to execute the frame loss recovery method of each of the above embodiments, including all or part of the steps executed by the data sending device or the data receiving device.

[0166] Optionally, an embodiment of the present application further provides an application publishing platform, which is used to publish a computer program product, wherein when the computer program product runs on a computer, the computer executes the frame loss recovery method of each embodiment as described above, and all or part of the steps performed by the data sending device or the data receiving device.

[0167] It should be noted that the apparatus provided in the above embodiments, when controlling a data transmission device, is merely illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments are based on the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0168] The serial numbers of the above embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0169] Those skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or by a program to instruct the relevant hardware, and the program may be stored in a computer-readable storage medium, which may be a read-only memory, a disk, or an optical disk, etc.

[0170] The above description is merely an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A frame loss recovery method, characterized in that: Applied to a data sending device, the method includes: Sending at least one data frame to a data receiving device, wherein the at least one data frame includes a long reference frame LTR added with a target reference identifier; When receiving the frame loss recovery request sent by the data receiving device, a target LTR is selected from at least one LTR added with the target reference identifier, and an LTR-P frame is generated according to the target LTR and sent to the data receiving device.

2. The method according to claim 1, characterized in that After sending at least one data frame to the data receiving device, the method further includes: receiving feedback information from the data receiving device, where the feedback information is used to indicate that the data receiving device has received a long reference frame LTR with the target reference identifier added thereto; The selecting a target LTR from at least one LTR to which the target reference identifier is added comprises: A target LTR is selected from each LTR having the target reference identifier received and corresponding to the feedback information.

3. The method according to claim 2, characterized in that The selecting a target LTR from each LTR having the target reference identifier and receiving the feedback information includes: Get the current data frame; From among the LTRs having the target reference identifiers for which the feedback information is received, the LTR having the highest similarity to the current data frame is determined as the target LTR.

4. The method according to claim 2, characterized in that The selecting a target LTR from each LTR having the target reference identifier and receiving the feedback information includes: Get the current data frame; From among the LTRs with the target reference identifiers for which the feedback information is received, the LTR with the shortest time interval to the current data frame is determined as the target LTR.

5. The method according to claim 1, wherein Before sending at least one data frame to the data receiving device, the method further includes: The target reference identifier is added to the at least one data frame.

6. The method according to claim 5, characterized in that Adding the target reference identifier to the at least one data frame includes: According to a preset adding cycle, the target reference identifier is added to the data frames meeting the preset adding cycle in the at least one data frame.

7. The method according to claim 5, characterized in that Adding the target reference identifier to the at least one data frame includes: Acquire the degree of picture change between the current data frame and the previous long reference frame LTR to which the target reference identifier is added; When the degree of picture change is greater than or equal to a preset change threshold, the target reference identifier is added to the current data frame.

8. The method according to claim 7, characterized in that The obtaining of the degree of picture change between the current data frame and the previous long reference frame LTR to which the target reference identifier is added includes: Respectively obtaining histogram information of the current data frame and the previous long reference frame LTR to which the target reference identifier is added; Comparing the histogram information of the current data frame with the previous long reference frame LTR to which the target reference identifier is added, to obtain picture change information of the current data frame relative to the previous long reference frame LTR to which the target reference identifier is added; The picture change degree is acquired according to the picture change information.

9. The method according to any one of claims 1 to 8, characterized in that: Upon receiving the frame loss recovery request sent by the data receiving device, if the data sending device is unable to select the target LTR from the at least one LTR to which the target reference identifier is added, the method further includes: The I frame is retransmitted to the data receiving device in a manner of feeding back an Instantaneous Decoding Refresh (IDR) request.

10. A frame loss recovery method, characterized in that: Applied to a data receiving device, the method includes: receiving at least one data frame sent by a data sending device, wherein the at least one data frame includes a long reference frame LTR to which a target reference identifier is added; In the event of data frame loss, a frame loss recovery request is sent to the data sending device, where the frame loss recovery request is used to instruct the data sending device to select a target LTR from at least one LTR to which the target reference identifier is added, so that the data sending device generates an LTR-P frame based on the target LTR and sends the LTR-P frame to the data receiving device.

11. A lost frame recovery device, characterized in that: Applied to a data sending device, the apparatus comprises: A data sending module, configured to send at least one data frame to a data receiving device, wherein the at least one data frame includes a long reference frame LTR to which a target reference identifier is added; The reference frame selection module is used to select a target LTR from at least one LTR added with the target reference identifier when receiving a frame loss recovery request sent by the data receiving device, and generate an LTR-P frame according to the target LTR and send the LTR-P frame to the data receiving device.

12. A lost frame recovery device, characterized in that: Applied to a data receiving device, the apparatus comprises: A first receiving module is configured to receive at least one data frame sent by a data sending device, wherein the at least one data frame includes a long reference frame LTR to which a target reference identifier is added; A request sending module is used to send a frame loss recovery request to a data sending device in the event of data frame loss. The frame loss recovery request is used to instruct the data sending device to select a target LTR from at least one LTR to which the target reference identifier is added, so that the data sending device generates an LTR-P frame according to the target LTR and sends the LTR-P frame to the data receiving device.

13. A data sending device, characterized in that: The data sending device includes a processor and a memory, wherein a computer program is stored in the memory, and the computer program is executed by the processor to implement the frame loss recovery method according to any one of claims 1 to 9.

14. A data receiving device, characterized in that: The data receiving device includes a processor and a memory, wherein a computer program is stored in the memory, and the computer program is executed by the processor to implement the frame loss recovery method according to claim 10.

15. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and the computer program is executed by a processor to implement the frame loss recovery method according to any one of claims 1 to 9 or the frame loss recovery method according to claim 10.

16. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the lost frame recovery method according to any one of claims 1 to 9 or the lost frame recovery method according to claim 10 is implemented.

Images