Video encoding method, video decoding method, computing device and computer program product

Abstract

The embodiment of the invention provides a video coding method, a video decoding method, a video coding device, a video decoding device, computing equipment and a computer program product. The video coding method comprises the following steps: receiving feedback information from a decoder; in response to the feedback information, determining a first video frame corresponding to the first data packet from a coded video stream; marking a video frame between the first video frame and the second video frame as an invalid video frame; determining a target reference frame from a plurality of candidate reference frames which are pre-stored in an encoder cache and synchronized to the decoder; switching the reference frame of the third video frame from the second video frame to the target reference frame; and sending the second data packet to the decoder. According to the technical scheme provided by the embodiment of the invention, errors are prevented from being further propagated in the video stream.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image processing technology, and in particular to a video encoding method, a video decoding method, an apparatus, a computing device, and a computer program product. Background Art

[0002] In the process of real-time video encoding and decoding, data transmission between the encoder and the decoder usually relies on the Real-Time Communication (RTC) protocol, which transmits data packets through the network or other media. In order to ensure high real-time performance, these real-time transport protocols are often based on the User Datagram Protocol (UDP) for transmission, because UDP can provide fast transmission speed and is suitable for transmitting real-time data. However, UDP is an unreliable transmission protocol. It does not guarantee the orderly transmission of data, nor does it guarantee that data can reach the destination reliably, which brings the risk of data packet loss.

[0003] The loss of data packets may cause the decoder to be unable to correctly decode the video frame. This situation not only affects the current frame corresponding to the lost data packet, but also may affect all subsequent frames that directly or indirectly depend on this frame. In the worst case, the number of damaged frames may continue to increase until the end of the video.

[0004] Therefore, how to avoid the error propagation generated in related technologies has become a technical problem that needs to be solved urgently. Summary of the invention

[0005] Embodiments of the present application provide a video encoding method, a video decoding method, an apparatus, a computing device, and a computer program product.

[0006] In a first aspect, an embodiment of the present application provides a video encoding method, which is applied to an encoder, and the method includes:

[0007] receiving feedback information from a decoder, the feedback information being used to indicate a result that the decoder fails to decode a first data packet received from the encoder;

[0008] In response to the feedback information, determining a first video frame corresponding to the first data packet from the encoded video stream;

[0009] Marking a video frame between the first video frame and a second video frame as an invalid video frame, wherein the second video frame is a previous video frame of a third video frame currently encoded by the encoder;

[0010] Determine a target reference frame from a plurality of candidate reference frames pre-stored in an encoder buffer and synchronized to the decoder, wherein the plurality of candidate reference frames include candidate short-term reference frames and candidate long-term reference frames;

[0011] Switching the reference frame of the third video frame from the second video frame to the target reference frame, so as to encode the third video frame based on the target reference frame and generate a second data packet;

[0012] The second data packet is sent to the decoder so that the decoder decodes the second data packet based on the target reference frame stored in a decoder buffer.

[0013] In a second aspect, an embodiment of the present application provides a video decoding method, which is applied to a decoder, and the method includes:

[0014] receiving a first data packet from an encoder;

[0015] In case of a failure in decoding the first data packet, generating feedback information;

[0016] The feedback information is sent to the encoder so that the encoder determines a first video frame corresponding to the first data packet from the encoded video stream in response to the feedback information; marks a video frame between the first video frame and the second video frame as an invalid video frame, wherein the second video frame is a previous video frame of a third video frame currently encoded by the encoder; determines a target reference frame from a plurality of candidate reference frames pre-stored in the encoder cache and synchronized to the decoder, wherein the plurality of candidate reference frames include candidate short-term reference frames and candidate long-term reference frames; switches a reference frame of the third video frame from the second video frame to the target reference frame, so as to encode the third video frame based on the target reference frame and generate a second data packet;

[0017] receiving the second data packet sent by the encoder;

[0018] The second data packet is decoded based on the target reference frame stored in the decoder buffer.

[0019] In a third aspect, an embodiment of the present application provides a video encoding device, the device comprising:

[0020] An information receiving module, used for receiving feedback information from a decoder, wherein the feedback information is used for indicating a result that the decoder fails to decode a first data packet received from the encoder;

[0021] a video frame determination module, configured to determine, in response to the feedback information, a first video frame corresponding to the first data packet from the encoded video stream;

[0022] A marking module, configured to mark a video frame between the first video frame and a second video frame as an invalid video frame, wherein the second video frame is a previous video frame of a third video frame currently encoded by the encoder;

[0023] A reference frame determination module, configured to determine a target reference frame from a plurality of candidate reference frames pre-stored in an encoder cache and synchronized to the decoder, wherein the plurality of candidate reference frames include candidate short-term reference frames and candidate long-term reference frames;

[0024] an encoding module, configured to switch a reference frame of the third video frame from the second video frame to the target reference frame, so as to encode the third video frame based on the target reference frame and generate a second data packet;

[0025] The data packet sending module is used to send the second data packet to the decoder, so that the decoder decodes the second data packet based on the target reference frame stored in the decoder buffer.

[0026] In a fourth aspect, an embodiment of the present application provides a video decoding device, the device comprising:

[0027] A data packet receiving module, used for receiving a first data packet from an encoder;

[0028] A feedback information generating module, configured to generate feedback information when decoding of the first data packet fails;

[0029] A feedback information sending module, configured to send the feedback information to the encoder, so that the encoder determines a first video frame corresponding to the first data packet from the encoded video stream in response to the feedback information; marks a video frame between the first video frame and the second video frame as an invalid video frame, wherein the second video frame is a previous video frame of a third video frame currently encoded by the encoder; determines a target reference frame from a plurality of candidate reference frames pre-stored in an encoder cache and synchronized to the decoder, wherein the plurality of candidate reference frames include candidate short-term reference frames and candidate long-term reference frames; switches a reference frame of the third video frame from the second video frame to the target reference frame, so as to encode the third video frame based on the target reference frame and generate a second data packet;

[0030] A data packet receiving module, used for receiving the second data packet sent by the encoder;

[0031] A decoding module is used to decode the second data packet based on the target reference frame stored in the decoder buffer.

[0032] In a fifth aspect, an embodiment of the present application provides a computing device, including a processing component and a storage component;

[0033] The storage component stores one or more computer instructions; the one or more computer instructions are used to be called and executed by the processing component to implement the video encoding method provided in the embodiment of the present application, or to implement the video decoding method provided in the embodiment of the present application.

[0034] In a sixth aspect, a computer program product is provided in an embodiment of the present application, and the computer program product includes a computer program code. When the computer program code is executed by a computer, the video encoding method provided in the embodiment of the present application is implemented, or the video decoding method provided in the embodiment of the present application is implemented.

[0035] In an embodiment of the present application, the following steps are adopted: receiving feedback information from a decoder, the feedback information being used to characterize a result of a decoder failing to decode a first data packet received from an encoder; determining a first video frame corresponding to the first data packet from an encoded video stream in response to the feedback information; marking a video frame between the first video frame and the second video frame as an invalid video frame, wherein the second video frame is a previous video frame of a third video frame currently encoded by the encoder; determining a target reference frame from a plurality of candidate reference frames pre-stored in an encoder cache and synchronized to a decoder, wherein the plurality of candidate reference frames include a candidate short-term reference frame and a candidate long-term reference frame; and switching a reference frame of the third video frame from the second video frame to A target reference frame is used to encode a third video frame based on the target reference frame to generate a second data packet; the second data packet is sent to a decoder so that the decoder decodes the second data packet based on the target reference frame stored in a decoder cache. After receiving the decoding failure information fed back by the decoder, the encoder marks all video frames that may be affected as invalid frames, and redetermines the reference frame for the currently encoded intended video frame, encodes the third video frame based on the redetermined reference frame, and, because the redetermined reference frame has been synchronized to the decoder, the decoder can normally decode the second data packet based on the reference frame, thereby avoiding further propagation of errors in the video stream.

[0036] These and other aspects of the present application will become more apparent from the description of the following embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0038] Figure 1 A schematic diagram of video encoding and decoding in the related art is shown;

[0039] Figure 2 A flowchart of a video encoding method provided by an embodiment of the present application is shown;

[0040] Figure 3 A schematic diagram showing a decoder sending feedback information to an encoder;

[0041] Figure 4 A schematic diagram showing an encoder marking invalid video frames;

[0042] Figure 5 A schematic diagram showing an encoder switching reference frame;

[0043] Figure 6 A schematic diagram of a video encoding method provided by an embodiment of the present application is shown;

[0044] Figure 7 A schematic diagram showing an encoder updating a reference frame;

[0045] Figure 8 A flowchart of a video decoding method provided by an embodiment of the present application is shown;

[0046] Fig. 9 A block diagram of a video encoding device provided in an embodiment of the present application is shown;

[0047] Fig.10 A block diagram of a video decoding device provided in an embodiment of the present application is shown;

[0048] Fig.11 A block diagram of a computing device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0049] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0050] In some of the processes described in the specification and claims of this application and the above-mentioned figures, multiple operations that appear in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this article or executed in parallel. The serial numbers of the operations, such as 101, 102, etc., are only used to distinguish between different operations, and the serial numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this article are used to distinguish different messages, devices, modules, etc., do not represent the order of precedence, and do not limit the "first" and "second" to be different types.

[0051] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0052] In the process of real-time video encoding and decoding, data transmission between the encoder and the decoder usually relies on the Real-Time Communication (RTC), which transmits data packets through the network or other media. In order to ensure high real-time performance, these real-time transport protocols are often based on the User Datagram Protocol (UDP) for transmission, because UDP can provide fast transmission speed and is suitable for transmitting real-time data. However, UDP is an unreliable transmission protocol. It does not guarantee the orderly transmission of data, nor does it guarantee that data can reach the destination reliably, which brings the risk of data packet loss.

[0053] The loss of data packets may cause the decoder to be unable to correctly decode the video frame. This situation affects not only the current frame corresponding to the lost data packet, but also all subsequent frames that directly or indirectly depend on this frame. As the number of damaged frames increases, the quality of the video will gradually decrease until the encoder generates a new key frame (Key Frame) or intra-frame reference frame (Intra Frame or I-Frame), which can end the dependency between frames and prevent the number of damaged frames from increasing further.

[0054] In the worst case, if the encoder does not regularly insert keyframes or intra-reference frames, the number of damaged frames may increase until the end of the video. Figure 1As shown in the figure, if a critical data packet (such as sequence number 2, POC 2, i.e., picture order count 2) is lost, the decoder will not be able to correctly decode this frame, and all subsequent frames that directly or indirectly reference this frame (such as sequence numbers 3, 4, 5, etc.) will also be damaged. This will cause freezes, screen distortion, or other quality issues during video playback, seriously affecting the viewing experience. Therefore, in order to ensure the reliability and quality of video encoding and decoding, the real-time transport protocol needs to take additional measures to reduce the impact of packet loss, such as enhancing the stability of data transmission through retransmission mechanisms and error detection and correction technologies.

[0055] In related technologies, in order to solve the video quality problem caused by data packet loss during real-time video encoding and decoding, the following methods are usually adopted:

[0056] First, the encoder can periodically insert key frames or intra-frame reference frames. Although this method can repair damaged frames, it also has two major problems. The first problem is that the number of bits of key frames or intra-frame reference frames is usually high, which will cause an instantaneous bit rate surge and is difficult to adapt to low-bandwidth transmission conditions. The second problem is that the interval between key frames may be large, such as several seconds or even longer, which means that users will still see picture problems before the next key frame is reached.

[0057] Secondly, the encoder can choose not to insert key frames or intra-frame reference frames, but use intra-frame prediction row by row or column by column in several frames through intra-frame coding refresh (Intra Refresh or Gradual Decoding Refresh). This method avoids the problem of instantaneous bit rate surge, but the higher number of bits generated by the intra-frame prediction module still consumes a lot of bandwidth, and the damaged picture takes several seconds or more to recover.

[0058] The third method is that the encoder does not insert key frames or intra-frame reference frames. Only when the decoder reports data packet loss, it inserts key frames to repair the damaged picture. Although this method can avoid the problem of damaged pictures being unable to be restored for a long time, the problem of instantaneous bit rate surge still exists.

[0059] Finally, error-tolerance information can be added to the transmission data packets, and the debugging and error correction functions of the transmission protocol can be used to prevent data packet loss. Although this method can reduce video quality issues caused by data packet loss, it will increase bandwidth, latency, and computing resource overhead, especially in real-time transmission scenarios, where the increase in latency may be unacceptable.

[0060] In order to solve the technical problems existing in the related art, an embodiment of the present application provides a video encoding method, which adopts: receiving feedback information from a decoder, the feedback information is used to characterize the result of the decoder failing to decode a first data packet received from an encoder; in response to the feedback information, determining a first video frame corresponding to the first data packet from an encoded video stream; marking a video frame between the first video frame and the second video frame as an invalid video frame, wherein the second video frame is a previous video frame of a third video frame currently encoded by the encoder; determining a target reference frame from a plurality of candidate reference frames pre-stored in an encoder cache and synchronized to a decoder, wherein the plurality of candidate reference frames include candidate short-term reference frames and candidate long-term reference frames; marking the third video frame The reference frame of the video frame is switched from the second video frame to the target reference frame, so as to encode the third video frame based on the target reference frame and generate a second data packet; the second data packet is sent to the decoder so that the decoder decodes the second data packet based on the target reference frame stored in the decoder cache. After receiving the decoding failure information fed back by the decoder, the encoder marks all the video frames that may be affected as invalid frames, and re-determines the reference frame for the intended video frame currently encoded, and encodes the third video frame based on the re-determined reference frame, and, since the re-determined reference frame has been synchronized to the decoder, the decoder can decode the second data packet normally based on the reference frame, thereby avoiding the error from propagating further in the video stream. In addition, the video encoding method provided in the embodiment of the present application uses the reference frame to deal with the problem of data packet loss, without relying on the encoder to generate key frames or intra-frame prediction modules, thereby avoiding the problem of consuming a large amount of bandwidth. In addition, there is no need to add additional delays, and the consumption of computing resources will not be increased. Because the reference frame already in the decoder cache is used, the encoder is not required to generate new key frames every time. In this way, even in the case of limited bandwidth, the impact of data packet loss on video quality can be effectively reduced, while maintaining the real-time and smoothness of video transmission.

[0061] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0062] The implementation details of the technical solution of the embodiment of the present application are elaborated in detail below.

[0063] Figure 2 A flowchart of a video encoding method provided in an embodiment of the present application is shown. The video encoding method can be applied to an encoder, such as Figure 2 As shown, the video encoding method may specifically include the following steps:

[0064] 201 : Receive feedback information from a decoder, where the feedback information is used to indicate a result that the decoder fails to decode a first data packet received from an encoder.

[0065] 202. In response to feedback information, determine a first video frame corresponding to a first data packet from an encoded video stream.

[0066] The failure of the decoder to decode the first data packet may be caused by packet loss generated by a data transmission protocol between the encoder and the decoder.

[0067] In an embodiment of the present application, a specific data transmission protocol can be used to transfer data packets between the encoder and the decoder. For example, the encoder and the decoder can use a real-time transport protocol based on UDP to transfer data packets. The UDP protocol has the advantage of high real-time performance and is suitable for real-time video transmission, but it is an unreliable transmission protocol, so during the transmission of data packets, packet loss may occur. For example, in the case of network congestion, signal interference or network equipment failure, the data packet may not be completely transmitted from the encoder to the decoder.

[0068] For the decoder, each data packet contains part or all of the information of the video frame. If the first data packet is lost, the decoder may not be able to obtain the complete video frame information, resulting in decoding failure.

[0069] After the decoder fails to decode the first data packet, feedback information may be sent to the encoder to inform the encoder of the problem generated during the data transmission process.

[0070] In one embodiment of the present application, the feedback information may include a data packet sequence number;

[0071] The step of determining, in response to the feedback information, a first video frame corresponding to the first data packet from the encoded video stream comprises:

[0072] Determine, based on the data packet sequence number, position information of the first data packet in the encoded video stream;

[0073] The first video frame is determined based on the position information.

[0074] Among them, the data packet sequence number can be used to uniquely identify the position of the first data packet in the entire transmission sequence. For example, in a video stream, the data packets are numbered in a certain order, for example, they can be numbered sequentially starting from 1. When the decoder feedbacks that the sequence number of the first data packet that cannot be decoded is 5, the encoder can accurately locate the video frame corresponding to the problem data packet.

[0075] In another embodiment of the present application, the feedback information may further include timestamp information, which may accurately indicate the time position of the video segment to which the first data packet belongs, so that the encoder can quickly find the video frame corresponding to the timestamp information.

[0076] After the encoder receives the feedback information sent by the decoder, it can read the feedback information and determine the first video frame from the encoded video stream based on the data packet sequence number and / or timestamp information included in the feedback information.

[0077] Figure 3 A schematic diagram showing a decoder sending feedback information to an encoder.

[0078] like Figure 3 As shown, 301 may represent an encoder, 3011 may represent an encoded video stream, 302 may represent a decoder, and 3021 may represent a decoded video stream.

[0079] The encoded video stream 3011 may include video frames with sequence numbers 0, 1, 2, and 3, respectively. At this time, the encoder 301 may have completed the encoding of the video frame with sequence number 3.

[0080] The decoded video stream 3021 may include video frames with serial numbers 0, 1, and 2, respectively. The decoder 302 may generate a decoding failure when decoding the video frame with serial number 2.

[0081] In this case, the decoder 302 may send feedback information 203 to the encoder 301 to inform the encoder 301 of the result of the decoding failure of the data packet corresponding to the video frame of sequence number 2 through the feedback information 203 .

[0082] 203 : Mark a video frame between a first video frame and a second video frame as an invalid video frame, wherein the second video frame is a previous video frame of a third video frame currently encoded by the encoder.

[0083] In the embodiment of the present application, since there is often a reference relationship between video frames, subsequent video frames may refer to previous video frames for encoding and decoding. Therefore, if the first video frame fails to be decoded, the subsequent video frames that rely on the first video frame may not be able to obtain the correct reference information, resulting in a decoding error.

[0084] The encoder can prevent the error from being further propagated in the video stream by marking the video frames starting from the video frame where the error occurs and before the currently encoded video frame as invalid video frames.

[0085] Figure 4 A schematic diagram showing an encoder marking invalid video frames.

[0086] like Figure 4As shown, 401 can represent a coded video stream, and the coded video stream 401 can include video frames with serial numbers 0, 1, 2, 3, and 4, wherein serial number 2 can be the first video frame, and serial number 4 can be the third video frame currently being prepared for encoding, and thus, serial number 3 is the second video frame.

[0087] In this case, the encoder may mark both sequence number 2 and sequence number 3 as invalid video frames.

[0088] 204 , determine a target reference frame from a plurality of candidate reference frames pre-stored in the encoder cache and synchronized to the decoder.

[0089] In one embodiment of the present application, the multiple candidate reference frames may include candidate short-term reference frames and candidate long-term reference frames.

[0090] During the process of the encoder encoding the video stream, certain video frames can be dynamically configured as reference frames based on preset rules, and the video frames configured as reference frames can be stored in the encoder cache. In addition, in order to enable the decoder to decode accurately, these reference frames will also be synchronized with the decoder, so that the decoder cache will also store the same reference frames as those in the encoder cache.

[0091] After marking the invalid video frame, the encoder can reselect the appropriate target reference frame from the encoder cache to encode the subsequent video frames. This can break the original encoding chain that may be affected by errors, re-establish a reliable encoding foundation, and improve the quality and stability of the video.

[0092] 205 : Switch the reference frame of the third video frame from the second video frame to the target reference frame, so as to encode the third video frame based on the target reference frame and generate a second data packet.

[0093] In an embodiment of the present application, after the decoder fails to decode the first data packet, in order to prevent error propagation, the encoder marks the videos between the first video frame and the second video frame as invalid video frames. In this case, the second video frame that was originally used as a reference frame for the third video frame is no longer available. Therefore, the encoder can determine a new target reference frame from multiple candidate reference frames in the encoder cache, and switch the reference frame of the third video frame from the second video frame to the re-determined target reference frame.

[0094] In an embodiment of the present application, after determining the target reference frame as the reference frame of the third video frame, the encoder can recalculate the difference information between the third video frame and the target reference frame, and encode the third video frame according to the difference information to generate a second data packet.

[0095] After generating the second data packet, the encoder may send the second data packet to the decoder.

[0096] Figure 5 A schematic diagram showing an encoder switching reference frame.

[0097] like Figure 5 As shown, 501 may represent a coded video stream, and the coded video stream 501 may include video frames with serial numbers 0, 1, 2, 3, and 4, wherein serial number 2 may be the first video frame, serial number 3 may be the second video frame, and serial number 4 may be the third video frame currently being prepared for encoding.

[0098] Since sequence number 2 and sequence number 3 are both marked as invalid video frames, assuming that sequence number 0 is stored as a long-term reference frame in the encoder cache, the reference frame of sequence number 4 can be switched from sequence number 3 to sequence number 0.

[0099] 106 , sending the second data packet to the decoder, so that the decoder decodes the second data packet based on the target reference frame stored in the decoder buffer.

[0100] After receiving the second data packet, the decoder can decode the second data packet based on the target reference frame stored in its cache. The decoder can first read the information in the data packet, and then restore the content of the third video frame in combination with the target reference frame in the cache to achieve correct decoding of the second data packet.

[0101] In an embodiment of the present application, the following steps are adopted: receiving feedback information from a decoder, the feedback information being used to characterize a result of a decoder failing to decode a first data packet received from an encoder; determining a first video frame corresponding to the first data packet from an encoded video stream in response to the feedback information; marking a video frame between the first video frame and the second video frame as an invalid video frame, wherein the second video frame is a previous video frame of a third video frame currently encoded by the encoder; determining a target reference frame from a plurality of candidate reference frames pre-stored in an encoder cache and synchronized to a decoder, wherein the plurality of candidate reference frames include a candidate short-term reference frame and a candidate long-term reference frame; and switching a reference frame of the third video frame from the second video frame to A target reference frame is used to encode a third video frame based on the target reference frame to generate a second data packet; the second data packet is sent to a decoder so that the decoder decodes the second data packet based on the target reference frame stored in a decoder cache. After receiving the decoding failure information fed back by the decoder, the encoder marks all video frames that may be affected as invalid frames, and redetermines the reference frame for the currently encoded intended video frame, encodes the third video frame based on the redetermined reference frame, and, because the redetermined reference frame has been synchronized to the decoder, the decoder can normally decode the second data packet based on the reference frame, thereby avoiding further propagation of errors in the video stream.

[0102] Figure 6A schematic diagram of a video encoding method provided by an embodiment of the present application is shown.

[0103] As shown in 6, 601 may represent an encoder, 6011 may represent an encoded video stream, 602 may represent a decoder, and 6021 may represent a decoded video stream.

[0104] The encoded video stream 6011 may include video frames numbered 0, 1, 2, 3, 4, and 5, and the decoded video stream 6021 may include video frames numbered 0, 1, 2, 3, 4, and 5.

[0105] Among them, the decoder 602 can notify the encoder through feedback information that its decoding failed when decoding sequence number 2. Therefore, after receiving the feedback information, the encoder 601 marks sequence numbers 2 and 3 as invalid video frames, and determines the reference video frame of sequence number 4 as the long-term reference frame sequence number 1, that is, sequence number 0, and encodes sequence number 4 according to the long-term reference frame sequence number 1, generates a second data packet and sends it to the decoder 602.

[0106] Therefore, after receiving the second data packet, the decoder 602 can decode the second data packet based on the long-term reference frame stored in the decoder cache to obtain sequence number 4. Further, it can continue to decode with reference to sequence number 4 to obtain sequence number 5.

[0107] Through the video encoding method provided in the embodiment of the present application, when the decoder fails to decode, the error can be prevented from further propagating. Specifically, in this example, only sequence numbers 2 and 3 fail to decode, and sequence numbers 4, 5 and thereafter are not affected.

[0108] In some embodiments, determining the target reference frame from a plurality of candidate reference frames pre-stored in the encoder cache and synchronized to the decoder may be specifically implemented as follows:

[0109] Screening out invalid reference video frames corresponding to invalid video frames from a plurality of candidate reference video frames to determine at least one valid reference video frame;

[0110] A target reference frame is determined from at least one valid reference video frame.

[0111] After the video frame between the first video frame and the second video frame is marked as an invalid video frame, the corresponding reference video frames may also become unreliable, and the reference video frames corresponding to the invalid video frame may be determined as invalid reference video frames. For example, if an invalid video frame is a change frame of a key scene, and some reference video frames are predictively encoded based on the invalid video frame, then these reference video frames may no longer be applicable in the current situation.

[0112] In some embodiments, the step of filtering out the invalid reference video frames corresponding to the invalid video frames from the multiple candidate reference video frames and determining at least one valid reference video frame may be specifically implemented as follows:

[0113] Traversing the multiple candidate reference video frames in sequence, and determining whether any candidate reference video frame has a corresponding relationship with any of the invalid video frames;

[0114] Eliminating the candidate reference video frame corresponding to any of the invalid video frames from the encoder buffer;

[0115] The candidate reference video frames remaining in the encoder buffer are determined as the valid reference video frames.

[0116] In an embodiment of the present application, the encoder may check multiple candidate reference video frames in the encoder cache one by one to determine whether each candidate reference video frame is an invalid reference video frame. If it is an invalid reference video frame, it is filtered out from the encoder cache.

[0117] After the invalid reference video frames are screened out, at least one remaining candidate reference frame can be a valid reference video frame. These valid reference video frames are not affected by the invalid video frames and can be used as a basis for determining the target reference frame.

[0118] In some embodiments, the video encoding method may further include:

[0119] During the encoding process of the encoder for the video stream, determining based on a preset rule whether to configure the third video frame currently being encoded as a candidate long-term reference frame;

[0120] If so, write the third video frame into the encoder cache as a candidate long-term reference frame, mark the third data packet generated by encoding the third video frame as a long-term reference frame, and send the third data packet to the decoder, so that the decoder writes the third video frame into the decoder cache.

[0121] In an embodiment of the present application, if it is determined according to a preset rule that the third video frame can be used as a candidate long-term reference frame, the encoder can write it into the encoder cache. For example, the encoder can store the complete data or key information of the third video frame in the cache, and record its related encoding parameters and attributes, such as frame sequence number, timestamp, etc. Thus, when it is necessary to reselect the reference frame or a decoding error occurs and needs to be recovered, the encoder can quickly obtain these candidate long-term reference frames from the cache, thereby improving processing efficiency.

[0122] After the encoder encodes the third video frame to generate a third data packet, the third data packet may be marked as a long-term reference frame. This marking information may be sent to the decoder along with the third data packet to inform the decoder that the video frame corresponding to the third data packet is a long-term reference frame.

[0123] After receiving the third data packet, the decoder can write the third video frame into the decoder cache according to the marking information. In this way, the decoder can also use this long-term reference frame to improve the accuracy and stability of decoding in the subsequent decoding process. For example, when a data packet is lost or a decoding error occurs, the decoder can use the long-term reference frame as an alternative reference to resume the normal decoding process.

[0124] In some embodiments, the preset rule includes a preset spacing between adjacent long-term reference frames.

[0125] In some embodiments, determining whether to configure the third video frame currently being encoded as a candidate long-term reference frame based on a preset rule may be specifically implemented as follows:

[0126] determining a first distance between a third video frame and a previous fourth video frame configured as a candidate reference frame;

[0127] When the first interval reaches a preset interval, the third video frame is determined as a candidate long-term reference frame.

[0128] When determining whether to configure the third video frame currently being encoded as a candidate long-term reference frame, the encoder can first determine the first spacing between the third video frame currently being encoded and the fourth video frame previously configured as a candidate reference frame. In an embodiment of the present application, the first spacing can be measured in a variety of ways, such as time intervals (such as the length of time the video is played), frame number intervals, etc. For example, if the video is played at a frame rate of 30 frames per second, the spacing can be determined by calculating the frame number difference between the third video frame and the fourth video frame. Assuming that the fourth video frame appears at the 10th second of the video playback, the third video frame appears at the 20th second, and the frame rate is 30 frames per second, then the frame number difference between them is (20-10)×30=300 frames, and this frame number difference is the first spacing.

[0129] Next, the encoder can compare the calculated first spacing with the preset spacing. The preset spacing is a value pre-set based on factors such as video characteristics, encoding requirements, and transmission environment. If the first spacing reaches the preset spacing, it means that the distance between the third video frame and the previous long-term reference frame is appropriate, and the third video frame can be determined as a candidate long-term reference frame. For example, if the preset spacing is 200 frames, and the calculated first spacing is 300 frames, then the conditions for determining the third video frame as a candidate long-term reference frame are met.

[0130] In an embodiment of the present application, by setting a preset spacing as a preset rule for determining a long-term reference frame, the efficiency and quality of video encoding can be better balanced. If the spacing between adjacent long-term reference frames is too small, the selection of long-term reference frames may be too frequent during the encoding process, increasing the complexity and amount of calculation of the encoding. At the same time, the long-term reference frames may not be able to fully play their role because of the high similarity between the long-term reference frames. On the contrary, if the spacing is too large, there may be a lack of long-term reference frames that are close enough for effective recovery when data packets are lost or decoding errors occur, thereby affecting the quality and stability of the video.

[0131] In another embodiment of the present application, the preset rule may further include whether the third video frame is a scene switching marker frame;

[0132] In some embodiments, determining whether to configure the third video frame currently being encoded as a candidate long-term reference frame based on a preset rule includes:

[0133] Determining whether the third video frame indicates that the video content corresponding to the video stream switches from the first scene to the second scene;

[0134] If so, the third video frame is determined as a candidate long-term reference frame.

[0135] In an embodiment of the present application, since in the video content, the picture content of the second scene after the scene switch is usually significantly different from the picture content of the first scene before the scene switch, by setting the third video frame marking the scene switch as a candidate long-term reference frame, the candidate long-term reference frame can be used to provide a reference for subsequent frame encoding in the scene.

[0136] In an embodiment of the present application, for example, when the third video frame is the first frame, or the first few frames, of the second scene, it can be determined that the third video frame is a scene switching frame.

[0137] In other embodiments of the present application, the third video frame can also be determined as a candidate long-term reference frame when the third video frame indicates that an important object appears (for example, the protagonist appears) or disappears (for example, the object is blocked or moves out of the screen) in the video content corresponding to the video stream.

[0138] In some embodiments, after determining the third video frame as a candidate long-term reference frame, the video encoding method may further include:

[0139] Determining whether the number of candidate long-term reference frames stored in the encoder buffer reaches a preset number threshold;

[0140] If so, the candidate long-term reference frame with the earliest generation time is deleted from the encoder cache, and the candidate long-term reference frame corresponding to the third video frame is written into the encoder cache.

[0141] In an embodiment of the present application, the cache space of the encoder is limited, and too many long-term reference frames will occupy a large amount of storage resources, which may affect the performance and efficiency of the encoder. In addition, if the number of long-term reference frames is too large, the computational complexity may be increased when selecting reference frames, which may reduce the encoding efficiency. By setting a preset number threshold, it is possible to ensure that the number of long-term reference frames in the cache is within a manageable range, avoid excessive resource consumption, and improve encoding efficiency.

[0142] When the encoder determines that the number of long-term reference frames stored in the encoder cache reaches a preset number threshold, the candidate long-term reference frame with the earliest generation time can be deleted from the encoder cache. This is because the earlier generated reference frames are usually less relevant to the current video content and are farther away in time, and their value as reference frames is relatively small.

[0143] In some embodiments, the deleting the candidate reference frame with the earliest generation time from the encoder cache may be specifically implemented as follows:

[0144] Sorting multiple candidate long-term reference frames according to generation time to generate a long-term reference frame list;

[0145] The candidate long-term reference frame corresponding to the first element in the long-term reference frame list is deleted.

[0146] In an embodiment of the present application, a long-term reference frame list can be maintained by sorting multiple candidate long-term reference frames according to generation time. When deletion is required, the first element in the list, that is, the candidate reference frame with the earliest generation time, can be directly removed.

[0147] After deleting the candidate reference frame with the earliest generation time, the candidate long-term reference frame corresponding to the third video frame can be written into the encoder cache, so as to ensure that the latest long-term reference frame with higher reference value is always retained in the cache.

[0148] The writing process includes storing relevant information of the third video frame (such as frame data, encoding parameters, etc.) into the cache and updating the management data structure of the cache so that the new long-term reference frame can be quickly accessed and used in the subsequent encoding process.

[0149] By limiting the number of candidate long-term reference frames, older candidate long-term reference frames can be eliminated in a timely manner, ensuring that the candidate long-term reference frames in the cache have high validity and reference value.

[0150] Figure 7 A schematic diagram showing an encoder updating a reference frame.

[0151] like Figure 7As shown, the encoded video stream may include video frames with sequence numbers 0, 1, 2, 3, 4, 5, and 6, wherein sequence numbers 0 and 2 may be set as candidate long-term reference frames by default.

[0152] In this example, the preset number is, for example, 2, and the preset interval is, for example, 1. When the encoder encodes to sequence number 4, since the first interval between sequence number 4 and sequence number 2 is 1, and the first interval reaches the preset interval at this time, sequence number 4 can be determined as a candidate long-term reference frame, and since the preset number is 2, sequence number 0 with the earliest generation time can be removed from the encoder cache, and after removing sequence number 0, sequence number 4 can be written into the encoder cache. Further, when the encoder encodes to sequence number 6, sequence number 2 can be removed, and sequence number 6 can be written into the encoder cache.

[0153] In some embodiments, the feedback information is used to characterize a result of a decoder failing to decode a plurality of first data packets received by the encoder;

[0154] In some embodiments, in response to the feedback information, determining the first video frame corresponding to the first data packet from the encoded video stream may be specifically implemented as follows:

[0155] In response to the feedback information, determining from the encoded video stream a plurality of candidate video frames respectively corresponding to the plurality of first data packets;

[0156] Based on the timestamps of the multiple candidate video frames, a candidate video frame with the earliest encoding time is determined as the first video frame.

[0157] As before, a plurality of candidate video frames respectively corresponding to a plurality of first data packets may be determined from the encoded video stream based on the data packet sequence number and / or timestamp information carried in the feedback information.

[0158] During the video encoding process, the earlier encoded video frame is the reference basis for the subsequent video frames. If there is a problem with this frame, it may cause the decoding of multiple subsequent frames to fail. By determining the candidate video frame with the earliest encoding time as the first video frame, the problem can be solved from the source to prevent the error from spreading further in the video stream.

[0159] For example, if there are multiple candidate video frames with timestamps of t1, t2, t3, etc., the video frame with the smallest timestamp may be selected as the first video frame, ie, the frame with the earliest encoding time.

[0160] Figure 8 A flowchart of a video decoding method provided by an embodiment of the present application is shown. The video decoding method can be applied to a decoder, such as Figure 8 , the video decoding method may specifically include the following steps:

[0161] 801, receiving a first data packet from an encoder;

[0162] 802, generating feedback information when decoding of the first data packet fails;

[0163] 803, sending feedback information to the encoder so that the encoder determines, in response to the feedback information, a first video frame corresponding to the first data packet from the encoded video stream; marking a video frame between the first video frame and the second video frame as an invalid video frame, wherein the second video frame is a previous video frame of a third video frame currently encoded by the encoder; determining a target reference frame from a plurality of candidate reference frames pre-stored in the encoder cache and synchronized to the decoder, wherein the plurality of candidate reference frames include candidate short-term reference frames and candidate long-term reference frames; switching a reference frame of the third video frame from the second video frame to the target reference frame, so as to encode the third video frame based on the target reference frame and generate a second data packet;

[0164] 804, receiving a second data packet sent by the encoder;

[0165] 805 , decode the second data packet based on the target reference frame stored in the decoder buffer.

[0166] Figure 8 The generation of feedback information in the video decoding method shown and the implementation process of generating the second datagram based on the feedback information can refer to Figure 1 The video encoding method shown will not be described in detail here.

[0167] In some embodiments, the video decoding method further includes:

[0168] receiving a third data packet sent by the encoder, wherein the third data packet is generated by encoding the target reference frame, and the third data packet is marked as a candidate long-term reference frame;

[0169] Decoding the third data packet to obtain the target reference frame;

[0170] The target reference frame is stored in the decoder buffer.

[0171] In some embodiments, decoding the second data packet based on the target reference frame stored in the decoder cache may be specifically implemented as follows:

[0172] Acquire the pre-stored target reference frame from the decoder buffer;

[0173] The second data packet is decoded based on the target reference frame.

[0174] In an embodiment of the present application, the third data packet may be generated after the encoder encodes the target reference frame and is marked as a candidate long-term reference frame. This means that the third data packet contains the encoding information of the target reference frame and its identification as a candidate long-term reference frame.

[0175] For example, during video transmission, after the encoder encodes the target reference frame, it transmits the third data packet to the decoder via the network. After receiving the data packet, the decoder starts to prepare for decoding operation.

[0176] After the decoder fails to decode the first data packet, the encoder may encode the third video frame based on the pre-stored target reference frame, and send the second data packet generated by the encoding to the decoder.

[0177] Since the second data packet is encoded based on the target reference frame, when decoding the second data packet, it is also necessary to decode based on the target reference frame. And since the decoder has obtained the target reference frame by decoding the third data packet and stored the target reference frame in the decoder cache, after obtaining the second data packet, the second data packet can be decoded based on the target reference frame obtained from the decoder cache.

[0178] Fig. 9 A block diagram of a video encoding device provided in an embodiment of the present application is shown. Fig. 9 As shown, the video encoding device may specifically include:

[0179] An information receiving module 901 is used to receive feedback information from a decoder, where the feedback information is used to indicate a result that the decoder fails to decode a first data packet received from the encoder;

[0180] A video frame determination module 902, configured to determine, in response to the feedback information, a first video frame corresponding to the first data packet from the encoded video stream;

[0181] A marking module 903, configured to mark a video frame between the first video frame and a second video frame as an invalid video frame, wherein the second video frame is a previous video frame of a third video frame currently encoded by the encoder;

[0182] A reference frame determination module 904, configured to determine a target reference frame from a plurality of candidate reference frames pre-stored in the encoder cache and synchronized to the decoder;

[0183] The encoding module 905 is used to switch the reference frame of the third video frame from the second video frame to the target reference frame, so as to encode the third video frame based on the target reference frame and generate a second data packet;

[0184] The data packet sending module 906 is configured to send the second data packet to the decoder so that the decoder decodes the second data packet based on the target reference frame stored in the decoder buffer.

[0185] In some embodiments, the reference frame determination module 904 may be specifically configured to:

[0186] Screening out invalid reference video frames corresponding to the invalid video frames from the multiple candidate reference video frames to determine at least one valid reference video frame;

[0187] The target reference frame is determined from the at least one valid reference video frame.

[0188] In some embodiments, the video encoding apparatus further comprises:

[0189] A reference frame selection module is used to determine whether to configure a third video frame currently being encoded as a candidate long-term reference frame based on preset rules during the encoding process of the encoder for the video stream; if so, write the third video frame into the encoder cache as a candidate long-term reference frame, mark a third data packet generated by encoding the third video frame as a long-term reference frame, and send the third data packet to the decoder so that the decoder writes the third video frame into the decoder cache.

[0190] In some embodiments, the preset rule includes a preset spacing between adjacent long-term reference frames;

[0191] In some embodiments, the reference frame selection module is specifically used to:

[0192] determining a first distance between the third video frame and a previous fourth video frame configured as a candidate reference frame;

[0193] When the first interval reaches the preset interval, the third video frame is determined as a candidate long-term reference frame.

[0194] In some embodiments, the reference frame selection module is specifically used to:

[0195] Determining whether the number of long-term reference frames stored in the encoder buffer reaches a preset number threshold;

[0196] If so, the candidate reference frame with the earliest generation time is deleted from the encoder cache, and the candidate long-term reference frame corresponding to the third video frame is written into the encoder cache.

[0197] In some embodiments, the feedback information is used to represent a result of a decoding failure of a plurality of first data packets received by the decoder from the encoder.

[0198] In some embodiments, the information receiving module 901 is specifically used to:

[0199] In response to the feedback information, determining, from the encoded video stream, a plurality of candidate video frames respectively corresponding to the plurality of first data packets;

[0200] Based on the timestamps of the multiple candidate video frames, a candidate video frame with the earliest encoding time is determined as the first video frame.

[0201] In some embodiments, the preset rule includes whether the third video frame is a scene switching marker frame.

[0202] In some embodiments, the reference frame selection module is specifically used to:

[0203] Determining whether the third video frame indicates that the video content corresponding to the video stream switches from the first scene to the second scene;

[0204] If so, the third video frame is determined as a candidate long-term reference frame.

[0205] In some embodiments, the plurality of candidate reference frames include candidate short-term reference frames and candidate long-term reference frames.

[0206] In some embodiments, the feedback information includes a data packet sequence number;

[0207] In some embodiments, the video frame determination module 902 is specifically configured to:

[0208] Determine, based on the data packet sequence number, position information of the first data packet in the encoded video stream;

[0209] The first video frame is determined based on the position information.

[0210] In some embodiments, the reference frame determination module 904 may be specifically configured to:

[0211] Traversing the multiple candidate reference video frames in sequence, and determining whether any candidate reference video frame has a corresponding relationship with any of the invalid video frames;

[0212] Eliminating the candidate reference video frame corresponding to any of the invalid video frames from the encoder buffer;

[0213] The candidate reference video frames remaining in the encoder buffer are determined as the valid reference video frames.

[0214] In some embodiments, the reference frame selection module is specifically used to:

[0215] Sorting multiple candidate long-term reference frames according to generation time to generate a long-term reference frame list;

[0216] The candidate long-term reference frame corresponding to the first element in the long-term reference frame list is deleted.

[0217] In some embodiments, the feedback information may include a data packet sequence number; the video frame determination module 902 is specifically configured to:

[0218] Determine, based on the data packet sequence number, position information of the first data packet in the encoded video stream;

[0219] The first video frame is determined based on the position information.

[0220] Figure 8 The video encoding device can perform Figure 2 The implementation principle and technical effect of the video encoding method described in the embodiment are not described in detail. The specific way in which each module and unit performs operations in the video encoding device in the above embodiment has been described in detail in the embodiment of the method, and will not be described in detail here.

[0221] Fig.10 A block diagram of a video decoding device provided in an embodiment of the present application is shown. Fig.10 As shown, the video decoding device may specifically include:

[0222] The data packet receiving module 1001 is used to receive a first data packet from an encoder;

[0223] A feedback information generating module 1002, configured to generate feedback information when decoding of the first data packet fails;

[0224] A feedback information sending module 1003 is used to send the feedback information to the encoder, so that the encoder determines the first video frame corresponding to the first data packet from the encoded video stream in response to the feedback information; marks the video frame between the first video frame and the second video frame as an invalid video frame, wherein the second video frame is a previous video frame of the third video frame currently encoded by the encoder; determines a target reference frame from a plurality of candidate reference frames pre-stored in the encoder cache and synchronized to the decoder, wherein the plurality of candidate reference frames include candidate short-term reference frames and candidate long-term reference frames; switches the reference frame of the third video frame from the second video frame to the target reference frame, so as to encode the third video frame based on the target reference frame and generate a second data packet;

[0225] The data packet receiving module 1004 is used to receive the second data packet sent by the encoder;

[0226] The decoding module 1005 is configured to decode the second data packet based on the target reference frame stored in the decoder buffer.

[0227] In some embodiments, the video decoding apparatus may further include:

[0228] A third data packet receiving module, configured to receive a third data packet sent by the encoder, wherein the third data packet is generated by encoding the target reference frame, and the third data packet is marked as a candidate long-term reference frame;

[0229] A third data packet decoding module, configured to decode the third data packet to obtain the target reference frame;

[0230] A reference frame buffer module, used for storing the target reference frame in the decoder buffer;

[0231] In some embodiments, the decoding module 1005 is specifically used to: obtain the pre-stored target reference frame from the decoder cache;

[0232] The second data packet is decoded based on the target reference frame.

[0233] Fig.10 The video decoding device can perform Figure 8 The implementation principle and technical effect of the video decoding method described in the embodiment are not described in detail. The specific way in which each module and unit performs operations in the video decoding device in the above embodiment has been described in detail in the embodiment of the method, and will not be described in detail here.

[0234] In a possible design, the video encoding device and the video decoding device provided in the embodiments of the present application can be implemented as a computing device, such as Fig.11 As shown, the computing device may include a storage component 1101 and a processing component 1102;

[0235] The storage component 1101 stores one or more computer instructions, wherein the one or more computer instructions are called and executed by the processing component 1102 to implement the video encoding method and the video decoding method provided in the embodiments of the present application.

[0236] Of course, the computing device may also include other components, such as input / output interfaces, communication components, etc. The input / output interface provides an interface between the processing component and the peripheral interface module, which may be an output device, an input device, etc. The communication component is configured to facilitate wired or wireless communication between the computing device and other devices.

[0237] Among them, the computing device can be a physical device or an elastic computing host provided by a cloud computing platform, etc. In this case, the computing device can refer to a cloud server, and the above-mentioned processing components, storage components, etc. can be basic server resources rented or purchased from the cloud computing platform.

[0238] When the computing device is a physical device, it can be implemented as a distributed cluster consisting of multiple servers or terminal devices, or as a single server or a single terminal device.

[0239] The embodiment of the present application further provides a computer-readable storage medium storing a computer program, which, when executed by a computer, can implement the video encoding method and the video decoding method provided in the embodiment of the present application.

[0240] The embodiment of the present application also provides a computer program product, including a computer program, which, when executed by a computer, can implement the video encoding method and the video decoding method provided in the embodiment of the present application.

[0241] The processing components in the above corresponding embodiments may include one or more processors to execute computer instructions to complete all or part of the steps in the above method. Of course, the processing components may also be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above method.

[0242] The storage component is configured to store various types of data to support operations in the device. The storage component can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0243] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0244] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0245] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0246] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A video encoding method, characterized in that: The method comprises: receiving feedback information from a decoder, the feedback information being used to indicate a result that the decoder fails to decode a first data packet received from the encoder; In response to the feedback information, determining a first video frame corresponding to the first data packet from the encoded video stream; Marking a video frame between the first video frame and a second video frame as an invalid video frame, wherein the second video frame is a previous video frame of a third video frame currently encoded by the encoder; Determine a target reference frame from a plurality of candidate reference frames pre-stored in an encoder buffer and synchronized to the decoder; Switching the reference frame of the third video frame from the second video frame to the target reference frame, so as to encode the third video frame based on the target reference frame and generate a second data packet; The second data packet is sent to the decoder so that the decoder decodes the second data packet based on the target reference frame stored in a decoder buffer.

2. The method according to claim 1, characterized in that Determining a target reference frame from a plurality of candidate reference frames pre-stored in the encoder cache and synchronized to the decoder comprises: Screening out invalid reference video frames corresponding to the invalid video frames from the multiple candidate reference video frames to determine at least one valid reference video frame; The target reference frame is determined from the at least one valid reference video frame.

3. The method according to claim 1, characterized in that The method further comprises: During the encoding process of the video stream by the encoder, determining whether to configure the third video frame currently being encoded as a candidate long-term reference frame based on a preset rule; If so, the third video frame is written into the encoder cache as a candidate long-term reference frame, and the third data packet generated by encoding the third video frame is marked as a long-term reference frame, and the third data packet is sent to the decoder, so that the decoder writes the third video frame into the decoder cache.

4. The method according to claim 3, characterized in that The preset rule includes a preset spacing between adjacent long-term reference frames; The determining, based on a preset rule, whether to configure the third video frame currently being encoded as a candidate long-term reference frame comprises: determining a first distance between the third video frame and a previous fourth video frame configured as a candidate reference frame; When the first interval reaches the preset interval, the third video frame is determined as a candidate long-term reference frame.

5. The method according to claim 4, characterized in that After determining the third video frame as a candidate long-term reference frame, the method further includes: Determining whether the number of candidate long-term reference frames stored in the encoder cache reaches a preset number threshold; If so, the candidate long-term reference frame with the earliest generation time is deleted from the encoder cache, and the candidate long-term reference frame corresponding to the third video frame is written into the encoder cache.

6. The method according to claim 1, characterized in that The feedback information is used to represent the result of the decoder failing to decode the multiple first data packets received from the encoder; The step of determining, in response to the feedback information, a first video frame corresponding to the first data packet from the encoded video stream comprises: In response to the feedback information, determining, from the encoded video stream, a plurality of candidate video frames respectively corresponding to the plurality of first data packets; Based on the timestamps of the multiple candidate video frames, a candidate video frame with the earliest encoding time is determined as the first video frame.

7. The method according to claim 3, characterized in that The preset rule includes whether the third video frame is a scene switching marker frame; The determining, based on a preset rule, whether to configure the third video frame currently being encoded as a candidate long-term reference frame comprises: Determining whether the third video frame indicates that the video content corresponding to the video stream switches from the first scene to the second scene; If so, the third video frame is determined as a candidate long-term reference frame.

8. The method according to claim 1, characterized in that The plurality of candidate reference frames include candidate short-term reference frames and candidate long-term reference frames.

9. The method according to claim 1, characterized in that: The feedback information includes a data packet sequence number; The step of determining, in response to the feedback information, a first video frame corresponding to the first data packet from the encoded video stream comprises: Determine, based on the data packet sequence number, position information of the first data packet in the encoded video stream; The first video frame is determined based on the position information.

10. The method according to claim 2, characterized in that The step of screening out invalid reference video frames corresponding to the invalid video frames from the plurality of candidate reference video frames and determining at least one valid reference video frame comprises: Traversing the multiple candidate reference video frames in sequence, and determining whether any candidate reference video frame has a corresponding relationship with any of the invalid video frames; Eliminating the candidate reference video frame corresponding to any of the invalid video frames from the encoder buffer; The candidate reference video frames remaining in the encoder buffer are determined as the valid reference video frames.

11. The method according to claim 5, characterized in that Deleting the candidate reference frame with the earliest generation time from the encoder cache comprises: Sorting multiple candidate long-term reference frames according to generation time to generate a long-term reference frame list; The candidate long-term reference frame corresponding to the first element in the long-term reference frame list is deleted.

12. The method according to claim 1, characterized in that The feedback information includes a data packet sequence number; The step of determining, in response to the feedback information, a first video frame corresponding to the first data packet from the encoded video stream comprises: Determine, based on the data packet sequence number, position information of the first data packet in the encoded video stream; The first video frame is determined based on the position information.

13. A video decoding method, characterized in that: Applied to a decoder, the method comprises: receiving a first data packet from an encoder; In case of a failure in decoding the first data packet, generating feedback information; The feedback information is sent to the encoder, so that the encoder determines a first video frame corresponding to the first data packet from the encoded video stream in response to the feedback information; marks a video frame between the first video frame and the second video frame as an invalid video frame, wherein the second video frame is a previous video frame of a third video frame currently encoded by the encoder; determines a target reference frame from a plurality of candidate reference frames pre-stored in the encoder cache and synchronized to the decoder; switches a reference frame of the third video frame from the second video frame to the target reference frame, so as to encode the third video frame based on the target reference frame and generate a second data packet; receiving the second data packet sent by the encoder; The second data packet is decoded based on the target reference frame stored in the decoder buffer.

14. The method according to claim 13, characterized in that The method further comprises: receiving a third data packet sent by the encoder, wherein the third data packet is generated by encoding the target reference frame, and the third data packet is marked as a candidate long-term reference frame; Decoding the third data packet to obtain the target reference frame; storing the target reference frame in the decoder buffer; The decoding of the second data packet based on the target reference frame stored in the decoder buffer comprises: Acquire the pre-stored target reference frame from the decoder buffer; The second data packet is decoded based on the target reference frame.

15. A computing device, characterized in that: including a processing component and a storage component; The storage component stores one or more computer instructions; the one or more computer instructions are used to be called and executed by the processing component to implement the video encoding method as described in any one of claims 1 to 12, or to implement the video decoding method as described in any one of claims 13 to 14.

16. A computer program product, characterized in that The computer program product comprises a computer program code, and when the computer program code is executed by a computer, the video encoding method according to any one of claims 1 to 12 is implemented, or the video decoding method according to any one of claims 13 to 14 is implemented.

Images