Video encoding method, video decoding method, computing device and computer program product

Abstract

The embodiments of the present application provide a video encoding method, a video decoding method, an apparatus, a computing device, and a computer program product. The video encoding method includes: receiving feedback information from a decoder; determining, in response to the feedback information, a first video frame corresponding to the first data packet from the encoded video stream; marking the video frame between the first video frame and the second video frame as an invalid video frame; determining a target reference frame from a plurality of candidate reference frames pre-stored in the encoder cache and synchronized to the decoder; switching the reference frame of the third video frame from the second video frame to the target reference frame; and sending the second data packet to the decoder. The technical solution provided by the embodiments of the present application prevents errors from further propagating in the video stream.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image processing technology, and in particular to a video encoding method, a video decoding method, an apparatus, a computing device, and a computer program product. Background Art

[0002] During real-time video encoding and decoding, data transmission between the encoder and decoder typically relies on the Real-Time Communication (RTC) protocol, which transmits data packets over a network or other medium. To ensure high real-time performance, these real-time transport protocols are often based on the User Datagram Protocol (UDP), as UDP offers fast transmission speeds and is suitable for transmitting real-time data. However, UDP is an unreliable transport protocol that does not guarantee the orderly transmission of data or that data will reliably reach its destination, which introduces the risk of packet loss.

[0003] Packet loss can cause the decoder to be unable to correctly decode a video frame. This affects not only the current frame corresponding to the lost packet, but also all subsequent frames that directly or indirectly depend on it. In the worst case, the number of damaged frames may continue to increase until the end of the video.

[0004] Therefore, how to avoid the error propagation generated in related technologies has become a technical problem that needs to be solved urgently. Summary of the Invention

[0005] Embodiments of the present application provide a video encoding method, a video decoding method, an apparatus, a computing device, and a computer program product.

[0006] In a first aspect, an embodiment of the present application provides a video encoding method, applied to an encoder, the method comprising:

[0007] receiving feedback information from a decoder, the feedback information being used to indicate a result that the decoder fails to decode a first data packet received from the encoder;

[0008] In response to the feedback information, determining a first video frame corresponding to the first data packet from the encoded video stream;

[0009] Marking a video frame between the first video frame and a second video frame as an invalid video frame, wherein the second video frame is a video frame preceding a third video frame currently encoded by the encoder;

[0010] Determining a target reference frame from a plurality of candidate reference frames pre-stored in an encoder buffer and synchronized to the decoder, wherein the plurality of candidate reference frames include candidate short-term reference frames and candidate long-term reference frames;

[0011] Switching a reference frame of the third video frame from the second video frame to the target reference frame, encoding the third video frame based on the target reference frame, and generating a second data packet;

[0012] The second data packet is sent to the decoder so that the decoder decodes the second data packet based on the target reference frame stored in a decoder buffer.

[0013] In a second aspect, an embodiment of the present application provides a video decoding method, which is applied to a decoder, and the method includes:

[0014] receiving a first data packet from an encoder;

[0015] generating feedback information in case decoding of the first data packet fails;

[0016] The feedback information is sent to the encoder so that the encoder determines, in response to the feedback information, a first video frame corresponding to the first data packet from the encoded video stream; marks a video frame between the first video frame and a second video frame as an invalid video frame, wherein the second video frame is a video frame preceding a third video frame currently encoded by the encoder; determines a target reference frame from a plurality of candidate reference frames pre-stored in an encoder cache and synchronized to the decoder, wherein the plurality of candidate reference frames include candidate short-term reference frames and candidate long-term reference frames; and switches a reference frame of the third video frame from the second video frame to the target reference frame, so as to encode the third video frame based on the target reference frame and generate a second data packet;

[0017] receiving the second data packet sent by the encoder;

[0018] The second data packet is decoded based on the target reference frame stored in the decoder buffer.

[0019] In a third aspect, an embodiment of the present application provides a video encoding device, the device comprising:

[0020] An information receiving module, configured to receive feedback information from a decoder, wherein the feedback information is used to indicate a result that the decoder fails to decode a first data packet received from the encoder;

[0021] a video frame determining module, configured to determine, in response to the feedback information, a first video frame corresponding to the first data packet from the encoded video stream;

[0022] a marking module, configured to mark a video frame between the first video frame and a second video frame as an invalid video frame, wherein the second video frame is a video frame preceding a third video frame currently encoded by the encoder;

[0023] a reference frame determination module, configured to determine a target reference frame from a plurality of candidate reference frames pre-stored in an encoder cache and synchronized to the decoder, wherein the plurality of candidate reference frames include candidate short-term reference frames and candidate long-term reference frames;

[0024] an encoding module, configured to switch a reference frame of the third video frame from the second video frame to the target reference frame, so as to encode the third video frame based on the target reference frame and generate a second data packet;

[0025] The data packet sending module is configured to send the second data packet to the decoder so that the decoder decodes the second data packet based on the target reference frame stored in a decoder buffer.

[0026] In a fourth aspect, an embodiment of the present application provides a video decoding device, the device comprising:

[0027] A data packet receiving module, configured to receive a first data packet from an encoder;

[0028] A feedback information generating module, configured to generate feedback information when decoding of the first data packet fails;

[0029] a feedback information sending module, configured to send the feedback information to the encoder, so that the encoder, in response to the feedback information, determines a first video frame corresponding to the first data packet from the encoded video stream; marks a video frame between the first video frame and a second video frame as an invalid video frame, wherein the second video frame is a video frame preceding a third video frame currently encoded by the encoder; determines a target reference frame from a plurality of candidate reference frames pre-stored in an encoder cache and synchronized to the decoder, wherein the plurality of candidate reference frames include candidate short-term reference frames and candidate long-term reference frames; and switches a reference frame of the third video frame from the second video frame to the target reference frame, so as to encode the third video frame based on the target reference frame and generate a second data packet;

[0030] A data packet receiving module, configured to receive the second data packet sent by the encoder;

[0031] A decoding module is configured to decode the second data packet based on a target reference frame stored in a decoder buffer.

[0032] In a fifth aspect, an embodiment of the present application provides a computing device, including a processing component and a storage component;

[0033] The storage component stores one or more computer instructions; the one or more computer instructions are used to be called and executed by the processing component to implement the video encoding method provided in the embodiment of the present application, or to implement the video decoding method provided in the embodiment of the present application.

[0034] In a sixth aspect, a computer program product is provided in an embodiment of the present application, and the computer program product includes a computer program code. When the computer program code is executed by a computer, it implements the video encoding method provided in the embodiment of the present application, or implements the video decoding method provided in the embodiment of the present application.

[0035] In an embodiment of the present application, the following steps are adopted: receiving feedback information from a decoder, the feedback information being used to characterize a result of a decoder failing to decode a first data packet received from an encoder; determining a first video frame corresponding to the first data packet from an encoded video stream in response to the feedback information; marking a video frame between the first video frame and the second video frame as an invalid video frame, wherein the second video frame is a previous video frame of a third video frame currently encoded by the encoder; determining a target reference frame from a plurality of candidate reference frames pre-stored in an encoder cache and synchronized to a decoder, wherein the plurality of candidate reference frames include a candidate short-term reference frame and a candidate long-term reference frame; switching a reference frame of the third video frame from the second video frame to A target reference frame is used to encode a third video frame based on the target reference frame to generate a second data packet; the second data packet is sent to the decoder so that the decoder decodes the second data packet based on the target reference frame stored in the decoder cache. After receiving the decoding failure information fed back by the decoder, the encoder marks all video frames that may be affected as invalid frames, and redetermines the reference frame for the currently encoded intended video frame, encodes the third video frame based on the redetermined reference frame, and, since the redetermined reference frame has been synchronized to the decoder, the decoder can normally decode the second data packet based on the reference frame, thereby avoiding further propagation of errors in the video stream.

[0036] These and other aspects of the present application will become more readily apparent from the description of the following embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0038] Figure 1 A schematic diagram of video encoding and decoding in related art is shown;

[0039] Figure 2 A flowchart of a video encoding method provided by an embodiment of the present application is shown;

[0040] Figure 3 A schematic diagram showing a decoder sending feedback information to an encoder;

[0041] Figure 4 A schematic diagram showing an encoder marking invalid video frames;

[0042] Figure 5 A schematic diagram showing an encoder switching reference frame;

[0043] Figure 6 A schematic diagram of a video encoding method provided by an embodiment of the present application is shown;

[0044] Figure 7 A schematic diagram showing an encoder updating a reference frame;

[0045] Figure 8 A flowchart of a video decoding method provided by one embodiment of the present application is shown;

[0046] Figure 9 A block diagram of a video encoding device provided in an embodiment of the present application is shown;

[0047] Figure 10 A block diagram of a video decoding device provided in an embodiment of the present application is shown;

[0048] Figure 11 A block diagram of a computing device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0049] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0050] In some of the processes described in the specification and claims of this application and the above-mentioned figures, multiple operations that appear in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this document or may be executed in parallel. The serial numbers of the operations, such as 101, 102, etc., are only used to distinguish between different operations, and the serial numbers themselves do not represent any order of execution. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this document are used to distinguish different messages, devices, modules, etc., and do not represent a sequential order, nor do they limit "first" and "second" to being different types.

[0051] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0052] During real-time video encoding and decoding, data transmission between the encoder and decoder typically relies on the Real-Time Communication (RTC) protocol, which transmits data packets over a network or other medium. To ensure high real-time performance, these real-time transport protocols are often based on the User Datagram Protocol (UDP), as UDP offers fast transmission speeds and is suitable for transmitting real-time data. However, UDP is an unreliable transport protocol that does not guarantee the orderly transmission of data or that data will reliably reach its destination, which introduces the risk of packet loss.

[0053] Packet loss can cause the decoder to be unable to correctly decode video frames. This affects not only the current frame corresponding to the lost packet, but also all subsequent frames that directly or indirectly depend on it. As the number of damaged frames increases, video quality gradually degrades until the encoder generates a new key frame (Key Frame) or intra-frame reference frame (Intra Frame or I-Frame). This frame can end the dependency between frames and prevent further increase in the number of damaged frames.

[0054] In the worst case, if the encoder does not regularly insert keyframes or intra-frame reference frames, the number of damaged frames may continue to increase until the end of the video. Figure 1As shown in the figure, if a critical data packet (e.g., sequence number 2, POC 2, i.e., picture order count 2) is lost, the decoder will not be able to correctly decode this frame, and all subsequent frames that directly or indirectly reference this frame (e.g., sequence numbers 3, 4, 5, etc.) will also be damaged. This can cause video playback to experience freezes, screen distortion, or other quality issues, seriously affecting the viewing experience. Therefore, to ensure the reliability and quality of video encoding and decoding, the Real-time Transport Protocol needs to take additional measures to reduce the impact of packet loss, such as using retransmission mechanisms and error detection and correction technologies to enhance the stability of data transmission.

[0055] In related technologies, in order to solve the video quality problem caused by data packet loss during real-time video encoding and decoding, the following methods are usually adopted:

[0056] First, the encoder can periodically insert key frames or intra-frame reference frames. While this method can repair damaged frames, it also has two major problems. The first problem is that key frames or intra-frame reference frames are usually high in bit count, which causes a sudden surge in bit rate and is difficult to adapt to low-bandwidth transmission conditions. The second problem is that the interval between key frames can be large, such as several seconds or even longer, which means that the user will still see the picture problem before the next key frame is reached.

[0057] Secondly, the encoder can choose not to insert key frames or intra-frame reference frames, but instead use intra-frame prediction row by row or column by column across several frames through intra-frame coding refresh (Intra Refresh or Gradual Decoding Refresh). This method avoids the problem of instantaneous bit rate spikes, but the higher bit count generated by the intra-frame prediction module still consumes a lot of bandwidth, and damaged images may take several seconds or longer to recover.

[0058] The third method is to not insert key frames or intra-frame reference frames in the encoder. Key frames are inserted to repair damaged images only when the decoder reports data packet loss. While this method can avoid the problem of damaged images being unable to be restored for a long time, the problem of instantaneous bitrate surges still exists.

[0059] Finally, error-tolerance information can be added to the transmitted data packets, and the debugging and error-correction capabilities of the transmission protocol can be leveraged to prevent packet loss. While this approach can mitigate video quality issues caused by packet loss, it increases bandwidth, latency, and computing resources. This increase in latency can be unacceptable, especially in real-time transmission scenarios.

[0060] In order to solve the technical problems existing in the related art, an embodiment of the present application provides a video encoding method, which adopts: receiving feedback information from a decoder, the feedback information is used to characterize the result of the decoder failing to decode a first data packet received from the encoder; in response to the feedback information, determining a first video frame corresponding to the first data packet from the encoded video stream; marking the video frame between the first video frame and the second video frame as an invalid video frame, wherein the second video frame is a previous video frame of the third video frame currently encoded by the encoder; determining a target reference frame from a plurality of candidate reference frames pre-stored in the encoder cache and synchronized to the decoder, wherein the plurality of candidate reference frames include candidate short-term reference frames and candidate long-term reference frames; marking the third video frame The reference frame of the video frame is switched from the second video frame to the target reference frame, and the third video frame is encoded based on the target reference frame to generate a second data packet; the second data packet is sent to the decoder so that the decoder decodes the second data packet based on the target reference frame stored in the decoder cache. After receiving the decoding failure information fed back by the decoder, the encoder marks all video frames that may be affected as invalid frames, and re-determines the reference frame for the currently encoded intended video frame, and encodes the third video frame based on the re-determined reference frame. Moreover, since the re-determined reference frame has been synchronized to the decoder, the decoder can decode the second data packet normally based on the reference frame, thereby avoiding the error from propagating further in the video stream. In addition, the video encoding method provided in the embodiment of the present application uses reference frames to deal with the problem of data packet loss, without relying on the encoder to generate key frames or intra-frame prediction modules, thereby avoiding the problem of consuming a large amount of bandwidth. In addition, there is no need to add additional delay and the consumption of computing resources will not be increased. Since the reference frame already in the decoder cache is used, the encoder does not need to generate new key frames every time. In this way, even in the case of limited bandwidth, the impact of data packet loss on video quality can be effectively reduced while maintaining the real-time and smoothness of video transmission.

[0061] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without making creative efforts are within the scope of protection of this application.

[0062] The implementation details of the technical solution of the embodiment of the present application are described in detail below.

[0063] Figure 2 A flow chart of a video encoding method provided by an embodiment of the present application is shown. The video encoding method can be applied to an encoder, such as Figure 2 As shown, the video encoding method may specifically include the following steps:

[0064] 201 : Receive feedback information from a decoder, where the feedback information is used to indicate a result that the decoder fails to decode a first data packet received from an encoder.

[0065] 202. In response to feedback information, determine a first video frame corresponding to a first data packet from the encoded video stream.

[0066] The failure of the decoder to decode the first data packet may be caused by packet loss generated by the data transmission protocol between the encoder and the decoder.

[0067] In an embodiment of the present application, a specific data transmission protocol may be used to transmit data packets between the encoder and the decoder. For example, the encoder and the decoder may utilize a real-time transport protocol based on UDP to transmit data packets. The UDP protocol has the advantage of high real-time performance and is suitable for real-time video transmission. However, it is an unreliable transmission protocol, and therefore, packet loss may occur during the transmission of data packets. For example, in the event of network congestion, signal interference, or network device failure, the data packet may not be fully transmitted from the encoder to the decoder.

[0068] For the decoder, each data packet contains part or all of the information of the video frame. If the first data packet is lost, the decoder may not be able to obtain the complete video frame information, resulting in decoding failure.

[0069] After the decoder fails to decode the first data packet, feedback information may be sent to the encoder to inform the encoder of problems generated during data transmission.

[0070] In one embodiment of the present application, the feedback information may include a data packet sequence number;

[0071] Determining, in response to the feedback information, a first video frame corresponding to the first data packet from the encoded video stream includes:

[0072] Determining position information of the first data packet in the encoded video stream based on the data packet sequence number;

[0073] The first video frame is determined based on the position information.

[0074] Among them, the data packet sequence number can be used to uniquely identify the position of the first data packet in the entire transmission sequence. For example, in a video stream, the data packets are numbered in a certain order, for example, they can be numbered sequentially starting from 1. When the decoder feedbacks that the sequence number of the first data packet that cannot be decoded is 5, the encoder can accurately locate the video frame corresponding to the problem data packet.

[0075] In another embodiment of the present application, the feedback information may further include timestamp information, which may accurately indicate the time position of the video segment to which the first data packet belongs, so that the encoder can quickly find the video frame corresponding to the timestamp information.

[0076] After receiving the feedback information sent by the decoder, the encoder can read the feedback information and determine the first video frame from the encoded video stream based on the data packet sequence number and / or timestamp information included in the feedback information.

[0077] Figure 3 A schematic diagram showing a decoder sending feedback information to an encoder.

[0078] like Figure 3 As shown, 301 may represent an encoder, 3011 may represent an encoded video stream, 302 may represent a decoder, and 3021 may represent a decoded video stream.

[0079] The encoded video stream 3011 may include video frames with sequence numbers 0, 1, 2, and 3, and the encoder 301 may have completed encoding of the video frame with sequence number 3.

[0080] The decoded video stream 3021 may include video frames with sequence numbers 0, 1, and 2, and the decoder 302 may generate a decoding failure when decoding the video frame with sequence number 2.

[0081] In this case, the decoder 302 may send feedback information 203 to the encoder 301 to inform the encoder 301 of the result of decoding failure of the data packet corresponding to the video frame with sequence number 2 through the feedback information 203 .

[0082] 203 : Mark the video frames between the first video frame and the second video frame as invalid video frames, where the second video frame is a video frame preceding the third video frame currently being encoded by the encoder.

[0083] In the embodiments of the present application, since there is often a reference relationship between video frames, subsequent video frames may refer to previous video frames for encoding and decoding. Therefore, if the decoding of the first video frame fails, the subsequent video frames that rely on the first video frame may not obtain the correct reference information, resulting in decoding errors.

[0084] The encoder can prevent the error from propagating further in the video stream by marking the video frames starting from the video frame that generates the error and before the currently encoded video frame as invalid video frames.

[0085] Figure 4 A schematic diagram showing an encoder marking invalid video frames.

[0086] like Figure 4As shown, 401 can represent a coded video stream, and the coded video stream 401 can include video frames with serial numbers 0, 1, 2, 3, and 4, wherein serial number 2 can be the first video frame, and serial number 4 can be the third video frame currently being prepared for encoding. Thus, serial number 3 is the second video frame.

[0087] In this case, the encoder may mark both sequence number 2 and sequence number 3 as invalid video frames.

[0088] 204 , determine a target reference frame from a plurality of candidate reference frames pre-stored in the encoder buffer and synchronized to the decoder.

[0089] In one embodiment of the present application, the multiple candidate reference frames may include candidate short-term reference frames and candidate long-term reference frames.

[0090] During the process of encoding the video stream by the encoder, certain video frames can be dynamically configured as reference frames based on preset rules, and the video frames configured as reference frames can be stored in the encoder cache. In addition, in order to enable the decoder to decode accurately, these reference frames will also be synchronized with the decoder, so that the decoder cache will also store the same reference frames as those in the encoder cache.

[0091] After marking invalid video frames, the encoder can reselect appropriate target reference frames from the encoder cache to encode subsequent video frames. This can break the encoding chain that may be affected by errors, rebuild a reliable encoding foundation, and improve video quality and stability.

[0092] 205 : Switch the reference frame of the third video frame from the second video frame to the target reference frame, and encode the third video frame based on the target reference frame to generate a second data packet.

[0093] In an embodiment of the present application, after the decoder fails to decode the first data packet, in order to prevent error propagation, the encoder marks the videos between the first video frame and the second video frame as invalid video frames. In this case, the second video frame that was originally used as the reference frame of the third video frame is no longer available. Therefore, the encoder can determine a new target reference frame from multiple candidate reference frames in the encoder cache, and switch the reference frame of the third video frame from the second video frame to the re-determined target reference frame.

[0094] In an embodiment of the present application, after determining the target reference frame as the reference frame of the third video frame, the encoder can recalculate the difference information between the third video frame and the target reference frame, and encode the third video frame according to the difference information to generate a second data packet.

[0095] After generating the second data packet, the encoder may send the second data packet to the decoder.

[0096] Figure 5 A schematic diagram showing an encoder switching reference frame.

[0097] like Figure 5 As shown, 501 can represent an encoded video stream, and the encoded video stream 501 can include video frames with serial numbers 0, 1, 2, 3, and 4, where serial number 2 can be the first video frame, serial number 3 can be the second video frame, and serial number 4 can be the third video frame currently being prepared for encoding.

[0098] Since sequence numbers 2 and 3 are both marked as invalid video frames, assuming that sequence number 0 is stored as a long-term reference frame in the encoder cache, the reference frame of sequence number 4 can be switched from sequence number 3 to sequence number 0.

[0099] 106 : Send the second data packet to the decoder, so that the decoder decodes the second data packet based on the target reference frame stored in the decoder buffer.

[0100] After receiving the second data packet, the decoder can decode the second data packet based on the target reference frame stored in its cache. The decoder can first read the information in the data packet and then, in combination with the target reference frame in the cache, restore the content of the third video frame to correctly decode the second data packet.

[0101] In an embodiment of the present application, the following steps are adopted: receiving feedback information from a decoder, the feedback information being used to characterize a result of a decoder failing to decode a first data packet received from an encoder; determining a first video frame corresponding to the first data packet from an encoded video stream in response to the feedback information; marking a video frame between the first video frame and the second video frame as an invalid video frame, wherein the second video frame is a previous video frame of a third video frame currently encoded by the encoder; determining a target reference frame from a plurality of candidate reference frames pre-stored in an encoder cache and synchronized to a decoder, wherein the plurality of candidate reference frames include a candidate short-term reference frame and a candidate long-term reference frame; switching a reference frame of the third video frame from the second video frame to A target reference frame is used to encode a third video frame based on the target reference frame to generate a second data packet; the second data packet is sent to the decoder so that the decoder decodes the second data packet based on the target reference frame stored in the decoder cache. After receiving the decoding failure information fed back by the decoder, the encoder marks all video frames that may be affected as invalid frames, and redetermines the reference frame for the currently encoded intended video frame, encodes the third video frame based on the redetermined reference frame, and, since the redetermined reference frame has been synchronized to the decoder, the decoder can normally decode the second data packet based on the reference frame, thereby avoiding further propagation of errors in the video stream.

[0102] Figure 6A schematic diagram of a video encoding method provided in an embodiment of the present application is shown.

[0103] As shown in 6, 601 may represent an encoder, 6011 may represent an encoded video stream, 602 may represent a decoder, and 6021 may represent a decoded video stream.

[0104] The encoded video stream 6011 may include video frames numbered 0, 1, 2, 3, 4, and 5, and the decoded video stream 6021 may include video frames numbered 0, 1, 2, 3, 4, and 5.

[0105] Among them, the decoder 602 can notify the encoder through feedback information that it failed to decode sequence number 2. Therefore, after receiving the feedback information, the encoder 601 marks sequence number 2 and sequence number 3 as invalid video frames, and determines the reference video frame of sequence number 4 as the long-term reference frame sequence number 1, that is, sequence number 0, and encodes sequence number 4 according to the long-term reference frame sequence number 1, generates a second data packet and sends it to the decoder 602.

[0106] Therefore, after receiving the second data packet, the decoder 602 can decode the second data packet based on the long-term reference frame stored in the decoder cache to obtain sequence number 4. Further, it can continue to decode with reference to sequence number 4 to obtain sequence number 5.

[0107] The video encoding method provided by the embodiment of the present application can prevent the error from further propagating when the decoder fails to decode. Specifically, in this example, only sequence numbers 2 and 3 fail to decode, and sequence numbers 4, 5 and subsequent ones will not be affected.

[0108] In some embodiments, determining the target reference frame from a plurality of candidate reference frames pre-stored in the encoder cache and synchronized to the decoder may be specifically implemented as follows:

[0109] Screening out invalid reference video frames corresponding to invalid video frames from a plurality of candidate reference video frames to determine at least one valid reference video frame;

[0110] A target reference frame is determined from at least one valid reference video frame.

[0111] After the video frames between the first and second video frames are marked as invalid video frames, the corresponding reference video frames may also become unreliable. These reference video frames corresponding to the invalid video frames may be determined as invalid reference video frames. For example, if an invalid video frame is a frame that represents a key scene change, and some reference video frames are predictively encoded based on this invalid video frame, then these reference video frames may no longer be applicable in the current situation.

[0112] In some embodiments, filtering out invalid reference video frames corresponding to the invalid video frames from the multiple candidate reference video frames and determining at least one valid reference video frame may be specifically implemented as follows:

[0113] Traversing the plurality of candidate reference video frames in sequence, and determining whether any candidate reference video frame has a corresponding relationship with any of the invalid video frames;

[0114] Eliminating a candidate reference video frame corresponding to any of the invalid video frames from the encoder cache;

[0115] The candidate reference video frames remaining in the encoder buffer are determined as the valid reference video frames.

[0116] In an embodiment of the present application, the encoder may check multiple candidate reference video frames in the encoder cache one by one to determine whether each candidate reference video frame is an invalid reference video frame. If it is an invalid reference video frame, it is filtered out from the encoder cache.

[0117] After filtering out invalid reference video frames, at least one remaining candidate reference frame can be a valid reference video frame. These valid reference video frames are not affected by invalid video frames and can serve as the basis for determining the target reference frame.

[0118] In some embodiments, the video encoding method may further include:

[0119] During encoding of the video stream by the encoder, determining, based on a preset rule, whether to configure a third video frame currently being encoded as a candidate long-term reference frame;

[0120] If so, write the third video frame into the encoder cache as a candidate long-term reference frame, mark the third data packet generated by encoding the third video frame as a long-term reference frame, and send the third data packet to the decoder, so that the decoder writes the third video frame into the decoder cache.

[0121] In an embodiment of the present application, if a third video frame is determined to be a candidate long-term reference frame according to a preset rule, the encoder can write it to the encoder cache. For example, the encoder can store the complete data or key information of the third video frame in the cache, while also recording its related encoding parameters and attributes, such as the frame number and timestamp. Thus, when a reference frame needs to be reselected or a decoding error requires recovery, the encoder can quickly retrieve these candidate long-term reference frames from the cache, improving processing efficiency.

[0122] After encoding the third video frame to generate a third data packet, the encoder may mark the third data packet as a long-term reference frame. This marking information may be sent along with the third data packet to the decoder to inform the decoder that the video frame corresponding to the third data packet is a long-term reference frame.

[0123] After receiving the third data packet, the decoder can write the third video frame into the decoder cache based on the marker information. This allows the decoder to use this long-term reference frame during subsequent decoding to improve decoding accuracy and stability. For example, if data packet loss or decoding errors occur, the decoder can use the long-term reference frame as an alternative reference to resume normal decoding.

[0124] In some embodiments, the preset rule includes a preset distance between adjacent long-term reference frames.

[0125] In some embodiments, determining whether to configure the third video frame currently being encoded as a candidate long-term reference frame based on a preset rule may be specifically implemented as follows:

[0126] determining a first distance between a third video frame and a previous fourth video frame configured as a candidate reference frame;

[0127] When the first distance reaches a preset distance, the third video frame is determined as a candidate long-term reference frame.

[0128] When the encoder determines whether to configure the currently encoded third video frame as a candidate long-term reference frame, it can first determine the first distance between the currently encoded third video frame and the previous fourth video frame configured as a candidate reference frame. In an embodiment of the present application, the first distance can be measured in a variety of ways, such as a time interval (such as the length of time the video is played), a frame interval, etc. For example, if the video is played at a frame rate of 30 frames per second, the distance can be determined by calculating the frame difference between the third video frame and the fourth video frame. Assuming that the fourth video frame appears at the 10th second of the video playback and the third video frame appears at the 20th second, and the frame rate is 30 frames per second, then the frame difference between them is (20-10)×30=300 frames, and this frame difference is the first distance.

[0129] Next, the encoder can compare the calculated first spacing with a preset spacing. The preset spacing is a value pre-set based on factors such as video characteristics, encoding requirements, and transmission environment. If the first spacing reaches the preset spacing, it means that the distance between the third video frame and the previous long-term reference frame is appropriate, and the third video frame can be determined as a candidate long-term reference frame. For example, if the preset spacing is 200 frames and the calculated first spacing is 300 frames, then the conditions for determining the third video frame as a candidate long-term reference frame are met.

[0130] In an embodiment of the present application, by setting a preset spacing as a preset rule for determining a long-term reference frame, the efficiency and quality of video encoding can be better balanced. If the spacing between adjacent long-term reference frames is too small, the selection of long-term reference frames may be too frequent during the encoding process, increasing the complexity and computational complexity of the encoding. At the same time, the long-term reference frames may not be able to fully play their role due to the high similarity between the long-term reference frames. On the contrary, if the spacing is too large, there may be a lack of long-term reference frames that are close enough for effective recovery when data packets are lost or decoding errors occur, thereby affecting the quality and stability of the video.

[0131] In another embodiment of the present application, the preset rule may further include whether the third video frame is a scene switching marker frame;

[0132] In some embodiments, determining whether to configure the third video frame currently being encoded as a candidate long-term reference frame based on a preset rule includes:

[0133] determining whether the third video frame indicates a switch from a first scene to a second scene in video content corresponding to the video stream;

[0134] If so, the third video frame is determined as a candidate long-term reference frame.

[0135] In an embodiment of the present application, since in the video content, the picture content of the second scene after the scene switch is usually significantly different from the picture content of the first scene before the scene switch, by setting the third video frame marking the scene switch as a candidate long-term reference frame, the candidate long-term reference frame can be used to provide a reference for subsequent frame encoding in the scene.

[0136] In an embodiment of the present application, for example, when the third video frame is the first frame, or the first few frames, of the second scene, it can be determined that the third video frame is a scene switching frame.

[0137] In other embodiments of the present application, the third video frame can also be determined as a candidate long-term reference frame when the third video frame indicates that an important object appears (for example, the protagonist appears) or disappears (for example, the object is blocked or moves out of the screen) in the video content corresponding to the video stream.

[0138] In some embodiments, after determining the third video frame as a candidate long-term reference frame, the video encoding method may further include:

[0139] determining whether the number of candidate long-term reference frames stored in the encoder buffer reaches a preset number threshold;

[0140] If so, the candidate long-term reference frame with the earliest generation time is deleted from the encoder cache, and the candidate long-term reference frame corresponding to the third video frame is written into the encoder cache.

[0141] In the embodiments of the present application, the encoder's cache space is limited. Excessive long-term reference frames will consume a significant amount of storage resources, potentially impacting encoder performance and efficiency. Furthermore, an excessive number of long-term reference frames may increase computational complexity when selecting reference frames, reducing encoding efficiency. By setting a preset threshold, the number of long-term reference frames in the cache can be kept within a manageable range, avoiding excessive resource consumption and improving encoding efficiency.

[0142] When the encoder determines that the number of long-term reference frames stored in the encoder cache has reached a preset threshold, the candidate long-term reference frame with the earliest generation time may be deleted from the encoder cache. This is because earlier generated reference frames are generally less relevant to the current video content and are further away in time, making them less valuable as reference frames.

[0143] In some embodiments, deleting the candidate reference frame with the earliest generation time from the encoder cache may be specifically implemented as follows:

[0144] Sort multiple candidate long-term reference frames according to generation time to generate a long-term reference frame list;

[0145] The candidate long-term reference frame corresponding to the first element in the long-term reference frame list is deleted.

[0146] In an embodiment of the present application, a long-term reference frame list can be maintained by sorting multiple candidate long-term reference frames according to generation time. When deletion is required, the first element in the list, that is, the candidate reference frame with the earliest generation time, can be directly removed.

[0147] After deleting the candidate reference frame with the earliest generation time, the candidate long-term reference frame corresponding to the third video frame can be written into the encoder cache. This ensures that the latest long-term reference frame with higher reference value is always retained in the cache.

[0148] The writing process includes storing relevant information of the third video frame (such as frame data, encoding parameters, etc.) into the cache and updating the cache management data structure so that this new long-term reference frame can be quickly accessed and used in subsequent encoding processes.

[0149] By limiting the number of candidate long-term reference frames, older candidate long-term reference frames can be eliminated in a timely manner, ensuring that the candidate long-term reference frames in the cache have high validity and reference value.

[0150] Figure 7 A schematic diagram showing an encoder updating a reference frame.

[0151] like Figure 7As shown, the encoded video stream may include video frames with sequence numbers 0, 1, 2, 3, 4, 5, and 6, wherein sequence numbers 0 and 2 may be set as candidate long-term reference frames by default.

[0152] In this example, the preset number is, for example, 2, and the preset interval is, for example, 1. When the encoder encodes to sequence number 4, since the first interval between sequence number 4 and sequence number 2 is 1, the first interval reaches the preset interval, and sequence number 4 can be determined as a candidate long-term reference frame. In addition, since the preset number is 2, sequence number 0, which has the earliest generation time, can be removed from the encoder cache. After removing sequence number 0, sequence number 4 can be written to the encoder cache. Furthermore, when the encoder encodes to sequence number 6, sequence number 2 can be removed, and sequence number 6 can be written to the encoder cache.

[0153] In some embodiments, the feedback information is used to represent a result of a decoder failing to decode a plurality of first data packets received by the decoder from the encoder;

[0154] In some embodiments, in response to the feedback information, determining the first video frame corresponding to the first data packet from the encoded video stream may be specifically implemented as follows:

[0155] In response to the feedback information, determining, from the encoded video stream, a plurality of candidate video frames corresponding to the plurality of first data packets respectively;

[0156] Based on the timestamps of the multiple candidate video frames, the candidate video frame with the earliest encoding time is determined as the first video frame.

[0157] As mentioned above, multiple candidate video frames respectively corresponding to the multiple first data packets may be determined from the encoded video stream based on the data packet sequence number and / or timestamp information carried in the feedback information.

[0158] During video encoding, earlier encoded frames serve as a reference for subsequent frames. If a problem with one frame occurs, it can cause decoding failures for multiple subsequent frames. By identifying the candidate frame with the earliest encoding time as the first frame, we can address the problem at its source and prevent errors from propagating further into the video stream.

[0159] For example, if there are multiple candidate video frames with timestamps of t1, t2, t3, etc., the video frame with the smallest timestamp may be selected as the first video frame, ie, the frame with the earliest encoding time.

[0160] Figure 8 A flowchart of a video decoding method provided by an embodiment of the present application is shown. The video decoding method can be applied to a decoder, such as Figure 8 , the video decoding method may specifically include the following steps:

[0161] 801, receiving a first data packet from an encoder;

[0162] 802. Generate feedback information when decoding of the first data packet fails.

[0163] 803. Send feedback information to the encoder, so that the encoder, in response to the feedback information, determines a first video frame corresponding to the first data packet from the encoded video stream; marks a video frame between the first video frame and the second video frame as an invalid video frame, wherein the second video frame is a video frame preceding a third video frame currently encoded by the encoder; determines a target reference frame from a plurality of candidate reference frames pre-stored in the encoder cache and synchronized to the decoder, wherein the plurality of candidate reference frames include candidate short-term reference frames and candidate long-term reference frames; switches a reference frame of the third video frame from the second video frame to the target reference frame, and encodes the third video frame based on the target reference frame to generate a second data packet;

[0164] 804, receiving a second data packet sent by the encoder;

[0165] 805 : Decode the second data packet based on the target reference frame stored in the decoder buffer.

[0166] Figure 8 The generation of feedback information in the video decoding method shown and the implementation process of generating the second datagram based on the feedback information can be referred to Figure 1 The video encoding method shown is not described in detail here.

[0167] In some embodiments, the video decoding method further includes:

[0168] receiving a third data packet sent by the encoder, wherein the third data packet is generated by encoding the target reference frame, and the third data packet is marked as a candidate long-term reference frame;

[0169] Decoding the third data packet to obtain the target reference frame;

[0170] The target reference frame is stored in the decoder buffer.

[0171] In some embodiments, decoding the second data packet based on the target reference frame stored in the decoder buffer may be specifically implemented as follows:

[0172] Obtaining the pre-stored target reference frame from the decoder buffer;

[0173] The second data packet is decoded based on the target reference frame.

[0174] In an embodiment of the present application, the third data packet may be generated after the encoder encodes the target reference frame and is marked as a candidate long-term reference frame. This means that the third data packet contains the encoding information of the target reference frame and its identification as a candidate long-term reference frame.

[0175] For example, during video transmission, after encoding the target reference frame, the encoder transmits the third data packet to the decoder via the network. After receiving the data packet, the decoder begins to prepare for decoding.

[0176] After the decoder fails to decode the first data packet, the encoder may encode the third video frame based on the pre-stored target reference frame, and send a second data packet generated by the encoding to the decoder.

[0177] Because the second data packet is encoded based on the target reference frame, decoding of the second data packet also needs to be performed based on the target reference frame. Furthermore, because the decoder has already obtained the target reference frame by decoding the third data packet and stored it in the decoder cache, after obtaining the second data packet, decoding of the second data packet can be performed based on the target reference frame obtained from the decoder cache.

[0178] Figure 9 A block diagram of a video encoding device provided in an embodiment of the present application is shown. Figure 9 As shown, the video encoding device may specifically include:

[0179] An information receiving module 901 is configured to receive feedback information from a decoder, wherein the feedback information is used to indicate a result that the decoder fails to decode a first data packet received from the encoder;

[0180] a video frame determining module 902, configured to determine, in response to the feedback information, a first video frame corresponding to the first data packet from the encoded video stream;

[0181] a marking module 903, configured to mark a video frame between the first video frame and a second video frame as an invalid video frame, wherein the second video frame is a video frame preceding a third video frame currently encoded by the encoder;

[0182] a reference frame determination module 904 for determining a target reference frame from a plurality of candidate reference frames pre-stored in an encoder buffer and synchronized to the decoder;

[0183] an encoding module 905 configured to switch a reference frame of the third video frame from the second video frame to the target reference frame, so as to encode the third video frame based on the target reference frame and generate a second data packet;

[0184] The data packet sending module 906 is configured to send the second data packet to the decoder, so that the decoder decodes the second data packet based on the target reference frame stored in the decoder buffer.

[0185] In some embodiments, the reference frame determination module 904 may be specifically configured to:

[0186] Screening out invalid reference video frames corresponding to the invalid video frames from the multiple candidate reference video frames to determine at least one valid reference video frame;

[0187] The target reference frame is determined from the at least one valid reference video frame.

[0188] In some embodiments, the video encoding apparatus further comprises:

[0189] A reference frame selection module is used to determine, based on preset rules, whether to configure the third video frame currently being encoded as a candidate long-term reference frame during the encoding process of the encoder for the video stream; if so, write the third video frame as a candidate long-term reference frame into the encoder cache, mark the third data packet generated by encoding the third video frame as a long-term reference frame, and send the third data packet to the decoder so that the decoder writes the third video frame into the decoder cache.

[0190] In some embodiments, the preset rule includes a preset spacing between adjacent long-term reference frames;

[0191] In some embodiments, the reference frame selection module is specifically configured to:

[0192] determining a first distance between the third video frame and a previous fourth video frame configured as a candidate reference frame;

[0193] When the first distance reaches the preset distance, the third video frame is determined as a candidate long-term reference frame.

[0194] In some embodiments, the reference frame selection module is specifically configured to:

[0195] Determining whether the number of long-term reference frames stored in the encoder buffer reaches a preset number threshold;

[0196] If so, the candidate reference frame with the earliest generation time is deleted from the encoder cache, and the candidate long-term reference frame corresponding to the third video frame is written into the encoder cache.

[0197] In some embodiments, the feedback information is used to represent a result of failure of the decoder to decode a plurality of first data packets received from the encoder.

[0198] In some embodiments, the information receiving module 901 is specifically configured to:

[0199] In response to the feedback information, determining, from the encoded video stream, a plurality of candidate video frames corresponding to the plurality of first data packets respectively;

[0200] Based on the timestamps of the multiple candidate video frames, a candidate video frame with the earliest encoding time is determined as the first video frame.

[0201] In some embodiments, the preset rule includes whether the third video frame is a scene switching marker frame.

[0202] In some embodiments, the reference frame selection module is specifically configured to:

[0203] determining whether the third video frame indicates a switch from a first scene to a second scene in video content corresponding to the video stream;

[0204] If so, the third video frame is determined as a candidate long-term reference frame.

[0205] In some embodiments, the plurality of candidate reference frames include candidate short-term reference frames and candidate long-term reference frames.

[0206] In some embodiments, the feedback information includes a data packet sequence number;

[0207] In some embodiments, the video frame determination module 902 is specifically configured to:

[0208] Determining position information of the first data packet in the encoded video stream based on the data packet sequence number;

[0209] The first video frame is determined based on the position information.

[0210] In some embodiments, the reference frame determination module 904 may be specifically configured to:

[0211] Traversing the plurality of candidate reference video frames in sequence, and determining whether any candidate reference video frame has a corresponding relationship with any of the invalid video frames;

[0212] Eliminating a candidate reference video frame corresponding to any of the invalid video frames from the encoder cache;

[0213] The candidate reference video frames remaining in the encoder buffer are determined as the valid reference video frames.

[0214] In some embodiments, the reference frame selection module is specifically configured to:

[0215] Sort multiple candidate long-term reference frames according to generation time to generate a long-term reference frame list;

[0216] The candidate long-term reference frame corresponding to the first element in the long-term reference frame list is deleted.

[0217] In some embodiments, the feedback information may include a data packet sequence number; the video frame determination module 902 is specifically configured to:

[0218] Determining position information of the first data packet in the encoded video stream based on the data packet sequence number;

[0219] The first video frame is determined based on the position information.

[0220] Figure 8 The video encoding device can perform Figure 2 The implementation principle and technical effects of the video encoding method described in the embodiment are not described in detail here. The specific manner in which each module and unit performs operations in the video encoding device in the above embodiment has been described in detail in the embodiment of the method, and will not be elaborated here.

[0221] Figure 10 A block diagram of a video decoding device provided in an embodiment of the present application is shown. Figure 10 As shown, the video decoding device may specifically include:

[0222] The data packet receiving module 1001 is configured to receive a first data packet from an encoder;

[0223] A feedback information generating module 1002, configured to generate feedback information when decoding of the first data packet fails;

[0224] A feedback information sending module 1003 is configured to send the feedback information to the encoder, so that the encoder, in response to the feedback information, determines a first video frame corresponding to the first data packet from the encoded video stream; marks a video frame between the first video frame and a second video frame as an invalid video frame, wherein the second video frame is a video frame preceding a third video frame currently encoded by the encoder; determines a target reference frame from a plurality of candidate reference frames pre-stored in an encoder cache and synchronized to the decoder, wherein the plurality of candidate reference frames include candidate short-term reference frames and candidate long-term reference frames; and switches a reference frame of the third video frame from the second video frame to the target reference frame, so as to encode the third video frame based on the target reference frame and generate a second data packet;

[0225] The data packet receiving module 1004 is configured to receive the second data packet sent by the encoder;

[0226] The decoding module 1005 is configured to decode the second data packet based on the target reference frame stored in the decoder buffer.

[0227] In some embodiments, the video decoding apparatus may further include:

[0228] a third data packet receiving module, configured to receive a third data packet sent by the encoder, wherein the third data packet is generated by encoding the target reference frame, and the third data packet is marked as a candidate long-term reference frame;

[0229] A third data packet decoding module, configured to decode the third data packet to obtain the target reference frame;

[0230] A reference frame buffer module, configured to store the target reference frame in the decoder buffer;

[0231] In some embodiments, the decoding module 1005 is specifically configured to: obtain the pre-stored target reference frame from the decoder cache;

[0232] The second data packet is decoded based on the target reference frame.

[0233] Figure 10 The video decoding device can perform Figure 8 The implementation principle and technical effects of the video decoding method described in the embodiment are not described in detail here. The specific manner in which each module and unit performs operations in the video decoding device in the above embodiment has been described in detail in the embodiment of the method, and will not be elaborated here.

[0234] In one possible design, the video encoding device and video decoding device provided in the embodiments of the present application can be implemented as a computing device, such as Figure 11 As shown, the computing device may include a storage component 1101 and a processing component 1102;

[0235] The storage component 1101 stores one or more computer instructions, wherein the one or more computer instructions are called and executed by the processing component 1102 to implement the video encoding method and video decoding method provided in the embodiments of the present application.

[0236] Of course, a computing device may also include other components, such as input / output interfaces and communication components. The input / output interface provides an interface between the processing component and peripheral interface modules, such as output devices and input devices. The communication component is configured to facilitate wired or wireless communication between the computing device and other devices.

[0237] Among them, the computing device can be a physical device or an elastic computing host provided by a cloud computing platform, etc. In this case, the computing device can refer to a cloud server, and the above-mentioned processing components, storage components, etc. can be basic server resources rented or purchased from the cloud computing platform.

[0238] When the computing device is a physical device, it can be implemented as a distributed cluster consisting of multiple servers or terminal devices, or as a single server or a single terminal device.

[0239] An embodiment of the present application further provides a computer-readable storage medium storing a computer program. When the computer program is executed by a computer, the video encoding method and the video decoding method provided in the embodiment of the present application can be implemented.

[0240] An embodiment of the present application further provides a computer program product, including a computer program, which, when executed by a computer, can implement the video encoding method and the video decoding method provided in the embodiment of the present application.

[0241] The processing components in the above embodiments may include one or more processors to execute computer instructions to complete all or part of the steps in the above methods. Of course, the processing components may also be implemented as one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0242] The memory component is configured to store various types of data to support operations in the device. The memory component can be implemented by any type of volatile or non-volatile memory device, or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0243] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0244] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0245] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, or of course, by hardware. Based on this understanding, the essence of the above technical solution or the part that contributes to the existing technology can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or certain parts of the embodiments.

[0246] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A video encoding method, characterized in that: The method comprises: receiving feedback information from a decoder, the feedback information being used to indicate a result that the decoder fails to decode a first data packet received from an encoder; In response to the feedback information, determining a first video frame corresponding to the first data packet from the encoded video stream; Marking a video frame between the first video frame and a second video frame as an invalid video frame, wherein the second video frame is a video frame preceding a third video frame currently encoded by the encoder; Determine a target reference frame from a plurality of candidate reference frames pre-stored in an encoder buffer and synchronized to the decoder; Switching a reference frame of the third video frame from the second video frame to the target reference frame, encoding the third video frame based on the target reference frame, and generating a second data packet; sending the second data packet to the decoder so that the decoder decodes the second data packet based on the target reference frame stored in a decoder buffer; The determining of a target reference frame from a plurality of candidate reference frames pre-stored in an encoder buffer and synchronized to the decoder comprises: Eliminate invalid reference video frames corresponding to the invalid video frames from the multiple candidate reference frames to determine at least one valid reference video frame; The target reference frame is determined from the at least one valid reference video frame.

2. The method according to claim 1, characterized in that The method further comprises: During encoding of the video stream by the encoder, determining based on a preset rule whether to configure a third video frame currently being encoded as a candidate long-term reference frame; If so, the third video frame is written into the encoder cache as a candidate long-term reference frame, and the third data packet generated by encoding the third video frame is marked as a long-term reference frame, and the third data packet is sent to the decoder, so that the decoder writes the third video frame into the decoder cache.

3. The method according to claim 2, characterized in that The preset rule includes a preset spacing between adjacent long-term reference frames; The determining, based on a preset rule, whether to configure the currently encoded third video frame as a candidate long-term reference frame includes: determining a first distance between the third video frame and a previous fourth video frame configured as a candidate reference frame; When the first distance reaches the preset distance, the third video frame is determined as a candidate long-term reference frame.

4. The method according to claim 3, characterized in that After determining the third video frame as a candidate long-term reference frame, the method further includes: determining whether the number of candidate long-term reference frames stored in the encoder buffer reaches a preset number threshold; If so, the candidate long-term reference frame with the earliest generation time is deleted from the encoder cache, and the candidate long-term reference frame corresponding to the third video frame is written into the encoder cache.

5. The method according to claim 1, characterized in that The feedback information is used to represent a result of failure of the decoder to decode a plurality of first data packets received from the encoder; Determining, in response to the feedback information, a first video frame corresponding to the first data packet from the encoded video stream includes: In response to the feedback information, determining, from the encoded video stream, a plurality of candidate video frames corresponding to the plurality of first data packets respectively; Based on the timestamps of the multiple candidate video frames, a candidate video frame with the earliest encoding time is determined as the first video frame.

6. The method according to claim 2, characterized in that The preset rule includes whether the third video frame is a scene switching marker frame; The determining, based on a preset rule, whether to configure the currently encoded third video frame as a candidate long-term reference frame includes: determining whether the third video frame indicates a switch from a first scene to a second scene in video content corresponding to the video stream; If so, the third video frame is determined as a candidate long-term reference frame.

7. The method according to claim 1, characterized in that The plurality of candidate reference frames include candidate short-term reference frames and candidate long-term reference frames.

8. The method according to claim 1, characterized in that The feedback information includes a data packet sequence number; Determining, in response to the feedback information, a first video frame corresponding to the first data packet from the encoded video stream includes: Determining position information of the first data packet in the encoded video stream based on the data packet sequence number; The first video frame is determined based on the position information.

9. The method according to claim 1, characterized in that Screening out invalid reference video frames corresponding to the invalid video frames from the multiple candidate reference frames to determine at least one valid reference video frame comprises: Traversing the plurality of candidate reference frames in sequence, and determining whether any candidate reference frame has a corresponding relationship with any of the invalid video frames; Eliminating a candidate reference frame corresponding to any of the invalid video frames from the encoder buffer; The candidate reference frames remaining in the encoder buffer are determined as the valid reference video frames.

10. The method according to claim 4, characterized in that Deleting the candidate reference frame with the earliest generation time from the encoder cache includes: Sort multiple candidate long-term reference frames according to generation time to generate a long-term reference frame list; The candidate long-term reference frame corresponding to the first element in the long-term reference frame list is deleted.

11. A video decoding method, characterized in that: Applied to a decoder, the method comprises: receiving a first data packet from an encoder; generating feedback information in case decoding of the first data packet fails; The feedback information is sent to the encoder so that the encoder determines, in response to the feedback information, a first video frame corresponding to the first data packet from the encoded video stream; marks a video frame between the first video frame and the second video frame as an invalid video frame, wherein the second video frame is a previous video frame of a third video frame currently encoded by the encoder; determines a target reference frame from a plurality of candidate reference frames pre-stored in the encoder cache and synchronized to the decoder; switches the reference frame of the third video frame from the second video frame to the target reference frame, so as to encode the third video frame based on the target reference frame and generate a second data packet; determining the target reference frame from the plurality of candidate reference frames pre-stored in the encoder cache and synchronized to the decoder includes: filtering out the invalid reference video frame corresponding to the invalid video frame from the plurality of candidate reference frames to determine at least one valid reference video frame; and determining the target reference frame from the at least one valid reference video frame; receiving the second data packet sent by the encoder; The second data packet is decoded based on the target reference frame stored in the decoder buffer.

12. The method according to claim 11, characterized in that The method further comprises: receiving a third data packet sent by the encoder, wherein the third data packet is generated by encoding the target reference frame, and the third data packet is marked as a candidate long-term reference frame; Decoding the third data packet to obtain the target reference frame; Storing the target reference frame in the decoder buffer; The decoding of the second data packet based on the target reference frame stored in the decoder buffer includes: Obtaining the pre-stored target reference frame from the decoder buffer; The second data packet is decoded based on the target reference frame.

13. A computing device, characterized in that including processing components and storage components; The storage component stores one or more computer instructions; the one or more computer instructions are used to be called and executed by the processing component to implement the video encoding method according to any one of claims 1 to 10, or to implement the video decoding method according to any one of claims 11 to 12.

14. A computer program product, characterized in that The computer program product includes computer program code, and when the computer program code is executed by a computer, it implements the video encoding method according to any one of claims 1 to 10, or implements the video decoding method according to any one of claims 11 to 12.

Images