A video frame encoding control method, apparatus and device

Abstract

The embodiments of the present specification provide a video frame encoding control method, device and equipment. The scheme comprises: judging whether the number of logical slots in the reference frame buffer queue which continuously do not receive the confirmation message reaches the preset number according to the confirmation state of each reference frame in the reference frame buffer queue; if not, encoding the to-be-encoded video frame based on the available reference frame in the reference frame buffer queue to obtain the encoded video frame; continuing to judge whether the to-be-encoded video frame meets the preset sparse storage strategy; if yes, storing the encoded video frame into the reference frame buffer queue; wherein the preset sparse storage strategy is used to indicate that part of the video frames are selected discontinuously from the sequence of continuously encoded video frames as the reference frames and stored into the reference frame buffer queue.

Description

Technical Field

[0001] This specification relates to the field of video coding technology in one or more embodiments, and particularly to a video frame coding control method, apparatus and device. Background Technology

[0002] In the H.265 / HEVC encoding standard, caching and retaining specific encoded video frames for cross-temporal reference prediction in subsequent encoded frames effectively improves video compression coding efficiency and enhances the stream's resilience against packet loss. The encoder typically configures a fixed-capacity reference frame buffer queue (e.g., capable of storing 14 frames). When multiple consecutive frames in the reference frame buffer queue fail to receive an ACK acknowledgment message from the receiver, the encoder determines that the inter-frame reference link is broken and forcibly inserts an I-frame intra-frame coding keyframe to re-establish timing synchronization between the encoder / decoder and the reference link.

[0003] In ultra-low latency sensitive service scenarios such as real-time audio and video communication and multimodal real-time interaction, high network round-trip latency (e.g., RTT greater than 500ms) or lost ACK packet transmission are common, which can easily cause the encoder to erroneously trigger reference frame waiting timeouts, thus unnecessarily forcing the insertion of I-frames. Since the I-frame bitstream is usually 5 to 10 times the size of a regular P-frame, frequent and unnecessary insertion of I-frames can cause sudden network bandwidth consumption, video playback stuttering and jitter, and significantly reduce the overall video encoding and compression efficiency.

[0004] Therefore, how to provide a video frame coding control method that can effectively extend the reference frame waiting time and reduce the insertion of unnecessary intra-frame coding keyframes has become an urgent technical problem to be solved. Summary of the Invention

[0005] In view of this, one or more embodiments of this specification provide a video frame coding control method, apparatus, and device to effectively extend the reference frame waiting time and reduce unnecessary intra-frame coding keyframe insertion.

[0006] According to a first aspect of one or more embodiments of this specification, a video frame encoding control method is provided, comprising: Based on the acknowledgment status of each reference frame in the reference frame buffer queue, determine whether the number of logical slots in the reference frame buffer queue that have not received consecutive acknowledgment messages has reached a preset number. If the number of logical slots that have not received a confirmation message for a consecutive period does not reach the preset number, the video frame to be encoded is encoded based on the available reference frames in the reference frame buffer queue to obtain the encoded video frame. Determine whether the video frame to be encoded conforms to the preset sparse storage strategy and obtain the determination result; the preset sparse storage strategy is used to indicate that a portion of video frames are selected non-continuously from the continuously encoded video frame sequence as reference frames and stored in the reference frame buffer queue. If the determination result indicates that the video frame to be encoded conforms to the preset sparse storage strategy, then the encoded video frame is stored in the reference frame buffer queue.

[0007] According to a second aspect of one or more embodiments of this specification, a video frame encoding control apparatus is provided, comprising: The first judgment module is used to determine whether the number of logical slots in the reference frame cache queue that have not received a continuous acknowledgment message has reached a preset number based on the acknowledgment status of each reference frame in the reference frame cache queue. The first decision module is used to encode the video frame to be encoded based on the available reference frames in the reference frame buffer queue if the number of logical slots that have not received a confirmation message for a consecutive time does not reach the preset number, so as to obtain the encoded video frame. The second judgment module is used to judge whether the video frame to be encoded conforms to the preset sparse storage strategy and obtain the judgment result; the preset sparse storage strategy is used to instruct to select a portion of video frames non-continuously from the continuously encoded video frame sequence as reference frames and store them in the reference frame buffer queue. The second decision module is used to store the encoded video frame into the reference frame cache queue if the judgment result indicates that the video frame to be encoded conforms to the preset sparse storage strategy.

[0008] According to a third aspect of one or more embodiments of this specification, a computing device is provided, including a memory, a processor, and computer instructions stored in the memory and executable on the processor, wherein the processor, when executing the computer instructions, implements the steps of the video frame encoding control method.

[0009] One embodiment of this specification achieves at least the following beneficial effects: Based on the acknowledgment status of each reference frame in the reference frame buffer queue, it is determined whether the number of logical slots in the reference frame buffer queue that have not received consecutive acknowledgment messages has reached a preset number; if not, the video frame to be encoded is encoded based on the available reference frames in the reference frame buffer queue to obtain the encoded video frame; further, it is determined whether the video frame to be encoded conforms to a preset sparse storage strategy; if it does, the encoded video frame is stored in the reference frame buffer queue; wherein, the preset sparse storage strategy is used to instruct the non-continuous selection of a portion of video frames from the continuously encoded video frame sequence as reference frames to be stored in the reference frame buffer queue. This scheme, through sparse storage, makes each logical slot correspond to a more distant physical frame number, thereby significantly extending the time window covered by the reference frames without increasing the buffer capacity. Under the same buffer capacity, it effectively extends the tolerance time for the encoding end to wait for the receiving end's acknowledgment message, significantly reduces the probability of mistakenly triggering the insertion of intra-frame keyframes due to high latency or lost acknowledgment messages, and ultimately reduces the insertion of unnecessary intra-frame keyframes. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in the embodiments or prior art of this specification, the drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is a schematic diagram illustrating an application scenario of a video frame encoding control method provided in the embodiments of this specification; Figure 2 This is a flowchart illustrating a video frame encoding control method provided in the embodiments of this specification; Figure 3 This is a flowchart of a video frame encoding control method provided in the embodiments of this specification; Figure 4 This is a schematic diagram of the structure of a video frame encoding control device provided in the embodiments of this specification; Figure 5 This is a schematic diagram of the structure of a computer device provided in the embodiments of this specification. Detailed Implementation

[0012] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0013] This specification uses specific terms to describe embodiments thereof. Terms such as "an embodiment," "one embodiment," and / or "some embodiments" refer to a particular feature, structure, or characteristic associated with at least one embodiment of this specification. Therefore, it should be emphasized and noted that references to "an embodiment," "one embodiment," or "an alternative embodiment" in different locations throughout this specification do not necessarily refer to the same embodiment. Furthermore, those skilled in the art can combine and integrate the different embodiments or examples described herein, as well as the features of those different embodiments or examples, without contradiction.

[0014] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “an,” “an,” “the,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification includes any or all possible combinations of one or more associated listed items.

[0015] The terms “comprising,” “including,” or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, product, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, product, or apparatus. Without further limitation, the presence of additional identical or equivalent elements in the process, method, product, or apparatus that includes said elements is not excluded.

[0016] Although the terms "first," "second," etc., may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, "first" may also be referred to as "second," and similarly, "second" may also be referred to as "first," without departing from the scope of one or more embodiments of this specification. Ordinal numbers such as "first," "second," etc., do not necessarily indicate order; often they are used to facilitate the distinction of objects. For example, "first server" and "second server" usually refer to two servers. To distinguish these two servers, they are described as "first server" and "second server." Of course, sometimes these two servers may be the same server.

[0017] Depending on the context, the word "if" as used here can be interpreted as "when," "when," or "in response to determination."

[0018] In this specification, unless explicitly stated otherwise, "receiving and sending data" does not necessarily mean direct receiving and sending; it can also mean indirect receiving and sending. For example, A receiving data sent by B can be understood as A directly receiving the data sent by B, or it can be understood as A indirectly receiving the data sent by B through other entities such as C. Similarly, B sending data to A can be understood as B sending the data directly to A, or it can be understood as B indirectly sending the data to A through other entities such as C. Here, C can be one entity, or it can be two or more entities.

[0019] In this specification, unless explicitly stated otherwise, the relationships between structures can be direct or indirect. For example, when describing "A is connected to B," unless it is explicitly stated that A and B are directly connected, it should be understood that A can be directly connected to B or indirectly connected to B. Similarly, when describing "A is on top of B," unless it is explicitly stated that A is directly above B (AB is adjacent and A is above B), it should be understood that A can be directly above B or indirectly above B (AB is separated by other elements, and A is above B). And so on.

[0020] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties. The collection, use and processing of related data shall comply with the relevant laws, regulations and standards of the relevant regions, and corresponding operation entry points shall be provided for users to choose to authorize or refuse.

[0021] The following explains the terms and concepts used in one or more embodiments of this specification.

[0022] Intra-coded keyframes (I-frames) are independent intra-coded keyframes in video coding standards such as H.265 / HEVC. They use pure intra-predictive coding and can independently complete decoding and reconstruction without relying on any preceding or following encoded reference frames.

[0023] Forward predictive frames (P-frames) are inter-frame coded frames in the H.265 / HEVC coding standard. They use only reference frames encoded before the current frame for temporal prediction and achieve high compression coding by removing inter-frame redundancy.

[0024] Bidirectional predictive frames (B-frames) are inter-frame coded frames in the H.265 / HEVC video coding standard.

[0025] A reference frame is a baseline frame used in video coding to predict other frames, thereby improving compression efficiency by eliminating temporal redundancy.

[0026] The reference frame buffer queue, also known as the reference frame buffer or reference frame buffer pool, is a pre-allocated, fixed-capacity dedicated storage resource within the H.265 / HEVC encoding end. The reference frame buffer queue can be used to cache encoded historical video frames that can be used for inter-frame prediction in subsequent frames; these historical video frames are the reference frames.

[0027] An acknowledgment message, also known as an ACK message, is part of a computer network communication protocol and is a message sent by a device or process to acknowledge receipt of data.

[0028] H.265 / HEVC (High Efficiency Video Coding) is a widely used next-generation video frame compression standard. H.265 / HEVC offers approximately 50% higher coding efficiency compared to H.264, better supporting real-time transmission of high-definition and ultra-high-definition video frames. In end-to-end latency-sensitive applications such as real-time audio / video frame communication (RTC), cloud gaming, and multimodal interaction, H.265 encoders typically employ a Long-Term Reference (LTR) mechanism to enhance packet loss resilience and compression efficiency.

[0029] The LTR mechanism allows the encoder to mark some encoded frames (usually P-frames or I-frames) as "long-term valid" and continue to use them as reference frames for a longer time window (e.g., after dozens or even hundreds of frames), thus avoiding frequent requests for intra-frame encoding keyframes due to the loss of short-term reference frames. Specifically, the encoder maintains a fixed-size LTR reference frame buffer (e.g., storing 7 or 14 frames), storing the most recently sent frames sequentially. The receiver returns ACK (acknowledgment) or NACK (negation) messages through feedback channels such as RTCP to inform the sender which frames have been successfully received. If the encoder finds that several consecutive video frames (e.g., all 14 frames) in the LTR buffer have not received ACK messages, it considers the reference chain broken, and the encoder needs to forcibly insert an I-frame to rebuild decoding synchronization.

[0030] However, the LTR mechanism described above is prone to triggering frequent I-frame insertions in weak network environments with high latency and high packet loss rates, leading to performance degradation.

[0031] To address the shortcomings in related technologies, this solution provides the following embodiments.

[0032] Figure 1 This is a schematic diagram illustrating an application scenario of a video frame encoding control method provided in the embodiments of this specification.

[0033] The video frame encoding control method provided in this application embodiment can be applied to, for example, Figure 1 The application scenarios shown are not limited to these. In, for example... Figure 1 In the application scenario shown, the video encoding server can establish a data communication connection with one or more terminal devices via a local area network (LAN), wide area network (WAN), the Internet, or other types of data networks. These terminal devices include video sending terminals and video receiving terminals. These terminal devices may include, but are not limited to, smartphones, tablets, laptops, desktop computers, smart wearable devices, in-vehicle terminals, real-time conferencing terminals, security monitoring equipment, and multimodal interactive terminals.

[0034] like Figure 1As shown, the scheme may include a video encoding server 101, a video sending terminal 102, and a video receiving terminal 103. The video sending terminal 102 can acquire raw video frames and send them to the video encoding server 101. The video encoding server 101 maintains a reference frame buffer queue. Based on the reception confirmation status of each video frame in the reference frame buffer queue, it determines whether the number of logical slots in the reference frame buffer queue that have not received consecutive confirmation messages has reached a preset number. If the number of logical slots that have not received consecutive confirmation messages has not reached the preset number, the video encoding server 101 encodes the video frame to be encoded based on the existing reference frames in the reference frame buffer queue to obtain the encoded video frame. The system determines whether the video frame to be encoded conforms to a preset sparse storage strategy, obtaining a judgment result. The preset sparse storage strategy, under the premise of ensuring image quality meets the basic requirements of real-time video playback, normal image detail recognition, and no significant visual distortion in business interactions, selects a portion of video frames from a continuously encoded video frame sequence and stores them in a reference frame buffer queue. If the judgment result indicates that the video frame to be encoded conforms to the preset sparse storage strategy, the encoded video frame is stored in the reference frame buffer queue for inter-frame reference encoding of subsequent video frames to be encoded. The video encoding server 101 sends the encoded video stream to the video receiving terminal 103 via a data network, where the video receiving terminal 103 performs decoding, rendering, and playback.

[0035] like Figure 1 In the scenario shown, the video encoding server can include, but is not limited to, any device, equipment, server platform, or server cluster that has video encoding processing, data caching, and network transmission capabilities.

[0036] This application provides a video frame encoding control method. This application also relates to a video frame encoding control device and a computing device, which will be described in detail in the following embodiments.

[0037] Figure 2 This is a flowchart illustrating a video frame encoding control method provided in an embodiment of this specification.

[0038] From a programming perspective, the entity executing the process can be a video encoding server. It's understandable that this method can be executed by any device, equipment, or cluster of devices with computing and processing capabilities.

[0039] like Figure 2 As shown, the process may include the following steps.

[0040] Step 202: Based on the acknowledgment status of each reference frame in the reference frame buffer queue, determine whether the number of logical slots in the reference frame buffer queue that have not received a continuous acknowledgment message has reached a preset number.

[0041] The reference frame buffer queue is maintained by the encoder during the encoding process. It serves as a buffer area for caching reference frames that have been encoded and can be used by subsequent video frames to be encoded for inter-frame prediction. In the embodiments of this specification, the reference frame buffer queue can be a long-term reference frame buffer queue, which may include the most recently sent reference frames to the receiver. In practical applications, the long-term reference frame buffer queue may also include reference frames marked as long-term valid for use within a longer subsequent time window. For ease of explanation, the following description will use an example where the long-term reference frame buffer queue only includes the most recently sent reference frames to the receiver.

[0042] The reference frame buffer queue consists of a finite number of logical slots, each of which can be used to store one encoded video frame selected as the reference frame.

[0043] A logical slot can refer to a storage location identifier in the reference frame buffer queue used to store reference frames. Each logical slot can correspond to a fixed buffer sequence number, such as slot 1, slot 2, ..., slot N. In one or more embodiments of this specification, each logical slot can independently maintain the acknowledgment status of the corresponding reference frame. The acknowledgment status can be "acknowledged" or "unacknowledged". If the encoding end receives an acknowledgment message (ACK) from the receiving end for a certain video frame, it can update the acknowledgment status of the logical slot containing that video frame to "acknowledged". If the encoding end has not yet received an acknowledgment message from the receiving end for that video frame or has received a negative acknowledgment (NACK) message, it can continue to maintain or set it to "unacknowledged".

[0044] When the encoding end receives an acknowledgment message from the receiving end for a certain video frame, it indicates that the receiving end has successfully received and correctly parsed the video frame. When the encoding end receives a negative acknowledgment from the receiving end for a certain video frame, it indicates that the receiving end has detected that the video frame is lost or corrupted and requests the sending end to retransmit it or mark it as unavailable.

[0045] In one or more embodiments of this specification, if the number of logical slots for which no acknowledgment messages have been received consecutively reaches a preset number, it indicates that the number of reference frames in the reference frame buffer queue that have not been acknowledged by the receiver has accumulated to the point of triggering a reference chain break determination, thus requiring the forced insertion of an I-frame to rebuild decoding synchronization. Similarly, if the number of logical slots for which no acknowledgment messages have been received consecutively has not yet reached the preset number, it indicates that the number of reference frames in the reference frame buffer queue that have not been acknowledged by the receiver has not yet accumulated to the point of triggering a reference chain break determination, therefore, it is not necessary to forcibly insert an I-frame to rebuild decoding synchronization at this time.

[0046] The preset quantity can be a fixed integer set in advance, or it can be a variable that is dynamically adjusted according to the network status.

[0047] As one implementation, the preset number can be configured to be equal to the total number of logical slots in the reference frame buffer queue, such as 14, to maximize the fault tolerance of the reference frame buffer. That is, the reference chain is considered broken and intra-frame coding keyframe insertion is triggered only when consecutive unacknowledged logical slots cover the entire buffer queue. The above setting can maximize the waiting time and effectively avoid inserting unnecessary I-frames due to local acknowledgment delays.

[0048] Step 204: If the number of logical slots that have not received a confirmation message for a consecutive period does not reach the preset number, then the video frame to be encoded is encoded based on the available reference frames in the reference frame buffer queue to obtain the encoded video frame; the available reference frames are the reference frames in the reference frame buffer queue that have received the confirmation message or the intra-frame encoded keyframes.

[0049] In practical applications, the number of logical slots that have not received a confirmation message for several consecutive times may fall short of the preset number in the following situations: 1. The queue is not full, and none of the receivers of the reference frames in the reference frame buffer queue have sent back a confirmation message; 2. The queue is not full, and at least one receiver of the reference frames in the reference frame buffer queue has sent back a confirmation message; 3. The queue is full, and at least one receiver of the reference frames in the reference frame buffer queue has sent back a confirmation message.

[0050] If the number of logical slots that have not received a continuous acknowledgment message has not yet reached the preset number, it can be said that the number of reference frames that have not been acknowledged by the receiving end in the reference frame buffer queue has not yet accumulated to the point of triggering the reference chain breakage judgment. At this time, the video to be encoded can be encoded based on the available reference frames in the reference buffer queue.

[0051] In this context, a usable reference frame refers to a reference frame that can be safely referenced by the current video frame to be encoded. Specifically, if there is a reference frame in the reference frame queue that has received an acknowledgment message from the receiver, the usable reference frame can be that reference frame in the queue that has received the acknowledgment message. This usable reference frame has been correctly received by the receiver and can serve as a reliable basis for prediction. If there are no reference frames in the reference frame queue that have received an acknowledgment message from the receiver, the usable reference frame can be an intra-frame coding keyframe, i.e., an I-frame. Because an I-frame itself does not depend on other frames for decoding, even if the receiver has not yet confirmed its delivery, using an I-frame as a reference for encoding can minimize the risk of decoding distortion caused by the lack of a reference frame.

[0052] In the embodiments of this specification, if the reference chain is not completely broken, that is, the number of consecutive unconfirmed logical slot bits has not reached the threshold, it means that at least some reference frames are known to the receiver. Using these secure reference frames for predictive coding can maintain a high compression ratio (the size of a P-frame is much smaller than that of an I-frame) and ensure the correct recovery at the decoding end, avoiding bandwidth waste caused by blindly inserting I-frames.

[0053] Step 206: Determine whether the video frame to be encoded conforms to the preset sparse storage strategy and obtain the determination result; the preset sparse storage strategy is used to indicate that a portion of video frames are selected non-continuously from the continuously encoded video frame sequence as reference frames and stored in the reference frame buffer queue.

[0054] In one or more embodiments of this specification, the preset sparse storage strategy means that, while ensuring that the image quality meets basic requirements, the encoding end does not store every frame in the reference frame buffer. Instead, it selects a portion of frames as reference frames from a continuous video frame sequence according to a skip rule. The basic requirements may refer to image quality that meets the requirements for real-time video playback, normal recognition of image details, or no significant visual distortion in business interactions.

[0055] In practical applications, the above-mentioned non-contiguous selection can be achieved by setting a positive integer sparse step size M. The sparse step size is a key parameter used to control the sparsity of the reference frame buffer. It defines how many frames in a continuously encoded video frame sequence are selected as reference frames and stored in the reference frame buffer queue. For example, when M=2, it means taking one frame every two frames, with the reference frames stored in the buffer being an even-numbered frame sequence: 2, 4, 6, 8, 10, 12, 14…; when M=3, it means taking one frame every three frames, with the reference frame sequence being 3, 6, 9, 12…

[0056] In practical applications, in 30fps video encoding, the time interval between two adjacent frames is approximately 1 / 30th of a second (≈33.33ms). Assuming the reference frame buffer queue supports a maximum of 14 logical slots, in the traditional contiguous storage method (where each frame is stored in the reference frame buffer queue), when all 14 logical slots are full and contain consecutive physical frames, the frame interval between the start and end frames is 14. A 1 = 13 frame interval. The corresponding time span is 13 × 33.33 ms ≈ 433.3 ms, which can usually be simplified to a time window of 14 consecutive frames, approximately 14 × 33.33 ms ≈ 467 ms (calculated from the boundary between the first frame being stored and the last frame awaiting confirmation). In the sparse storage method of one or more embodiments of this specification, taking a sparse step size M = 2 as an example, the 14 logical slots correspond to frames 2, 4, 6, ..., 28, and the 14 logical slots correspond to actual physical frame numbers from 2 to 28. The frame interval between the starting frame 2 and the ending frame 28 is 28. 2 = 26 frame intervals. The corresponding time span is 26 × 33.33 ms ≈ 867 ms, approximately 900 ms. If M = 3 is used, that is, one frame is taken every 3 frames, and the reference frame sequence is 3, 6, ..., 42, the corresponding time span can reach (42) 3)×33.33≈1300ms.

[0057] In one or more embodiments of this specification, sparse storage is used, where each logical slot corresponds to a more distant physical frame number. This extends the reference frame coverage time window from approximately 467ms (at 30fps) with continuous storage to nearly 900ms or even longer without increasing the buffer capacity. This extended time window effectively extends the tolerance time for the encoder to wait for the receiver's ACK message with the same buffer capacity, significantly reducing the probability of erroneously triggering I-frame insertion due to high latency or lost ACK messages, ultimately reducing unnecessary I-frame insertions.

[0058] Step 208: If the judgment result indicates that the video frame to be encoded conforms to the preset sparse storage strategy, then the encoded video frame is stored in the reference frame buffer queue.

[0059] If the current video frame to be encoded is determined to conform to the preset sparse storage strategy, the encoded video frame can be added to the reference frame buffer queue so that it can be used as a reference frame for other video frames to be encoded in the subsequent encoding process.

[0060] In the embodiments described in this specification, the encoded video frame is the output data generated after the video frame to be encoded has been compressed and encoded by the encoder. The encoded video frame can be stored in the reference frame buffer queue or sent to the receiving end.

[0061] In practical applications, if the judgment result indicates that the video frame to be encoded does not conform to the preset sparse storage strategy, the encoded video frame can be sent directly to the receiving end, but not stored in the reference frame buffer queue.

[0062] Understandably, after sending the encoded video frame to the receiving end, the next video frame to be encoded can be obtained, and steps 204 to 208 can be executed for the next video frame to be encoded.

[0063] In one or more embodiments of this specification, if the number of logical slots that have not received consecutive acknowledgment messages reaches a preset number, it indicates that the reference chain has been broken. The receiving end may have lost multiple consecutive reference frames. If any unacknowledged reference frames are used for predictive encoding, the receiving end will be unable to decode correctly due to the lack of necessary reference frames, resulting in screen distortion or continuous stuttering. To address this situation, the present invention provides a recovery mechanism after a reference chain break.

[0064] Optionally, after determining whether the number of logical slots in the reference frame buffer queue that have not received a consecutive acknowledgment message has reached a preset number, the method may further include: If the number of logical slots that have not received a confirmation message for a consecutive period reaches the preset number, the video frame to be encoded will be encoded as an intra-frame coded keyframe.

[0065] The reference frame buffer queue is reset to obtain the reset reference frame buffer queue.

[0066] The intra-frame coded keyframe is stored in the reset reference frame buffer queue.

[0067] In one or more embodiments of this specification, when the number of logical slots that have not received consecutive acknowledgment messages has reached a preset number, i.e., the reference chain is broken, the current video frame to be encoded can be directly encoded as an intra-frame coding keyframe, i.e., an I-frame. An intra-frame coding keyframe is a frame type that does not rely on any other frame and only uses its own pixel information for compression. Forcing I-frame encoding when the reference chain is broken can rebuild decoding synchronization. Because I-frames do not depend on historical reference frames, the receiving end can restore the correct decoding starting point upon receiving it.

[0068] As one implementation method, the reference frame buffer queue can be reset by clearing all logical slots in the reference frame buffer queue, that is, discarding all reference frame data and their acknowledgment status stored in each logical slot, so as to restore the queue to its initial empty state.

[0069] As another implementation, queue reset can be achieved by adding a "valid" flag to each logical slot. Specifically, when resetting the reference frame buffer queue, the valid flags of all logical slots in the reference frame buffer queue can be set to "invalid," indicating that the reference frame data in the corresponding logical slot has expired. Subsequently, when a new encoded video frame needs to be stored, a slot with an "invalid" flag can be selected according to the slot order or a predefined rule, the new encoded video frame can be written to that logical slot, and the flag of that logical slot can be updated to "valid." The above implementation can significantly reduce memory erase and write operations on the actual data storage area in the slot, thereby improving cache management efficiency.

[0070] In one or more embodiments of this specification, resetting the reference frame buffer queue is intended to discard all unacknowledged reference frames after a reference chain break, completely eliminating the potential interference of old, unacknowledged reference frames with subsequent encoding, and preventing newly encoded frames from continuing to reference frames that may have been lost by the receiver. After resetting, the newly encoded intra-frame keyframe (I-frame) will be stored as the first reference frame in the reference frame buffer queue, thereby re-establishing the reference chain.

[0071] In one or more embodiments of this specification, by resetting the reference frame buffer queue and actively inserting I-frames, the reference chain deadlock caused by the continuous loss of ACK messages can be broken, providing a new synchronization point for the encoder and decoder. This promptly blocks the propagation of errors and avoids continuous decoding failures due to missing reference frames. The reset buffer retains only one independently decodeable I-frame, making the reference relationship of subsequent frames clear and reliable. It is understood that the sparse storage strategy of this scheme will continue to be executed after the reset, meaning that subsequent video frames to be encoded must still meet the sparse storage strategy before being stored, thus maintaining the long window advantage under weak network conditions while restoring synchronization.

[0072] In practical applications, frames that are temporally close in a video sequence usually have strong temporal correlation. Selecting a nearby confirmed reference frame can minimize motion residuals, thereby achieving a higher compression ratio and improving coding efficiency.

[0073] Based on this, optionally, if there are reference frames in the reference frame buffer queue that have received the acknowledgment message, other than the intra-coding keyframes, the available reference frames are the reference frames that have received the acknowledgment message, excluding the intra-coding keyframes. Encoding the video frame to be encoded based on the available reference frames in the reference frame buffer queue may specifically include: From the available reference frames, determine the target reference frame whose frame number is closest to the frame number of the video frame to be encoded.

[0074] Encoding is performed on the video frame to be encoded based on the target reference frame.

[0075] In one or more embodiments of this specification, if the number of logical slots that have not received a continuous acknowledgment message does not reach a preset threshold, it indicates that the reference chain is not completely broken, and the system can encode the video frame to be encoded based on the available reference frames. If there is at least one reference frame that has received an acknowledgment message in the reference frame buffer queue, excluding intra-frame coding keyframes, then these acknowledged reference frames constitute a safe prediction candidate set.

[0076] In practical applications, video sequences exhibit strong temporal correlation; frames that are closer in time exhibit less change in image content and smaller motion vectors. Typically, the intra-frame coding keyframe (I-frame) is the first frame in the reference frame buffer queue, or the first frame stored after a reset. Subsequent reference frames that receive acknowledgment messages are often temporally closer to the current video frame to be encoded than the intra-frame coding keyframe. Therefore, regardless of whether the system receives an acknowledgment message for an I-frame, if there are other reference frames with acknowledgment messages besides the I-frame, one of these acknowledgment frames can be selected for inter-frame predictive coding to achieve better prediction results. Alternatively, the reference frame with the closest frame number to the current video frame to be encoded can be selected as the target reference frame, and then used for inter-frame predictive coding of the video frame to be encoded. The target reference frame has the least change in image content and the smallest motion vector compared to the video frame to be encoded. Encoding the video frame to be encoded based on this reference frame minimizes motion residual energy, thereby achieving a higher compression ratio and coding efficiency.

[0077] For example, suppose there are three reference frames in the reference frame buffer queue that have received acknowledgment messages, with frame numbers 4, 6, and 7 respectively; the frame number of the current video frame to be encoded is 16. The system can calculate the difference between the frame number of the video frame to be encoded and each of the reference frames that have received acknowledgment messages: |16-4|=12, |16-6|=10, |16-7|=9. Among them, frame 7 has the smallest difference, so frame 7 is selected as the target reference frame. The encoder performs P-frame encoding on frame 16 with frame 7 as the target reference. Since the two are close in time, the motion residual is smaller compared to other reference frames that have received acknowledgment messages.

[0078] In real-world weak network environments, another situation may occur: although the number of consecutive unacknowledged slots in the reference frame buffer queue has not yet reached the preset number, no logical slot in the reference frame buffer queue has received an acknowledgment message, such as at the initial stage of encoding, or when ACK messages for all recent frames have not yet returned due to network latency. In this case, there are no acknowledged "safe" reference frames. To continue performing inter-frame predictive coding, this solution provides a robust implementation method.

[0079] Optionally, if there is no reference frame in the reference frame buffer queue that has received the acknowledgment message, or if there are no reference frames other than the intra-frame coding keyframe that have received the acknowledgment message, the available reference frame is the intra-frame coding keyframe in the reference frame buffer queue. Encoding the video frame to be encoded based on the available reference frames in the reference frame buffer queue may specifically include: The video frame to be encoded is encoded based on the intra-frame coding keyframes in the reference frame buffer queue.

[0080] In one or more embodiments of this specification, regardless of whether the system receives an acknowledgment message about the intra-frame coding keyframe, as long as there is no reference frame in the reference frame buffer queue that has received an acknowledgment message other than the intra-frame coding keyframe, the system can use the video intra-frame coding keyframe as a usable reference frame to encode the video frame to be encoded.

[0081] Specifically, the encoder can retrieve the I-frame (usually the only keyframe in the reference frame buffer queue) from the reference frame buffer queue, use the I-frame as the prediction reference source, perform motion estimation, motion compensation and residual coding, and generate the encoded video frame.

[0082] In one or more embodiments of this specification, since the I-frame itself has independent decoding capabilities, even if the acknowledgment status of the I-frame is unacknowledged, using it as a reference for predictive coding will not introduce a higher risk than directly using other reference frames that have not received acknowledgment messages for coding. At the same time, compared to encoding the frame to be encoded as an I-frame, using an I-frame as a reference for output encoding can significantly reduce the bit rate of the video frame, thereby maintaining stable bandwidth usage and coding efficiency in the absence of safe reference frames.

[0083] For ease of understanding, the embodiments in this specification also provide specific methods for storing encoded video frames into a reference frame buffer queue.

[0084] Optionally, storing the encoded video frame in the reference frame buffer queue may specifically include: If the reference frame buffer queue is not full, the encoded video frame is stored in the next free logical slot.

[0085] If the reference frame buffer queue is full and the reference frame buffer queue includes at least one unavailable reference frame that has not been marked as a usable reference frame, then one of the unavailable reference frames is replaced with the encoded video frame.

[0086] If the reference frame buffer queue is full and all reference frames are available, the earliest stored reference frame is replaced with the encoded video frame according to the first-in-first-out rule.

[0087] In one or more embodiments of this specification, the next free logical slot can refer to the next free logical slot relative to the currently occupied logical slot set. Specifically, the next free logical slot can be the first unoccupied logical slot found by searching backwards from the head of the queue (the slot with the smallest sequence number) according to the fixed sequence number of the logical slots, such as slot 1, slot 2, ..., slot N. In practical applications, each logical slot in the reference frame buffer queue can have a unique sequence number, and the system maintains the set of currently occupied logical slots. When a new frame needs to be stored, the encoder checks the occupancy flag of each logical slot sequentially, starting from the first logical slot. Once a logical slot is found to be unoccupied, it is the next free logical slot. For example, if logical slots 1, 2, and 3 are occupied, and logical slot 4 is free, then logical slot 4 is the "next free logical slot". This sequential search method is simple to implement, highly deterministic, and avoids management chaos caused by skipped-order storage.

[0088] In practical applications, if the reference frame buffer queue is not full, the system can look for the next available logical slot. The encoded video frame is then stored in the available logical slot, and the confirmation status of the logical slot is initialized to "unconfirmed".

[0089] A non-usable reference frame is a reference frame that has not been marked as a usable reference frame. In one or more embodiments of this specification, usable reference frames include two categories: ① reference frames that have received an acknowledgment message; ② intra-frame encoded keyframes (I-frames) in the absence of any acknowledgment reference frames. Conversely, if a reference frame has neither received an acknowledgment message nor is an I-frame, it is considered a non-usable reference frame.

[0090] In practical applications, if the reference frame buffer queue is full and contains unusable reference frames, the system can prioritize replacing these unusable reference frames to protect potentially valuable security reference frames. One implementation method is to replace them according to the "oldest first" principle, that is, selecting the oldest unusable reference frame, discarding the old video frame data in the logical slot containing that frame, and writing the new encoded video frame. For example, if the queue is full (all 14 logical slots are full), logical slots 1-3 contain usable reference frames, and logical slots 4-14 contain unusable reference frames, slot 4 can be selected for replacement. Another implementation method is to randomly select an unusable reference frame for replacement, that is, randomly selecting one unusable reference frame, discarding the old video frame data in the logical slot containing that frame, and writing the new encoded video frame.

[0091] In practical applications, if the reference frame buffer queue is full but all reference frames in the queue are available (meaning each reference frame in the queue has either received an acknowledgment message or is an I-frame), and there are no unavailable reference frames to replace them, the system can replace the oldest reference frame in the queue with a new encoded video frame, based on a first-in, first-out (FIFO) rule, to ensure that the encoded video frame can be successfully stored in the reference frame buffer queue without blocking the encoding process. Furthermore, by discarding the oldest video frame data, relatively newer reference frames are objectively preserved, which helps ensure the prediction efficiency of subsequent encoding.

[0092] When storing encoded video frames into the reference frame buffer queue or replacing old video frame data in the reference frame buffer queue, the system can write information such as the frame number, encoding data, and reference frame type of the encoded video frame into the target logical slot and reset the confirmation status of that logical slot to "unconfirmed". If the logical slot has a valid flag, the valid flag corresponding to that logical slot can also be set to "valid".

[0093] For ease of understanding, this specification also provides a specific method for determining whether the number of logical slots in the reference frame buffer queue that have not received a continuous acknowledgment message has reached a preset number.

[0094] Optionally, determining whether the number of logical slots in the reference frame buffer queue that have not received a consecutive acknowledgment message has reached a preset number may specifically include: According to the logical slot order in the reference frame buffer queue, count the number of consecutive logical slots for which the corresponding reference frame has not received an acknowledgment message.

[0095] Determine whether the quantity has reached the preset quantity.

[0096] In one or more embodiments of this specification, the preset quantity can typically be set to be equal to the total number of logical slots in the reference frame buffer queue, such as 14, in order to maximize the fault tolerance capability of the reference frame buffer.

[0097] Based on this, if the preset number is equal to the total number of logical slots in the reference frame buffer queue, then it is determined whether the number of logical slots in the reference frame buffer queue that have not received a continuous acknowledgment message has reached the preset number, which may specifically include: Starting from the head of the reference frame buffer queue, each logical slot is checked sequentially to see if an acknowledgment message has been received for its corresponding reference frame. If a logical slot that has received an acknowledgment message is encountered during the check, it is determined that the number of logical slots that have not received an acknowledgment message consecutively has not reached the preset number; if no logical slot that has received an acknowledgment message is found after checking all logical slots, it is determined that the number of logical slots that have not received an acknowledgment message consecutively has reached the preset number.

[0098] In one implementation, the encoder can maintain a counter, initially set to 0. Starting with the first logical slot, the confirmation status of each logical slot is checked sequentially: if the current slot's status is "unconfirmed," the counter is incremented by 1, and the process continues checking the next logical slot; if the current logical slot's status is "confirmed," the checking stops. The counter value then represents the number of consecutive unconfirmed slots. If no confirmed slot is encountered after checking all logical slots, the counter value equals the total number of occupied logical slots in the current queue (when the reference frame buffer queue is not full) or the total capacity of logical slots (when the reference frame buffer queue is full). Finally, the counter value is compared to a preset number. If the counter value is greater than or equal to the preset number, a reference chain break determination is triggered (subsequent I-frames are encoded and the buffer is reset); otherwise, normal encoding continues.

[0099] In one or more embodiments of this specification, the preset quantity may also be set to a value less than the total number of logical slots in the reference frame buffer queue.

[0100] Based on this, if the preset number is less than the total number of logical slots in the reference frame buffer queue; determining whether the number of logical slots in the reference frame buffer queue that have not received a continuous acknowledgment message has reached the preset number may specifically include: In the reference frame buffer queue, find the logical slot that received the last acknowledgment message.

[0101] If there exists a logical slot that has received the last acknowledgment message, then count the number of logical slots that have not received an acknowledgment message consecutively after that logical slot; if there are no logical slots that have received an acknowledgment message, then count the number of logical slots that have not received an acknowledgment message consecutively in the entire reference frame buffer queue; determine whether the counted number of logical slots reaches the preset number.

[0102] If the preset number is configured to be less than the total number of logical slots in the reference frame buffer queue, for example, if the total capacity is 14 and the preset number is 10, the condition for triggering the insertion of an I-frame is no longer that all logical slots have not received acknowledgment information, but that there is a continuous segment of unacknowledged logical slots.

[0103] In cases where the preset number is less than the total number of logical slots in the reference frame buffer queue, in one or more embodiments of this specification, the system may first search for the last logical slot in the reference frame buffer queue that has received an acknowledgment message. If the logical slot is found, the system counts the number of consecutive logical slots that have not received an acknowledgment message, starting from the next logical slot, sequentially until the end of the reference frame buffer queue. This count is the current number of consecutive logical slots that have not received an acknowledgment message. If there are no reference frames in the reference frame buffer queue that have received an acknowledgment message, the system counts the number of consecutive unacknowledgment slots starting from the head of the queue, i.e., the total number of all occupied slots in the queue. Finally, the counted number is compared with the preset number. If the preset number is reached or exceeded, a reference chain break is triggered, and an I-frame is forcibly inserted; otherwise, no break is triggered.

[0104] In practical applications, the number of logical slots that have not received a confirmed message consecutively can also be used to determine whether the number of such slots has reached a preset limit. The specific method is as follows: Obtain the first frame number of the video frame to be encoded; if there is a reference frame in the reference frame buffer queue that has received the confirmed message, obtain the second frame number of the reference frame that has received the confirmed message and is closest to the frame number of the video frame to be encoded; use the difference between the first frame number and the second frame number, divide by the sparse step size, to obtain the number of logical slots that have not received a confirmed message consecutively; determine whether the number of logical slots has reached the preset limit.

[0105] Specifically, the second frame number of the reference frame that has received an acknowledgment message closest to the frame number of the video frame to be encoded can be obtained, denoted as Fack; the first frame number of the video frame to be encoded can be obtained, denoted as Fcur; and the preset sparse step size M can be obtained. If there exists a reference frame that has received an acknowledgment message, the number of logical slots that have not received an acknowledgment message consecutively, N = (Fcur) / (Fcur) can be calculated. Fack) / M. If there are no reference frames that have received an acknowledgment message, then the number N of logical slots that have not received an acknowledgment message consecutively is equal to the total number of logical slots currently occupied in the reference frame buffer queue or the total number of logical slots. If the reference frame buffer queue is not full, the number is the number of occupied slots; if the reference frame buffer queue is full, the number is the total number of logical slots. Determine whether N has reached the preset number.

[0106] In one or more embodiments of this specification, under a stable sparse storage strategy, the slot order in the reference frame buffer queue corresponds to a sparse sequence with increasing frame numbers. Once a video frame is confirmed, all subsequent sparse frames are stored sequentially in the following logical slots. The confirmation status of these logical slots is either "unconfirmed" or possibly partially confirmed; however, since we take the confirmed frame Fack closest to the video frame to be encoded, Fack is the last confirmed sparse frame, and all subsequent sparse frames must be in the "unconfirmed" state. Therefore, by calculating the quotient of the frame number difference and M, the number of consecutive unconfirmed logical slots after the last confirmed frame can be accurately obtained. If there are no confirmed frames, then all slots in the entire queue are unconfirmed, and the number of occupied slots is taken.

[0107] To facilitate understanding, a concrete example is given below. Assume the sparse stride M = 2, and the total capacity of the reference frame buffer queue is 14. The current video frame to be encoded is frame number 30, and the frame number of the most recently confirmed reference frame is 20. Calculate (30... 20) / 2=5, meaning that 5 sparse frames should be stored between frame 20 and frame 30: frames 22, 24, 26, 28, and 30. None of these 5 consecutive logical slots received an ACK. If the preset quantity is 5, then N=5, reaching the preset quantity, triggering the insertion of an I-frame; if the preset quantity is 6, it does not trigger. If there are no acknowledged frames, and the current queue already occupies 10 slots (e.g., frames 2, 4, ..., 20), then N is 10, and if the preset quantity is ≤10, the insertion of an I-frame is triggered. If the preset quantity is the total capacity of the reference frame buffer queue, which is 14, it does not trigger.

[0108] In one or more embodiments of this specification, the number of logical slots that have not received a confirmed message consecutively is determined based on the frame number. This method has low computational complexity, requiring only a few integer operations, and avoids the CPU overhead caused by traversing the queue, making it suitable for high frame rate real-time encoding.

[0109] For ease of understanding, one or more embodiments of this specification also provide specific details on determining whether the video frame to be encoded conforms to a preset sparse storage strategy.

[0110] Optionally, the result can be obtained by calculating the modulo operation between the frame number of the video frame to be encoded and the sparse step size.

[0111] If the modulo operation result is 0, then the video frame to be encoded is determined to conform to the preset sparse storage strategy.

[0112] In one or more embodiments of this specification, the sparsity step size is a key parameter used to control the sparsity of the reference frame buffer. The sparsity step size defines how many frames in a continuously encoded video frame sequence are selected as reference frames and stored in the reference frame buffer queue.

[0113] In one or more embodiments of this specification, sparse selection of video frames can be achieved through modulo operations. Specifically, the encoding end obtains the frame number of the current video frame to be encoded and reads the currently configured sparse step size M. Then, it calculates the remainder of "frame number mod M". Here, modulo operation is a mathematical operation of integer division to find the remainder. A result of 0 indicates that the current frame number is divisible by M. If the remainder is 0, it can be determined that the video frame conforms to the preset sparse storage strategy, and the video frame can be stored in the reference frame buffer queue. If the remainder is not 0, it means that the video frame does not conform to the preset sparse storage strategy, will not be selected as a reference frame, and will not be stored in the reference frame buffer queue.

[0114] The method of determining whether a video frame to be encoded conforms to a preset sparse storage strategy based on modulo arithmetic is extremely lightweight, requiring only one integer division and comparison operation, without the need for additional buffering or complex calculations. The sparse step size M determines the frame density (1 / M) stored in the buffer. For example, M=2 means one frame is taken every two frames, and M=3 means one frame is taken every three frames. Since the video frame rate is constant, this uniform step size ensures that the reference frames in the buffer are distributed at approximately equal intervals along the time axis, so that the physical time span covered by the fixed-capacity logical slots is approximately proportional to M. In addition, the determinism of modulo arithmetic allows the encoder and receiver (i.e., the decoder) to predict which frames will become reference frames, facilitating the synchronization of the reference frame lists between the two parties.

[0115] As one implementation method, it is also possible to determine whether the video frame to be encoded conforms to the preset sparse storage strategy based on the timestamp interval. Specifically, the timestamp T_last of the previous frame stored in the reference frame buffer queue can be recorded. If the difference between the timestamp T_curr of the current frame and T_last exceeds the preset time threshold Δt (e.g., 50ms), then the current frame is determined to conform to the sparse storage strategy.

[0116] As another implementation, the frame interval counter can be used to determine whether the video frame to be encoded conforms to the preset sparse storage strategy. Specifically, a counter can be maintained, which increments by 1 for each frame encoded. When the counter reaches a preset step size M, the current frame is determined to conform to the preset sparse storage strategy, and the counter is reset to zero for the next counting and determination.

[0117] In one or more embodiments of this specification, the sparse step size can be configured as a parameter that is dynamically adjusted according to the network state to adapt to different network environments and balance compression efficiency and robustness in weak networks.

[0118] Accordingly, optionally, the sparse step size is a dynamically adjusted parameter, and the method may further include: Monitor network transmission quality parameters; the network transmission quality parameters include at least one of round-trip time (RTT), packet loss rate, or receiver feedback delay.

[0119] If the network transmission quality parameter indicates an improvement in network transmission quality, then the sparsity step size is reduced.

[0120] If the network transmission quality parameter indicates a decline in network transmission quality, then the sparsity step size is increased.

[0121] In one or more embodiments of this specification, network transmission quality parameters can refer to quantitative indicators that reflect the current network channel conditions, including at least one of round-trip time (RTT), packet loss rate, and receiver feedback delay. RTT refers to the total time it takes for a data packet to travel from the sender to the receiver, and for the acknowledgment data packet returned by the receiver to the sender. Packet loss rate refers to the proportion of data packets lost per unit time. Receiver feedback delay refers to the time difference between the receiver sending an acknowledgment message and the sender receiving it.

[0122] In practical applications, the sparsity step size is a positive integer, which can be used to control the density of reference frames stored in the reference frame buffer queue. The larger the sparsity step size, the sparser the reference frame storage, the longer the time window covered by the same buffer capacity, and the longer the tolerance time for the encoding end to wait for acknowledgment message feedback; the smaller the sparsity step size, the denser the reference frame storage, and the higher the compression efficiency.

[0123] In one or more embodiments of this specification, the encoding end can continuously collect or periodically sample network transmission quality parameters. For example, it can obtain round-trip time and packet loss rate through RTCP feedback messages, or confirm the arrival interval of messages through local timing statistics.

[0124] An improvement in network transmission quality can refer to a favorable change in parameter values, such as a decrease in round-trip time, a reduction in packet loss rate, or a shortening of feedback delay; a decrease in network transmission quality can refer to an unfavorable change in parameter values, such as an increase in round-trip time, an increase in packet loss rate, or a lengthening of feedback delay.

[0125] In practical applications, the encoder periodically evaluates network parameters, for example, every 100ms or every 30 frames. If an increase in transmission quality is detected, the sparse step size is reduced to make the reference frame storage denser and improve compression efficiency. If a decrease in transmission quality is detected, the sparse step size is increased to make the storage sparser, extend the time window for waiting for acknowledgment messages, and enhance the ability to withstand packet loss and high latency. Adjusting the sparse step size must ensure that it is not less than 1 and does not exceed a preset maximum value to ensure that the encoder adjusts within a reasonable range.

[0126] In one or more embodiments of this specification, when the network quality is good, the acknowledgment message is returned quickly, and acknowledgment can be obtained within a short time window. Therefore, denser reference frame storage (small sparse step size) can be used to improve prediction accuracy and compression ratio. When the network quality is poor, the risk of acknowledgment message delay or loss is high, requiring a longer tolerance time. By increasing the sparse step size to lengthen the number of physical frames covered by each logical slot, the time required for consecutive unacknowledged slots to reach the threshold is longer, thereby reducing unnecessary I-frame insertion.

[0127] In practical applications, during video encoding and post-encoded video transmission, the system will continuously receive acknowledgment messages (ACK messages) or negative responses (NACK messages) returned by the receiver through RTCP or other feedback channels to determine whether each reference frame has been correctly received.

[0128] Accordingly, the method may optionally include: Get the feedback message from the receiving end.

[0129] If the feedback message contains an acknowledgment message for a specified reference frame, then according to the acknowledgment message, the acknowledgment status of the logical slot corresponding to the specified reference frame in the reference frame cache queue is updated to "acknowledged".

[0130] Feedback messages can refer to signaling messages sent by the receiving end to the encoding end to inform the encoding end of the video frame reception status. In actual real-time communication systems, after receiving one or more video frames, the receiving end can send back a status report through the Real-time Transmission Control Protocol (RTP, RTCP) or other feedback channels. Feedback messages typically include: acknowledgment messages or negative acknowledgment messages; acknowledgment messages, i.e., ACK messages, indicate that the specified video frame has been successfully received and correctly decoded; negative acknowledgment messages, i.e., NACK messages, indicate that the specified video frame has been lost or corrupted, requesting the sending end to retransmit. In practical applications, the feedback message received by the encoding end can be either an ACK message or a NACK message.

[0131] In practical applications, feedback messages typically carry a frame number. Upon receiving a feedback message, the system can parse it to obtain the frame number. Based on this frame number, it can determine whether the video frame is a reference frame. If not, the feedback message can be discarded. If it is, it can be further checked whether the feedback message is an acknowledgment message. If so, it indicates that the receiving end has successfully received the video frame.

[0132] In one or more embodiments of this specification, the specified reference frame may refer to any reference frame in the current reference frame buffer queue, specifically the reference frame corresponding to the frame number carried in the acknowledgment message.

[0133] If parsing the feedback message yields a confirmation message, the logical slot corresponding to that video frame number can be found in the reference frame buffer queue based on the frame number in the confirmation message. The confirmation status of that logical slot is then updated from "unconfirmed" to "confirmed." Once the logical slot is marked as "confirmed," the reference frame stored in that logical slot is considered a safe and usable prediction source, which can be referenced by subsequent video frames to be encoded. In practical applications, the logical slot corresponding to a frame number can be determined through the mapping relationship between frame numbers and logical slots.

[0134] In practical applications, if the received feedback message is a negative acknowledgment message, the acknowledgment status of the corresponding logical slot of the video frame can be kept as "unacknowledged" without updating the status. This allows the number of consecutive unacknowledged slots to continue accumulating, triggering I-frame reconstruction synchronization when necessary. Furthermore, the encoding end can initiate a retransmission mechanism based on the negative acknowledgment message, resending the video frame to the receiving end to reduce the impact of packet loss on decoding quality.

[0135] In one or more embodiments of this specification, the encoder can ensure that the reference frame synchronization information between the encoder and decoder is up-to-date by acquiring and processing the acknowledgment message fed back by the receiver, thus avoiding making incorrect encoding decisions due to outdated status. In addition, by processing the ACK in a timely manner, the continuous unacknowledged count can be interrupted, preventing the I-frame from being mistakenly triggered due to delayed ACK.

[0136] The following is in conjunction with the appendix Figure 3 Taking a reference frame buffer queue with a capacity of 14 logical slots, a sparse step size of 2, and an example where the number of logical slots that trigger the insertion of an I-frame without receiving an acknowledgment message equals the total number of logical slots, the video frame encoding control method will be further explained. Figure 3 A flowchart of a video frame encoding control method according to an embodiment of this specification is shown, including the following steps.

[0137] Step 302: Obtain the feedback message from the receiving end.

[0138] The encoding end can receive feedback messages from the receiving end via the network. These feedback messages can be acknowledgment messages or negative response messages.

[0139] Step 304: Determine whether the feedback message contains an acknowledgment message for the specified reference frame; if yes, proceed to step 306; otherwise, proceed to step 308.

[0140] In practical applications, the system can parse the received feedback message and check whether it is an acknowledgment message (ACK) for a specific reference frame. The specific process has been detailed above and will not be repeated here.

[0141] Step 306: Based on the confirmation message, update the confirmation status of the logical slot corresponding to the specified reference frame in the reference frame buffer queue to "confirmed". Return to step 302, and continue executing step 310.

[0142] In practical applications, acknowledgment messages typically carry a frame number to indicate that the receiving end has successfully received the video frame. Therefore, based on the frame number in the acknowledgment message, the logical slot corresponding to that video frame number can be found in the reference frame buffer queue, and the acknowledgment status of that logical slot is updated from "unacknowledged" to "acknowledged".

[0143] Step 308: Keep the confirmation status of the corresponding logical slot as unconfirmed. Then return to step 302.

[0144] Understandably, steps 302 to 308 are processed simultaneously with other steps in the following text, that is, steps 302 to 308 run continuously.

[0145] Step 310: Obtain the video frames to be encoded.

[0146] Step 312: Determine whether the number of logical slots in the reference frame buffer queue that have not received a continuous acknowledgment message has reached the preset number; if not, proceed to step 314; if yes, proceed to step 320.

[0147] In practical applications, the number of logical slots in the reference frame buffer queue that have not received a continuous acknowledgment message can be determined based on the acknowledgment status of each reference frame in the reference frame buffer queue.

[0148] Specifically, starting from the head of the reference frame buffer queue, count the number of logical slots that have not received a consecutive acknowledgment message. If the number is ≥14, the reference chain is determined to be broken, and the process jumps to step 320; otherwise, step 314 is executed.

[0149] Step 314: Encode the video frame to be encoded based on the available reference frames in the reference frame buffer queue to obtain the encoded video frame.

[0150] In practical applications, if there are reference frames in the reference frame buffer queue that have received acknowledgment messages, other than intra-coding keyframes, then the available reference frames are those that have received acknowledgment messages, excluding intra-coding keyframes. The target reference frame whose frame number is closest to the frame number of the video frame to be encoded can be determined from the available reference frames. Encoding is then performed on the video frame to be encoded based on the target reference frame.

[0151] If there is no reference frame in the reference frame buffer queue that has received an acknowledgment message, or if there is no reference frame in the reference frame buffer queue that has received an acknowledgment message except for the intra-frame coding keyframe, then the reference frame can be used as the intra-frame coding keyframe in the reference frame buffer queue; the video frame to be encoded can be encoded based on the intra-frame coding keyframe.

[0152] Step 316: Determine whether the video frame to be encoded conforms to the preset sparse storage strategy; if yes, proceed to step 318; otherwise, proceed to step 324.

[0153] In one or more embodiments of this specification, a preset sparse storage strategy is used to instruct the non-contiguous selection of a portion of video frames from a continuously encoded video frame sequence as reference frames to be stored in a reference frame buffer queue.

[0154] In practical applications, the modulo operation result of the frame number and sparse step size of the video frame to be encoded can be calculated. If the modulo operation result is 0, the video frame to be encoded is determined to conform to the preset sparse storage strategy. If the modulo operation result is not 0, the video frame to be encoded is determined to not conform to the preset sparse storage strategy.

[0155] For example, with a sparse step size of 2, the current frame number mod2 can be calculated. If the result is 0, it can be determined that the video frame conforms to the preset sparse storage strategy, such as even-numbered frames like frames 2, 4, and 6; otherwise, it can be determined that the video frame does not conform to the preset sparse storage strategy.

[0156] After storing the encoded video frame in the reference frame buffer queue, the next frame can be processed.

[0157] Step 318: Store the encoded video frames into the reference frame buffer queue.

[0158] Step 320: Encode the video frame to be encoded into an intra-frame coded keyframe.

[0159] Step 322: Reset the reference frame buffer queue and store the intra-frame encoded keyframes into the reset reference frame buffer queue.

[0160] Step 324: Send intra-frame encoded keyframes or encoded video frames to the receiving end. Return to step 310.

[0161] Based on the same idea, embodiments of this specification also provide apparatus corresponding to the above methods.

[0162] Figure 4 This is a schematic diagram of a video frame encoding control device provided in the embodiments of this specification.

[0163] like Figure 4 As shown, the device may include: The first judgment module 402 is used to determine whether the number of logical slots in the reference frame cache queue that have not received a continuous acknowledgment message has reached a preset number based on the acknowledgment status of each reference frame in the reference frame cache queue.

[0164] The first decision module 404 is used to encode the video frame to be encoded based on the available reference frames in the reference frame buffer queue if the number of logical slots that have not received a confirmation message for a consecutive time does not reach the preset number, so as to obtain the encoded video frame.

[0165] The second judgment module 406 is used to judge whether the video frame to be encoded conforms to the preset sparse storage strategy and obtain the judgment result; the preset sparse storage strategy is used to instruct to select a portion of video frames non-continuously from the continuously encoded video frame sequence as reference frames and store them in the reference frame buffer queue.

[0166] The second decision module 408 is used to store the encoded video frame into the reference frame cache queue if the judgment result indicates that the video frame to be encoded conforms to the preset sparse storage strategy.

[0167] Optional, such as Figure 4 The apparatus shown may further include: The third decision module is used to encode the video frame to be encoded as an intra-frame keyframe if the number of logical slots that have not received a confirmation message for a consecutive period reaches the preset number.

[0168] The reset module is used to reset the reference frame buffer queue to obtain the reset reference frame buffer queue.

[0169] The storage module is used to store the intra-frame encoded keyframes into the reset reference frame buffer queue.

[0170] Optionally, if there are reference frames in the reference frame buffer queue that have received the acknowledgment message, excluding the intra-coded keyframes, the available reference frames are those that have received the acknowledgment message, excluding the intra-coded keyframes; the first decision module 404 can specifically be used for: From the available reference frames, determine the target reference frame whose frame number is closest to the frame number of the video frame to be encoded.

[0171] Encoding is performed on the video frame to be encoded based on the target reference frame.

[0172] Optionally, if there is no reference frame in the reference frame buffer queue that has received the acknowledgment message, or if there is no reference frame other than the intra-coded keyframe that has received the acknowledgment message, the available reference frame is the intra-coded keyframe in the reference frame buffer queue; the first decision module 404 can be specifically used for: The video frame to be encoded is encoded based on the intra-frame coding keyframes in the reference frame buffer queue.

[0173] Optionally, the second decision module 408 can be specifically used for: If the reference frame buffer queue is not full, the encoded video frame is stored in the next free logical slot.

[0174] If the reference frame buffer queue is full and the reference frame buffer queue includes at least one unavailable reference frame that has not been marked as a usable reference frame, then one of the unavailable reference frames is replaced with the encoded video frame.

[0175] If the reference frame buffer queue is full and all reference frames are available, the earliest stored reference frame is replaced with the encoded video frame according to the first-in-first-out rule.

[0176] Optionally, the first judgment module 402 can be specifically used for: According to the logical slot order in the reference frame buffer queue, count the number of consecutive logical slots for which the corresponding reference frame has not received an acknowledgment message.

[0177] Determine whether the quantity has reached the preset quantity.

[0178] Optionally, the second judgment module 406 can be specifically used for: Calculate the result of the modulo operation between the frame number and the sparse step size of the video frame to be encoded.

[0179] If the modulo operation result is 0, then the video frame to be encoded is determined to conform to the preset sparse storage strategy.

[0180] Optional, such as Figure 4 The apparatus shown may further include: The monitoring module is used to monitor network transmission quality parameters; the network transmission quality parameters include at least one of round-trip time, packet loss rate, or receiver feedback delay.

[0181] The first sparse step size adjustment module is used to reduce the sparse step size if the network transmission quality parameter indicates an improvement in network transmission quality.

[0182] The second sparse step size adjustment module is used to increase the sparse step size if the network transmission quality parameter indicates a decrease in network transmission quality.

[0183] Optional, such as Figure 4 The apparatus shown may further include: The feedback message acquisition module is used to acquire feedback messages from the receiving end.

[0184] The status update module is used to update the confirmation status of the logical slot corresponding to the specified reference frame in the reference frame cache queue to "confirmed" according to the confirmation message if the feedback message contains a confirmation message for the specified reference frame.

[0185] The above is a schematic scheme of a video frame encoding control device according to this embodiment. It should be noted that the technical solution of this video frame encoding control device and the technical solution of the video frame encoding control method described above belong to the same concept. For details not described in detail in the technical solution of the video frame encoding control device, please refer to the description of the technical solution of the video frame encoding control method described above.

[0186] Based on the same idea, this specification also provides devices corresponding to the above methods in its embodiments.

[0187] Figure 5 A schematic diagram of the structure of a computing device 500 provided according to an embodiment of this specification is shown.

[0188] The computing device 500 includes: Memory 510 and processor 520; The memory 510 is used to store computer programs / instructions, and the processor 520 is used to execute the computer programs / instructions, which, when executed by the processor 520, implement the steps of the video frame encoding control method.

[0189] Specifically, the components of the computing device 500 include, but are not limited to, a memory 510 and a processor 520. The processor 520 is connected to the memory 510 via a bus 530, and the database 550 is used to store data.

[0190] The computing device 500 also includes an access device 540, which enables the computing device 500 to communicate via one or more networks 560. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. The access device 540 may include one or more of any type of wired or wireless network interface (e.g., a network interface card (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, a Wi-MAX (Worldwide Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC) interface, and so on.

[0191] In one embodiment of this specification, the aforementioned components of the computing device 500 and Figure 5 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 5 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this application. Those skilled in the art can add or replace other components as needed.

[0192] The computing device 500 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). The computing device 500 can also be a mobile or stationary server.

[0193] The processor 520 executes the computer instructions to implement the steps of the video frame encoding control method.

[0194] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device and the technical solution of the video frame encoding control method described above belong to the same concept. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solution of the video frame encoding control method described above.

[0195] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for apparatuses, devices, and embodiments, since they are basically similar to the method embodiments, the descriptions are relatively simple, and relevant parts can be referred to the descriptions of the method embodiments. The apparatuses, devices, and methods provided in the embodiments of this specification are corresponding to each other, and therefore the apparatuses and devices also have similar beneficial technical effects as the corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the corresponding apparatuses and devices will not be repeated here.

[0196] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0197] In the 1990s, improvements to a technology could be clearly distinguished as either hardware improvements (e.g., improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to methodology). However, with technological advancements, many methodological improvements today can be considered direct improvements to hardware circuit structures. Designers almost always obtain the corresponding hardware circuit structure by programming the improved methodology into the hardware circuit. Therefore, it cannot be said that a methodological improvement cannot be implemented using hardware physical modules. For example, a Programmable Logic Device (PLD) (such as a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logic function is determined by the user programming the device. Designers can program a digital system themselves to "integrate" it onto a PLD, without needing chip manufacturers to design and manufacture dedicated integrated circuit chips. Furthermore, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly implemented using "logic compiler" software. Similar to the software compiler used in program development, the original code before compilation must also be written in a specific programming language, called a Hardware Description Language (HDL). There are many HDLs, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, the most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also understand that by simply performing some logic programming on the method flow using one of these hardware description languages ​​and programming it into an integrated circuit, the hardware circuit implementing the logical method flow can be easily obtained.

[0198] The controller can be implemented in any suitable manner. For example, it can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code form, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0199] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0200] For ease of description, the above devices are described separately by function as various units. Of course, in implementing this application, the functions of each unit can be implemented in one or more software and / or hardware.

[0201] Those skilled in the art will understand that one or more embodiments of this specification can be provided as a method, system, or computer program product. Therefore, the invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0202] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0203] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0204] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0205] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0206] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0207] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital character versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0208] This application can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0209] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A video frame coding control method, comprising: Based on the acknowledgment status of each reference frame in the reference frame buffer queue, determine whether the number of logical slots in the reference frame buffer queue that have not received consecutive acknowledgment messages has reached a preset number. If the number of logical slots that have not received a confirmation message for a consecutive period does not reach the preset number, the video frame to be encoded is encoded based on the available reference frames in the reference frame buffer queue to obtain the encoded video frame; the available reference frames are the reference frames in the reference frame buffer queue that have received the confirmation message or the intra-frame encoded keyframes. Determine whether the video frame to be encoded conforms to the preset sparse storage strategy, and obtain the determination result; The preset sparse storage strategy is used to instruct the selection of a portion of video frames from a continuously encoded video frame sequence as reference frames and store them in the reference frame buffer queue. If the determination result indicates that the video frame to be encoded conforms to the preset sparse storage strategy, then the encoded video frame is stored in the reference frame buffer queue.

2. The method as described in claim 1, further comprising, after determining whether the number of logical slots in the reference frame buffer queue that have not received a consecutive acknowledgment message has reached a preset number: If the number of logical slots that have not received a confirmation message for a consecutive period reaches the preset number, the video frame to be encoded will be encoded as an intra-frame coded keyframe. Reset the reference frame buffer queue to obtain the reset reference frame buffer queue; The intra-frame coded keyframe is stored in the reset reference frame buffer queue.

3. The method as described in claim 1, wherein if there are reference frames in the reference frame buffer queue that have received the acknowledgment message, excluding the intra-coding keyframes, the available reference frames are the reference frames that have received the acknowledgment message, excluding the intra-coding keyframes; the encoding of the video frame to be encoded based on the available reference frames in the reference frame buffer queue specifically includes: Determine the target reference frame from the available reference frames whose frame number is closest to the frame number of the video frame to be encoded; Encoding is performed on the video frame to be encoded based on the target reference frame.

4. The method as described in claim 1, wherein if there is no reference frame in the reference frame buffer queue that has received the acknowledgment message, or if there is no reference frame other than the intra-frame coding keyframe that has received the acknowledgment message, the available reference frame is the intra-frame coding keyframe in the reference frame buffer queue; the encoding of the video frame to be encoded based on the available reference frames in the reference frame buffer queue specifically includes: The video frame to be encoded is encoded based on the intra-frame coding keyframes in the reference frame buffer queue.

5. The method as described in claim 1, wherein storing the encoded video frame into the reference frame buffer queue specifically includes: If the reference frame buffer queue is not full, the encoded video frame is stored in the next free logical slot. If the reference frame buffer queue is full and the reference frame buffer queue includes at least one unavailable reference frame that has not been marked as a usable reference frame, then one of the unavailable reference frames is replaced with the encoded video frame. If the reference frame buffer queue is full and all reference frames are available, the earliest stored reference frame is replaced with the encoded video frame according to the first-in-first-out rule.

6. The method as described in claim 1, wherein determining whether the number of logical slots in the reference frame buffer queue that have not received consecutive acknowledgment messages has reached a preset number specifically includes: According to the logical slot order in the reference frame buffer queue, count the number of consecutive logical slots for which the corresponding reference frame has not received an acknowledgment message; Determine whether the quantity has reached the preset quantity.

7. The method as described in claim 1, wherein determining whether the video frame to be encoded conforms to a preset sparse storage strategy specifically includes: Calculate the result of the modulo operation between the frame number and the sparse step size of the video frame to be encoded; If the modulo operation result is 0, then the video frame to be encoded is determined to conform to the preset sparse storage strategy.

8. The method as described in claim 1, wherein the preset sparse storage strategy is implemented by a sparse step size, wherein the sparse step size is a positive integer, used to indicate that a frame is selected from the continuously encoded video frame sequence at intervals of the sparse step size minus 1 frame as a reference frame and stored in the reference frame buffer queue.

9. The method of claim 8, wherein the sparse step size is a dynamically adjusted parameter, and the method further comprises: Monitor network transmission quality parameters; The network transmission quality parameters include at least one of round-trip time, packet loss rate, or receiver feedback delay; If the network transmission quality parameter indicates an improvement in network transmission quality, then the sparsity step size is reduced. If the network transmission quality parameter indicates a decline in network transmission quality, then the sparsity step size is increased.

10. The method as described in claim 7 or 8, wherein determining whether the number of logical slots in the reference frame buffer queue that have not received a consecutive acknowledgment message has reached a preset number specifically includes: Obtain the first frame number of the video frame to be encoded; If there is a reference frame in the reference frame buffer queue that has received the acknowledgment message, then obtain the second frame number of the reference frame that has received the acknowledgment message and is closest to the frame number of the video frame to be encoded. The number of logical slots that have not received a continuous acknowledgment message is obtained by dividing the difference between the first frame number and the second frame number by the sparse step size. Determine whether the number of logical slots has reached the preset number.

11. The method of claim 1, further comprising: Get the feedback message from the receiving end; If the feedback message contains an acknowledgment message for a specified reference frame, then according to the acknowledgment message, the acknowledgment status of the logical slot corresponding to the specified reference frame in the reference frame cache queue is updated to acknowledged.

12. A video frame encoding control device, comprising: The first judgment module is used to determine whether the number of logical slots in the reference frame cache queue that have not received a continuous acknowledgment message has reached a preset number based on the acknowledgment status of each reference frame in the reference frame cache queue. The first decision module is used to encode the video frame to be encoded based on the available reference frames in the reference frame buffer queue if the number of logical slots that have not received a confirmation message for a consecutive time does not reach the preset number, so as to obtain the encoded video frame. The second judgment module is used to determine whether the video frame to be encoded conforms to the preset sparse storage strategy and obtain the judgment result. The preset sparse storage strategy is used to instruct the selection of a portion of video frames from a continuously encoded video frame sequence as reference frames and store them in the reference frame buffer queue. The second decision module is used to store the encoded video frame into the reference frame cache queue if the judgment result indicates that the video frame to be encoded conforms to the preset sparse storage strategy.

13. A computing device, comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the method according to any one of claims 1 to 11.

Images