A frame loss method, device, server and medium
By determining the total number of frames to be dropped and the priority of discarding in the video frame queue, and selecting the target discarding frame processing, the problem of video stuttering under weak network conditions is solved, and bandwidth utilization and user experience are improved.
Patent Information
- Application Number
- CN202210207969.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-03-04
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2042-03-04
AI Technical Summary
Under weak network conditions, live videos are stuck due to slow transmission speed, and the existing collective frame loss method will waste bandwidth and reduce user experience.
By obtaining the timeout frames and the number of undropped frames in the video frame queue, the total number of frames to be dropped is determined, and the discard priority is determined based on the decoding reference information, thereby selecting the target discarded frame for processing, avoiding collective frame drops and ensuring low latency and video Continuity.
Reduces the number of frame drops, improves bandwidth utilization and video playback coherence, and improves user viewing experience.
Smart Images

Figure CN114567796B_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present disclosure relate to Internet technology, and in particular to a frame loss method, device, server and medium. Background Art
[0002] With the rapid development of Internet technology, more and more users like to watch live videos on the client. When the network quality is good, stable low-latency live video can be obtained. However, in weak network conditions, due to the slow transmission speed of live audio and video data, the video will be stuck and waiting, which will increase the delay. At present, in order to reduce latency and avoid screen distortion, the continuous video frames from the current timeout to the next key frame can be collectively dropped. However, this collective frame loss method often causes the server to have no data to send for a period of time, which wastes bandwidth and also causes the video to be stuck for a long time, reducing the user's viewing experience. Summary of the invention
[0003] The embodiments of the present disclosure provide a frame loss method, device, server and medium to reduce the number of frame losses and avoid the situation where no data can be sent while ensuring low latency and avoiding screen distortion, thereby improving bandwidth utilization and the continuity of video playback and enhancing the user viewing experience.
[0004] In a first aspect, an embodiment of the present disclosure provides a frame dropping method, including:
[0005] Get the current video frame queue to be sent;
[0006] Determine the timed-out video frames in the current video frame queue, and determine the total number of current to-be-lost frames corresponding to the current video frame queue based on the number of the timed-out video frames and the current number of unlost frames;
[0007] Determining discardable video frames in the current video frame queue;
[0008] Based on the current total number of frames to be discarded and the discarding priority corresponding to the discardable video frames, a target discarded video frame is determined from the discardable video frames, and the target discarded video frame is subjected to frame discarding processing, wherein the discarding priority is determined based on the decoding reference information corresponding to the discarded video frame.
[0009] In a second aspect, the embodiment of the present disclosure further provides a frame loss device, including:
[0010] The current video frame queue acquisition module is used to acquire the current video frame queue to be sent;
[0011] A current total number of frames to be lost determining module, used to determine the timed-out video frames in the current video frame queue, and based on the number of the timed-out video frames and the current number of non-lost frames, determine the current total number of frames to be lost corresponding to the current video frame queue;
[0012] A discardable video frame determination module, used to determine the discardable video frames in the current video frame queue;
[0013] A frame loss processing module is used to determine a target discarded video frame from the discarded video frames based on the current total number of frames to be lost and the discarding priority corresponding to the discarded video frames, and perform frame loss processing on the target discarded video frames, wherein the discarding priority is determined based on the decoding reference information corresponding to the discarded video frames.
[0014] In a third aspect, an embodiment of the present disclosure further provides a server, the server comprising:
[0015] one or more processors;
[0016] A memory for storing one or more programs;
[0017] When the one or more programs are executed by the one or more processors, the one or more processors implement the frame dropping method provided in any embodiment of the present disclosure.
[0018] In a fourth aspect, an embodiment of the present disclosure further provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, a frame dropping method provided in any embodiment of the present disclosure is implemented.
[0019] The disclosed embodiment does not directly drop frames after determining the timed-out video frames in the current video frame queue to be sent, but determines the total number of frames to be dropped corresponding to the current video frame queue based on the number of timed-out video frames and the current number of non-dropped frames, that is, the total number of frames that need to be dropped to ensure low latency. By determining the discardable video frames in the current video frame queue, and determining the discarding priority and the current total number of frames to be dropped based on the decoding reference information corresponding to the discardable video frames, the target discarded video frames to be discarded are determined from each discardable video frame without collective frame dropping, thereby reducing the number of dropped frames while ensuring low latency and avoiding screen distortion, avoiding the situation where there is no data to be sent, thereby improving bandwidth utilization and the continuity of video playback, and enhancing the user viewing experience. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.
[0021] Figure 1 is a flow chart of a frame dropping method provided in Embodiment 1 of the present disclosure;
[0022] Figure 2 This is an example of determining a discard priority involved in the first embodiment of the present disclosure;
[0023] Figure 3 is a flow chart of a frame dropping method provided in Embodiment 2 of the present disclosure;
[0024] Figure 4 is a flow chart of a frame dropping method provided in Embodiment 3 of the present disclosure;
[0025] Figure 5 is a structural diagram of a frame dropping device provided in Embodiment 4 of the present disclosure;
[0026] Figure 6 It is a structural diagram of a server provided in Embodiment 5 of the present disclosure. DETAILED DESCRIPTION
[0027] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.
[0028] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.
[0029] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.
[0030] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.
[0031] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".
[0032] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.
[0033] Embodiment 1
[0034] Figure 1 This is a flowchart of a frame dropping method provided in the first embodiment of the present disclosure. The present embodiment of the present disclosure can be applied to the situation where frame dropping is performed during video transmission to ensure low latency, especially in the scenario where frame dropping is performed on the live video to be transmitted in a live broadcasting scenario. The method can be performed by a frame dropping device, which can be implemented by software and / or hardware and integrated in a server. Figure 1 As shown, the method specifically comprises the following steps:
[0035] S110: Obtain a current video frame queue to be sent.
[0036] Specifically, the server can use a thread A to encode the current video to be sent, and write the encoded video frames into the video frame queue to be sent in sequence. For example, in a live broadcast scenario, the live video received in real time can be encoded, and the encoded video frames can be written into the video frame queue to be sent in sequence. In the disclosed embodiment, the video frame queue to be sent changes dynamically over time. The server can use another thread B to obtain the current video frame queue to be sent at the current moment in real time or periodically, so as to send each video frame in the current video frame queue to the client in sequence, and remove the successfully sent video frames from the video frame queue, and perform subsequent frame loss processing on the video frames that have not been sent due to poor network conditions to ensure low latency of the video.
[0037] S120: Determine the timed-out video frames in the current video frame queue, and determine the total number of current to-be-lost frames corresponding to the current video frame queue based on the number of timed-out video frames and the current number of non-lost frames.
[0038] Among them, the timed-out video frame may refer to a video frame whose time between the current moment and the moment of starting to send exceeds the preset delay time. The preset delay time may be the maximum allowed video delay time pre-set based on business needs and scenarios. The moment of starting to send may refer to the moment when the video frame is allowed to start sending, so as to maintain real-time synchronization of the video. The current number of unlost frames may refer to the number of frames that should have been lost but have not been lost as of the current moment. For example, the previous video frame queue should have lost 10 frames, but actually only 3 frames were lost. At this time, the current number of unlost frames is 7 frames, so 7 more frames need to be lost in the current video frame queue to ensure low latency. The current total number of frames to be lost may refer to the total number of video frames that need to be discarded in the current video frame queue to avoid the video delay time exceeding the preset delay time and ensure low latency of the video.
[0039] Specifically, for each video frame to be sent in the current video frame queue, it can be detected whether the current time exceeds the current timeout time corresponding to the video frame. If it exceeds, the video frame is determined to be a timeout video frame. The initial value of the current timeout time corresponding to the video frame can be determined based on the start sending time of the video frame and the preset delay time. The number of all timeout video frames in the current video frame queue and the current number of non-lost frames are added, and the obtained addition result is determined as the current total number of frames to be lost corresponding to the current video frame queue.
[0040] S130: Determine the discardable video frames in the current video frame queue.
[0041] The discardable video frames may refer to video frames in the current video frame queue that can be discarded in order to avoid screen distortion.
[0042] Specifically, when encoding a video frame, the video frame can be encoded as a key frame (i.e., an I frame), a forward predictive coded frame (i.e., a P frame), or a bidirectional predictive coded frame (i.e., a B frame), wherein when decoding an I frame, only the information of the key frame itself needs to be used, without referring to other video frames. The interval between two I frames is relatively long. When decoding a P frame, it is necessary to refer to the information of the video frame before the current frame in order to decode it. If the video frame to which it refers is lost, a distorted screen phenomenon will result. When decoding a B frame, it is necessary to refer to the video frames before and after the current frame in order to decode it. If the video frame to which it refers is lost, a distorted screen phenomenon will also result. Based on this characteristic of video frame encoding, the disclosed embodiment can detect whether there is at least one key frame in the current video frame queue, and whether the last key frame in the queue is the first frame of the queue, and determine the discardable video frames in the current video frame queue based on the detection result.
[0043] For example, if it is detected that there is at least one key frame in the current video frame queue and the last key frame is not the first frame of the queue, each video frame before the last key frame in the current video frame queue can be used as a discardable video frame; if it is detected that there is no key frame in the current video frame queue, or the only key frame that exists is the first frame of the queue, then the operation of step S110 can be returned to wait until it is detected that there is at least one key frame in the current video frame queue and the last key frame is not the first frame of the queue, so as to start the subsequent frame dropping operation after the next I frame is written to the queue.
[0044] For example, if it is detected that there is no key frame in the current video frame queue, or the only key frame that exists is the first frame of the queue, frame drop can be performed in advance by predicting the number of remaining video frames in the current image group GOP (Group of Pictures) in the current video frame queue that have not been written into the queue, that is, the subsequent frame drop operation is triggered before the next I frame is written into the queue.
[0045] S140. Based on the total number of frames to be discarded and the discarding priority corresponding to the discardable video frames, determine the target discarded video frames from the discardable video frames, and perform frame discarding processing on the target discarded video frames, wherein the discarding priority is determined based on the decoding reference information corresponding to the discarded video frames.
[0046] Among them, the discarding priority can be used to characterize the discarding order corresponding to the discardable video frame, so as to minimize the number of discarded frames while avoiding the occurrence of a distorted screen. The disclosed embodiment can use a numerical value to characterize the level of the discarding priority. For example, the lower the discarding priority of the video frame, that is, the smaller the numerical value, the higher the priority of the discarding of the video frame. The decoding reference information corresponding to the discardable video frame can refer to whether the information of the discardable video frame needs to be referenced when decoding other video frames. For example, if the information of the discardable video frame does not need to be referenced when decoding other video frames, the discardable video frame is determined to be a non-reference frame. At this time, if the non-reference frame is discarded, it will not affect the decoding of other video frames, and thus the distorted screen will not appear. If the information of the discardable video frame needs to be referenced when decoding other video frames, the discardable video frame is determined to be a reference frame. At this time, if the reference frame is discarded, it will affect the decoding of other video frames, but the reference frame is often referenced by the video frame behind it, so that the discarding priority corresponding to the reference frame located behind can be set lower, so that the reference frame behind can be discarded more preferentially, thereby avoiding the occurrence of a distorted screen.
[0047] Specifically, for congestion control algorithms based on bandwidth detection, such as the BBR algorithm, collective frame loss often results in no data being sent for a period of time, which will also reduce the bandwidth detection result, thereby reducing the data transmission rate, and the lower data transmission rate will trigger frame loss again when less data is sent, which may eventually lead to an avalanche, greatly affecting the user's viewing experience. The disclosed embodiment can determine the target discarded video frame that needs to be discarded this time from each discardable video frame based on the discard priority corresponding to each discardable video frame, and perform frame loss processing on each target discarded video frame, thereby reducing the number of frame losses while ensuring low latency and avoiding screen distortion, without the need for collective frame loss, while also avoiding the situation where there is no data to send, avoiding the situation of avalanche, and improving bandwidth utilization, thereby also improving the continuity of video playback and enhancing the user's viewing experience.
[0048] It should be noted that if the number of discardable video frames in the current video frame queue is less than the total number of frames to be discarded, it means that the total number of video frames cannot be discarded in this frame loss, that is, the current video frame queue cannot meet the frame loss requirements. At this time, the remaining frames that have not been discarded can be discarded in the next video frame queue to ensure low latency of the video.
[0049] Exemplarily, after S140, the method may further include: determining and updating the current number of unlost frames based on the current total number of frames to be lost and the target number of discarded video frames. Specifically, the difference between the current total number of frames to be lost and the target number of discarded video frames may be determined as the current number of unlost frames, so that frame loss processing is performed when a frame is lost next time, thereby ensuring low latency of the video.
[0050] The technical solution of the disclosed embodiment does not directly drop frames after determining the timed-out video frames in the current video frame queue to be sent, but determines the total number of frames to be dropped corresponding to the current video frame queue based on the number of timed-out video frames and the current number of frames not dropped, that is, the total number of frames that need to be dropped to ensure low latency. By determining the discardable video frames in the current video frame queue, and based on the discarding priority and the current total number of frames to be dropped determined based on the decoding reference information corresponding to the discardable video frames, the target discarded video frames are determined from each discardable video frame and the frame dropping process is performed, so that there is no need for collective frame dropping. While ensuring low latency and avoiding screen distortion, the number of frame drops is reduced, avoiding the situation where there is no data to send, and improving bandwidth utilization, thereby improving the continuity of video playback and enhancing the user viewing experience.
[0051] On the basis of the above technical solution, after "determining the timed-out video frame in the current video frame queue" in step S120, it may also include: based on a preset delay duration, delaying the current timeout moment corresponding to the timed-out video frame, and updating the current timeout moment.
[0052] Specifically, the current timeout moment corresponding to each timed-out video frame can be delayed by a preset delay duration, and the delayed moment can be used as the updated current timeout moment, so that the timed-out video frame can be changed from a timed-out state to a non-timed-out state, so that the timed-out video frames in the current video frame queue are not discarded as target discarded video frames, that is, when these timed-out video frames still exist in the next video frame queue, by delaying the current timeout moment, whether these video frames are timed-out video frames can be re-determined in the next video frame queue, thereby obtaining the total number of frames that need to be dropped while ensuring low latency, thereby achieving low latency of the video.
[0053] On the basis of the above-mentioned technical solutions, the embodiment of the present disclosure can determine the corresponding discarding priority in real time based on the decoding reference information corresponding to each discarding video frame after determining the discardable video frame in the current video frame queue in thread B. If the discarding priority corresponding to a discarding video frame has been determined when the frame was discarded last time, it is not necessary to re-determine the discarding priority corresponding to the discarding video frame this time. The embodiment of the present disclosure can also uniformly determine the discarding priority corresponding to each video frame in advance based on the decoding reference information corresponding to the encoded video frame before writing the encoded video frame into the video frame queue to be sent in thread A, and obtain the discarding priority corresponding to each discarding video frame in the current video frame queue based on the discarding priority corresponding to each video frame determined in advance.
[0054] The triggering moments of the above two discarding priority determination operations are different, but the determination process is the same. The embodiment of the present disclosure takes the first triggering method as an example to describe in detail the discarding priority determination process. Exemplarily, determining the discarding priority based on the decoded reference information corresponding to the discardable video frame may include: if the discardable video frame is a key frame, then based on a preset key frame priority determination method, determining the discarding priority corresponding to the discardable video frame; if the discardable video frame is a reference frame, then based on a preset reference frame priority determination method, determining the discarding priority corresponding to the discardable video frame; if the discardable video frame is a non-reference frame, then based on a preset non-reference frame priority determination method, determining the discarding priority corresponding to the discardable video frame. Among them, a reference frame may refer to a video frame that is referenced by other video frames when decoding other video frames. A non-reference frame may refer to a video frame that is not referenced by other video frames when decoding other video frames.
[0055] Specifically, each discardable video frame can be detected in turn according to the order in which the discardable video frames are arranged in the queue, whether it is a key frame, a reference frame or a non-reference frame, and the discarding priority corresponding to each discardable video frame can be determined based on the determination method corresponding to the detection result.
[0056] Exemplarily, the discarding priority corresponding to the key frame in the same image group is the highest, the discarding priority corresponding to the reference frame is the second highest, and the discarding priority corresponding to the non-reference frame is the lowest; the discarding priority corresponding to each reference frame in the same image group decreases, and the discarding priority corresponding to each non-reference frame increases; the discarding priority corresponding to the key frames in different image groups increases. For example, for each reference frame in the same image group, the corresponding discarding priority can be decreased in sequence based on the order of arrangement. For each non-reference frame in the same image group, the corresponding discarding priority can be increased in sequence based on the order of arrangement. For key frames in different image groups, the corresponding discarding priority can be increased in sequence based on the order of arrangement of the image groups. Through the above-mentioned setting method of discarding priority, it can be ensured that non-reference frames are discarded first in the same image group GOP, and then reference frames are discarded, and frame loss is made to occur at the end of each image group GOP as much as possible, so that the front video frames in the image group GOP have more opportunities to be sent, avoiding the situation of no data transmission, and trying to avoid the situation of flower screen, thereby improving the user viewing experience.
[0057] Exemplarily, determining the discarding priority corresponding to the discardable video frame based on a preset key frame priority determination method may include: determining the discarding priority corresponding to the discardable video frame based on the current number of image groups corresponding to the discardable video frame, a first preset reference value and a second preset reference value.
[0058] Among them, the current number of picture groups GOPCount may refer to the number of picture groups GOP generated before the discardable video frame (i.e., key frame), that is, the number of key frames generated before the discardable video frame (i.e., key frame). The initial value of the current number of picture groups is 0, and the current number of picture groups can be counted using a counter. The first preset reference value GOPPriority may be pre-set and used to characterize the difference between two discard priorities corresponding to the key frames in two adjacent picture groups GOP. The second preset reference value KeyframePriority may be pre-set and used to characterize the initial value of the discard priority corresponding to the key frame, and the initial value may refer to the discard priority corresponding to the key frame in the first picture group. The first preset reference value GOPPriority and the second preset reference value KeyframePriority can both be determined based on the number of video frames contained in the picture group GOP, and the first preset reference value is greater than the second preset reference value to avoid determining the same discard priority.
[0059] Specifically, the current number of picture groups GOPCount may be multiplied by the first preset reference value GOPPriority, and the multiplication result may be added to the second preset reference value KeyframePriority, and the obtained addition result (i.e., GOPCount×GOPPriority+KeyframePriority) may be used as the discarding priority corresponding to the discardable video frame. When determining the discarding priority corresponding to the next key frame, the current number of picture groups GOPCount may be added by 1.
[0060] Exemplarily, determining the discarding priority corresponding to the discardable video frame based on a preset reference frame priority determination method may include: determining the discarding priority corresponding to the discardable video frame based on the current image group number corresponding to the discardable video frame, the frame sequence number in the image group GOP where the discardable video frame is located, a first preset reference value and a third preset reference value.
[0061] Among them, the current number of picture groups GOPCount may refer to the number of picture groups GOP generated before the picture group GOP where the discardable video frame is located, that is, the current number of picture groups GOPCount corresponding to the key frame in the picture group GOP where the discardable video frame is located. The frame sequence number FrameInGOP in the picture group GOP where the discardable video frame is located can be used to characterize the video frame number in the picture group GOP. The initial value of the frame sequence number may be 0, and the frame sequence number may be counted using a counter. The third preset reference value RefFramePriority may be pre-set and used to characterize the initial value of the discard priority corresponding to the reference frame, and the initial value may refer to the discard priority corresponding to the first reference frame in the first picture group. The third preset reference value may be determined based on the number of reference frames contained in the picture group GOP to avoid determining repeated discard priorities.
[0062] Specifically, the current number of image groups GOPCount and the first preset reference value GOPPriority may be multiplied, and the multiplication result and the third preset reference value RefFramePriority may be added, and the addition result and the frame number FrameInGOP may be subtracted, and the difference (i.e., GOPCount×GOPPriority+RefFramePriority-FrameInGOP) obtained may be used as the discarding priority corresponding to the discardable video frame. When determining the discarding priority corresponding to the next video frame in the GOP, the frame number FrameInGOP may be incremented by 1.
[0063] Exemplarily, determining the discarding priority corresponding to the discardable video frame based on a preset non-reference frame priority determination method may include: determining the discarding priority corresponding to the discardable video frame based on the current image group number corresponding to the discardable video frame, the frame sequence number in the image group GOP where the discardable video frame is located, and a first preset reference value.
[0064] Specifically, the current number of image groups GOPCount and the first preset reference value GOPPriority may be multiplied, and the multiplication result may be added to the frame number FrameInGOP, and the obtained addition result (i.e., GOPCount×GOPPriority+FrameInGOP) may be used as the discarding priority corresponding to the discardable video frame. When determining the discarding priority corresponding to the next video frame in the GOP, the frame number FrameInGOP may be added by 1.
[0065] For example, Figure 2 An example of determining the discard priority is given. Figure 2 As shown, the first video frame is a key frame in the first generated image group. When the first preset reference value GOPPriority is 1000, the second preset reference value KeyframePriority is 999, and the third preset reference value RefFramePriority is 500, the discarding priority corresponding to each video frame in the queue determined by the above method is as follows: Figure 2 By performing frame loss processing based on the determined drop priority, frame loss can be made to occur at the end of each picture group GOP as much as possible, so that the front video frames in the picture group GOP have more opportunities to be sent, avoiding the situation of no data transmission and avoiding the situation of screen distortion as much as possible, thereby improving the user viewing experience.
[0066] Embodiment 2
[0067] Figure 3 This is a flowchart of a frame dropping method provided in Embodiment 2 of the present disclosure. Based on the above embodiments, the present disclosure triggers a frame dropping operation after the next key frame is written into the queue, and describes the frame dropping process in detail. The explanations of the terms that are the same or corresponding to the above embodiments are not repeated here.
[0068] See also Figure 3 The frame loss method provided in the embodiment of the present disclosure specifically includes the following steps:
[0069] S310: Obtain a current video frame queue to be sent.
[0070] S320: Determine the timed-out video frames in the current video frame queue, and determine the total number of current to-be-lost frames corresponding to the current video frame queue based on the number of timed-out video frames and the current number of non-lost frames.
[0071] S330, check whether there is at least one key frame in the current video frame queue, and whether the last key frame is not the first frame of the queue. If so, proceed to step S340; if not, proceed to step S310.
[0072] Specifically, by detecting whether there is at least one key frame in the current video frame queue and whether the last key frame is not the first frame of the queue, it is possible to determine whether the next key frame is written into the current video frame queue, that is, whether the current image group in the current video frame queue has been written, thereby determining whether a subsequent frame drop operation can be triggered. For example, if it is detected that there is no key frame in the current video frame queue, or the only key frame that exists is the first frame of the queue, it is possible to return to execute the operation of step S310 and wait until it is detected that there is at least one key frame in the current video frame queue and the last key frame is not the first frame of the queue, so that the subsequent frame drop operation can be started after the next I frame is written into the queue, thereby avoiding the situation where a distorted screen is caused by frame drop.
[0073] S340: Take each video frame before the last key frame in the current video frame queue as a discardable video frame.
[0074] Specifically, when it is detected that there is at least one key frame in the current video frame queue and the last key frame is not the first frame of the queue, it indicates that there are at least two key frames in the current video frame queue, or when there is only one key frame, the key frame is not the first frame of the queue. At this time, in order to avoid increasing delays and causing screen distortion, each video frame before the last key frame in the current video frame queue can be treated as a discardable video frame for subsequent frame drop operations.
[0075] S350, detecting whether the number of discardable video frames is greater than the total number of frames to be discarded, if so, proceeding to step S360; if not, proceeding to step S370.
[0076] S360: According to the discarding priorities corresponding to the discardable video frames and the total number of frames to be discarded, target discarded video frames are determined from the discardable video frames, and frame discarding processing is performed on the target discarded video frames.
[0077] Specifically, when the number of discardable video frames is greater than the total number of frames to be discarded, based on the discard priority corresponding to each discardable video frame, the total number of frames to be discarded that need to be discarded first is determined from each discardable video frame, and frame discard processing is performed on each target discarded video frame. In this way, while ensuring low latency and avoiding screen distortion, the number of lost frames is reduced, the situation where no data can be sent is avoided, the bandwidth utilization rate is improved, and the continuity of video playback is improved, thereby enhancing the user viewing experience.
[0078] Exemplarily, S360 may include: arranging the discarding order of the discardable video frames based on the discarding priorities corresponding to the discardable video frames, and taking the first discarded discardable video frames in the total number of frames to be discarded as target discarded video frames.
[0079] Specifically, the discarding priorities corresponding to the discardable video frames can be arranged in ascending order from low to high to obtain the discardable video frames whose discarding orders are arranged in sequence, and the discardable video frames before the total number of frames to be discarded after arrangement are used as target discarded video frames for frame discarding. The disclosed embodiment can also remove the discardable video frames with the highest discarding priority from the discardable video frame set to become video frames that do not need to be discarded this time, and update the number of current discardable video frames, and return to perform the operation of S350 based on the updated number of discardable video frames until the number of discardable video frames is equal to the total number of frames to be discarded, thereby screening out the target discarded video frames that need to be discarded this time.
[0080] S370: Take each discardable video frame as a target discarded video frame, and perform frame discarding processing on the target discarded video frame.
[0081] Specifically, when the number of discardable video frames is less than or equal to the total number of frames to be discarded, each discardable video frame in the current video frame queue can be directly used as a target discarded video frame for frame discarding processing to ensure low video latency.
[0082] It should be noted that when the number of discardable video frames is less than the total number of frames to be discarded, it indicates that the current frame loss cannot discard the total number of video frames, that is, the current video frame queue cannot meet the frame loss requirements. At this time, the remaining frames that have not been discarded can be processed as frame drops in the next video frame queue to achieve a low-latency video effect.
[0083] The technical solution of the disclosed embodiment is as follows: when it is detected that there is at least one key frame in the current video frame queue and the last key frame is not the first frame of the queue, each video frame before the last key frame in the current video frame queue is used as a discardable video frame, and based on the number of discardable video frames and the total number of frames to be discarded, a target discarded video frame is determined from each discardable video frame and a frame discard process is performed, so that the frame drop operation is started after the next key frame is written to the queue. In this way, the number of frame drops is reduced while ensuring low latency and avoiding screen distortion, avoiding the situation where no data can be sent, improving bandwidth utilization, and further improving the continuity of video playback, thereby enhancing the user viewing experience.
[0084] Embodiment 3
[0085] Figure 4 This is a flowchart of a frame dropping method provided in Embodiment 3 of the present disclosure. Based on the above disclosed embodiments, the present disclosed embodiment triggers a frame dropping operation before the next key frame is written into the queue, and describes in detail the process of early frame dropping. The explanations of the terms that are the same or corresponding to the above embodiments are not repeated here.
[0086] See also Figure 4 The frame loss method provided in the embodiment of the present disclosure specifically includes the following steps:
[0087] S410: Obtain a current video frame queue to be sent.
[0088] S420: Determine the timed-out video frames in the current video frame queue, and determine the total number of current to-be-lost frames corresponding to the current video frame queue based on the number of timed-out video frames and the current number of non-lost frames.
[0089] S430, detecting whether there is at least one key frame in the current video frame queue, and whether the last key frame is not the first frame of the queue, if so, proceeding to step S440; if not, proceeding to step S480.
[0090] S440: Take each video frame before the last key frame in the current video frame queue as a discardable video frame.
[0091] Specifically, when it is detected that there is at least one key frame in the current video frame queue and the last key frame is not the first frame of the queue, it indicates that the next key frame has been written into the queue. At this time, the frame dropping operation can be performed by executing steps S440-S470.
[0092] S450, detecting whether the number of discardable video frames is greater than the total number of frames to be discarded, if so, proceeding to step S460; if not, proceeding to step S470.
[0093] S460: Determine target discarded video frames from the discarded video frames according to the discarding priorities corresponding to the discarded video frames and the total number of frames to be discarded, and perform frame discarding processing on the target discarded video frames.
[0094] S470: Take each discardable video frame as a target discarded video frame, and perform frame discarding processing on the target discarded video frame.
[0095] S480: Take each video frame in the current video frame queue as a discardable video frame.
[0096] Specifically, in the same video stream, such as the same live stream, the number of video frames in each group of pictures GOP is the same, so the number of remaining video frames that have not been written to the group of pictures GOP can be predicted to extract and drop frames before the next key frame is written into the queue. When dropping frames in advance, all video frames that have not been written in the current group of pictures GOP need to be discarded until the next key frame is written.
[0097] For example, when it is detected that there is no key frame in the current video frame queue, or the only key frame that exists is the first frame of the queue, it indicates that the next key frame has not been written into the queue, that is, the current image group GOP in the current video frame queue has not been written yet. At this time, each video frame in the current video frame queue can be used as a discardable video frame to enable early frame discard.
[0098] S490: Based on the number of video frames included in the picture group, determine the number of remaining video frames that have not been written into the current picture group in the current video frame queue.
[0099] Specifically, the number of video frames included in the group of pictures GOP may be subtracted from the number of video frames already written into the current group of pictures in the current video frame queue, and the obtained difference is determined as the number of remaining video frames not yet written into the current group of pictures.
[0100] It should be noted that this embodiment does not limit the execution order of steps S480 and S490. For example, step S480 may be executed first and then S490, or step S490 may be executed first and then S480.
[0101] S491. If the number of remaining video frames is less than the total number of frames to be discarded, the number of remaining frames to be discarded is determined, and based on the discarding priorities corresponding to the discardable video frames, target discarded video frames corresponding to the remaining number of frames to be discarded are determined from the discardable video frames, and the target discarded video frames are subjected to frame discarding processing, and then enter S492.
[0102] Specifically, by detecting whether the number of remaining video frames that have not been written to the current image group GOP is less than the total number of frames to be discarded, it can be determined whether it is necessary to discard frames in the current video frame queue after discarding the remaining video frames as target discarded video frames, so as to ensure low latency of the video. When the number of remaining video frames is less than the total number of frames to be discarded, it indicates that additional frames need to be discarded in the current video frame queue. At this time, the total number of frames to be discarded can be subtracted from the number of remaining video frames, and the difference obtained is determined as the number of remaining frames to be discarded, that is, the number of frames that need to be discarded in the current video frame queue. Based on the discard priority corresponding to each discardable video frame in the current video frame queue, the remaining number of target discarded frames to be discarded that need to be discarded first is determined from each discardable video frame, and frame discarding processing is performed on each target discarded video frame. The process of determining the target discarded video frame based on the discard priority can refer to the relevant description of the above embodiment, which will not be repeated here.
[0103] It should be noted that if the number of remaining video frames is equal to the total number of frames to be dropped, it indicates that the low-latency effect can be achieved by discarding the remaining video frames. At this time, all the remaining video frames can be discarded as target video frames by executing S492. If the number of remaining video frames is greater than the total number of frames to be dropped, it indicates that there are many remaining video frames that have not been written. At this time, the current frame drop can be stopped, and the next frame drop can be performed by returning to execute S410 to avoid the possible screen distortion caused by the direct frame drop this time, further improving the continuity of video playback.
[0104] S492: Take the remaining video frames corresponding to the current image group as target discarded video frames for frame discarding processing.
[0105] Specifically, by directly determining the remaining video frames that have not yet been written to the current image group as target discarded video frames and performing frame discard processing, early frame discarding is achieved without waiting for the next key frame to be written to the queue, thereby further improving the frame discarding efficiency, and thus improving the continuity of video playback and enhancing the user viewing experience.
[0106] Exemplarily, S492 may include: determining the maximum discarding priority among the target discarded video frames corresponding to the remaining number of frames to be discarded; if the discarding priority corresponding to the newly received video frame is less than the maximum discarding priority, determining that the video frame is the remaining video frame corresponding to the current image group, and prohibiting the video frame from being written into the queue of video frames to be sent.
[0107] Specifically, the remaining video frames that have not been written to the current image group can be determined by labeling the remaining video frames that have not been written or based on the discarding priority, and the remaining video frames can be subjected to frame discarding processing by prohibiting writing to the queue. For example, if the discarding priority corresponding to each video frame is determined before the video frame is written to the queue (such as implemented in thread A), the maximum discarding priority in the target discarded video frame corresponding to the number of remaining frames to be discarded can be determined when the frame is discarded in S491 (such as implemented in thread B). At this time, in thread A, every time a newly encoded video frame is received and the discarding priority of the video frame is determined, it can be detected whether the discarding priority corresponding to the video frame is less than the maximum discarding priority. If so, it indicates that the video frame is the remaining video frame that has not been written to the current image group. At this time, the frame discarding processing can be more conveniently implemented by prohibiting the video frame from being written to the video frame queue to be sent, thereby ensuring low latency of the video. If the discarding priority corresponding to the video frame is greater than the maximum discarding priority, it indicates that the video frame is not a remaining video frame. At this time, the video frame can be written to the video frame queue to be sent for the next frame discarding processing.
[0108] The technical solution of the disclosed embodiment is as follows: when it is detected that there is no key frame in the current video frame queue, or that only one key frame exists as the first frame of the queue, each video frame in the current video frame queue is used as a discardable video frame, and when the number of remaining video frames that have not been written into the current image group is less than the total number of frames to be discarded, based on the discarding priority corresponding to the discarded video frames, the target discarded video frames corresponding to the remaining number of frames to be discarded are determined from the discarded video frames, and the target discarded video frames are subjected to frame discarding processing, and the remaining video frames corresponding to the current image group are used as target discarded video frames for frame discarding processing, thereby realizing early frame discarding of video frames without waiting for the next key frame to be written into the queue, further improving the frame dropping efficiency, thereby also improving the continuity of video playback, and enhancing the user viewing experience.
[0109] The following is an embodiment of a frame dropping device provided in an embodiment of the present disclosure. The device and the frame dropping method of the above embodiment belong to the same inventive concept. For details not described in detail in the embodiment of the frame dropping device, reference can be made to the frame dropping method of the above embodiment.
[0110] Embodiment 4
[0111] Figure 5 This is a schematic diagram of the structure of a frame dropping device provided in the fourth embodiment of the present disclosure. This embodiment can be applied to the situation where frame dropping is performed during video transmission to ensure low latency, especially in the live broadcast scene where frame dropping is performed on the live video to be transmitted. Figure 5As shown, the device specifically includes: a current video frame queue acquisition module 510, a current total number of frames to be lost determination module 520, a discardable video frame determination module 530 and a frame loss processing module 540.
[0112] Among them, the current video frame queue acquisition module 510 is used to obtain the current video frame queue to be sent; the current total number of frames to be lost determination module 520 is used to determine the timed video frames in the current video frame queue, and based on the number of timed video frames and the current number of non-lost frames, determine the current total number of frames to be lost corresponding to the current video frame queue; the discardable video frame determination module 530 is used to determine the discardable video frames in the current video frame queue; the frame loss processing module 540 is used to determine the target discarded video frame from the discardable video frames based on the current total number of frames to be lost and the discard priority corresponding to the discarded video frames, and perform frame loss processing on the target discarded video frames, wherein the discard priority is determined based on the decoding reference information corresponding to the discarded video frame.
[0113] The technical solution of the disclosed embodiment does not directly drop frames after determining the timed-out video frames in the current video frame queue to be sent, but determines the total number of frames to be dropped corresponding to the current video frame queue based on the number of timed-out video frames and the current number of non-dropped frames, that is, the total number of frames that need to be dropped to ensure low latency. By determining the discardable video frames in the current video frame queue, and based on the discarding priority and the current total number of frames to be dropped determined based on the decoding reference information corresponding to the discardable video frames, the target discarded video frames to be discarded are determined from each discardable video frame without collective frame dropping, thereby reducing the number of dropped frames while ensuring low latency and avoiding screen distortion, avoiding the situation where there is no data to be sent, thereby improving bandwidth utilization and the continuity of video playback, and enhancing the user viewing experience.
[0114] On the basis of the above technical solution, the device also includes:
[0115] The priority determination module is specifically used to: if the discardable video frame is a key frame, determine the discarding priority corresponding to the discardable video frame based on a preset key frame priority determination method; if the discardable video frame is a reference frame, determine the discarding priority corresponding to the discardable video frame based on a preset reference frame priority determination method, wherein the reference frame refers to a video frame that is referenced by other video frames when decoding other video frames; if the discardable video frame is a non-reference frame, determine the discarding priority corresponding to the discardable video frame based on a preset non-reference frame priority determination method, wherein the non-reference frame refers to a video frame that is not referenced by other video frames when decoding other video frames.
[0116] Based on the above technical solutions, the discarding priority corresponding to the key frames in the same image group is the highest, the discarding priority corresponding to the reference frames is the second highest, and the discarding priority corresponding to the non-reference frames is the lowest; the discarding priority corresponding to each reference frame in the same image group decreases, and the discarding priority corresponding to each non-reference frame increases; the discarding priority corresponding to the key frames in different image groups increases.
[0117] On the basis of the above technical solutions, the priority determination module is also specifically used for:
[0118] Based on the current number of image groups corresponding to the discardable video frame, the first preset reference value and the second preset reference value, the discarding priority corresponding to the discardable video frame is determined; wherein the first preset reference value is used to represent the difference between two discarding priorities corresponding to key frames in two adjacent image groups; the second preset reference value is used to represent the initial value of the discarding priority corresponding to the key frame.
[0119] On the basis of the above technical solutions, the video frame determination module 530 may be discarded, which is specifically used for:
[0120] If it is detected that there is at least one key frame in the current video frame queue, and the last key frame is not the first frame of the queue, each video frame before the last key frame in the current video frame queue is used as a discardable video frame.
[0121] Based on the above technical solutions, the frame loss processing module 540 includes:
[0122] A first determining unit is configured to determine a target discarded video frame from the discarded video frames according to a discarding priority corresponding to the discarded video frames and the current total number of frames to be discarded if it is detected that the number of discarded video frames is greater than the current total number of frames to be discarded;
[0123] The second determining unit is configured to use each discardable video frame as a target discarded video frame if it is detected that the number of discardable video frames is less than or equal to the total number of frames currently to be discarded.
[0124] On the basis of the above technical solutions, the first determining unit is specifically used to:
[0125] Based on the discarding priorities corresponding to the discardable video frames, the discarding order of the discardable video frames is arranged, and each of the discardable video frames that is discarded first and the total number of frames to be discarded is used as the target discarded video frame.
[0126] On the basis of the above technical solutions, the discardable video frame determination module 530 is further specifically used for:
[0127] If it is detected that there is no key frame in the current video frame queue, or the only key frame that exists is the first frame of the queue, each video frame in the current video frame queue is used as a discardable video frame.
[0128] Based on the above technical solutions, the frame loss processing module 540 further includes:
[0129] A remaining video frame number determining unit, used to determine the number of remaining video frames that have not been written into the current image group in the current video frame queue based on the number of video frames included in the image group;
[0130] A first frame dropping unit is configured to determine the number of remaining frames to be dropped if the number of remaining video frames is less than the total number of frames to be dropped, and determine target dropped video frames corresponding to the number of remaining frames to be dropped from the discardable video frames based on the drop priorities corresponding to the discardable video frames, and perform frame dropping processing on the target dropped video frames;
[0131] The second frame dropping unit is used to take the remaining video frames corresponding to the current image group as target dropped video frames for frame dropping processing.
[0132] On the basis of the above technical solutions, the second frame dropping unit is specifically used for:
[0133] Determine the maximum discarding priority among the target discarded video frames corresponding to the remaining number of frames to be discarded; if the discarding priority corresponding to the newly received video frame is less than the maximum discarding priority, determine that the video frame is the remaining video frame corresponding to the current image group, and prohibit the video frame from being written into the queue of video frames to be sent.
[0134] On the basis of the above technical solutions, the device also includes:
[0135] The delay module is used to, after determining the timed-out video frame in the current video frame queue, delay the current timeout moment corresponding to the timed-out video frame based on a preset delay duration to update the current timeout moment.
[0136] On the basis of the above technical solutions, the device also includes:
[0137] The current non-lost frame number updating module is used to determine and update the current non-lost frame number based on the current total number of frames to be lost and the number of target discarded video frames after the target discarded video frames are subjected to frame loss processing.
[0138] The frame dropping device provided in the embodiments of the present disclosure can execute the frame dropping method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects for executing the frame dropping method.
[0139] It is worth noting that in the embodiment of the above-mentioned frame loss device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.
[0140] Embodiment 5
[0141] Reference below Figure 6 , which shows a structural diagram of a server 900 suitable for implementing an embodiment of the present disclosure. Figure 6 The server shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.
[0142] like Figure 6 As shown, the server 900 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage device 908 to a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the server 900 are also stored. The processing device 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.
[0143] Typically, the following devices may be connected to the I / O interface 905: input devices 906 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 907 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 908 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 909. The communication devices 909 may allow the server 900 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 6 The server 900 is shown with various devices, but it should be understood that it is not required to implement or have all the devices shown. More or fewer devices may be implemented or have instead.
[0144] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 909, or installed from a storage device 908, or installed from a ROM 902. When the computer program is executed by the processing device 901, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.
[0145] The server provided in the embodiment of the present disclosure and the frame loss method provided in the above embodiment belong to the same inventive concept. The technical details not fully described in the embodiment of the present disclosure can be referred to the above embodiment, and the embodiment of the present disclosure has the same beneficial effects as the above embodiment.
[0146] Embodiment 6
[0147] The embodiment of the present disclosure provides a computer storage medium on which a computer program is stored. When the program is executed by a processor, the frame dropping method provided in the above embodiment is implemented.
[0148] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.
[0149] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.
[0150] The computer-readable medium may be included in the server, or may exist independently without being installed in the server.
[0151] The above-mentioned computer-readable medium carries one or more programs. When the above-mentioned one or more programs are executed by the server, the server is enabled to: obtain the current video frame queue to be sent; determine the timed-out video frames in the current video frame queue, and based on the number of the timed-out video frames and the current number of non-lost frames, determine the current total number of frames to be lost corresponding to the current video frame queue; determine the discardable video frames in the current video frame queue; based on the current total number of frames to be lost and the discarding priority corresponding to the discarding video frames, determine the target discarding video frames from the discarding video frames, and perform frame discarding processing on the target discarding video frames, wherein the discarding priority is determined based on the decoding reference information corresponding to the discarding video frames.
[0152] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).
[0153] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0154] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware. The name of a unit does not limit the unit itself in some cases. For example, an editable content display unit may also be described as an "editing unit".
[0155] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.
[0156] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0157] According to one or more embodiments of the present disclosure, [Example 1] provides a frame dropping method, including:
[0158] Get the current video frame queue to be sent;
[0159] Determine the timed-out video frames in the current video frame queue, and determine the total number of current to-be-lost frames corresponding to the current video frame queue based on the number of the timed-out video frames and the current number of unlost frames;
[0160] Determining discardable video frames in the current video frame queue;
[0161] Based on the current total number of frames to be discarded and the discarding priority corresponding to the discardable video frames, a target discarded video frame is determined from the discardable video frames, and the target discarded video frame is subjected to frame discarding processing, wherein the discarding priority is determined based on the decoding reference information corresponding to the discarded video frame.
[0162] According to one or more embodiments of the present disclosure, [Example 2] provides a frame dropping method, further comprising:
[0163] Optionally, determining the discarding priority based on the decoding reference information corresponding to the discardable video frame includes:
[0164] If the discardable video frame is a key frame, determining a discarding priority corresponding to the discardable video frame based on a preset key frame priority determination method;
[0165] If the discardable video frame is a reference frame, determining a discarding priority corresponding to the discardable video frame based on a preset reference frame priority determination method, wherein the reference frame refers to a video frame that is referenced by other video frames when decoding other video frames;
[0166] If the discardable video frame is a non-reference frame, the discarding priority corresponding to the discardable video frame is determined based on a preset non-reference frame priority determination method, wherein the non-reference frame refers to a video frame that is not referenced by other video frames when other video frames are decoded.
[0167] According to one or more embodiments of the present disclosure, [Example 3] provides a frame dropping method, further comprising:
[0168] Optionally, the key frames in the same image group have the highest discarding priority, the reference frames have the second highest discarding priority, and the non-reference frames have the lowest discarding priority;
[0169] The discarding priority corresponding to each reference frame in the same picture group decreases, and the discarding priority corresponding to each non-reference frame increases;
[0170] The discarding priorities corresponding to key frames in different image groups are increased.
[0171] According to one or more embodiments of the present disclosure, [Example 4] provides a frame dropping method, further comprising:
[0172] Optionally, determining the discarding priority corresponding to the discardable video frame based on a preset key frame priority determination method includes:
[0173] Determining a discarding priority corresponding to the discardable video frame based on the current number of picture groups corresponding to the discardable video frame, a first preset reference value, and a second preset reference value;
[0174] The first preset reference value is used to represent the difference between two discarding priorities corresponding to key frames in two adjacent image groups; and the second preset reference value is used to represent the initial value of the discarding priority corresponding to the key frame.
[0175] According to one or more embodiments of the present disclosure, [Example 5] provides a frame dropping method, further comprising:
[0176] Optionally, the determining the discardable video frames in the current video frame queue includes:
[0177] If it is detected that there is at least one key frame in the current video frame queue and the last key frame is not the first frame of the queue, each video frame before the last key frame in the current video frame queue is used as a discardable video frame.
[0178] According to one or more embodiments of the present disclosure, [Example 6] provides a frame dropping method, further comprising:
[0179] Optionally, the determining a target discarded video frame from the discardable video frames based on the current total number of frames to be discarded and the discarding priorities corresponding to the discardable video frames includes:
[0180] If it is detected that the number of the discardable video frames is greater than the total number of frames to be discarded, determining target discarded video frames from the discardable video frames according to the discarding priorities corresponding to the discardable video frames and the total number of frames to be discarded;
[0181] If it is detected that the number of the discardable video frames is less than or equal to the total number of frames to be discarded, each of the discardable video frames is used as a target discarded video frame.
[0182] According to one or more embodiments of the present disclosure, [Example 7] provides a frame dropping method, further comprising:
[0183] Optionally, determining a target discarded video frame from the discardable video frames according to the discarding priority corresponding to the discardable video frame and the total number of frames to be discarded currently includes:
[0184] The discarding order of the discardable video frames is arranged based on the discarding priorities corresponding to the discardable video frames, and each of the discardable video frames of the total number of the current frames to be discarded that are discarded first is used as the target discarded video frame.
[0185] According to one or more embodiments of the present disclosure, [Example 8] provides a frame dropping method, further comprising:
[0186] Optionally, the determining the discardable video frames in the current video frame queue includes:
[0187] If it is detected that there is no key frame in the current video frame queue, or the only key frame that exists is the first frame of the queue, each video frame in the current video frame queue is used as a discardable video frame.
[0188] According to one or more embodiments of the present disclosure, [Example 9] provides a frame dropping method, further comprising:
[0189] Optionally, the determining a target discarded video frame from the discardable video frames based on the current total number of frames to be discarded and the discarding priorities corresponding to the discardable video frames includes:
[0190] Based on the number of video frames included in the group of pictures, determining the number of remaining video frames that have not been written into the current group of pictures in the current video frame queue;
[0191] If the number of the remaining video frames is less than the total number of frames to be discarded, the number of remaining frames to be discarded is determined, and based on the discarding priorities corresponding to the discardable video frames, target discarded video frames corresponding to the remaining number of frames to be discarded are determined from the discardable video frames, and the target discarded video frames are subjected to frame discarding processing;
[0192] The remaining video frames corresponding to the current image group are used as target discarded video frames for frame discarding processing.
[0193] According to one or more embodiments of the present disclosure, [Example 10] provides a frame dropping method, further comprising:
[0194] Optionally, the taking the remaining video frames corresponding to the current group of images as target discarded video frames for frame discarding processing includes:
[0195] Determine the maximum discarding priority among the target discarded video frames corresponding to the remaining number of frames to be discarded;
[0196] If the discarding priority corresponding to the newly received video frame is less than the maximum discarding priority, the video frame is determined to be a remaining video frame corresponding to the current picture group, and the video frame is prohibited from being written into the queue of video frames to be sent.
[0197] According to one or more embodiments of the present disclosure, [Example 11] provides a frame dropping method, further comprising:
[0198] Optionally, after determining the timed-out video frame in the current video frame queue, the method further includes:
[0199] Based on the preset delay time, the current timeout moment corresponding to the timeout video frame is delayed to update the current timeout moment.
[0200] According to one or more embodiments of the present disclosure, [Example 12] provides a frame dropping method, further comprising:
[0201] Optionally, after performing frame discarding processing on the target discarded video frame, the method further includes:
[0202] Based on the current total number of frames to be dropped and the target number of dropped video frames, the current number of non-dropped frames is determined and updated.
[0203] According to one or more embodiments of the present disclosure, [Example 13] provides a frame dropping device, including:
[0204] The current video frame queue acquisition module is used to acquire the current video frame queue to be sent;
[0205] A current total number of frames to be lost determining module, used to determine the timed-out video frames in the current video frame queue, and based on the number of the timed-out video frames and the current number of non-lost frames, determine the current total number of frames to be lost corresponding to the current video frame queue;
[0206] A discardable video frame determination module, used to determine the discardable video frames in the current video frame queue;
[0207] A frame loss processing module is used to determine a target discarded video frame from the discarded video frames based on the current total number of frames to be lost and the discarding priority corresponding to the discarded video frames, and perform frame loss processing on the target discarded video frames, wherein the discarding priority is determined based on the decoding reference information corresponding to the discarded video frames.
[0208] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.
[0209] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.
[0210] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.
Claims
1. A frame dropping method, characterized in that: include: Get the current video frame queue to be sent; Determine the timed-out video frames in the current video frame queue, and determine the total number of current to-be-lost frames corresponding to the current video frame queue based on the number of the timed-out video frames and the current number of unlost frames; Determining discardable video frames in the current video frame queue; Based on the current total number of frames to be discarded and the discarding priority corresponding to the discardable video frame, a target discarded video frame is determined from the discarded video frames, and the target discarded video frame is subjected to frame discarding processing, wherein the discarding priority is determined based on decoding reference information corresponding to the discarded video frame, and the decoding reference information is information on whether the discarded video frame needs to be referenced when decoding other video frames.
2. The method according to claim 1, characterized in that Determining the discarding priority based on the decoding reference information corresponding to the discardable video frame includes: If the discardable video frame is a key frame, determining a discarding priority corresponding to the discardable video frame based on a preset key frame priority determination method; If the discardable video frame is a reference frame, determining a discarding priority corresponding to the discardable video frame based on a preset reference frame priority determination method, wherein the reference frame refers to a video frame that is referenced by other video frames when decoding other video frames; If the discardable video frame is a non-reference frame, the discarding priority corresponding to the discardable video frame is determined based on a preset non-reference frame priority determination method, wherein the non-reference frame refers to a video frame that is not referenced by other video frames when other video frames are decoded.
3. The method according to claim 2, characterized in that The key frames in the same image group have the highest discarding priority, the reference frames have the second highest discarding priority, and the non-reference frames have the lowest discarding priority. The discarding priority corresponding to each reference frame in the same picture group decreases, and the discarding priority corresponding to each non-reference frame increases; The discarding priorities corresponding to key frames in different image groups are increased.
4. The method according to claim 2, characterized in that: The determining the discarding priority corresponding to the discardable video frame based on the preset key frame priority determination method includes: Determining a discarding priority corresponding to the discardable video frame based on the current number of picture groups corresponding to the discardable video frame, a first preset reference value, and a second preset reference value; The first preset reference value is used to represent the difference between two discarding priorities corresponding to key frames in two adjacent image groups; and the second preset reference value is used to represent the initial value of the discarding priority corresponding to the key frame.
5. The method according to claim 1, characterized in that The determining of the discardable video frames in the current video frame queue comprises: If it is detected that there is at least one key frame in the current video frame queue and the last key frame is not the first frame of the queue, each video frame before the last key frame in the current video frame queue is used as a discardable video frame.
6. The method according to claim 5, characterized in that The step of determining a target discarded video frame from the discardable video frames based on the current total number of frames to be discarded and the discarding priorities corresponding to the discardable video frames includes: If it is detected that the number of the discardable video frames is greater than the total number of frames to be discarded, determining target discarded video frames from the discardable video frames according to the discarding priorities corresponding to the discardable video frames and the total number of frames to be discarded; If it is detected that the number of the discardable video frames is less than or equal to the total number of frames currently to be discarded, each of the discardable video frames is used as a target discarded video frame.
7. The method according to claim 6, characterized in that The step of determining a target discarded video frame from the discarded video frames according to the discarding priorities corresponding to the discarded video frames and the total number of frames to be discarded, comprises: The discarding order of the discardable video frames is arranged based on the discarding priorities corresponding to the discardable video frames, and each of the discardable video frames of the total number of the current frames to be discarded that are discarded first is used as the target discarded video frame.
8. The method according to claim 1, characterized in that The determining of the discardable video frames in the current video frame queue comprises: If it is detected that there is no key frame in the current video frame queue, or the only key frame that exists is the first frame of the queue, each video frame in the current video frame queue is used as a discardable video frame.
9. The method according to claim 8, characterized in that The step of determining a target discarded video frame from the discardable video frames based on the current total number of frames to be discarded and the discarding priorities corresponding to the discardable video frames includes: Based on the number of video frames included in the group of pictures, determining the number of remaining video frames that have not been written into the current group of pictures in the current video frame queue; If the number of the remaining video frames is less than the total number of frames to be discarded, the number of remaining frames to be discarded is determined, and based on the discarding priorities corresponding to the discardable video frames, target discarded video frames corresponding to the remaining number of frames to be discarded are determined from the discardable video frames, and the target discarded video frames are subjected to frame discarding processing; The remaining video frames corresponding to the current image group are used as target discarded video frames for frame discarding processing.
10. The method according to claim 9, characterized in that The step of taking the remaining video frames corresponding to the current group of images as target discarded video frames for frame discarding processing includes: Determine the maximum discarding priority among the target discarded video frames corresponding to the remaining number of frames to be discarded; If the discarding priority corresponding to the newly received video frame is less than the maximum discarding priority, the video frame is determined to be a remaining video frame corresponding to the current picture group, and the video frame is prohibited from being written into the queue of video frames to be sent.
11. The method according to claim 1, characterized in that: After determining the timeout video frame in the current video frame queue, the method further includes: Based on the preset delay duration, the current timeout moment corresponding to the timeout video frame is delayed to update the current timeout moment.
12. The method according to any one of claims 1 to 11, characterized in that: After the target discarded video frame is subjected to frame discarding processing, the method further includes: Based on the current total number of frames to be dropped and the target number of dropped video frames, the current number of non-dropped frames is determined and updated.
13. A frame dropping device, characterized in that: include: The current video frame queue acquisition module is used to acquire the current video frame queue to be sent; A current total number of frames to be lost determining module, used to determine the timed-out video frames in the current video frame queue, and based on the number of the timed-out video frames and the current number of non-lost frames, determine the current total number of frames to be lost corresponding to the current video frame queue; A discardable video frame determination module, used to determine the discardable video frames in the current video frame queue; A frame loss processing module is used to determine a target discarded video frame from the discarded video frames based on the current total number of frames to be discarded and the discarding priority corresponding to the discarded video frames, and perform frame loss processing on the target discarded video frames, wherein the discarding priority is determined based on decoding reference information corresponding to the discarded video frames, and the decoding reference information is information on whether the discarded video frames need to be referenced when decoding other video frames.
14. A server, characterized in that: The server comprises: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the frame dropping method as described in any one of claims 1-12.
15. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the frame dropping method as described in any one of claims 1 to 12 is implemented.
Citation Information
Cited By
Frame dropping method, server and medium
US12739477B2
