Video frame processing method, electronic device and storage medium
By determining the packet loss position and transmission duration in video frame processing, and combining RTT to determine the frame loss or retransmission operation, the video playback delay problem is solved and the video fluency is improved.
Patent Information
- Application Number
- CN202210966572.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-08-12
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2042-08-12
AI Technical Summary
The prior art requires waiting for the next key frame to arrive after packet loss confirmation during the video frame transmission, resulting in a long delay in video playback.
By determining the position of the lost packet in the picture group GOP and the transmission time of the corresponding video frame to the end of the GOP frame, combined with the round trip delay RTT, it is determined whether to perform a frame drop operation or a packet drop retransmission operation.
Reduces real-time video delay and lag, and improves video fluency.
Smart Images

Figure CN115473875B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of video frame processing, and in particular to a video frame processing method, electronic device and storage medium. Background Art
[0002] In the prior art, when sending and receiving real-time video frames, after confirming packet loss, the receiving end will send a FIR (Full Intra Request, key frame retransmission request) request packet to request the key frame in real time, and clear the buffer of the current video frame until the next key frame arrives. It can be seen that in the prior art, the process includes packet loss confirmation, the receiving end sending the FIR request packet to the sending end, and the receiving end waiting for the next key frame to come. That is to say, the processing after confirming packet loss in the prior art requires the FIR request packet and waiting for the next key frame. The process takes a long time, resulting in a long delay in video playback. Summary of the invention
[0003] The present application provides a video frame processing method, electronic device and storage medium to solve the problem in the prior art that after confirming packet loss, an FIR request packet and waiting for the next key frame are required, which takes a long time and causes a long delay in video playback.
[0004] In a first aspect, the present application provides a method for processing video frames, comprising: in a case where it is determined that a group of pictures (GOP) in a video stream has packet loss, determining a position of the lost data packet in the group of pictures (GOP); determining a first duration based on the position of the lost data packet in the GOP, wherein the first duration is the transmission duration from the video frame corresponding to the lost data packet to the end of the frame of the GOP; and performing a frame loss operation or a packet loss retransmission operation on the video frame based on a comparison result of a round-trip delay (RTT) and the first duration.
[0005] In a second aspect, the present application provides a video frame processing device, comprising: a first determination module, used to determine the position of the lost data packet in the picture group GOP when it is determined that packet loss occurs in the GOP in the video stream; a second determination module, used to determine a first duration based on the position of the lost data packet in the GOP, wherein the first duration is the transmission duration from the video frame corresponding to the lost data packet to the end of the frame of the GOP; a first processing module, used to perform a frame loss operation or a packet loss retransmission operation on the video frame based on a comparison result of the round-trip delay RTT and the first duration.
[0006] In a third aspect, an electronic device is provided, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;
[0007] Memory, used to store computer programs;
[0008] The processor is used to implement the method steps described in any embodiment of the first aspect when executing the program stored in the memory.
[0009] In a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method steps described in any embodiment of the first aspect are implemented.
[0010] The above technical solution provided by the embodiment of the present application has the following advantages compared with the prior art:
[0011] The method provided by the embodiment of the present application can determine whether to perform a frame loss operation or a packet loss retransmission operation next based on the comparison result between the transmission time from the video frame corresponding to the lost data packet to the frame end of the GOP and the RTT when it is determined that packet loss occurs in the picture group GOP in the video stream. Compared with the prior art method in which an FIR request packet is required after confirming packet loss and waiting for the next key frame, which is a time-consuming process, real-time video delay and jamming are reduced and real-time video fluency is improved. BRIEF DESCRIPTION OF THE DRAWINGS
[0012] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.
[0013] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.
[0014] Figure 1 One of the flowcharts of a method for processing a video frame provided in an embodiment of the present application;
[0015] Figure 2 It is a schematic diagram of the processing flow after packet loss occurs in the prior art;
[0016] Figure 3 A second flowchart of a method for processing a video frame provided in an embodiment of the present application;
[0017] Figure 4 A flowchart of a method for reducing video delay provided in an embodiment of the present application;
[0018] Figure 5 A schematic diagram of the structure of a video frame processing device provided in an embodiment of the present application;
[0019] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0020] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.
[0021] The video frames encoded by the video sending device can be key frames or non-key frames. A picture group GOP is a group of continuous pictures, starting with a key frame and ending with the next key frame, with several non-key frames between two key frames. A key frame records a complete frame of video, and a non-key frame records the different parts of one frame of video from another frame of video. If a video frame is a non-key frame, the video playback device needs to decode the video based on the reference frame of the video to obtain a complete video picture. A non-key frame can reference a key frame as a reference frame, or it can reference another non-key frame as a reference frame. The number of non-key frames only affects the smoothness of the picture, and does not affect the image restoration and clarity. Based on this, the present application provides the following video frame processing method.
[0022] Figure 1 A flowchart of a method for processing a video frame provided in an embodiment of the present application is shown as follows: Figure 1 As shown, the method steps of the embodiment of the present application include:
[0023] Step 102, when it is determined that a group of pictures GOP in the video stream has packet loss, determine the position of the lost data packet in the GOP;
[0024] It should be noted that the position of the data packet in the GOP refers to further determining the position of the data packet in the current video frame after determining the position of the video frame where the data packet is located in the GOP.
[0025] Step 104, determining a first duration based on the position of the lost data packet in the GOP, wherein the first duration is a transmission duration from a video frame corresponding to the lost data packet to the end of a frame of the GOP;
[0026] Step 106: Based on the comparison result of the round-trip delay RTT and the first duration, a frame dropping operation or a packet dropping retransmission operation is performed on the video frame.
[0027] Through the above steps 102 to 106, when it is determined that packet loss occurs in the group of pictures GOP in the video stream, it can be determined whether to perform a frame loss operation or a packet loss retransmission operation next based on the comparison result between the transmission time from the video frame corresponding to the lost data packet to the end of the frame of the GOP and the RTT.
[0028] The existing technology such as Figure 2 As shown in the figure, the processing flow after packet loss includes:
[0029] Step 201, confirming that packet loss has occurred;
[0030] Step 202, determine whether a data packet is received within 20ms, if received, execute step 206, if not received, execute step 203;
[0031] Step 203, sending a NACK packet loss request;
[0032] Step 204, determining whether the number of times of sending NACK packet loss requests exceeds a preset threshold; if not, executing step 206; if exceeded, executing step 205;
[0033] Step 205, clear the buffer data and send a FIR key frame request;
[0034] Step 206, group video frames for playback.
[0035] It can be seen from the above steps 201 to 206 that in the prior art, when packet loss occurs, if no data packet is received within 20ms, a NACK packet loss request is sent. If the corresponding data packet is still not received after sending the NACK packet loss request, a NACK packet loss request is sent at a fixed time interval. If the number of NACK requests sent exceeds the preset number, the buffered data is cleared and a FIR key frame request is sent. The time it takes for the terminal to send the FIR key frame request and receive the key frame depends on the RTT (Round Trip Time). When the RTT is large, it will cause a large video delay.
[0036] Compared with the prior art, the present application takes into account that discarding some non-key frames only affects the smoothness of the video. Therefore, based on the comparison result between the transmission time from the video frame corresponding to the lost data packet to the frame end of the GOP and the RTT, it is determined whether to perform a frame drop operation or a packet drop retransmission operation next. This can reduce the real-time video delay and freeze and improve the real-time video smoothness to a certain extent.
[0037] In an optional implementation of the embodiment of the present application, the method for determining the position of the lost data packet in the GOP involved in the above step 102 may further include:
[0038] Step 11, based on the second target field in the extended packet of the real-time transport protocol (RTP) packet, determine the position of the lost data packet in the GOP, wherein the second target field includes the position information of the video frame where the data packet is located in the GOP.
[0039] The position of the lost data packet in the GOP can be further determined after the position of the video frame where the data packet is located in the GOP is determined, and then the position of the data packet in the current video frame is determined.
[0040] It should be noted that a video frame is composed of multiple data packets, which may be RTP packets in this application, and RTP includes RTP extension packets, which may include multiple fields, which are used to characterize the extension type of the extension packet, the total number of frames in the current GOP, the position of the current video frame in the GOP, the total number of data packets in the current video frame, and the position of the data packet in the current frame. Based on this, in a specific example, the fields in the extension packet in a specific application scenario may be as shown in Table 1:
[0041]
[0042] Table 1
[0043] Among them, Defined by Profile occupies 16 bits and is defined as 0x50, RTP 8-byte extension, with a length of 16 bits, which is the number of extension elements behind it, and is 1 in Table 1.
[0044] ID: The extension identifier occupies 8 bits and has a value of 0x01, indicating the extension type;
[0045] Gop-count occupies 8 bits and is the total number of frames in the current GOP;
[0046] Frame-index occupies 8 bits and is the position of the current video frame in the GOP;
[0047] Pack-count occupies 8 bits and is the total number of packets in the current video frame.
[0048] It can be seen that the second target field in the embodiment of the present application is Frame-index, which is used to characterize the position of the current video frame in the GOP. In other implementations of the embodiment of the present application, the position of the current video frame in the GOP can also be determined by the total GOP length and its video frame in the terminal history record, and the position of the data packet in the current frame can be determined by the sequence number of the data packet combined with Pack-count. In addition, the RTP extension packet itself also has a start packet and an end packet that mark a frame.
[0049] In another optional implementation of the embodiment of the present application, before determining the position of the lost data packet in the GOP, that is, before Figure 1 Based on this, the method steps of the embodiment of the present application are as follows Figure 3 As shown, it may also include:
[0050] Step 302, when the sequence numbers of the data packets are discontinuous, determining that a packet loss occurs in a GOP in the video stream;
[0051] Step 304, when it is determined that a packet loss occurs in the GOP and the RTT is less than a preset duration, a packet loss retransmission operation is performed;
[0052] Step 306 (Step 102), when it is determined that packet loss occurs in the GOP and the RTT is greater than or equal to the preset duration, an operation of determining the position of the lost data packet in the GOP is performed.
[0053] It can be seen from the above steps 302 to 306 that since each data packet in the RTP packet has a corresponding sequence number, and the sequence numbers of the data packets in the RTP packet are continuously increased based on the sorting position, that is, the sequence numbers of the data packets in the RTP packet are regularly set. Therefore, whether packet loss occurs can be determined by whether the sequence numbers are continuous. For example, the sequence numbers in the data packets are numbered with numbers. If there are 1000 data packets in the current RTP, they are numbered from 1 to 1000. If it is found that the sequence number 100 does not exist, it indicates that the data packet corresponding to the sequence number is lost, that is, packet loss occurs in the GOP in the video stream.
[0054] In addition, for the above Figure 3 In the specific example of the method steps, taking the preset duration of 100ms as an example, if the RTT is less than 100ms, it means that the round-trip time for sending the request to execute the packet loss retransmission operation is short, that is, the packet loss retransmission operation can be performed directly without considering whether other operations need to be performed, which can also reduce the video delay to a certain extent. Of course, if the RTT is greater than 100ms, it means that the round-trip time for sending the request to execute the packet loss retransmission operation is long. At this time, it is necessary to further determine the location of the lost data packet, and determine whether to execute the frame loss operation or the packet loss retransmission operation based on the location. It should be noted that the 100ms is preset according to needs, and in other application scenarios it can also be 80ms, 90ms, 110ms, etc., that is, it can be set accordingly according to actual needs, and this is not limited in the present application.
[0055] Based on the above-mentioned extension packet and each field in the extension packet, in another optional implementation manner of the embodiment of the present application, the method for determining the first duration based on the position of the lost data packet in the GOP involved in the above-mentioned step 104 may further include:
[0056] Step 21, determining a unit transmission time required to transmit one video frame based on the total number of video frames in the GOP;
[0057] Step 22, determining the number of video frames from the current video frame where the lost data packet is located to the end of the frame in the GOP as the target number;
[0058] Step 23: determine the product of the unit transmission duration and the target number as the first duration.
[0059] For the above steps 21 to 23, since the extension packet includes fields that can characterize the total number of frames of the current GOP, the position of the current video frame in the GOP, and the total number of data packets of the current video frame, the position of the video frame where the currently lost data packet is located in the GOP can be determined from the extension packet. In this regard, in the embodiment of the present application, the GOP takes 30 frames as an example, and the average time interval of one frame in the 30-frame GOP is 1000 / 30≈33.3ms, that is, the transmission time of one video frame is 33.3ms. If the video corresponding to the lost data packet is the 28th frame of the current GOP, the first time length is the transmission time from the 28th frame to the end of the frame (the 30th frame) is 33.3*3=99.9ms, and if the video corresponding to the currently lost data packet is the 25th frame of the current GOP, the first time length is the transmission time from the 25th frame to the end of the frame (the 30th frame) is 33.3*6=166.5ms.
[0060] Based on this, as to whether to perform a frame drop operation or a packet drop retransmission operation, in an optional implementation manner of an embodiment of the present application, for the comparison result of the round-trip delay RTT and the first duration involved in the above step 106, the frame drop operation or the packet drop retransmission operation on the video frame may further include:
[0061] Step 31, when the first duration is less than the RTT, performing a frame drop operation;
[0062] Step 32: When the first time duration is greater than or equal to the RTT, a packet loss retransmission operation is performed.
[0063] For step 31 and step 32, in a specific example, since the transmission time of each frame is 33.3ms, if the RTT is 100ms, the transmission time of 3 frames is 99.9ms, and when the RTT is 100ms, the maximum number of lost frames is 3. If the RTT is 167ms, the maximum number of lost frames is 5 (33.3*5=166.5ms). If the RTT is 200ms, the maximum number of lost frames is 6. That is to say, when the RTT is 100ms, if the frame where the lost data packet is located (set as the Nth frame) falls between the 28th and 30th frames of the GOP, the frame drop operation is performed, that is, the Nth to 30th frames are dropped; if the frame where the lost data packet is located (set as the Nth frame) falls on a frame before the 28th frame of the GOP, the packet loss retransmission operation is performed; if the RTT is 200ms, if the frame where the lost data packet is located (set as the Mth frame) falls between the 25th and 30th frames of the GOP, the Mth to 30th frames are dropped; if the frame where the lost data packet is located falls on a frame before the 25th frame of the GOP, the packet loss data retransmission operation is performed. That is to say, if the video frame where the lost data packet is located is relatively late, then the time to receive the next key frame after losing these frames is shorter than the time to receive the next key frame after sending the packet loss retransmission operation request, then the frame loss operation can be performed. If the time to receive the next key frame after losing these frames is longer than the time to receive the next key frame after sending the packet loss retransmission operation request, then the packet loss retransmission operation is performed. The maximum number of frames that can be lost can be determined by the specific value of RTT in this application, because if the transmission time of the lost video frame is greater than RTT, there is no need to perform the frame loss operation, but it is more efficient to perform packet loss retransmission. Only when the transmission time of the lost video frame is less than RTT, the frame loss operation will be performed, which can further reduce the real-time video delay and jamming and improve the overall video fluency.
[0064] In an implementation of an embodiment of the present application, the frame drop operation involved in the above step 31 may further be one of the following: performing frame drop processing on the current video frame in the GOP to the end of the frame where the lost data packet is located; or performing frame drop processing on all video frames after the current video frame in the GOP where the lost data packet is located.
[0065] It can be seen that in this application, if the first duration is less than RTT, a frame drop operation can be performed. There are two ways of the frame drop operation, namely, whether to discard the video frame where the lost data packet is located. As for which specific method to choose, corresponding settings can be made according to actual needs. For example, after discarding the video frame where the lost data packet is located, if the smoothness of the video playback is not greatly affected, the video frame where the lost data packet is located can be discarded. If the smoothness of the video playback is greatly affected after discarding the video frame where the lost data packet is located, the video frame where the lost data packet is located is retained. The frame drop operation involved in step 31 of the claim may include the following steps:
[0066] Step 41, when the lost data packet is located within a preset range of the current video frame in the GOP, a frame loss process is performed from the current video frame to the end of the frame in the GOP where the lost data packet is located;
[0067] Step 42: When the lost data packet is not located within a preset range of the current video frame in the GOP, all video frames after the current video frame in the GOP where the lost data packet is located are subjected to frame loss processing.
[0068] It can be known from the above steps 41 and 42 that when performing the frame loss processing operation, if the position of the current frame (1 video frame is composed of several data packets) where the currently lost data packet is located, if the lost data packet is relatively at the back of the current video frame, then this frame can be retained, and if the lost data packet is relatively at the front of the current video, then the current video frame can be discarded until the next key frame arrives. Because if the lost data packet is relatively at the back of the current video frame, continuing to play the video frame will have little effect on the video fluency, and the video frame can continue to be played even if the data packet is lost. If the lost data packet is relatively at the front of the current video, then continuing to play the video frame will have a greater impact on the video fluency, and then the video frame can be discarded. In a specific example, the preset range can refer to the last 1 / 3 area, 1 / 4 area, etc. of the current frame.
[0069] Based on the above steps 41 and 42, before performing the frame dropping operation, the method steps of the embodiment of the present application may further include:
[0070] Step 51, based on the position of the lost data packet in the current video frame in the GOP, determine whether the lost data packet is located in a preset range, wherein the preset range is the area from the last data packet in the video frame to the target data packet, and the area from the last data packet to the target data packet is 1 / 3 of the area occupied by the video frame.
[0071] It should be noted that 1 / 3 is only an example to indicate the position relatively later in the current video frame, and may also be other relatively later ranges, such as 1 / 4, 1 / 5, etc., as long as it is a relatively later range. In other words, the preset range can be adjusted again as needed and is not limited to 1 / 3.
[0072] In an embodiment of the present application, the position of the lost data packet in the current video frame in the GOP is determined in the following manner: based on the serial number corresponding to the lost data packet and the total number of data packets in the current video frame in the GOP, the position of the lost data packet in the current video frame in the GOP is determined, wherein one video frame includes multiple data packets, the serial number corresponding to each data packet is continuously incremented based on the sorting position, and the total number of data packets in the current video frame where the data packet is located is determined by the first target field in the lost data packet.
[0073] Since each video frame includes multiple data packets, and each data packet has a corresponding serial number, which is continuously incremented based on the sorting position, after obtaining the total data packets of the video frame, the position of the corresponding data packet can be found based on the serial number. For example, the current video frame has 30 data packets, and the corresponding serial numbers are 1 to 30. If the 25th data packet is lost, the position of the data packet in the video frame can be determined.
[0074] It can be seen that since the data packet carries a field that can represent the position information of the data packet in the current video frame in the GOP, the position of the data packet in the video frame can be directly determined through this field.
[0075] The present application is described below with reference to the specific implementation methods of the present application. Figure 4 As shown, this specific implementation provides a method for reducing video delay, the steps of the method include:
[0076] Step 401, determining that packet loss occurs in a video stream, wherein the video stream includes a plurality of GOPs, and data packets in video frames in the GOPs are RTP packets;
[0077] Step 402, determine whether the RTT is greater than or equal to 100ms; if yes, execute step 404, if not, execute step 403;
[0078] If the RTT is not greater than 100ms, it indicates that the current time for sending a packet loss retransmission request is short. Based on the packet loss retransmission, the lost data packet can be retrieved. When the RTT is short, the smoothness of the video stream playback is not greatly affected. If the RTT is greater than 100ms, it indicates that the current round-trip time for packet loss retransmission is long. If continuing to perform packet loss retransmission has a greater impact on the smoothness of the video stream playback, the frame loss processing can be performed directly without packet loss retransmission. It should be noted that the 100ms is preset according to needs. In other application scenarios, it can also be 80ms, 90ms, 110ms, etc., that is, it can be set accordingly according to actual needs, and this is not limited in this application.
[0079] Step 403, performing packet loss retransmission;
[0080] Step 404, obtaining the position of the current frame in the GOP;
[0081] Step 405, determine whether the transmission time from the frame where the currently lost data packet is located to the end of the frame in the GOP is less than or equal to the RTT; if yes, execute step 406, if not, execute step 403;
[0082] Step 406, determine whether the currently lost data packet falls within the last 1 / 3 area of the current frame, if not, execute step 407, if yes, execute step 408;
[0083] Step 407, performing frame drop processing from the current frame to the end of the frame;
[0084] Step 408: Perform frame drop processing on the frames after the current frame to the end of the frame in the GOP.
[0085] In this specific implementation, when performing a frame loss processing operation, if the current lost data packet is located in the current frame (1 video frame is composed of several data packets), if the lost data packet is located relatively later in the current video frame, then this frame can be retained; if the lost data packet is located relatively earlier in the current video, then the current video frame can be dropped until the next key frame arrives. Because if the lost data packet is located relatively later in the current video frame, continuing to play the video frame will have little effect on the video fluency, and the video frame can continue to be played even if the data packet is lost; if the lost data packet is located relatively earlier in the current video frame, then continuing to play the video frame will have a greater impact on the video fluency, and then the video frame can be dropped. That is, in this specific implementation, the RTT value determines the area where frame loss can be performed, and then decides whether to perform frame loss or packet loss retransmission based on whether the video frame where the lost data packet falls is in this area, thereby improving the real-time video fluency and reducing real-time video delay and freeze, thereby improving the fluency of the entire video.
[0086] Corresponding to the above Figure 1 In the method steps, the embodiment of the present application also provides a video frame processing device, such as Figure 5 As shown, the device comprises:
[0087] A first determination module 52 is used to determine the position of the lost data packet in the GOP when it is determined that the GOP in the video stream has packet loss;
[0088] A second determination module 54 is used to determine a first duration based on the position of the lost data packet in the GOP, wherein the first duration is a transmission duration from a video frame corresponding to the lost data packet to a frame end of the GOP;
[0089] The first processing module 56 is used to perform a frame dropping operation or a packet dropping retransmission operation on the video frame based on a comparison result of the round-trip delay RTT and the first duration.
[0090] Through the device of the embodiment of the present application, compared with the prior art, the present application takes into account that discarding some non-key frames only affects the smoothness of the video. Therefore, based on the comparison result between the transmission time from the video frame corresponding to the lost data packet to the frame end of the GOP and the RTT, it is determined whether to perform a frame drop operation or a packet drop retransmission operation next. This can reduce the real-time video delay and freeze and improve the real-time video smoothness to a certain extent.
[0091] Optionally, the first processing module 52 in the embodiment of the present application may further include: a first processing unit, used to perform a frame loss operation when the first time length is less than RTT; and a second processing unit, used to perform a packet loss retransmission operation when the first time length is greater than or equal to RTT.
[0092] Optionally, the first processing unit in the embodiment of the present application may further include: a first processing sub-unit, used to perform frame loss processing on the lost data packet from the current video frame to the end of the frame in the GOP; and a second processing sub-unit, used to perform frame loss processing on all video frames after the current video frame in the GOP where the lost data packet is located.
[0093] Furthermore, in the present application, the first processing subunit is also used to, when the lost data packet is located within a preset range of the current video frame in the GOP, perform frame loss processing on the current video frame to the end of the frame of the lost data packet in the GOP. The second processing subunit is also used to, when the lost data packet is not located within the preset range of the current video frame in the GOP, perform frame loss processing on all video frames after the current video frame of the lost data packet in the GOP.
[0094] Optionally, the device in the embodiment of the present application may also include: a third determination module, used to determine the position of the lost data packet in the current video frame in the GOP based on the serial number corresponding to the lost data packet and the total number of data packets in the current video frame in the GOP where the lost data packet is located, wherein one video frame includes multiple data packets, the serial number corresponding to each data packet is continuously incremented based on the sorting position, and the total number of data packets in the current video frame where the data packet is located is determined by the first target field in the lost data packet.
[0095] Optionally, the device in the embodiment of the present application may also include: a fourth determination module, used to determine whether the lost data packet is located within a preset range based on the position of the lost data packet in the current video frame in the GOP before performing the frame drop operation, wherein the preset range is from the last data packet in the video frame to the area where the target data packet is located, and the area from the last data packet to the target data packet is 1 / 3 of the area occupied by the video frame.
[0096] Optionally, the second determination module in the embodiment of the present application may further include: a first determination unit, used to determine the unit transmission time required to transmit one video frame based on the total number of video frames in the GOP; a second determination unit, used to determine the number of video frames from the current video frame where the lost data packet is located to the end of the frame in the GOP as the target number; and a third determination unit, used to determine the product of the unit transmission time and the target number as the first time.
[0097] Optionally, the first determination module 52 in the embodiment of the present application may further include: a fourth determination unit, which determines the position of the lost data packet in the GOP based on the second target field in the lost data packet, wherein the second target field includes the position information of the video frame where the data packet is located in the GOP.
[0098] like Figure 6 As shown, an embodiment of the present application provides an electronic device, including a processor 111, a communication interface 112, a memory 113 and a communication bus 114, wherein the processor 111, the communication interface 112, and the memory 113 communicate with each other through the communication bus 114;
[0099] Memory 113, used for storing computer programs;
[0100] In one embodiment of the present application, the processor 111 is used to execute the program stored in the memory 113 to implement the video frame processing method provided by any of the aforementioned method embodiments, and the technical effects achieved are similar and will not be repeated here.
[0101] An embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the video frame processing method provided in any one of the aforementioned method embodiments are implemented.
[0102] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0103] The foregoing is merely a specific embodiment of the present invention, which enables those skilled in the art to understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features claimed herein.
Claims
1. A method for processing a video frame, characterized in that: include: In the case of determining that a group of pictures GOP in the video stream has packet loss, determining a position of the lost data packet in the GOP; Determine a first duration based on the position of the lost data packet in the GOP, wherein the first duration is a transmission duration from a video frame corresponding to the lost data packet to an end of a frame of the GOP; Based on the comparison result of the round-trip delay RTT and the first duration, performing a frame drop operation or a packet loss retransmission operation on the video frame; The performing of a frame dropping operation or a packet loss retransmission operation on a video frame based on a comparison result of the round-trip delay RTT and the first duration includes: When the first duration is less than the RTT, performing the frame dropping operation; When the first time duration is greater than or equal to the RTT, performing the packet loss retransmission operation; The performing of the frame dropping operation comprises: Perform frame loss processing from the current video frame to the end of the frame where the lost data packet is located in the GOP; or, Perform frame loss processing on all video frames after the current video frame where the lost data packet is located in the GOP; The step of performing frame dropping processing on the current video frame to the end of the frame of the lost data packet in the GOP comprises: when the lost data packet is located within a preset range of the current video frame in the GOP, performing frame dropping processing on the current video frame to the end of the frame of the lost data packet in the GOP; The step of performing frame dropping processing on all video frames after the current video frame in the GOP where the lost data packet is located includes: when the lost data packet is not located within a preset range of the current video frame in the GOP, performing frame dropping processing on all video frames after the current video frame in the GOP where the lost data packet is located.
2. The method according to claim 1, characterized in that Before performing the frame dropping operation, the method further includes: Based on the position of the lost data packet in the current video frame in the GOP, determine whether the lost data packet is located in the preset range, wherein the preset range is the area from the last data packet in the video frame to the target data packet, and the area from the last data packet to the target data packet is 1 / 3 of the area occupied by the video frame.
3. The method according to claim 2, characterized in that The position of the lost data packet in the current video frame in the GOP is determined by: Based on the serial number corresponding to the lost data packet and the total number of data packets in the video frame where the lost data packet is currently located in the GOP, determine the position of the lost data packet in the video frame where the lost data packet is currently located in the GOP, wherein one video frame includes multiple data packets, the serial number corresponding to each data packet is continuously increased based on the sorting position, and the total number of data packets in the video frame where the data packet is currently located is determined by the first target field in the lost data packet.
4. The method according to claim 1, characterized in that: The determining of the first duration based on the position of the lost data packet in the GOP comprises: Determine the unit transmission time required to transmit one video frame based on the total number of video frames in the GOP; Determine the number of video frames from the current video frame where the lost data packet is located in the GOP to the end of the frame as the target number; The product of the unit transmission duration and the target number is determined as the first duration.
5. The method according to claim 1, characterized in that Determining the position of the lost data packet in the GOP includes: Based on the second target field in the lost data packet, the position of the lost data packet in the GOP is determined, wherein the second target field includes the position information of the video frame where the data packet is located in the GOP.
6. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing the method steps described in any one of claims 1 to 5 when executing a program stored in a memory.
7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method steps according to any one of claims 1 to 5 are implemented.
