Long-term reference frame determination method and device, equipment, storage medium and product

Abstract

The embodiment of the invention provides a long-term reference frame determination method and device, equipment, a storage medium and a product. According to the technical scheme provided by the embodiment of the invention, the long-term reference frame setting interval is determined according to the packet loss rate and the round-trip delay of data transmission, whether the coding frame is marked as the long-term reference frame is determined according to the long-term reference frame setting interval, and when it is determined that the coding frame is marked as the long-term reference frame, the coding frame is marked as the long-term reference frame. After a coding frame is sent to a target terminal, when a notification that the target terminal successfully decodes a coding frame marked as a long-term reference frame is received, the coding frame marked as the long-term reference frame is set as an effective long-term reference frame, and the coding frame is sent to the target terminal by adopting an unequal transmission protection strategy. The transmission of the coding frame marked as the long-term reference frame is emphatically protected, the transmission and decoding success rate of the long-term reference frame is improved, the recovery probability of the long-term reference frame is effectively improved, and the coding transmission effect and the subjective feeling of video playing are improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technology, and in particular, to a method, apparatus, device, storage medium and product for determining a long-term reference frame. Background Art

[0002] With the development of Internet technology and computer technology, streaming media application scenarios are becoming more and more extensive. Whether it is real-time video communication or live broadcast, it is attracting more and more attention and use from users, and user experience is particularly important.

[0003] Due to various unstable factors in the actual network, data loss and jitter may occur during video data transmission, resulting in video playback freezes and delays, which greatly affects the user's subjective experience. Generally, different anti-packet loss measures (such as automatic retransmission request, forward error correction, and a combination of the two) are used to improve data transmission effects, or by changing the reference relationship of video frames to improve data transmission effects, such as using long-term reference frames (LTR, Long-Term Reference) for video encoding. In existing video encoding schemes, long-term reference frames are generally determined at fixed intervals, which leads to a low probability of recovery of marked long-term reference frames in poor network conditions, affecting video encoding effects. Summary of the invention

[0004] The embodiments of the present application provide a long-term reference frame determination method, apparatus, device, storage medium and product to solve the technical problem in the related art that long-term reference frames are determined according to fixed intervals, resulting in a low probability of long-term reference frame recovery, which affects the video encoding effect. The method can effectively improve the probability of long-term reference frame recovery and enhance the video encoding and transmission effects.

[0005] In a first aspect, an embodiment of the present application provides a method for determining a long-term reference frame, including:

[0006] Acquire a packet loss rate and a round-trip delay of data transmission, and determine a long-term reference frame setting interval according to the packet loss rate and the round-trip delay;

[0007] Determining whether to mark the coded frame as a long-term reference frame according to the long-term reference frame setting interval, and marking the coded frame as a long-term reference frame if it is determined that the coded frame is marked as a long-term reference frame;

[0008] In a case where the coded frame is marked as a coded frame of a long-term reference frame, the coded frame is sent to the target terminal based on a first transmission protection strategy, and in a case where the coded frame is not marked as a coded frame of a long-term reference frame, the coded frame is sent to the target terminal based on a second transmission protection strategy, wherein the protection level of the second transmission protection strategy is lower than that of the first transmission protection strategy;

[0009] When receiving a notification that the target terminal successfully decodes a coded frame marked as a long-term reference frame, the coded frame marked as a long-term reference frame is set as a valid long-term reference frame, wherein the valid long-term reference frame is used as an optional reference frame for image coding.

[0010] In a second aspect, an embodiment of the present application provides a long-term reference frame determination device, including an interval determination module, a marking processing module, a data sending module and a validation processing module, wherein:

[0011] The interval determination module is configured to obtain a packet loss rate and a round trip delay of data transmission, and determine a long-term reference frame setting interval according to the packet loss rate and the round trip delay;

[0012] The marking processing module is configured to determine whether to mark the coded frame as a long-term reference frame according to the long-term reference frame setting interval, and if it is determined that the coded frame is marked as a long-term reference frame, mark the coded frame as a long-term reference frame;

[0013] The data sending module is configured to send the coded frame to the target terminal based on a first transmission protection strategy when the coded frame is marked as a coded frame of a long-term reference frame, and to send the coded frame to the target terminal based on a second transmission protection strategy when the coded frame is not marked as a coded frame of a long-term reference frame, wherein the protection level of the second transmission protection strategy is lower than that of the first transmission protection strategy;

[0014] The effectiveness processing module is configured to set the coded frame marked as a long-term reference frame as an effective long-term reference frame upon receiving a notification that the target terminal successfully decodes the coded frame marked as a long-term reference frame, wherein the effective long-term reference frame is used as an optional reference frame for image encoding.

[0015] In a third aspect, an embodiment of the present application provides a long-term reference frame determination device, including: a memory and one or more processors;

[0016] The memory is used to store one or more programs;

[0017] When the one or more programs are executed by the one or more processors, the one or more processors implement the long-term reference frame determination method as described in the first aspect.

[0018] In a fourth aspect, an embodiment of the present application provides a non-volatile storage medium storing computer executable instructions, which, when executed by a computer processor, are used to execute the long-term reference frame determination method as described in the first aspect.

[0019] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes a computer program stored in a computer-readable storage medium, and at least one processor of the device reads and executes the computer program from the computer-readable storage medium, so that the device performs the long-term reference frame determination method as described in the first aspect.

[0020] The embodiment of the present application determines a long-term reference frame setting interval according to a packet loss rate and a round-trip delay of data transmission, determines whether to mark a coded frame as a long-term reference frame according to the long-term reference frame setting interval, and marks the coded frame as a long-term reference frame when it is determined that the coded frame is marked as a long-term reference frame, and sends the coded frame to the target terminal based on a first transmission protection strategy when the coded frame is marked as a coded frame of the long-term reference frame, and sends the coded frame to the target terminal based on a second transmission protection strategy when the coded frame is not marked as a coded frame of the long-term reference frame, and sets the coded frame marked as the long-term reference frame as an effective long-term reference frame when a notification is received that the target terminal successfully decodes the coded frame marked as the long-term reference frame, wherein the effective long-term reference frame can be used as an optional reference frame for image coding, and the effective long-term reference frame can be used as an optional reference frame for image coding according to the packet loss rate and the round-trip delay. Actively determine the long-term reference frame setting interval, adaptively mark the long-term reference frames according to the network conditions, improve the transmission and decoding success rate of the long-term reference frames, and reduce the situation in which the long-term reference frame transmission effect is poor due to too many long-term reference frames in the case of poor network conditions, and the probability of long-term reference frame recovery decreases, which affects the video encoding quality. It can effectively improve the probability of long-term reference frame recovery, improve video encoding effect, transmission effect and subjective feeling of video playback. At the same time, based on the network conditions, the long-term reference frame setting interval is adaptively adjusted and the asymmetric protection strategy is adopted. The long-term reference frame interval is adaptively set and asymmetric protection is performed according to the network conditions. When the long-term reference frame function is used, the recovery rate after frame loss is maximized, which effectively improves the fluency and subjective feeling when the long-term reference frame function is used in scenarios such as video calls and live broadcasts, and improves the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 is a flow chart of a method for determining a long-term reference frame provided by an embodiment of the present application;

[0022] Figure 2 is a flowchart of another method for determining a long-term reference frame provided by an embodiment of the present application;

[0023] Figure 3 It is a structural diagram of a long-term reference frame determination device provided in an embodiment of the present application;

[0024] Figure 4 It is a structural diagram of a long-term reference frame determination device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical scheme and advantages of the present application clearer, the specific embodiments of the present application are further described in detail below in conjunction with the accompanying drawings. It is understood that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for the convenience of description, only the part related to the present application but not all the contents are shown in the accompanying drawings. Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow chart describes each operation (or step) as a sequential process, many of the operations therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of each operation can be rearranged. The above process can be terminated when its operation is completed, but it can also have additional steps not included in the accompanying drawings. The above process can correspond to a method, a function, a procedure, a subroutine, a subprogram, etc.

[0026] The long-term reference frame determination method provided in this application can be applied to streaming media transmission scenarios such as real-time video communication and live broadcast. It aims to flexibly determine the long-term reference frame setting interval based on the packet loss rate and round-trip delay, adaptively mark the long-term reference frame according to the network conditions, increase the probability of long-term reference frame recovery, improve video encoding and transmission effects, and effectively improve video smoothness and subjective feeling of video playback.

[0027] In the existing long-term reference frame determination scheme, the long-term reference frame is generally determined at a fixed interval. However, in the case of a poor network, it is easy for a large number of marked long-term reference frames to appear, and the target terminal receiving the coded frame is prone to being unable to normally receive the long-term reference frame due to network fluctuations, congestion, etc., and the probability of long-term reference frame recovery decreases, affecting video coding and transmission effects. Based on this, a long-term reference frame determination method of an embodiment of the present application is provided to solve the technical problem that the existing long-term reference frame is determined at a fixed interval, resulting in a low probability of long-term reference frame recovery, which affects video coding and transmission effects.

[0028] Figure 1 A flowchart of a long-term reference frame determination method provided in an embodiment of the present application is given. The long-term reference frame determination method provided in an embodiment of the present application can be executed by a long-term reference frame determination device, which can be implemented by hardware and / or software and integrated in a long-term reference frame determination device.

[0029] The following description is made by taking the long-term reference frame determination method performed by the long-term reference frame determination device as an example. Figure 1 , the long-term reference frame determination method comprises:

[0030] S110: Acquire a packet loss rate and a round-trip delay of data transmission, and determine a long-term reference frame setting interval according to the packet loss rate and the round-trip delay.

[0031] Exemplarily, the packet loss rate and round-trip delay of data transmitted to the target terminal are obtained, wherein the target terminal may be video information (e.g., video packets, which include coded frames) in streaming media transmission scenarios such as real-time video communication and live broadcast. After determining the packet loss rate and round-trip delay of data transmission with the target terminal, the long-term reference frame setting interval is determined according to the packet loss rate and the round-trip delay, wherein the long-term reference frame setting interval is positively correlated with the packet loss rate and the round-trip delay, that is, the greater the packet loss rate and / or the round-trip delay, the greater the long-term reference frame setting interval.

[0032] In one embodiment, the packet loss rate and the round-trip delay are determined based on the confirmation information (such as ACK information, NACK information, etc.) of the target terminal based on the received data packet. For example, the packet loss rate is determined based on the ratio of the number of received coded frames to the number of sent coded frames within a set time period, and the round-trip delay is determined based on the average time length from sending the coded frame to receiving the feedback of the corresponding coded frame. Optionally, when there are multiple target terminals, the corresponding packet loss rate and round-trip delay can be the average packet loss rate and round-trip delay corresponding to the multiple target terminals, or the maximum packet loss rate and maximum round-trip delay among the multiple target terminals. Optionally, when determining the packet loss rate, the time period for evaluating the packet loss rate can be moved forward by a set time length to reduce the impact of network jitter on the determination of the packet loss rate and improve the accuracy of packet loss rate evaluation.

[0033] In one embodiment, after obtaining the packet loss rate and the round-trip delay, it can be determined whether the current network is a lossless network according to the packet loss rate and the round-trip delay. When the current network is a lossless network, the preset interval is used as the long-term reference frame setting interval, and when the current network is a lossy network, the long-term reference frame setting interval is adaptively determined according to the packet loss rate and the round-trip delay. A lossless network can be understood as a network with no data loss or less data loss during data transmission, and a lossy network can be understood as a network with data loss or greater data loss during data transmission.

[0034] Optionally, when the packet loss rate is less than the first preset packet loss threshold or the packet loss rate is the second preset packet loss threshold, and the round-trip delay is less than the set delay threshold, the current network is considered to be a lossless network, and when the packet loss rate is greater than or equal to the first preset packet loss threshold or the packet loss rate is greater than the second preset packet loss threshold, or the round-trip delay is greater than or equal to the set delay threshold, the current network is considered to be a lossy network. For example, the second preset packet loss threshold is set to 0, and the set delay threshold is set to 200ms. When the packet loss rate is 0 and the round-trip delay is less than 200ms, the current network is considered to be a lossless network, otherwise the current network is considered to be a lossy network.

[0035] When the current network is a lossless network, the preset interval can be used as the long-term reference frame setting interval. For example, the preset interval is 2, that is, a long-term reference frame is marked every 3 frames. When the target terminal normally recovers the coded frame marked as the long-term reference frame, it will inform the long-term reference frame determination device through feedback information that the long-term reference frame has been decoded normally. The long-term reference frame determination device can set this long-term reference frame to be effective for subsequent frame reference encoding. When the frame to be encoded is subsequently encoded, the selected reference frame can be selected from the previous encoded frame and the most recently effective long-term reference frame in the encoder according to the residual value, etc.

[0036] When the current network is a lossy network, the long-term reference frame setting interval can be adaptively and gradually lengthened according to the packet loss rate and round-trip delay. When there is more packet loss or the network is poor, the possibility of long-term reference frame loss is greater, and the time for the target terminal to feedback will also be longer. Therefore, by lengthening the long-term reference frame setting interval and focusing on protecting the transmission of long-term reference frames, the recovery rate of long-term reference frames during calls will be significantly improved, and the recovery probability of coded frames obtained by encoding with reference to long-term reference frames will be increased, which can effectively improve the smoothness of video calls. Optionally, when encoding frames in a lossy network, short-term reference frames may not be used for encoding, but multiple long-term reference frames that have been recently set to take effect (the number of long-term reference frames can be set according to demand) are used as optional reference frames, and the coded reference frame is selected from the optional reference frames according to the residual value between the frame to be encoded and the optional reference frames. As long as the coded frame that refers to the coded reference frame can be completely received at the target terminal, the coded frame can be decoded and restored normally.

[0037] S120: Determine whether to mark the coded frame as a long-term reference frame according to the long-term reference frame setting interval, and if it is determined that the coded frame is to be marked as a long-term reference frame, mark the coded frame as a long-term reference frame.

[0038] Exemplarily, before sending the encoded frame obtained by encoding the frame to be encoded to the target terminal, it can be determined whether the encoded frame needs to be marked as a long-term reference frame according to the long-term reference frame setting interval determined above.

[0039] Among them, the long-term reference frame can be a standard function established in the encoding protocols such as H264 and H265, and can be used as a reference frame in the streaming media encoder. The encoding protocol records the frame type definition of the long-term reference frame, the structure and syntax information of the frame, as well as the marking and effective settings of the long-term reference frame, the field assignment and parsing of encoding and decoding, etc. In general, the sender will mark the frame that needs to be set as a long-term reference frame. When the frame is successfully decoded at the receiving end and the information is fed back to the sender, the sender will set the frame as an effective long-term reference frame, and it may be used as a reference for subsequent frames. In theory, when the frame with the long-term reference frame as the reference frame is received by the receiving end, it can be decoded normally as long as the frame itself is complete. In one embodiment, when there is a storage frame number for the encoder in the encoding protocol, the number of long-term reference frames and short-term reference frames (STR, Short-Term Reference) can be reasonably allocated to ensure the recovery rate of the encoded frames during the video data transmission process and the final call experience (when a frame is encoded, if it is necessary to refer to the long-term reference frame, it needs to be specified through information such as framenumber (frame number). If not specified, the previous frame encoding is referenced by default). For example, the maximum number of reference frames of the encoder is 16 frames, the number of long-term reference frames can be configured to 15 frames, and the short-term reference frame can be configured to 1 frame. This solution can adaptively adjust the setting interval of the long-term reference frame according to the network conditions, and can adaptively adjust the setting interval of the long-term reference frame under different network conditions, more reasonably select the encoded frame marked as the long-term reference frame, and select and confirm the reference frame of the frame to be encoded, effectively improve the recovery rate after the current frame packet loss, and improve the call experience.

[0040] Exemplarily, when it is determined that a coded frame is to be marked as a long-term reference frame, the coded frame is marked as a long-term reference frame and cached at a set position, and when it is determined that a coded frame is not to be marked as a long-term reference frame, it is not necessary to mark the coded frame, and the coded frame can be sent as an ordinary frame. For example, when the long-term reference frame setting interval is 2, when the frame number of the last frame marked as a long-term reference frame is 3, it can be determined that the frame number corresponding to the next coded frame to be marked as a long-term reference frame is 6 (with two ordinary frames in between), and the coded frames with frame numbers 4 and 5 do not need to be marked as long-term reference frames.

[0041] S130: When the coded frame is marked as a coded frame of a long-term reference frame, the coded frame is sent to the target terminal based on a first transmission protection strategy; when the coded frame is not marked as a coded frame of a long-term reference frame, the coded frame is sent to the target terminal based on a second transmission protection strategy.

[0042] In one embodiment, after determining whether to mark a coded frame as a long-term reference frame according to a long-term reference frame setting interval, and marking the coded frame as a long-term reference frame when it is determined that the coded frame is to be marked as a long-term reference frame, the coded frame can be sent to a target terminal through a network. After receiving and successfully decoding the coded frame, the target terminal sends a notification indicating successful decoding of the coded frame marked as a long-term reference frame to a long-term reference frame determination device (for example, ACK information corresponding to the coded frame is sent to the long-term reference frame determination device). When the target terminal fails to successfully receive the coded frame or fails to successfully decode the coded frame, it sends a notification indicating unsuccessful decoding of the coded frame marked as a long-term reference frame to the long-term reference frame determination device (for example, NACK information corresponding to the coded frame is sent to the long-term reference frame determination device).

[0043] The coded frame sent to the target terminal may be a coded frame marked as a long-term reference frame or a coded frame not marked as a long-term reference frame, and the scheme adopts an unequal transmission protection strategy to send the coded frame marked as a long-term reference frame and the coded frame not marked as a long-term reference frame to the target terminal respectively. The protection level of the second transmission protection strategy applied to the transmission of the coded frame not marked as a long-term reference frame is lower than the first transmission protection strategy applied to the transmission of the coded frame marked as a long-term reference frame.

[0044] Exemplarily, when a coded frame to be sent to a target terminal is marked as a long-term reference frame, the coded frame can be sent to the target terminal based on a preset first transmission protection strategy, and when the coded frame is not marked as a coded frame of a long-term reference frame, the coded frame can be sent to the target terminal based on a second transmission protection strategy. The second transmission protection strategy provided by this solution has a lower protection level than the first transmission protection strategy, that is, relative to ordinary coded frames, the transmission of coded frames marked as long-term reference frames is given priority protection.

[0045] Optionally, sending the coded frame to the target terminal based on the first transmission protection strategy may be sending the coded frame to the target terminal based on a preset number of retransmissions and / or a preset redundancy, wherein the preset number of retransmissions and / or the preset redundancy are higher than the number of retransmissions and / or the redundancy corresponding to the second transmission protection strategy (e.g., a preset reference number of retransmissions and / or a reference redundancy).

[0046] In one embodiment, the preset number of retransmissions is higher than the number of retransmissions of the normal coded frame by a set number amplitude (e.g., 1-2 times), and the preset redundancy is higher than the redundancy of the normal coded frame by a set percentage amplitude (e.g., 10%-20%). Optionally, the set number amplitude and / or the set percentage amplitude may be positively correlated with the long-term reference frame setting interval, that is, the larger the long-term reference frame setting interval is, the larger the corresponding set number amplitude and / or the set percentage amplitude is. For example, the long-term reference frame setting interval is divided into different ranges, and the corresponding set number amplitude and / or the set percentage amplitude are set for the long-term reference frame setting intervals of different ranges. The corresponding set number amplitude and / or the set percentage amplitude can be added to the number of retransmissions and / or the redundancy of the normal coded frame to obtain the preset number of retransmissions and / or the preset redundancy. This scheme implements unequal protection of different coded frames by focusing on the transmission of coded frames marked as long-term reference frames, thereby improving the transmission success rate of long-term reference frames and improving video coding and transmission effects.

[0047] In a possible embodiment, the long-term reference frame determination method provided by the present solution may further select a coding reference frame from optional reference frames before sending the coding frame to the target terminal, and encode the frame to be coded based on the coding reference frame to obtain the coding frame.

[0048] Exemplarily, when it is necessary to encode the frame to be encoded, a coding reference frame is selected from the optional reference frames, wherein the optional reference frame may include one or more effective long-term reference frames and may also include a short-term reference frame. Optionally, the residual value between each optional reference frame and the frame to be encoded may be calculated, and the optional reference frame corresponding to the minimum residual value may be used as the coding reference frame, and based on a preset coding algorithm, the coding reference frame is used as a reference to encode the frame to be encoded to obtain the encoded frame. In this scheme, a coding reference frame is selected from the optional reference frames to encode the frame to be encoded, and the long-term reference frame in the optional reference frame is selected based on the adaptively determined long-term reference frame setting interval, so that the decoding success rate of the encoded frame is higher, the video playback effect is improved, and the user's video call experience is enhanced.

[0049] S140: When receiving a notification that the target terminal successfully decodes the coded frame marked as the long-term reference frame, the coded frame marked as the long-term reference frame is set as a valid long-term reference frame.

[0050] Exemplarily, when a notification is received that a target terminal has successfully decoded a coded frame marked as a long-term reference frame, the coded frame marked as a long-term reference frame may be set as an effective long-term reference frame, wherein the effective long-term reference frame is used as an optional reference frame for image coding. It is necessary to set the coded frame marked as a long-term reference frame as an effective long-term reference frame before it can be used as an optional reference frame for image coding after determining that the coded frame has been successfully received and decoded.

[0051] Optionally, the optional reference frames provided by the present solution may include long-term reference frames (including the latest determined and previously determined long-term reference frames, a set number of which are set), and may also include short-term reference frames. In one embodiment, when encoding a frame to be encoded, a coding reference frame may be determined from a plurality of long-term reference frames, and the frame to be encoded may be encoded based on the coding reference frame to obtain a corresponding coding frame.

[0052] Optionally, the coding reference frame may be determined from the latest long-term reference frame, or may be determined from multiple long-term reference frames, or may be determined from one or more long-term reference frames and a short-term reference frame (e.g., the last coded frame). The coding reference frame may be determined based on the residual between the frame to be coded and the optional reference frame, for example, selecting the optional reference frame with the smallest residual as the coding reference frame.

[0053] In the above, by determining the long-term reference frame setting interval according to the packet loss rate and the round-trip delay of data transmission, determining whether to mark the coded frame as the long-term reference frame according to the long-term reference frame setting interval, and when it is determined that the coded frame is marked as the long-term reference frame, marking the coded frame as the long-term reference frame, and when the coded frame is marked as a coded frame of the long-term reference frame, sending the coded frame to the target terminal based on the first transmission protection strategy, and when the coded frame is not marked as a coded frame of the long-term reference frame, sending the coded frame to the target terminal based on the second transmission protection strategy, when receiving the notification that the target terminal successfully decodes the coded frame marked as the long-term reference frame, setting the coded frame marked as the long-term reference frame as the effective long-term reference frame, wherein the effective long-term reference frame can be used as an optional reference frame for image coding, and according to the packet loss rate and the round-trip delay, the effective long-term reference frame can be used as an optional reference frame for image coding. Actively determine the long-term reference frame setting interval, adaptively mark the long-term reference frame according to the network conditions, improve the transmission and decoding success rate of the long-term reference frame, and reduce the situation in which the long-term reference frame transmission effect is poor due to too many long-term reference frames in the case of poor network conditions, and the probability of long-term reference frame recovery decreases, which affects the video encoding quality. It can effectively improve the probability of long-term reference frame recovery and improve video encoding and transmission effects. At the same time, based on the network conditions, the long-term reference frame setting interval is adaptively adjusted and the asymmetric protection strategy is adopted. The long-term reference frame interval is adaptively set and asymmetric protection is performed according to the network conditions. When the long-term reference frame function is used, the recovery rate after frame loss is maximized, which effectively improves the fluency and subjective feeling when using the long-term reference frame function in scenarios such as video calls and live broadcasts, and improves the user experience.

[0054] Based on the above embodiments, Figure 2 A flowchart of another long-term reference frame determination method provided in an embodiment of the present application is given. The long-term reference frame determination method is a specific embodiment of the long-term reference frame determination method described above. Figure 2 , the long-term reference frame determination method comprises:

[0055] S210: Obtaining the packet loss rate and round-trip delay of data transmission.

[0056] S220: Determine a first interval according to the packet loss rate, determine a second interval according to the round-trip delay, and determine a long-term reference frame setting interval according to the first interval and the second interval.

[0057] Exemplarily, after determining the packet loss rate and round-trip delay of data transmission, a first interval can be determined according to the packet loss rate, and a second interval can be determined according to the round-trip delay, and a long-term reference frame setting interval can be determined according to the first interval and the second interval.

[0058] Optionally, the long-term reference frame setting interval may be one of the first interval and the second interval, or the sum of the first interval and the second interval, the average of the first interval and the second interval, the product of the first interval and the second interval, etc., or the weighted sum of the first interval and the second interval. In one embodiment, the long-term reference frame setting interval provided by the present solution is an integer. When the long-term reference frame setting interval is a non-integer, the non-integer long-term reference frame setting interval may be rounded up to obtain an integer long-term reference frame setting interval.

[0059] This solution determines the first interval and the second interval respectively by determining the packet loss rate and round-trip delay of data transmission, and accurately determines the long-term reference frame setting interval based on the first interval and the second interval. The long-term reference frame setting interval can be flexibly set according to the packet loss rate and round-trip delay, effectively improving the probability of long-term reference frame recovery and improving video encoding and transmission effects.

[0060] In a possible embodiment, the long-term reference frame determination method provided by the present solution determines the first interval according to the packet loss rate, and may determine the first interval according to the sum of a first set reference interval and a ratio of the packet loss rate and a first preset coefficient.

[0061] Exemplarily, the sum of the first set reference interval and the ratio of the packet loss rate to the first preset coefficient can be used as the first interval. Optionally, when the first interval is a non-integer, the first interval is rounded up. For example, the first interval can be determined by the following formula:

[0062] T_lost=a1+lossrate / b1

[0063] Wherein, T_lost is the first interval, lossrate is the value corresponding to the packet loss rate, a1 is the first set reference interval, for example, a1 = 2, and b1 is the first preset coefficient, for example, b1 = 10. This solution accurately determines the first interval under the influence of the packet loss rate according to the sum of the first set reference interval and the ratio of the packet loss rate to the first preset coefficient, improves the accuracy of the long-term reference frame setting interval, effectively improves the probability of long-term reference frame recovery, and improves video encoding and transmission effects.

[0064] In a possible embodiment, the long-term reference frame determination method provided by the present scheme determines the second interval based on the round-trip delay, which may be: when the round-trip delay is greater than a preset reference delay, the second interval is determined based on the second set reference interval and the sum of the ratio of the difference between the round-trip delay and the preset reference delay and the second preset coefficient; when the round-trip delay is less than or equal to the preset reference delay, the second interval is determined based on the third set reference interval.

[0065] Exemplarily, it is determined whether the round-trip delay is greater than the preset reference delay. When the round-trip delay is greater than the preset reference delay, the sum of the second preset reference interval and the ratio of the difference between the round-trip delay and the preset reference delay and the second preset coefficient is used as the second interval, and when the round-trip delay is less than or equal to the preset reference delay, the third preset reference interval is used as the second interval. Optionally, when the second interval is a non-integer, the second interval is rounded up. For example, the second interval can be determined by the following formula:

[0066] T_rtt=rtt>c? a2+(rtt-c) / b2:a3

[0067] Wherein, T_rtt is the second interval, rtt is the corresponding value of the round-trip delay, c is the preset reference delay, for example, c=200, a2 is the second preset reference interval, b2 is the second preset coefficient, for example, b2=20, and a3 is the third preset reference interval, for example, a3=2. This solution uses the sum of the ratio of the second preset reference interval and the difference between the round-trip delay and the preset reference delay and the second preset coefficient as the second interval when the round-trip delay is greater than the preset reference delay, and uses the third preset reference interval as the second interval when the round-trip delay is less than or equal to the preset reference delay, thereby improving the accuracy of the long-term reference frame setting interval, effectively improving the probability of long-term reference frame recovery, and improving video encoding and transmission effects.

[0068] In a possible embodiment, the long-term reference frame determination method provided by the present solution determines the long-term reference frame setting interval according to the first interval and the second interval, and the maximum value of the first interval and the second interval is used as the long-term reference frame setting interval.

[0069] Exemplarily, after determining the first interval and the second interval, the maximum value of the first interval and the second interval is used as the long-term reference frame setting interval. For example, the long-term reference frame setting interval can be determined by the following formula:

[0070] T = max(T_lost, T_rtt)

[0071] Among them, T is the long-term reference frame setting interval, T_lost is the first interval, and T_rtt is the second interval. This solution uses the maximum value of the first interval and the second interval as the long-term reference frame setting interval based on the impact of packet loss rate and round-trip delay on network conditions. The determination of the long-term reference frame setting interval is more suitable for the current network conditions, effectively improving the probability of long-term reference frame recovery and improving video encoding and transmission effects.

[0072] In one embodiment, after determining the long-term reference frame setting interval, it can be determined whether the long-term reference frame setting interval is within a preset interval range. If the long-term reference frame setting interval exceeds the preset interval range, the long-term reference frame setting interval is limited within the preset interval range. For example, when the long-term reference frame setting interval is less than the lower limit of the preset interval range, the long-term reference frame setting interval is set to the lower limit of the preset interval range, and when the long-term reference frame setting interval is greater than the upper limit of the preset interval range, the long-term reference frame setting interval is set to the upper limit of the preset interval range. For example, the lower limit and upper limit of the preset interval range can be set to 2 and 10 respectively, and the preset interval range is: [2, 10]. The long-term reference frame setting interval is limited within a suitable range by the preset interval range, so as to reduce the situation where the long-term reference frame is set too densely or too sparsely, and ensure the video encoding effect.

[0073] S230: Determine whether to mark the coded frame as a long-term reference frame according to the long-term reference frame setting interval, and if it is determined that the coded frame is to be marked as a long-term reference frame, mark the coded frame as a long-term reference frame.

[0074] S240: When the coded frame is marked as a coded frame of a long-term reference frame, the coded frame is sent to the target terminal based on a first transmission protection strategy; when the coded frame is not marked as a coded frame of a long-term reference frame, the coded frame is sent to the target terminal based on a second transmission protection strategy.

[0075] S250: When receiving a notification that the target terminal successfully decodes the coded frame marked as the long-term reference frame, the coded frame marked as the long-term reference frame is set as a valid long-term reference frame.

[0076] In the above, by determining the long-term reference frame setting interval according to the packet loss rate and the round-trip delay of data transmission, determining whether to mark the coded frame as the long-term reference frame according to the long-term reference frame setting interval, and when it is determined that the coded frame is marked as the long-term reference frame, marking the coded frame as the long-term reference frame, and when the coded frame is marked as a coded frame of the long-term reference frame, sending the coded frame to the target terminal based on the first transmission protection strategy, and when the coded frame is not marked as a coded frame of the long-term reference frame, sending the coded frame to the target terminal based on the second transmission protection strategy, when receiving the notification that the target terminal successfully decodes the coded frame marked as the long-term reference frame, setting the coded frame marked as the long-term reference frame as the effective long-term reference frame, wherein the effective long-term reference frame can be used as an optional reference frame for image coding, and according to the packet loss rate and the round-trip delay, the effective long-term reference frame can be used as an optional reference frame for image coding. Actively determine the long-term reference frame setting interval, adaptively mark the long-term reference frame according to the network conditions, improve the transmission and decoding success rate of the long-term reference frame, reduce the situation where the long-term reference frame transmission effect is poor due to too many long-term reference frames under poor network conditions, and the probability of long-term reference frame recovery decreases, which affects the video encoding quality. It can effectively improve the probability of long-term reference frame recovery and improve the video encoding and transmission effect. At the same time, based on the long-term reference frame setting interval adaptive adjustment and asymmetric protection strategy of the network condition, the long-term reference frame interval is adaptively set and asymmetric protection is performed according to the network condition, and the recovery rate after frame loss is maximized when the long-term reference frame function is used, effectively improving the smoothness and subjective feeling when the long-term reference frame function is used in scenes such as video calls and live broadcasts, and improving the user experience. In addition, the first interval and the second interval are determined respectively by determining the packet loss rate and round-trip delay of data transmission, and the long-term reference frame setting interval is accurately determined according to the first interval and the second interval. The long-term reference frame setting interval can be flexibly set according to the packet loss rate and round-trip delay, effectively improving the probability of long-term reference frame recovery and improving video encoding and transmission effects.

[0077] Figure 3 Schematic diagram of a long-term reference frame determination device provided by an embodiment of the present application. Figure 3 The long-term reference frame determination device includes an interval determination module 31, a marking processing module 32, a data sending module 33 and a validation processing module 34.

[0078] Among them, the interval determination module 31 is configured to obtain the packet loss rate and round-trip delay of data transmission, and determine the long-term reference frame setting interval according to the packet loss rate and the round-trip delay; the marking processing module 32 is configured to determine whether to mark the coded frame as a long-term reference frame according to the long-term reference frame setting interval, and mark the coded frame as a long-term reference frame when it is determined that the coded frame is marked as a long-term reference frame; the data sending module 33 is configured to send the coded frame to the target terminal based on the first transmission protection strategy when the coded frame is marked as a coded frame of the long-term reference frame, and to send the coded frame to the target terminal based on the second transmission protection strategy when the coded frame is not marked as a coded frame of the long-term reference frame, wherein the protection level of the second transmission protection strategy is lower than that of the first transmission protection strategy; the effectiveness processing module 34 is configured to set the coded frame marked as a long-term reference frame as an effective long-term reference frame when receiving a notification that the target terminal successfully decodes the coded frame marked as a long-term reference frame, wherein the effective long-term reference frame is used as an optional reference frame for image encoding.

[0079] In the above, by determining the long-term reference frame setting interval according to the packet loss rate and the round-trip delay of data transmission, determining whether to mark the coded frame as the long-term reference frame according to the long-term reference frame setting interval, and when it is determined that the coded frame is marked as the long-term reference frame, marking the coded frame as the long-term reference frame, and when the coded frame is marked as a coded frame of the long-term reference frame, sending the coded frame to the target terminal based on the first transmission protection strategy, and when the coded frame is not marked as a coded frame of the long-term reference frame, sending the coded frame to the target terminal based on the second transmission protection strategy, when receiving the notification that the target terminal successfully decodes the coded frame marked as the long-term reference frame, setting the coded frame marked as the long-term reference frame as the effective long-term reference frame, wherein the effective long-term reference frame can be used as an optional reference frame for image coding, and according to the packet loss rate and the round-trip delay, the effective long-term reference frame can be used as an optional reference frame for image coding. Actively determine the long-term reference frame setting interval, adaptively mark the long-term reference frame according to the network conditions, improve the transmission and decoding success rate of the long-term reference frame, and reduce the situation in which the long-term reference frame transmission effect is poor due to too many long-term reference frames in the case of poor network conditions, and the probability of long-term reference frame recovery decreases, which affects the video encoding quality. It can effectively improve the probability of long-term reference frame recovery and improve video encoding and transmission effects. At the same time, based on the network conditions, the long-term reference frame setting interval is adaptively adjusted and the asymmetric protection strategy is adopted. The long-term reference frame interval is adaptively set and asymmetric protection is performed according to the network conditions. When the long-term reference frame function is used, the recovery rate after frame loss is maximized, which effectively improves the fluency and subjective feeling when using the long-term reference frame function in scenarios such as video calls and live broadcasts, and improves the user experience.

[0080] In a possible embodiment, the interval determination module 31 determines the long-term reference frame setting interval according to the packet loss rate and the round-trip delay, and is configured as: determining the first interval according to the packet loss rate; determining the second interval according to the round-trip delay; determining the long-term reference frame setting interval according to the first interval and the second interval.

[0081] In a possible embodiment, the interval determination module 31 determines the first interval according to the packet loss rate, and is configured to determine the first interval according to the sum of the first set reference interval and the ratio of the packet loss rate and the first preset coefficient.

[0082] In a possible embodiment, the interval determination module 31 determines the second interval according to the round-trip delay, and is configured as follows:

[0083] When the round-trip delay is greater than the preset reference delay, determining the second interval according to the second preset reference interval and the sum of the ratio of the difference between the round-trip delay and the preset reference delay and the second preset coefficient;

[0084] When the round-trip delay is less than or equal to the preset reference delay, the second interval is determined according to the third set reference interval.

[0085] In a possible embodiment, the interval determination module 31 determines the long-term reference frame setting interval according to the first interval and the second interval, and is configured to use the maximum value of the first interval and the second interval as the long-term reference frame setting interval.

[0086] In a possible embodiment, the data sending module 33 sends the coded frame to the target terminal based on the first transmission protection strategy, and is configured to: send the coded frame to the target terminal based on a preset number of retransmissions and / or a preset redundancy.

[0087] In a possible embodiment, the long-term reference frame determination device further includes a coding processing module, which is configured to select a coding reference frame from the optional reference frames, and perform coding processing on the frame to be coded based on the coding reference frame to obtain a coded frame.

[0088] It is worth noting that in the above-mentioned embodiment of the long-term reference frame determination device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the protection scope of the embodiments of the present application.

[0089] The embodiment of the present application also provides a long-term reference frame determination device, which can integrate the long-term reference frame determination apparatus provided in the embodiment of the present application. Figure 4 is a schematic diagram of the structure of a long-term reference frame determination device provided in an embodiment of the present application. Figure 4The long-term reference frame determination device includes: an input device 43, an output device 44, a memory 42, and one or more processors 41; the memory 42 is used to store one or more programs; when the one or more programs are executed by the one or more processors 41, the one or more processors 41 implement the long-term reference frame determination method provided in the above embodiment. The long-term reference frame determination device, equipment and computer provided above can be used to execute the long-term reference frame determination method provided in any of the above embodiments, and have corresponding functions and beneficial effects.

[0090] The embodiments of the present application also provide a non-volatile storage medium storing computer executable instructions, which are used to execute the long-term reference frame determination method provided in the above embodiments when executed by a computer processor. Of course, the non-volatile storage medium storing computer executable instructions provided in the embodiments of the present application, whose computer executable instructions are not limited to the long-term reference frame determination method provided above, can also execute the related operations in the long-term reference frame determination method provided in any embodiment of the present application. The long-term reference frame determination apparatus, device and storage medium provided in the above embodiments can execute the long-term reference frame determination method provided in any embodiment of the present application. For technical details not described in detail in the above embodiments, please refer to the long-term reference frame determination method provided in any embodiment of the present application.

[0091] On the basis of the above embodiments, the embodiments of the present application also provide a computer program product. The technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer program product is stored in a storage medium, and includes a number of instructions for enabling a computer device, a mobile terminal or a processor therein to execute all or part of the steps of the long-term reference frame determination method provided in each embodiment of the present application.

Claims

1. A method for determining a long-term reference frame, characterized in that: include: Acquire a packet loss rate and a round-trip delay of data transmission, and determine a long-term reference frame setting interval according to the packet loss rate and the round-trip delay; Determining whether to mark the coded frame as a long-term reference frame according to the long-term reference frame setting interval, and marking the coded frame as a long-term reference frame if it is determined that the coded frame is marked as a long-term reference frame; In a case where the coded frame is marked as a coded frame of a long-term reference frame, the coded frame is sent to the target terminal based on a first transmission protection strategy, and in a case where the coded frame is not marked as a coded frame of a long-term reference frame, the coded frame is sent to the target terminal based on a second transmission protection strategy, wherein the protection level of the second transmission protection strategy is lower than that of the first transmission protection strategy; When receiving a notification that the target terminal successfully decodes a coded frame marked as a long-term reference frame, the coded frame marked as a long-term reference frame is set as a valid long-term reference frame, wherein the valid long-term reference frame is used as an optional reference frame for image coding.

2. The method for determining a long-term reference frame according to claim 1, characterized in that: The determining of the long-term reference frame setting interval according to the packet loss rate and the round-trip delay comprises: determining a first interval according to the packet loss rate; determining a second interval according to the round trip delay; A long-term reference frame setting interval is determined according to the first interval and the second interval.

3. The method for determining a long-term reference frame according to claim 2, characterized in that: The determining the first interval according to the packet loss rate includes: The first interval is determined according to the sum of the first set reference interval and the ratio of the packet loss rate to the first preset coefficient.

4. The method for determining a long-term reference frame according to claim 2, characterized in that: The determining the second interval according to the round trip delay comprises: In the case where the round-trip delay is greater than the preset reference delay, determining the second interval according to the second set reference interval and the sum of the ratio of the difference between the round-trip delay and the preset reference delay and the second preset coefficient; When the round-trip delay is less than or equal to the preset reference delay, the second interval is determined according to a third set reference interval.

5. The method for determining a long-term reference frame according to claim 2, wherein: The determining of the long-term reference frame setting interval according to the first interval and the second interval comprises: The maximum value of the first interval and the second interval is used as the long-term reference frame setting interval.

6. The method for determining a long-term reference frame according to claim 1, characterized in that: Sending the encoded frame to the target terminal based on the first transmission protection strategy includes: The coded frame is sent to a target terminal based on a preset number of retransmissions and / or a preset redundancy.

7. The method for determining a long-term reference frame according to claim 1, characterized in that: Before sending the coded frame to the target terminal, the method further includes: A coding reference frame is selected from the optional reference frames, and a coding process is performed on the frame to be coded based on the coding reference frame to obtain a coded frame.

8. A long-term reference frame determination device, characterized in that: It includes an interval determination module, a mark processing module, a data sending module and a validation processing module, wherein: The interval determination module is configured to obtain a packet loss rate and a round trip delay of data transmission, and determine a long-term reference frame setting interval according to the packet loss rate and the round trip delay; The marking processing module is configured to determine whether to mark the coded frame as a long-term reference frame according to the long-term reference frame setting interval, and if it is determined that the coded frame is marked as a long-term reference frame, mark the coded frame as a long-term reference frame; The data sending module is configured to send the coded frame to the target terminal based on a first transmission protection strategy when the coded frame is marked as a coded frame of a long-term reference frame, and to send the coded frame to the target terminal based on a second transmission protection strategy when the coded frame is not marked as a coded frame of a long-term reference frame, wherein the protection level of the second transmission protection strategy is lower than that of the first transmission protection strategy; The effectiveness processing module is configured to set the coded frame marked as a long-term reference frame as an effective long-term reference frame upon receiving a notification that the target terminal successfully decodes the coded frame marked as a long-term reference frame, wherein the effective long-term reference frame is used as an optional reference frame for image encoding.

9. A long-term reference frame determination device, characterized in that: include: memory and one or more processors; The memory is used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the long-term reference frame determination method as described in any one of claims 1 to 7.

10. A non-volatile storage medium storing computer executable instructions, characterized in that: When the computer executable instructions are executed by a computer processor, they are used to perform the long-term reference frame determination method according to any one of claims 1 to 7.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the long-term reference frame determination method according to any one of claims 1 to 7 is implemented.

Images