A video encoding method, device, storage medium and electronic device

Abstract

The embodiment of the present specification discloses a video encoding method, which monitors the current network quality when carrying out inter-frame prediction encoding based on STR frames. If the current network quality is poor, the method selects LTR frames from the encoded video frames according to the current network delay and the storage space for storing LTR frames, stores the selected LTR frames into the storage space, and starts inter-frame prediction encoding based on LTR frames according to the LTR frames stored in the storage space. Through the above method, inter-frame prediction encoding based on LTR can be enabled only when the network quality is poor, and LTR frames can be managed according to the current network delay and the size of the storage space for storing LTR frames, so that the encoding quality can be considered while the encoding cost is reduced as much as possible.

Description

Technical Field

[0001] This specification relates to the field of computer technology, and in particular to a video encoding method, apparatus, storage medium, and electronic device. Background Technology

[0002] Inter-frame predictive coding is one of the key technologies in modern video coding. When encoding video frames, this technique does not encode each frame completely; instead, it divides the frames into keyframes and prediction frames. Keyframes require complete encoding, while prediction frames only need to be encoded to address the differences between the prediction frame and the reference frame, significantly improving video coding efficiency.

[0003] Inter-frame predictive coding is further divided into inter-frame predictive coding based on Short-Term Reference (STR) frames and inter-frame predictive coding based on Long-Term Reference (LTR) frames.

[0004] Inter-frame predictive coding (IRC) based on STR (Synchronous Transmission) only allows the predicted frame to use its preceding frame as a reference frame. Motion estimation is then performed on the predicted frame and the reference frame, and the differences between them are encoded. IRC based on LTR (Long-Terminal Transmission) breaks this limitation, allowing earlier, non-contiguous video frames to be used as reference frames for encoding. Because LTR-based IRC has more reliable reference points during random access or network packet loss, it exhibits better robustness and is widely used in video coding for scenarios such as video conferencing and streaming media.

[0005] However, LTR-based inter-frame predictive coding is also more expensive. In practical applications, a method that flexibly switches between STR-based and LTR-based inter-frame predictive coding is often used for video coding. Therefore, how to control the enabling of LTR-based inter-frame predictive coding and manage LTR frames to balance coding performance and cost is an urgent problem to be solved. Summary of the Invention

[0006] This specification provides a video encoding method, apparatus, storage medium, and electronic device to partially solve the problems existing in the prior art.

[0007] The embodiments in this specification adopt the following technical solutions: This specification provides a video encoding method, the method comprising: When performing inter-frame predictive coding based on short-term reference STR frames on each video frame in the video to be sent, monitor the current network quality degradation characterization value. If the current network quality degradation value is greater than the preset degradation threshold, then the current network latency is determined. Based on the current network latency and the storage space available for storing Long-Term Reference (LTR) frames, select an LTR frame from the encoded video frames. The selected LTR frames are stored in the storage space, and based on the LTR frames stored in the storage space, inter-frame predictive coding based on LTR frames is performed on the uncoded video frames in the video to be sent.

[0008] This specification provides a video encoding device, the device comprising: The monitoring module is used to monitor the current network quality degradation characterization value when performing inter-frame predictive coding based on short-term reference STR frames on each video frame in the video to be sent. The determination module is used to determine the current network latency if the current network quality degradation characterization value is greater than a preset degradation threshold. The selection module is used to select an LTR frame from the encoded video frames based on the current network latency and the storage space used to store the Long-Term Reference (LTR) frame. An enabling module is used to store selected LTR frames into the storage space and perform LTR frame-based inter-frame predictive coding on the uncoded video frames in the video to be sent based on the LTR frames stored in the storage space.

[0009] This specification provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described video encoding method.

[0010] This specification provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the video encoding method described above.

[0011] This specification provides a computer program product, which includes a computer program that, when executed by a processor, implements the aforementioned video encoding method.

[0012] The above-described at least one technical solution adopted in the embodiments of this specification can achieve the following beneficial effects: This specification discloses a video coding method. When performing inter-frame predictive coding based on STR frames, the method monitors the current network quality. If the current network quality is poor, it first selects an LTR frame from the encoded video frames based on the current network latency and the available storage space for storing LTR frames. Then, it stores the selected LTR frame in the storage space and initiates inter-frame predictive coding based on the LTR frame stored therein. This method enables LTR-based inter-frame predictive coding only when network quality deteriorates, and it manages LTR frames based on the current network latency and the size of the storage space for storing LTR frames, thus minimizing coding costs while maintaining coding quality. Attached Figure Description

[0013] The accompanying drawings, which are included to provide a further understanding of this specification and form part of this specification, illustrate exemplary embodiments and are used to explain this specification, but do not constitute an undue limitation thereof. In the drawings: Figure 1 This is a schematic diagram of the video encoding method provided in the embodiments of this specification; Figure 2 A schematic diagram of the inter-frame predictive coding process based on STR frames provided in the embodiments of this specification; Figure 3 A schematic diagram of the inter-frame predictive coding process based on LTR frames provided in the embodiments of this specification; Figure 4 A schematic diagram illustrating the expected number of frames covered by network latency as provided in the embodiments of this specification; Figure 5 A schematic diagram of the encoding process after the transmitter disables inter-frame predictive coding based on LTR frames, as provided in the embodiments of this specification; Figure 6 A schematic diagram of a video encoding device provided in the embodiments of this specification; Figure 7 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this specification. Detailed Implementation

[0014] To make the objectives, technical solutions, and advantages of this specification clearer, the technical solutions of this specification will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of them. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this specification.

[0015] The technical solutions provided in the various embodiments of this specification are described in detail below with reference to the accompanying drawings.

[0016] Figure 1 This is a schematic flowchart of the video encoding method provided in the embodiments of this specification, which specifically includes the following steps: S100: When performing inter-frame predictive coding based on short-term reference STR frames for each video frame in the video to be transmitted, monitor the current network quality degradation characterization value.

[0017] In the embodiments described in this specification, the following are employed: Figure 1 The device used for video encoding by the method shown can be a sending end for sending the video to be sent. Specifically, the sending end can be a user terminal, including personal computers, mobile phones, tablets, etc., or a server, including servers or server clusters composed of multiple servers, etc. This specification does not limit this, and it will be referred to as the sending end below.

[0018] The sending end can first acquire the video to be sent. When sending the video to the receiving end, it needs to encode each video frame in the video according to the order of the video frames, and simultaneously transmit the encoded video frames to the receiving end in sequence. Notably, the sending end does not need to encode all video frames in the video before sending the encoded ones; for each video frame, it can be sent as soon as the encoding is complete, without waiting for other video frames to finish encoding.

[0019] To improve video encoding efficiency, the transmitting end can use inter-frame predictive coding to encode each video frame in the video to be transmitted. In the embodiments of this specification, the transmitting end defaults to using STR-based inter-frame predictive coding to encode each video frame in the video to be transmitted. That is, when encoding the nth frame, the (n-1)th frame is used as the reference frame, and only the difference between the nth frame and the reference frame is encoded, without needing to encode the entire nth frame. Figure 2 As shown.

[0020] Figure 2 This diagram illustrates the inter-frame predictive coding process based on STR frames, as provided in the embodiments of this specification. The first frame is a keyframe, the second frame is a predictive frame encoded using the first frame as a reference frame, the third frame is a predictive frame encoded using the second frame as a reference frame, and so on. After these predictive frames are transmitted to the receiving end, the receiving end also needs to decode the received predictive frames based on their reference frames.

[0021] However, when network quality deteriorates between the sender and receiver, the receiver may fail to receive a certain video frame, such as... Figure 2As shown, when the receiver fails to receive the 3rd frame, since the 4th frame is a predicted frame with the 3rd frame as the reference frame, and the 5th frame is a predicted frame with the 4th frame as the reference frame, even if the receiver receives the 4th frame, it will be unable to decode the 4th frame due to the 3rd frame being the reference frame. Consequently, it will also be unable to decode the 5th frame until the receiver successfully receives the next keyframe, at which point it can continue decoding correctly. Figure 2 Video frames that the receiver cannot receive or decode are represented by dashed lines. In other words, errors caused by inter-frame predictive coding based on STR frames will be propagated forward.

[0022] Therefore, when network quality deteriorates between the transmitter and receiver, it is necessary to switch to inter-frame predictive coding based on LTR frames, such as... Figure 3 As shown.

[0023] Figure 3 This diagram illustrates the inter-frame predictive coding process based on LTR frames, as provided in the embodiments of this specification. Frame 1 is the keyframe, frame 2 is a predictive frame encoded using frame 1 as the reference frame, frame 3 is also a predictive frame encoded using frame 1 as the reference frame, and frame 4 is a predictive frame encoded using frame 2 as the reference frame. Since LTR-based inter-frame predictive coding allows frame n to use any frame as the reference frame, therefore, in Figure 3 In this scenario, if the receiving end receives the first frame but not the second frame, it can still decode the third frame based on the first frame it has already received.

[0024] Therefore, in the embodiments of this specification, while the transmitting end uses STR-based inter-frame predictive coding to encode each video frame in the video to be transmitted, it also needs to monitor the current network quality degradation characterization value between itself and the receiving end in real time or periodically. This network quality degradation characterization value is used to represent the degree of network quality degradation between the transmitting end and the receiving end.

[0025] Specifically, various network degradation environments can be preset in the sending end, including but not limited to: weak network environment, suspected weak network environment, and slow feedback environment. The sending end can periodically send a high-precision periodic signal to the receiving end. After receiving the high-precision periodic signal, the receiving end sends it back to the sending end. The sending end can then measure the network quality between the sending and receiving ends based on the sent and received high-precision periodic signals. The measurements include, but are not limited to, network bandwidth, packet loss rate, network latency, and network jitter between the receiving ends. The period for sending the high-precision periodic signal can be a preset fixed period or a period that dynamically changes with the network quality degradation indicator value. For example, the period can be negatively correlated with the network quality degradation indicator value; that is, the better the network environment and the lower the network quality degradation indicator value, the longer the period, and vice versa.

[0026] Furthermore, a weak network environment refers to a situation where the network bandwidth between the sender and receiver is low, and / or the packet loss rate is high, and / or the network latency is high, and / or the network jitter is high. The sender can determine the confidence level, or probability, that the current network environment between the sender and receiver belongs to this weak network environment based on at least one of the measured current network bandwidth, packet loss rate, network latency, and network jitter. The confidence level of belonging to a weak network environment is negatively correlated with the network bandwidth measured by the sender, and positively correlated with the packet loss rate, network latency, and network jitter.

[0027] A suspected weak network environment refers to a network environment where the sending end sends the aforementioned high-precision periodic signal to the receiving end but fails to receive the returned high-precision periodic signal. The sending end can determine the confidence level of whether the current network environment belongs to this suspected weak network environment based on whether it received the returned high-precision periodic signal after the most recent transmission. Specifically, if the sending end has not received the returned high-precision periodic signal after its most recent transmission for measuring network latency, the confidence level, or probability, of belonging to this suspected weak network environment is determined based on the time elapsed between the most recent transmission and the current time. The confidence level of belonging to this suspected weak network environment is positively correlated with the time elapsed between the most recent transmission of the high-precision periodic signal and the current time.

[0028] A slow feedback environment refers to a network environment where, after the transmitter sends the aforementioned high-precision periodic signal to the receiver, although the receiver can receive the same high-precision periodic signal, the moment of receipt is significantly later than the expected reception time under ideal network conditions. The transmitter can then determine the confidence level, or probability, that the current network environment belongs to this slow feedback environment based on the current Round-Trip Time (RTT). The confidence level of belonging to this slow feedback environment is positively correlated with the RTT. The RTT is the time from the moment the transmitter sends the high-precision periodic signal to the moment the receiver returns the signal.

[0029] After the sending end determines the confidence level of the current network environment belonging to the three types of network degradation environments mentioned above, it can determine the current network quality degradation characterization value based on the confidence level of the current network environment belonging to each type of network degradation environment and the weights pre-set for each type of network degradation environment. That is, the current network quality degradation characterization value can be obtained by weighted summing of the confidence levels of the current network environment belonging to each type of network degradation environment according to the pre-set weights. Specifically, the network quality degradation characterization value can be calculated using the formula S1=w1×I1+w2×I2+w3×I3, where S1 is the network quality degradation characterization value, I1, I2, and I3 are the confidence levels of the current network environment belonging to the weak network environment, suspected weak network environment, and slow feedback environment, respectively, and w1, w2, and w3 are the pre-set weights for the weak network environment, suspected weak network environment, and slow feedback environment, respectively. A larger network quality degradation characterization value indicates a worse current network quality, and vice versa.

[0030] Those skilled in the art should understand that the above description is only based on three preset network degradation environments: weak network environment, suspected weak network environment, and slow feedback environment. Different network degradation environments can be set arbitrarily according to actual needs, which does not constitute a limitation of this application.

[0031] S101: If the current network quality degradation value is greater than the preset degradation threshold, then determine the current network latency.

[0032] When the transmitting end detects that the current network quality degradation indicator value is greater than the preset degradation threshold, it indicates that the network quality is already poor. If STR-based inter-frame predictive coding continues to encode the unencoded video frames in the video to be transmitted, the receiving end will generate a large number of decoding errors. Therefore, it is necessary to switch to LTR-based inter-frame predictive coding to encode the unencoded video frames in the video to be transmitted. Of course, the transmitting end can also switch to LTR-based inter-frame predictive coding to encode the unencoded video frames in the video to be transmitted when the current network quality degradation indicator value is greater than the preset degradation threshold, and the time elapsed since the last time LTR-based inter-frame predictive coding was turned off is greater than a specified duration. Alternatively, it can switch to LTR-based inter-frame predictive coding to encode the unencoded video frames in the video to be transmitted when the number of times the current network quality degradation indicator value is greater than the preset degradation threshold exceeds a specified number of times, or when the duration of the continuous detection of the current network quality degradation indicator value being greater than the preset degradation threshold exceeds a specified duration. This specification does not impose any restrictions on this.

[0033] To employ LTR-based inter-frame predictive coding, LTR frames need to be managed. This is because LTR-based inter-frame predictive coding references long-term reference frames, i.e., LTR frames. For a video frame to be encoded, all encoded video frames preceding it in the transmitted video may serve as LTR frames. However, the storage space available for buffering LTR frames at the transmitting end is limited, and typically not all encoded video frames can be stored in this space as LTR frames. Therefore, LTR frame management is necessary.

[0034] Managing LTR frames requires first determining the current network latency between the sender and receiver. This is because, theoretically, all encoded video frames preceding the video frame to be encoded could be used as LTR frames for the video frame to be encoded. However, selecting frames closer to the video frame to be encoded as reference frames yields better encoding results. Therefore, it is best to store the first few frames of the video frame to be encoded as LTR frames in the aforementioned storage space. However, the first few frames of the video frame to be encoded will inevitably be overwritten by the current network latency. Therefore, we need to determine the current network latency first, and then manage the LTR frames based on this network latency.

[0035] It should be noted that the network latency described in this specification refers to the time it takes for a signal or information to travel from the sender to the receiver, and for the receiver to respond and then return to the sender, under the current network environment between the sender and receiver. Therefore, in the embodiments of this specification, the method by which the sender determines the current network latency is to take the sum of the current RTT, the target latency, and the redundancy latency as the current network latency. Here, RTT mainly measures the time it takes for a signal or information to travel during transmission; the target latency is the expected network latency under ideal network conditions, i.e., only considering the encoding latency, transmission buffer latency, decoding latency, and reception buffer latency of the signal or information; and the redundancy latency is the waiting time actively introduced at both the sender and receiver to cope with network jitter and packet loss.

[0036] S102: Select an LTR frame from the encoded video frames based on the current network latency and the storage space available for storing the Long Term Reference (LTR) frame.

[0037] Once the sending end determines the aforementioned network latency, it can determine the number of frames within the time frame corresponding to the current network latency, based on the current network latency and the frame rate of the video to be sent, as the expected number of frames. Figure 4 As shown.

[0038] Figure 4This diagram illustrates the expected number of frames covered by network latency as provided in the embodiments of this specification. Assuming the frame rate of the video to be sent is 30fps (30 frames per second), and the current network latency is 500ms, then within these 500ms, the video to be sent will play 15 frames at normal speed. These 15 frames represent the number of frames within the time corresponding to the network latency. In other words, the expected number of frames described in this specification is the number of frames required for the video to be sent to play at normal speed within the time corresponding to the current network latency.

[0039] exist Figure 4 In this scenario, the first n-1 frames of the video to be sent have already been encoded and transmitted. The current video frame to be encoded is the nth frame. If the frame rate of the video to be sent is 30fps and the current network latency is 500ms, then if the current network quality remains unchanged, during the encoding of the nth frame, the sending end can only receive a response message from the receiving end indicating that it has received the n-15th frame. Since the closer to the nth frame, the better the encoding effect of the nth frame, the video frame used as the reference frame for the nth frame should be generated from the 15 frames from the n-15th frame to the n-1th frame as much as possible. In other words, these 15 frames should be used as LTR frames as much as possible.

[0040] At this point, the sending end needs to determine whether the expected number of frames is not greater than the maximum number of frames that the storage space used to store LTR frames can store. Assuming that the storage space can store a maximum of 30 frames, the sending end can use the number of encoded video frames preceding the current video frame to be encoded in the video to be sent as the selected LTR frames. That is, the 15 frames from frame n-15 to frame n-1 are all stored as selected LTR frames in the above storage space, so that when encoding the nth frame, a reference frame can be selected from the LTR frames stored in this storage space, and the nth frame can be encoded based on the reference frame.

[0041] If the maximum number of frames that the storage space can store is less than the expected number of frames, for example, assuming that the storage space can store a maximum of 8 frames, then the sending end must select the maximum number of encoded video frames from the number of encoded video frames that are several frames ahead of the current video frame to be encoded, as the selected LTR frames. That is, 8 frames are selected from the 15 encoded video frames from frame n-15 to frame n-1 as LTR frames and stored in the storage space.

[0042] When selecting 8 frames as LTR frames from the aforementioned 15 frames, the transmitting end can determine the importance characterization values ​​of these 15 frames and select the 8 frames with the highest importance characterization values ​​as the selected LTR frames. The importance characterization values ​​described in this specification are used to characterize the importance of an encoded video frame in encoding the current video frame to be encoded, or in other words, its suitability as a reference frame for the current video frame to be encoded. A higher importance characterization value for an encoded video frame indicates that the encoded video frame is more important for encoding the current video frame to be encoded and is more suitable as a reference frame for the current video frame to be encoded; conversely, a lower importance characterization value indicates that the encoded video frame is less important for encoding the current video frame to be encoded and is less suitable as a reference frame for the current video frame to be encoded.

[0043] When determining the importance characterization value of an encoded video frame, the transmitting end can determine the importance characterization value of the encoded video frame based on at least one of the following: the distance from the encoded video frame to the video frame to be encoded, the transmission cost of transmitting the encoded video frame, the image complexity of the encoded video frame, and the image quality of the encoded video frame.

[0044] The distance between an encoded video frame and a video frame to be encoded refers to the number of frames between the encoded video frame and the video frame to be encoded in the video to be sent. The importance value of the encoded video frame is negatively correlated with this distance.

[0045] The transmission cost of transmitting the encoded video frame refers to the number of data packets encoded into the encoded video frame, or the amount of data corresponding to the encoded video frame. The larger the amount of data, the more difficult it is for the receiving end to recover the completed encoded video frame once packet loss occurs. Therefore, the importance value of the encoded video frame is negatively correlated with the transmission cost.

[0046] The image complexity of the encoded video frame refers to the complexity of the texture in the image corresponding to the encoded video frame. The more complex the texture, the easier it is to encode the difference between the video frame to be encoded and the encoded video frame. Therefore, the importance characterization value of the encoded video frame is positively correlated with the image complexity.

[0047] The image quality of the encoded video frame refers to the quality of the image after encoding. The higher the image quality, the clearer the image obtained by the receiver after decoding the encoded video frame, and the richer the details retained. This image quality can be characterized by the qp value or other parameters, and this specification does not limit this. Therefore, the importance value of the encoded video frame is positively correlated with its image quality.

[0048] For a given encoded video frame, after the sending end determines the distance from the encoded video frame to the video frame to be encoded, the transmission cost of transmitting the encoded video frame, the image complexity of the encoded video frame, and the image quality of the encoded video frame using the methods described above, it can determine a first dimension representation value based on the distance from the encoded video frame to the video frame to be encoded, a second dimension representation value based on the transmission cost of transmitting the encoded video frame, a third dimension representation value based on the image complexity of the encoded video frame, and a fourth dimension representation value based on the image quality of the encoded video frame. Specifically, the first dimension representation value is negatively correlated with the distance from the encoded video frame to the video frame to be encoded, the second dimension representation value is negatively correlated with the transmission cost of transmitting the encoded video frame, the third dimension representation value is positively correlated with the image complexity of the encoded video frame, and the fourth dimension representation value is positively correlated with the image quality of the encoded video frame.

[0049] After determining the above four dimension representation values, the sending end can weight the first dimension representation value, the second dimension representation value, the third dimension representation value, and the fourth dimension representation value according to the preset weights for the first dimension representation value, the second dimension representation value, the third dimension representation value, and the fourth dimension representation value respectively, to obtain the importance representation value of the encoded video frame. That is, the importance representation value of the encoded video frame is determined by the formula S2=w1'×I1'+w2'×I2'+w3'×I3'+w4'×I4', where S2 is the importance representation value of the encoded video frame, I1'~I4' are the above four dimension representation values ​​respectively, and w1'~w4' are the weights preset for the above four dimension representation values.

[0050] For the 15 encoded video frames from n-15 to n-1 in the example above, after the sending end determines the importance representation value of these 15 frames, it can select 8 video frames in descending order of importance representation value and store them as LTR frames in the above storage space.

[0051] In addition, to cope with extreme network degradation, the aforementioned storage space needs to store at least one encoded video frame that has been received by the receiver as an LTR frame (each time the receiver receives an encoded video frame, it will return a response message to the sender confirming that the encoded video frame has been received). Therefore, after the sender selects the largest number of encoded video frames with the highest importance characterization value using the above method, it can also determine whether the currently selected encoded video frames contain encoded video frames that have already been received by the receiver. If they do, the currently selected encoded video frames can be stored as LTR frames in the aforementioned storage space. If they do not, the encoded video frame with the lowest importance characterization value among the currently selected encoded video frames needs to be removed, and the encoded video frame with the highest importance characterization value that has already been received by the receiver among the previous expected number of encoded video frames of the video frame to be encoded is used to replace the removed encoded video frame. The replaced encoded video frames are then used as the selected LTR frames.

[0052] Continuing with the previous example, after the transmitting end determines the importance representation values ​​of the 15 encoded video frames from n-15 to n-1, it selects the 8 video frames with the highest importance representation values. It then determines whether the selected 8 frames include video frames that have already been received by the receiving end. If so, these 8 frames are directly stored as LTR frames in the aforementioned storage space. Otherwise, the video frame with the lowest importance representation value among these 8 frames is removed, and the encoded video frame that has already been received by the receiving end and has the highest importance representation value is selected from these 15 frames to replace the removed video frame. Finally, the newly determined 8 frames are stored as LTR frames in the aforementioned storage space.

[0053] Similarly, when the expected number of frames is not greater than the maximum number of frames that the storage space used to store LTR frames can hold, at least one encoded video frame that has been received by the receiving end before the current video frame to be encoded can be stored in the storage space. Specifically, the least important video frame among the expected number of encoded video frames before the current video frame to be encoded can be replaced by the most recent encoded video frame received by the receiving end before the current video frame to be encoded, and stored in the storage space.

[0054] It should be noted that for each current video frame to be encoded, the sending end can first clear the aforementioned storage space, and then use the method shown in step S102 above to store the corresponding LTR frame for the current video frame to be encoded.

[0055] S103: Store the selected LTR frame into the storage space, and perform inter-frame predictive coding based on LTR frames on the uncoded video frames in the video to be sent, according to the LTR frames stored in the storage space.

[0056] After the transmitting end selects an LTR frame and stores it in the storage space through step S102, it can perform inter-frame predictive coding based on the LTR frame in the video to be transmitted and its subsequent video frames that need to be encoded, according to the LTR frame stored in the storage space. This specification does not restrict which LTR frame is specifically selected from the aforementioned storage space as the reference frame when performing inter-frame predictive coding based on the LTR frame for the current video frame to be encoded.

[0057] The above describes the method for the transmitter to switch from the default STR-based inter-frame predictive coding to LTR-based inter-frame predictive coding. When the transmitter detects that the current network quality degradation value is not greater than the preset degradation threshold, and the duration of LTR-based inter-frame predictive coding has exceeded the preset duration, it can switch back to the default STR-based inter-frame predictive coding to continue encoding the unencoded video frames in the video to be transmitted.

[0058] Since switching from LTR-based inter-frame predictive coding back to STR-based inter-frame predictive coding means that the first frame after the switch must satisfy the constraint that the predictive frame takes the previous frame as its reference frame, it is crucial to determine whether the first frame after the switch is a key frame or a predictive frame.

[0059] Figure 5 This is a schematic diagram of the encoding process after the transmitter disables inter-frame predictive coding based on LTR frames, as provided in the embodiments of this specification. Figure 5 As shown, the transmitting end can determine the first video frame to be encoded after disabling inter-frame predictive coding based on LTR frames, assuming it is the m-th frame. Next, the transmitting end can determine the difference between the amount of data encoded as a keyframe and the amount of data encoded as a prediction frame for this first video frame, that is, the difference in the amount of data between encoding the m-th frame as an I-frame (keyframe) and encoding it as a P-frame (prediction frame).

[0060] Encoding the m-th frame as an I-frame means that the m-th frame needs to be fully encoded, and its data volume is generally greater than that of encoding the m-th frame as a P-frame. Although encoding the m-th frame as an I-frame has a large data volume and high transmission cost, it no longer needs to use the last frame encoded before the handover (i.e., the m-1-th frame) as the reference frame, so the transmission reliability of this encoding is higher. Therefore, in order to balance transmission cost and transmission reliability, if the difference between the data volume of encoding the m-th frame as an I-frame and encoding it as a P-frame is greater than the preset data volume difference, that is, the transmission cost of encoding as an I-frame is much greater than that of encoding as a P-frame, the transmitting end can use the inter-frame predictive coding of STR frames, with the m-1-th frame (i.e., the last frame encoded by inter-frame predictive coding based on LTR frames before the handover) as the reference frame, to encode the m-th frame as a P-frame. If the difference in data volume between encoding the m-th frame as an I-frame and encoding it as a P-frame is no greater than the preset data volume difference, that is, the transmission cost of encoding it as an I-frame is roughly the same as the transmission cost of encoding it as a P-frame, then the sending end can directly encode the m-th frame as an I-frame based on the inter-frame predictive coding of the STR frame to take into account transmission reliability.

[0061] To further improve transmission reliability, when the transmitting end switches back to the default STR-based inter-frame predictive coding (IPC) from LTR-based to STR-based IPC, additional transmission protection is implemented for encoded video frames transmitted during a certain period of the switching process, such as forward error correction-based transmission protection. Specifically, still using... Figure 5 To illustrate, if the m-th frame is encoded as an I-frame, the sending end can transmit all video frames from the m-th frame to the terminating video frame in the video frame to be sent to the receiving end using this additional transmission protection method after encoding. The terminating video frame is the last video frame encoded by the sending end using the default STR-based inter-frame predictive coding when the m-th frame is received by the receiving end. Assume... Figure 5 When the m-th frame is received by the receiving end, the last video frame encoded by the sending end using the default STR-based inter-frame predictive coding is the m+2-th frame. Therefore, the sending end needs to transmit the three frames from the m-th frame to the m+2-th frame to the receiving end using the transmission protection method described above.

[0062] Specifically, after encoding the m-th frame, the sending end can transmit it to the receiving end using the above-mentioned transmission protection method. After performing inter-frame prediction coding based on STR frames on subsequent video frames, the sending end can also transmit them to the receiving end using the above-mentioned transmission protection method until it receives a response message from the receiving end indicating that the m-th frame has been received.

[0063] Still with Figure 5For example, if the m-th frame is encoded as a P-frame, the sending end can use the reference frame from the m-th frame in the video to be sent as the starting video frame. All video frames from this starting frame to the ending video frame will be transmitted to the receiving end using this additional transmission protection method after encoding. That is, the four frames from the (m-1)-th to the (m+2)-th frames in the video to be sent will be transmitted to the receiving end using the aforementioned transmission protection method. Specifically, after encoding the m-th frame, the sending end can construct a protection frame from the encoded (m-1)-th frame and the m-th frame and transmit it to the receiving end using the aforementioned transmission protection method. After performing inter-frame predictive coding based on STR frames on subsequent video frames, all will be transmitted to the receiving end using the aforementioned transmission protection method until a response message indicating that the m-th frame has been received is received from the receiving end.

[0064] Using the above method, the transmitting end can gradually switch from LTR-based inter-frame predictive coding back to the default STR-based inter-frame predictive coding in a progressive manner, balancing transmission cost and transmission reliability.

[0065] Furthermore, in the embodiments of this specification, the preset weights for determining the network quality degradation characterization values ​​in step S100 and the preset weights for determining the importance characterization values ​​of each encoded video frame in step S102 are adjustable. Specifically, the transmitting end can record network parameters and effect parameters for characterizing the coding effect during the inter-frame predictive coding process based on LTR frames. The network parameters can be used to characterize the current network environment of the transmitting end, including the aforementioned network bandwidth, packet loss rate, network latency, and network jitter, and can also be used to characterize the current geographical location, network segment, and network service provider of the transmitting end. The effect parameters may include the image quality corresponding to the encoded video frame and the correct decoding rate of the receiving end.

[0066] The sending end can upload the recorded network parameters and effect parameters to the server for adjusting the weights. The server can then fit the functional relationship between the weights, network parameters, and effect parameters based on the network parameters and effect parameters sent by multiple sending ends. For each sending end, the server determines the weights that maximize the effect parameters based on the sending end's network parameters and the fitted functional relationship, and then sends the determined weights back to that sending end. The sending end then uses the weights sent by the server as adjusted weights and reconfigures them in the video encoding process provided in this manual.

[0067] In this manual, when the server fits the above functional relationship and determines the weights that maximize the effect parameters, it can use a machine learning model or directly use optimization algorithms such as greedy algorithms. This manual does not restrict this.

[0068] The above is a video encoding method provided by the embodiments of this specification. Based on the same idea, this specification also provides corresponding devices, storage media and electronic devices.

[0069] Figure 6 This is a schematic diagram of a video encoding device provided in an embodiment of this specification. The device includes: The monitoring module 600 is used to monitor the current network quality degradation characterization value when performing inter-frame predictive coding based on short-term reference STR frames on each video frame in the video to be sent. The determination module 601 is used to determine the current network latency if the current network quality degradation characterization value is greater than a preset degradation threshold. Selection module 602 is used to select an LTR frame from the encoded video frames based on the current network latency and the storage space used to store the Long-Term Reference (LTR) frame. The enable module 603 is used to store the selected LTR frames into the storage space, and perform inter-frame predictive coding based on LTR frames on the uncoded video frames in the video to be sent, according to the LTR frames stored in the storage space.

[0070] Optionally, the monitoring module 600 is specifically used to: determine the confidence level of the current network environment belonging to each preset network degradation environment; and determine the current network quality degradation characterization value based on the confidence level of the current network environment belonging to each network degradation environment and the weights preset for each network degradation environment.

[0071] Optionally, the monitoring module 600 is specifically configured to: when the preset network degradation environment includes a weak network environment, determine the confidence level that the current network environment belongs to the weak network environment based on at least one of the current network bandwidth, packet loss rate, network latency, and network jitter; when the preset network degradation environment includes a suspected weak network environment, if the high-precision periodic signal used to measure network latency has not been received from the receiving end after the last transmission of the high-precision periodic signal, determine the confidence level that the current network environment belongs to the suspected weak network environment based on the time length from the last transmission of the high-precision periodic signal to the current time; when the preset network degradation environment includes a slow feedback environment, determine the confidence level that the current network environment belongs to the slow feedback environment based on the current round-trip time (RTT).

[0072] Optionally, the determining module 601 is specifically used to determine the current network latency based on the current round-trip latency, target latency, and redundancy latency.

[0073] Optionally, the selection module 602 is specifically configured to: determine the number of frames within the time corresponding to the current network latency, based on the current network latency and the frame rate of the video to be sent, as the expected number of frames; determine whether the expected number of frames is not greater than the maximum number of frames that the storage space used to store LTR frames can store; if so, select the number of encoded video frames preceding the expected number of frames of the current video frame to be encoded in the video to be sent as the selected LTR frames; otherwise, select the maximum number of encoded video frames among the number of encoded video frames preceding the expected number of frames of the current video frame to be encoded as the selected LTR frames.

[0074] Optionally, the selection module 602 is specifically configured to, for each encoded video frame among the several encoded video frames preceding the video frame to be encoded, determine an importance characterization value for the encoded video frame based on at least one of the following: distance from the encoded video frame to the video frame to be encoded, transmission cost of transmitting the encoded video frame, image complexity of the encoded video frame, and image quality of the encoded video frame; the importance characterization value is negatively correlated with the distance and the transmission cost, and positively correlated with the image complexity and the image quality; and select the several encoded video frames with the largest importance characterization value among the several encoded video frames preceding the current video frame to be encoded.

[0075] Optionally, the selection module 602 is specifically configured to: determine a first dimension representation value based on the distance from the encoded video frame to the video frame to be encoded; determine a second dimension representation value based on the transmission cost of transmitting the encoded video frame; determine a third dimension representation value based on the image complexity of the encoded video frame; determine a fourth dimension representation value based on the image quality of the encoded video frame; and weight the first dimension representation value, the second dimension representation value, the third dimension representation value, and the fourth dimension representation value according to preset weights for each of the first dimension representation value, the second dimension representation value, the third dimension representation value, and the fourth dimension representation value respectively, to obtain the importance representation value of the encoded video frame.

[0076] Optionally, the selection module 602 is further configured to, after selecting the largest number of encoded video frames with the highest importance characterization value, determine whether the selected largest number of encoded video frames with the highest importance characterization value contains encoded video frames that have already been received by the receiver; if so, then the selected largest number of encoded video frames with the highest importance characterization value are used as the selected LTR frames; otherwise, the encoded video frame with the lowest importance characterization value among the selected encoded video frames is removed, and the encoded video frame with the highest importance characterization value that has already been received by the receiver among the previous expected number of encoded video frames of the video frame to be encoded is used to replace the removed encoded video frame, and the selected encoded video frames after replacement are used as the selected LTR frames.

[0077] Optionally, the enabling module 603 is further configured to, after performing LTR-based inter-frame predictive coding on the unencoded video frames in the video to be sent, if the current network quality degradation characterization value is not greater than a preset degradation threshold and the duration of LTR-based inter-frame predictive coding has exceeded a preset duration, then disable LTR-based inter-frame predictive coding and perform STR-based inter-frame predictive coding on the currently unencoded video frames in the video to be sent.

[0078] Optionally, the enabling module 603 is specifically configured to: determine the first video frame to be encoded after disabling inter-frame predictive coding based on LTR frames; determine the difference between the amount of data used to encode the first video frame to be encoded as a keyframe and the amount of data used to encode it as a prediction frame; if the difference is greater than a preset data difference, then using the last encoded video frame before disabling inter-frame predictive coding based on LTR frames as a reference frame, and using inter-frame predictive coding based on STR frames to encode the first video frame to be encoded as a prediction frame; if the difference is not greater than a preset data difference, then using inter-frame predictive coding based on STR frames to encode the first video frame to be encoded as a keyframe.

[0079] Optionally, the enabling module 603 is further configured to, after performing STR-based inter-frame predictive coding on the currently uncoded video frames in the video to be sent, if the first video frame to be encoded after disabling LTR-based inter-frame predictive coding is encoded as a key frame, then all video frames in the video to be sent from the first video frame to the terminating video frame are transmitted to the receiving end using transmission protection; if the first video frame to be encoded after disabling LTR-based inter-frame predictive coding is encoded as a prediction frame, then a reference frame for the first video frame to be encoded is determined as the starting video frame, and all video frames in the video to be sent from the starting video frame to the terminating video frame are transmitted to the receiving end using transmission protection; wherein, the terminating video frame is the last video frame encoded by STR-based inter-frame predictive coding when the first video frame to be encoded is received by the receiving end.

[0080] Optionally, the device further includes: The configuration module 604 is used to record network parameters and effect parameters used to characterize the coding effect during the inter-frame predictive coding process based on LTR frames; upload the network parameters and the effect parameters to the server, so that the server adjusts the preset weights according to the network parameters and the effect parameters; and receive the adjusted weights sent by the server.

[0081] This specification also provides a computer-readable storage medium storing a computer program that, when executed by a processor, can be used to perform the video encoding method provided above.

[0082] This specification also provides a computer program product comprising a computer program that, when executed by a processor, implements the video encoding method described above.

[0083] based on Figure 1 The video encoding method shown in this specification is further illustrated in the embodiments. Figure 7 The diagram shows the structure of the electronic device. Figure 7 At the hardware level, the electronic device includes a processor, internal bus, network interface, memory, and non-volatile memory, and may also include other hardware required for the business. The processor reads the corresponding computer program from the non-volatile memory into memory and then runs it to implement the video encoding method described above.

[0084] The above description is merely an embodiment of this specification and is not intended to limit this specification. Various modifications and variations can be made to this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of the claims of this specification.

Claims

1. A video encoding method, the method comprising: When performing inter-frame predictive coding based on short-term reference STR frames on each video frame in the video to be sent, monitor the current network quality degradation characterization value. If the current network quality degradation value is greater than the preset degradation threshold, then the current network latency is determined. Based on the current network latency and the storage space available for storing Long-Term Reference (LTR) frames, select an LTR frame from the encoded video frames. The selected LTR frames are stored in the storage space, and based on the LTR frames stored in the storage space, inter-frame predictive coding based on LTR frames is performed on the uncoded video frames in the video to be sent.

2. The method as described in claim 1, specifically including: monitoring the current network quality degradation characteristic value, comprising: For each preset network degradation environment, determine the confidence level that the current network environment belongs to that type of network degradation environment; The current network quality degradation characterization value is determined based on the confidence level of the current network environment belonging to each type of network degradation environment and the weights pre-set for each type of network degradation environment.

3. The method as described in claim 2, determining the confidence level that the current network environment belongs to this type of network degradation environment, specifically includes: When the preset network degradation environment includes a weak network environment, the confidence level that the current network environment belongs to the weak network environment is determined based on at least one of the current network bandwidth, packet loss rate, network latency and network jitter. When the preset network degradation environment includes a suspected weak network environment, if the high-precision periodic signal used to measure network latency has not been received from the receiver after the last high-precision periodic signal was sent, the confidence level that the current network environment belongs to the suspected weak network environment is determined based on the time length from the last time the high-precision periodic signal was sent to the current time. When the preset network degradation environment includes a slow feedback environment, the confidence level that the current network environment belongs to the slow feedback environment is determined based on the current round-trip time (RTT).

4. The method as described in claim 1, wherein determining the current network latency specifically includes: Determine the current network latency based on the current round-trip latency, target latency, and redundancy latency.

5. The method of claim 1, wherein selecting an LTR frame from the encoded video frames based on the current network latency and the storage space for storing the Long Term Reference (LTR) frame, specifically includes: Based on the current network latency and the frame rate of the video to be sent, determine the number of frames within the time corresponding to the current network latency, as the expected number of frames; Determine whether the expected number of frames is not greater than the maximum number of frames that the storage space used to store LTR frames can store; If so, select several encoded video frames that are the expected number of frames before the current video frame to be encoded in the video to be sent as the selected LTR frame; Otherwise, from the number of encoded video frames preceding the current expected video frame to be encoded, the maximum number of encoded video frames are selected as the selected LTR frame.

6. The method as described in claim 5, wherein selecting the maximum number of encoded video frames from the expected number of encoded video frames preceding the current video frame to be encoded, specifically includes: For each encoded video frame among the several encoded video frames preceding the video frame to be encoded, an importance characterization value is determined based on at least one of the following: distance from the encoded video frame to the video frame to be encoded, transmission cost of transmitting the encoded video frame, image complexity of the encoded video frame, and image quality of the encoded video frame; the importance characterization value is negatively correlated with the distance and the transmission cost, and positively correlated with the image complexity and the image quality. Among the previously encoded video frames that are expected to be encoded in the current video frame to be encoded, select the largest number of encoded video frames with the highest importance characterization value.

7. The method of claim 6, wherein determining the importance characterization value of the encoded video frame based on at least one of the distance from the encoded video frame to the video frame to be encoded, the transmission cost of transmitting the encoded video frame, the image complexity of the encoded video frame, and the image quality of the encoded video frame, specifically includes: The first dimension representation value is determined based on the distance between the encoded video frame and the video frame to be encoded; The second dimension characterization value is determined based on the transmission cost of transmitting the encoded video frame; the third dimension characterization value is determined based on the image complexity of the encoded video frame; and the fourth dimension characterization value is determined based on the image quality of the encoded video frame. Based on the preset weights for the first dimension representation value, the second dimension representation value, the third dimension representation value, and the fourth dimension representation value, respectively, the importance representation value of the encoded video frame is obtained by weighting the first dimension representation value, the second dimension representation value, the third dimension representation value, and the fourth dimension representation value.

8. The method of claim 6, after selecting the largest number of encoded video frames with the highest importance characterization values, the method further includes: Determine whether the largest number of encoded video frames with the highest importance representation value contain encoded video frames that have already been received by the receiver; If so, the largest number of encoded video frames with the highest importance representation value will be selected as the LTR frames; Otherwise, the encoded video frame with the smallest importance characterization value among the selected encoded video frames is removed, and the encoded video frame with the largest importance characterization value that the receiver has already received from the previous expected number of encoded video frames of the video frame to be encoded is used to replace the removed encoded video frame. The selected encoded video frames after replacement are then used as the selected LTR frames.

9. The method of claim 1, further comprising, after performing LTR-based inter-frame predictive coding on the uncoded video frames in the video to be sent, the method further comprising: If the current network quality degradation value is not greater than the preset degradation threshold, and the duration of inter-frame predictive coding based on LTR frames has exceeded the preset duration, then inter-frame predictive coding based on LTR frames is turned off, and inter-frame predictive coding based on STR frames is performed on the currently uncoded video frames in the video to be sent.

10. The method as described in claim 9, wherein inter-frame predictive coding based on STR frames is performed on the currently unencoded video frames in the video to be sent, specifically including: Determine to disable the first video frame to be encoded after inter-frame predictive coding based on LTR frames; Determine the difference between the amount of data used to encode the first video frame to be encoded as a keyframe and the amount of data used to encode it as a prediction frame; If the difference is greater than the preset data volume difference, then the last encoded video frame before the inter-frame predictive coding based on LTR frames is turned off is used as the reference frame, and the first video frame to be encoded is encoded as a prediction frame based on STR frames. If the difference is not greater than the preset data volume difference, then the first video frame to be encoded is encoded as a key frame based on the inter-frame predictive coding of the STR frame.

11. The method of claim 9, further comprising, after performing STR-based inter-frame predictive coding on the currently uncoded video frames in the video to be sent, the method further comprising: If the first video frame to be encoded after disabling inter-frame predictive coding based on LTR frames is encoded as a key frame, then all video frames in the video to be sent, from the first video frame to the terminating video frame, are transmitted to the receiving end using transmission protection. If the first video frame to be encoded after disabling inter-frame predictive coding based on LTR frames is encoded as a predictive frame, then the reference frame of the first video frame to be encoded is determined as the starting video frame, and all video frames in the video to be sent from the starting video frame to the ending video frame are transmitted to the receiving end using transmission protection. The terminating video frame is the last video frame encoded by inter-frame predictive coding based on STR frames when the first video frame to be encoded is received by the receiving end.

12. The method of claim 2 or 7, further comprising: During the inter-frame predictive coding process based on LTR frames, network parameters and performance parameters used to characterize the coding effect are recorded. The network parameters and the effect parameters are uploaded to the server, so that the server can adjust the preset weights according to the network parameters and the effect parameters; Receive the adjusted weights sent by the server.

13. A video encoding apparatus, the apparatus comprising: The monitoring module is used to monitor the current network quality degradation characterization value when performing inter-frame predictive coding based on short-term reference STR frames on each video frame in the video to be sent. The determination module is used to determine the current network latency if the current network quality degradation characterization value is greater than a preset degradation threshold. The selection module is used to select an LTR frame from the encoded video frames based on the current network latency and the storage space used to store the Long-Term Reference (LTR) frame. An enabling module is used to store selected LTR frames into the storage space and perform LTR frame-based inter-frame predictive coding on the uncoded video frames in the video to be sent based on the LTR frames stored in the storage space.

14. A computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in any one of claims 1-12.

15. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the method according to any one of claims 1-12.

16. A computer program product comprising a computer program that, when executed by a processor, implements the method described in any one of claims 1-12.

Images