A video stream processing method and apparatus

By estimating the forwarding latency of video frames through network devices and instructing the sending device to reduce the bit rate, the problem of high latency in network transmission of video streams is solved, thus improving the user experience of video live streaming services.

CN123123619APending Publication Date: 2026-11-17HUAWEI TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202510638444.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-05-16
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

High latency in network-transmitted video streams causes video stuttering and negatively impacts user experience.

Method used

Network devices estimate the forwarding latency of video frames and send an indication message to the sending device when the latency exceeds a threshold, instructing it to reduce the bit rate of the video stream to alleviate bandwidth pressure and avoid congestion.

Benefits of technology

This effectively avoids video stream congestion at network devices, improving the user experience quality of live video streaming services.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123123619A_ABST
    Figure CN123123619A_ABST
Patent Text Reader

Abstract

This application provides a video stream processing method applied to a network device. The network device estimates the forwarding delay of a first video frame, which is a video frame sent from a sending device to the network device and belongs to the video stream. If the forwarding delay is greater than or equal to a first delay threshold, the network device sends indication information to the sending device. This indication information instructs the sending device to reduce the bitrate of the video stream starting from the first video frame, thereby alleviating the bandwidth pressure on the network device when forwarding the video stream and effectively preventing congestion at the network device, thus ensuring the transmission quality of the video stream in the network. In one example, in a scenario where the video stream is a live video service, the solution provided in this application can prevent users from experiencing video stuttering, thereby improving the QoE of the live video service.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of communications, and in particular to a video stream processing method and apparatus. Background Technology

[0002] With the development of computer and network technologies, various video streaming services have emerged, such as cloud gaming, cloud desktops, virtual reality (VR), and augmented reality (AR). For video streaming services, servers can send video streams to terminal devices, allowing users to watch the corresponding videos on their devices.

[0003] The video stream sent from the server to the terminal device needs to be transmitted over the network. Therefore, the quality of the video stream transmitted over the network will also affect the quality of experience (QoE) of the aforementioned live video services. For example, if the latency of the video stream transmitted over the network is high, users may experience video stuttering when watching the video, resulting in a poor viewing experience.

[0004] Ensuring the quality of video streams transmitted over the network is a problem that urgently needs to be solved. Summary of the Invention

[0005] This application provides a video stream processing method and apparatus that can ensure the quality of video streams transmitted over a network.

[0006] Firstly, this application provides a video stream processing method applicable to network devices. The network device can estimate its forwarding latency for a first video frame, which is a video frame sent from a sending device to the network device and belongs to a video stream. After estimating the forwarding latency of the first video frame, the network device can compare the forwarding latency with a first latency threshold. If the forwarding latency is greater than or equal to the first latency threshold, it indicates that the forwarding latency of the first video frame is relatively large, and the video stream is likely to experience congestion at the network device. In this case, the network device sends an indication message to the sending device, instructing the sending device to reduce the bitrate of the video stream starting from the first video frame, thereby alleviating the bandwidth pressure on the network device when forwarding the video stream and effectively preventing congestion at the network device, thus ensuring the transmission quality of the video stream in the network. In one example, in a scenario where the video stream corresponds to a live video service, this solution can prevent users from experiencing video stuttering, thereby improving the QoE of the live video service.

[0007] In one possible implementation, the indication information includes the frame number of the first video frame. This allows the transmitting device to determine from which video frame to begin reducing the video stream bitrate, based on the frame number. For example, if the frame number of the first video frame is k, then the indication information carries that frame number k, and accordingly, the transmitting device can begin reducing the video stream bitrate from the video frame with frame number k.

[0008] In one possible implementation, the indication information further includes activation information, which is also used to instruct the transmitting device to include the activation information in the video frames it sends to the network device after reducing the bitrate. In this way, the network device can distinguish whether the video frames it receives were sent by the transmitting device before the bitrate reduction or after the bitrate reduction.

[0009] In one possible implementation, the estimated forwarding latency of the network device for the first video frame can be calculated by the network device based on the latency of forwarding video frames queued before the first video frame and the latency of forwarding the unforwarded portion of the first video frame. After estimating its own forwarding latency for the first video frame, the network device can determine the congestion status of the video stream at the network device and thus decide on further processing measures.

[0010] In one possible implementation, the video frames queued before the first video frame are video frames belonging to the same video stream as the first video frame. In this scenario, the video frames queued before the first video frame can be video frames whose queuing position in the target queue is earlier than that of the first video frame. The target queue is the queue used by the network device to forward the video stream to which the first video frame belongs. In a specific example, assuming the current bandwidth used by the network device to forward the video stream is a first bandwidth, the latency for the network device to forward the video frames queued before the first video frame can be determined based on the first bandwidth and the length of the video frames queued before the first video frame in the target queue.

[0011] In one possible implementation, the delay of the network device forwarding the unforwarded portion of the first video frame is determined based on a first bandwidth, a second bandwidth, the frame rate of the first video frame, and a first ratio. The first ratio is the ratio between the length of the first video frame not yet forwarded by the network device and the total length of the first video frame. The first bandwidth is the bandwidth currently used by the network device to forward the video stream, and the second bandwidth is the bandwidth historically used by the network device to forward the video stream. Since the bandwidth used by the network device to forward the first video frame may change dynamically, determining the delay of the network device forwarding the unforwarded portion of the first video frame considers both the first and second bandwidths, thereby accurately determining the delay of the network device forwarding the unforwarded portion of the first video frame.

[0012] In one possible implementation, considering that in practical applications, a decrease in the bandwidth used by a network device to forward video streams can easily cause congestion at the network device, in one example, assuming that the aforementioned first bandwidth is less than the second bandwidth, and the first bandwidth is the bandwidth currently used by the network device to forward the video stream, the network device can, in response to the reduction of its bandwidth for forwarding the video stream from the second bandwidth to the first bandwidth, perform the step of estimating the forwarding delay of the first video frame. That is, the network device can estimate the forwarding delay of the first video frame when its own bandwidth for forwarding the video stream decreases.

[0013] In one possible implementation, when the bandwidth used by the network device to forward the video stream decreases from a second bandwidth to a first bandwidth, the network device can further determine whether the bandwidth decrease will cause congestion of the video stream at the network device. Specifically, the network device can determine the time required for the currently included video frames in the target queue to complete forwarding based on the first bandwidth and the current length of the target queue. If the time is greater than or equal to a preset time, it indicates that the time required for the already cached video frames in the target queue to complete forwarding is relatively long. In this case, the probability of video stream congestion at the network device is relatively high. Therefore, the network device can perform the step of estimating the forwarding delay of the first video frame when the time is greater than or equal to the preset time. That is, the network device can estimate the forwarding delay of the first video frame when its own bandwidth for forwarding the video stream decreases and the time required for the already cached video frames in the target queue to complete forwarding is relatively long.

[0014] In one possible implementation, the first latency threshold can be determined based on the time required to retransmit the first video frame after the bitrate reduction. This time can be determined based on the latency of the first video frame after the bitrate reduction being forwarded by the network device, and the round-trip time between the network device and the sending device. In other words, the first latency threshold is determined based on the latency of the first video frame after the bitrate reduction being forwarded by the network device, and the round-trip time between the network device and the sending device. For example, the first latency threshold may be greater than or equal to the sum of the latency of the first video frame after the bitrate reduction being forwarded by the network device and the round-trip time. In this scenario, if the forwarding latency of the first video frame is greater than or equal to the time required to retransmit the first video frame after the bitrate reduction, the network device can send an indication to the sending device to reduce congestion at the network device, reduce the transmission latency of the first video frame in the network, and thus improve the transmission quality of the video stream in the network.

[0015] In one possible implementation, if the forwarding delay is greater than or equal to a first delay threshold, the network device may further send queue length information and / or bandwidth information to the sending device. The queue length information indicates the length of the target queue, and the bandwidth information indicates the bandwidth used by the network device to forward the video stream. The queue length information and / or the bandwidth information are used by the sending device to determine the reduced bitrate of the video stream. Since the queue length information and / or bandwidth information are determined by the network device and sent to the sending device, this queue length information and / or bandwidth information can accurately characterize the network device's ability to forward the video stream, and / or the congestion status of the video stream at the network device. Accordingly, the sending device, based on this queue length information and / or bandwidth information, determines the reduced bitrate of the video stream, ensuring that the determined bitrate matches the network device's ability to forward the video stream, thereby effectively reducing congestion of the video stream at the network device.

[0016] In one possible implementation, the first video frame is a video frame that has already arrived at the network device. In this case, if the forwarding delay is greater than or equal to a first delay threshold, the network device, in addition to sending an indication message to the sending device, can also discard the first video frame buffered in the target queue. That is, the network device actively discards video frames in the target queue to avoid video frame congestion at the network device.

[0017] In one possible implementation, the first video frame is a video frame that has not yet reached the network device. In this case, the network device can also receive the first video frame sent by the sending device. If the received first video frame does not include the activation information, it indicates that the received first video frame was sent to the network device by the sending device before the bitrate was reduced. In this case, the network device can discard the received first video frame that does not include the activation information. If the first video frame includes the activation information, it indicates that the received first video frame was sent to the network device by the sending device after the bitrate was reduced. In this case, the network device can forward the received first video frame that includes the activation information.

[0018] Secondly, this application provides a video stream processing method that can be applied to a sending device. The sending device can receive indication information sent by a network device, the indication information instructing the sending device to reduce the bitrate of the video stream starting from a first video frame, where the first video frame belongs to the video stream. After receiving the indication information, the sending device can, in response to receiving the indication information, reduce the bitrate of the video stream starting from the first video frame and send the video frames with reduced bitrate to the network device. The video frames with reduced bitrate include at least the first video frame. Since the amount of data that needs to be transmitted is reduced compared to the video frames before the bitrate reduction (for example, before the bitrate reduction, the first video frame includes 100 packets, while after the bitrate reduction, the first video frame only includes 70 packets), the sending device reducing the bitrate of the video stream starting from the first video frame and sending the video frames with reduced bitrate to the network device can alleviate the bandwidth pressure on the network device for forwarding the video stream, effectively preventing congestion of the video stream at the network device, thereby ensuring the transmission quality of the video stream in the network. In one example, in a scenario where the video stream corresponds to a live video service, this solution can prevent users from experiencing video buffering, thereby improving the QoE of the live video service.

[0019] In one possible implementation, the indication information includes the frame number of the first video frame.

[0020] In one possible implementation, the indication information further includes activation information, which is also used to: instruct the sending device to carry the activation information in the video frame sent to the network device after reducing the bit rate, wherein the activation information is included in the first video frame after the bit rate reduction.

[0021] In one possible implementation, the method further includes: receiving queue length information and / or bandwidth information sent by the network device, wherein the queue length information is used to indicate the length of a target queue, the target queue being a queue used by the network device to forward the video stream, the bandwidth information is used to indicate the bandwidth used by the network device to forward the video stream, and the queue length information and / or the bandwidth information are used by the sending device to determine the reduced bitrate of the video stream.

[0022] In one possible implementation, reducing the bitrate of the video stream starting from the first video frame includes: determining the bitrate of the video stream after reduction based on the queue length information and / or the bandwidth information; and reducing the bitrate of the video stream to the reduced bitrate starting from the first video frame.

[0023] In one possible implementation, after receiving the indication information, the sending device, in addition to reducing the bitrate of the video stream, can also reduce the rate at which it sends the video stream to the network device, thereby further alleviating the bandwidth pressure on the network device for forwarding the video stream. In this scenario, when the sending device sends video frames with reduced bitrates to the network device, in a specific implementation, the sending device can send the video frames with reduced bitrates to the network device based on a reduced first rate.

[0024] In one possible implementation, considering the queue length information received by the transmitting device in two consecutive receptions and the time difference between those receptions, the magnitude of the change in the target queue length during the time interval between two consecutive receptions of the indication information can be determined. This magnitude of change in the target queue length also characterizes the magnitude of the change in the rate at which the network device forwards the video stream. Therefore, in one example, the first rate can be determined based on the queue length information received by the transmitting device in two consecutive receptions, the time difference between those receptions, and the second rate before the rate decrease.

[0025] In one possible implementation, considering that in practical applications, if the rate at which the sending device transmits the video stream is reduced too drastically, it will also affect the user experience, for example, making the video playback choppy. Therefore, in one example, a rate reduction threshold can be set to limit the sending device from indefinitely reducing its own video stream transmission rate. In this scenario, the first rate can be determined based on the aforementioned second rate and the rate reduction threshold.

[0026] In one possible implementation, the queue length information received by the sending device can be considered as the queue length forwarded by the network device within the aforementioned time difference. In this scenario, the sending device can determine the first rate based on the aforementioned time difference and the queue length information sent by the network device. That is, the first rate is determined based on the time difference and the queue length information sent by the network device.

[0027] Thirdly, this application provides a video stream processing apparatus applied to a network device. The apparatus includes: a processing unit configured to estimate the forwarding delay of the network device forwarding a first video frame, wherein the first video frame is a video frame sent by a sending device to the network device; and a sending unit configured to send indication information to the sending device if the forwarding delay is greater than or equal to a first delay threshold, wherein the indication information is used to instruct the sending device to reduce the bitrate of the video stream starting from the first video frame, wherein the first video frame belongs to the video stream.

[0028] In one possible implementation, the indication information includes the frame number of the first video frame.

[0029] In one possible implementation, the indication information further includes activation information, which is also used to instruct the sending device to carry the activation information in the video frames it sends to the network device after reducing the bit rate.

[0030] In one possible implementation, the processing unit is specifically configured to: estimate the forwarding delay of the network device for forwarding the first video frame based on the delay of the network device forwarding video frames queued before the first video frame and the delay of the network device forwarding the portion of the first video frame that has not yet been forwarded.

[0031] In one possible implementation, the delay of the network device forwarding video frames queued before the first video frame is determined based on a first bandwidth and the length of video frames queued before the first video frame in a target queue, wherein the first bandwidth is the bandwidth currently used by the network device to forward the video stream, and the target queue is the queue used by the network device to forward the video stream.

[0032] In one possible implementation, the delay of the network device forwarding the portion of the first video frame that has not yet been forwarded is determined based on a first bandwidth, a second bandwidth, the frame rate of the first video frame, and a first ratio. The first ratio is the ratio between the length of the first video frame that has not yet been forwarded by the network device and the total length of the first video frame. The first bandwidth is the bandwidth currently used by the network device to forward the video stream, and the second bandwidth is the bandwidth historically used by the network device to forward the video stream.

[0033] In one possible implementation, the processing unit is specifically configured to: estimate the forwarding delay of the network device for forwarding the first video frame in response to the network device reducing the bandwidth used for forwarding the video stream from the second bandwidth to the first bandwidth.

[0034] In one possible implementation, the processing unit is specifically configured to: in response to the network device reducing the bandwidth used for forwarding the video stream from the second bandwidth to the first bandwidth, determine the time required for the current video frames included in the target queue to complete forwarding based on the first bandwidth and the current length of the target queue, wherein the target queue is the queue used by the network device for forwarding the video stream; and if the time is greater than or equal to a preset time, estimate the forwarding delay of the network device for forwarding the first video frame.

[0035] In one possible implementation, the first latency threshold is determined based on the latency of the first video frame after the bitrate reduction being forwarded by the network device, and the round-trip time between the network device and the sending device.

[0036] In one possible implementation, if the forwarding delay is greater than or equal to a first delay threshold, the sending unit is further configured to: send queue length information and / or bandwidth information to the sending device, wherein the queue length information is used to indicate the length of a target queue, the target queue being a queue used by the network device to forward the video stream, the bandwidth information is used to indicate the bandwidth used by the network device to forward the video stream, and the queue length information and / or the bandwidth information are used by the sending device to determine the reduced bitrate of the video stream.

[0037] In one possible implementation, the first video frame is a video frame that has already arrived at the network device, and the processing unit is further configured to: discard the first video frame cached in the target queue used by the network device to forward the video stream.

[0038] In one possible implementation, the first video frame is a video frame that has not yet arrived at the network device, and the apparatus further includes: a receiving unit, configured to receive the first video frame sent by the sending device; the processing unit is further configured to discard the received first video frame that does not include the validity information if the received first video frame does not include the validity information; or, the sending unit is further configured to forward the received first video frame that includes the validity information if the first video frame includes the validity information.

[0039] Fourthly, this application provides a video stream processing apparatus applied to a transmitting device. The apparatus includes: a receiving unit for receiving indication information sent by a network device, the indication information being used to instruct the transmitting device to reduce the bitrate of a video stream starting from a first video frame, the first video frame belonging to the video stream; a processing unit for reducing the bitrate of the video stream starting from the first video frame in response to receiving the indication information; and a transmitting unit for transmitting a video frame with reduced bitrate to the network device, the video frame with reduced bitrate including the first video frame.

[0040] In one possible implementation, the indication information includes the frame number of the first video frame.

[0041] In one possible implementation, the indication information further includes activation information, which is also used to: instruct the sending device to carry the activation information in the video frame sent to the network device after reducing the bit rate, wherein the activation information is included in the first video frame after the bit rate reduction.

[0042] In one possible implementation, the receiving unit is further configured to: receive queue length information and / or bandwidth information sent by the network device, wherein the queue length information is used to indicate the length of a target queue, the target queue being a queue used by the network device to forward the video stream, the bandwidth information is used to indicate the bandwidth used by the network device to forward the video stream, and the queue length information and / or the bandwidth information are used by the sending device to determine the reduced bitrate of the video stream.

[0043] In one possible implementation, reducing the bitrate of the video stream starting from the first video frame includes: determining the bitrate of the video stream after reduction based on the queue length information and / or the bandwidth information; and reducing the bitrate of the video stream to the reduced bitrate starting from the first video frame.

[0044] In one possible implementation, the processing unit is further configured to: reduce the rate at which the transmitting device transmits the video stream in response to receiving the indication information; the transmitting unit is specifically configured to: transmit video frames with reduced bitrate to the network device based on a first rate after the rate reduction.

[0045] In one possible implementation, the first rate is determined based on the queue length information received by the transmitting device in two consecutive transactions, the time difference between the two consecutive transactions of queue length information received by the transmitting device, and the second rate before the rate reduction; or, the first rate is determined based on the second rate and a rate reduction threshold; or, the first rate is determined based on the time difference and the queue length information sent by the network device.

[0046] Fifthly, this application provides an apparatus (e.g., a video streaming processing apparatus). The apparatus includes a processor and a memory. The memory is used to store instructions or computer programs. The processor is used to execute the instructions or computer programs in the memory to perform the methods described in the first aspect and any one thereof. Alternatively, the processor is used to execute the instructions or computer programs in the memory to perform the methods described in the second aspect and any one thereof.

[0047] Sixthly, this application provides a computer-readable storage medium including instructions or a computer program that, when run on a computer, causes the computer to perform the methods described in the first aspect and any one of the first aspects above, or causes the computer to perform the methods described in the second aspect and any one of the second aspects above.

[0048] In a seventh aspect, this application provides a computer program product comprising instructions or a computer program, which, when run on a computer, causes the computer to perform the method described in any one of the first aspects above, or causes the computer to perform the method described in the second aspect above and any one of the second aspects above.

[0049] Eighthly, this application provides a video stream processing system, the system comprising: a network device and a transmitting device, wherein the network device is configured to perform the method described in the first aspect above and any one of the first aspects above, and the transmitting device is configured to perform the method described in the second aspect above and any one of the second aspects above. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0051] Figure 1 This is a schematic diagram illustrating an exemplary application scenario provided in an embodiment of this application;

[0052] Figure 2 A signaling interaction diagram of a video stream processing method provided in an embodiment of this application;

[0053] Figure 3 This is a schematic diagram of the structure of a video stream processing device provided in an embodiment of this application;

[0054] Figure 4 A schematic diagram of the structure of another video stream processing device provided in the embodiments of this application;

[0055] Figure 5 This is a schematic diagram of the structure of a device provided in an embodiment of this application. Detailed Implementation

[0056] This application provides a video stream processing method and apparatus that can ensure the quality of video streams transmitted over a network. As a specific example, in scenarios where the video stream corresponds to a live video streaming service, this solution can improve the QoE of the live video streaming service.

[0057] See Figure 1 This figure is a schematic diagram of an exemplary application scenario provided by an embodiment of this application. Figure 1 As shown:

[0058] The sending device can send video streams to the terminal device via a network device. The sending device mentioned here may be, for example, a server. The network device mentioned here includes, but is not limited to, devices such as switches or routers; in one example, the network device may be an access device. The terminal device mentioned here includes, but is not limited to, mobile terminal devices such as mobile phones, tablets, or wearable smart devices, as well as terminal devices such as personal computers; these are not all listed here.

[0059] Currently, video streams sent from sending devices to terminal devices via network devices may experience congestion at the network devices, resulting in significant transmission delays. Consequently, terminal devices may not receive the video streams in a timely manner, impacting the user's video viewing experience.

[0060] Video stream congestion at network devices may be caused by insufficient bandwidth for forwarding the video stream. Research has found that reducing the video stream bitrate can alleviate the bandwidth pressure on network devices, thereby reducing congestion. The video stream bitrate can be understood as the encoding bit rate of the video stream.

[0061] Currently, the sending device can detect the bandwidth used by the network device to forward the video stream and adjust the video stream bitrate accordingly. However, the bandwidth detected by the sending device for forwarding the video stream is often inaccurate. Therefore, this method cannot effectively alleviate the congestion of the video stream at the network device.

[0062] In view of this, embodiments of this application provide a video stream processing method and apparatus, which can effectively reduce the congestion of video streams at network devices, thereby ensuring the quality of video streams transmitted over the network.

[0063] Next, with reference to the accompanying drawings, the video stream processing method provided in the embodiments of this application will be described.

[0064] See Figure 2 , Figure 2 This is a schematic diagram of signaling interaction for a video stream processing method provided in an embodiment of this application. Figure 2 The video stream processing method shown can be applied to Figure 1 The application scenarios shown.

[0065] Figure 2 The method shown includes the following steps S101-S105.

[0066] S101: The network device estimates the forwarding delay of the first video frame, which is a video frame sent by the sending device to the network device.

[0067] In this application, the first video frame is a video stream, which is a video stream sent by the sending device to the network device. As an example, the video stream may be a video stream corresponding to a live video streaming service. Live video streaming services mentioned in this application include, but are not limited to, cloud gaming, cloud desktops, VR, or AR services.

[0068] In one example, the first video frame is a video frame that has been sent by the sending device but has not yet arrived at the network device.

[0069] In another example, the first video frame is a video frame that has arrived at the network device. Specifically, the first video frame is a video frame that has arrived at the network device but has not yet been fully forwarded by the network device. In this scenario, some or all of the packets included in the first video frame are cached in the target queue of the network device, which is the queue used by the network device to forward the video stream. Regarding the relationship between video frames and packets, it should be noted that a video frame (e.g., the first video frame) may include multiple packets.

[0070] Because the first video frame is a video frame that has been sent by the sending device but has not yet reached the network device, or the first video frame is a video frame that has reached the network device but has not yet been fully forwarded by the network device, the actual forwarding delay of the network device in forwarding the first video frame is unknown. Therefore, the network device can estimate its own forwarding delay in forwarding the first video frame in order to assess the possibility or degree of congestion of the video stream at the network device.

[0071] In this application, the first video frame may include a plurality of messages. For any message included in the first video frame, the message may include the following information: the frame number of the video frame to which the message belongs (i.e., the first video frame), the total number of messages included in the video frame to which the message belongs, and the message number among all messages included in the first video frame.

[0072] In one example, S101 can be implemented in conjunction with a large model. For instance, at least one of the following information can be input into the large model: the bandwidth information of the network device, the characteristics of the video stream, and the characteristics of the first video frame. The large model can then estimate the forwarding latency of the network device for forwarding the first video frame.

[0073] In another example, in the specific implementation of S101, the network device can estimate the forwarding delay of the first video frame based on the delay of the network device forwarding the video frames queued before the first video frame and the delay of the network device forwarding the portion of the first video frame that has not yet been forwarded. As a specific example, the network device can determine the estimated forwarding delay of the first video frame as the sum of the delay of the network device forwarding the video frames queued before the first video frame and the delay of the network device forwarding the portion of the first video frame that has not yet been forwarded. This can be understood in conjunction with formula (1).

[0074] Formula (1) shows the calculation formula for the estimated forwarding delay of the network device forwarding the first video frame.

[0075] T k =T1+T2 formula (1)

[0076] In formula (1):

[0077] T k This represents the estimated forwarding delay of the network device in forwarding the first video frame, where the subscript k can be the frame number of the first video frame. In the following examples, "forwarding delay of the network device in forwarding the first video frame" refers to the estimated forwarding delay of the network device in forwarding the first video frame.

[0078] T1 represents the latency of the network device forwarding the video frames queued before the first video frame; T1 is an estimated value.

[0079] T2 represents the latency for the network device to forward the portion of the first video frame that has not yet been forwarded; T2 is also an estimated value.

[0080] In one example, the video frame queued before the first video frame is a video frame belonging to the same video stream as the first video frame. In this scenario, the video frame queued before the first video frame can be a video frame whose queuing position in the aforementioned target queue is earlier than that of the first video frame. In a specific example, assuming the current bandwidth used by the network device to forward the video stream is a first bandwidth, the latency for the network device to forward the video frame queued before the first video frame can be determined based on the first bandwidth and the length of the video frame queued before the first video frame in the target queue. For example, the latency for the network device to forward the video frame queued before the first video frame can be equal to the length of the video frame queued before the first video frame in the target queue divided by the first bandwidth. The "current bandwidth (i.e., the first bandwidth)" mentioned here refers to the bandwidth used by the network device to forward the first video frame. In the scenario where the first video frame is a video frame that has not yet arrived at the network device, the first bandwidth refers to the bandwidth used by the network device to forward the video stream when the first video frame arrives at the network device. In this case, the first bandwidth is the bandwidth at a future time, and the first bandwidth is predicted by the network device. In a scenario where the first video frame is a video frame that has already arrived at the network device, the first bandwidth is, for example, the bandwidth that the network device uses to forward the video frame at the "current moment".

[0081] You can refer to formula (2) for understanding. Formula (2) shows one formula for calculating T1.

[0082]

[0083] In formula (2):

[0084] QLen k This indicates the length of the video frames queued in the target queue before the first video frame.

[0085] Bw new This indicates the first bandwidth.

[0086] In one example, the portion of the first video frame that has not yet been forwarded by the network device can be the entire first video frame or a portion thereof. In this application, the latency for the network device to forward the portion of the first video frame that has not yet been forwarded can be determined by combining a first bandwidth, a second bandwidth, the frame rate of the first video frame, and a first ratio. Wherein:

[0087] The first ratio is the ratio between the length of the first video frame that has not yet been forwarded by the network device and the total length of the first video frame, and the value of the first ratio is less than or equal to 1. Specifically, when the portion of the first video frame that has not yet been forwarded by the network device is the entire first video frame, the value of the first ratio is 1; when the portion of the first video frame that has not yet been forwarded by the network device is only a part of the first video frame, the value of the first ratio is less than 1. As an example, the first ratio can be determined based on the total number of packets included in the first video frame and the number of packets in the first video frame that have not yet been forwarded by the network device.

[0088] The second bandwidth refers to the bandwidth historically used by the network device to forward the video stream. Alternatively, the second bandwidth can be understood as the bandwidth used by the network device to forward the portion of the first video frame that has already been forwarded. In one example, the network device can periodically detect its own bandwidth used for forwarding the video stream. The first bandwidth and the second bandwidth can be the bandwidth detected by the network device in two consecutive instances for forwarding the video stream. For example, the first bandwidth is the bandwidth currently detected by the network device for forwarding the video stream, and the second bandwidth is the bandwidth previously detected by the network device for forwarding the video stream. Another example is that the first bandwidth is the bandwidth predicted by the network device for forwarding the video stream when the first video frame arrives at the network device, and the second bandwidth is the bandwidth predicted by the network device for forwarding the video stream before the first video frame arrives at the network device.

[0089] The frame rate of the first video frame refers to the number of frames per second.

[0090] In this application, considering that the bandwidth of the network device is not constant when forwarding the first video frame, the latency of the network device forwarding the unforwarded portion of the first video frame can be estimated by combining the first bandwidth, the second bandwidth, the frame rate of the first video frame, and the first ratio. This can be understood in conjunction with formula (3), which shows a formula for calculating T3.

[0091]

[0092] In formula (3):

[0093] Bw old Indicates the second bandwidth;

[0094] Bw new Indicates the first bandwidth;

[0095] FR represents the frame rate of the first video frame;

[0096] F p This represents the first ratio.

[0097] The bandwidth of the video stream forwarded by the network device at any given time can be calculated according to the following formula (4).

[0098] Bw=(1-Dr)·C i Formula (4)

[0099] In formula (4):

[0100] Bw is the available bandwidth used by the network device to forward the video stream, for example, Bw. old Bw represents the available bandwidth historically used by network devices to forward the video stream. new The available bandwidth currently used by the network device to forward the video stream;

[0101] Dr represents the packet loss rate during which the network device forwards the video stream;

[0102] C i The bandwidth allocated by the network device for the video stream; in one example, C i It can be calculated using the following formula (5).

[0103] C i =C avg +C lef Formula (5)

[0104] In formula (5):

[0105] C avg The average bandwidth allocated by the network device for the video stream; in one example, Where C is the total bandwidth provided by the network device; m is the number of services that fully utilize the allocated average bandwidth; n is the number of services that do not fully utilize the allocated average bandwidth; and m+n represents the total number of services that the network device needs to forward.

[0106] C lef The bandwidth allocated to the video stream is from the unused bandwidth of other services; in one example... Where C′ represents the remaining bandwidth of the network device, and the meaning of m is explained above and will not be repeated here.

[0107] In one example, considering that in practical applications, a decrease in the bandwidth used by a network device to forward video streams can easily cause congestion at the network device, in one example, assuming that the aforementioned first bandwidth is less than the second bandwidth, and the first bandwidth is the bandwidth currently used by the network device to forward the video stream, the network device can execute S101 in response to the reduction of the bandwidth used by the network device to forward the video stream from the second bandwidth to the first bandwidth. That is, the network device can estimate the forwarding delay of the first video frame when its own bandwidth for forwarding the video stream decreases.

[0108] In one example, when the bandwidth used by the network device to forward the video stream is reduced from a second bandwidth to a first bandwidth, the network device can further determine whether the bandwidth reduction will cause congestion of the video stream at the network device. Specifically, the network device can determine the time required for the currently included video frames in the target queue to complete forwarding based on the first bandwidth and the current length of the target queue. For example, the current length of the target queue can be divided by the first bandwidth to obtain the time required for the currently included video frames in the target queue to complete forwarding. After obtaining the time required for the currently included video frames in the target queue to complete forwarding, this time can be compared with a preset time. If the time is greater than or equal to the preset time, it indicates that the time required for the already cached video frames in the target queue to complete forwarding is relatively long. In this case, the probability of video stream congestion at the network device is relatively high. Therefore, the network device can execute S101 when the time is greater than or equal to the preset time. That is, the network device can estimate the forwarding delay of the first video frame when its own bandwidth for forwarding the video stream decreases and the time required for the already cached video frames in the target queue to complete forwarding is relatively long.

[0109] S102: If the forwarding delay is greater than or equal to the first delay threshold, the network device sends an indication message to the sending device. The indication message is used to instruct the sending device to reduce the bit rate of the video stream starting from the first video frame, where the first video frame belongs to the video stream.

[0110] After estimating its forwarding latency for the first video frame, the network device can compare this latency with a first latency threshold to determine the likelihood of congestion at the network device. For example, if the network device's forwarding latency for the first video frame is greater than or equal to the first latency threshold, it indicates a relatively high latency and a greater likelihood of video stream congestion at the network device. In this case, the network device sends an indication to the sending device, instructing it to reduce the video stream bitrate starting from the first video frame. This alleviates the bandwidth pressure on the network device when forwarding the video stream, effectively preventing congestion and ensuring the transmission quality of the video stream throughout the network.

[0111] This application embodiment does not specifically limit the first latency threshold. The first latency threshold can be a preset value, such as an empirical value set by the user. The first latency threshold can also be a value calculated by the network device. As a specific example, the first latency threshold can be determined based on the time required to retransmit the first video frame after the bitrate reduction. The time required to retransmit the first video frame after the bitrate reduction can be determined based on the latency of the first video frame after the bitrate reduction being forwarded by the network device and the round-trip time between the network device and the sending device. For example, the first latency threshold is greater than or equal to the sum of the latency of the first video frame after the bitrate reduction being forwarded by the network device and the round-trip time. Refer to formula (6) for further understanding:

[0112] Formula (6) shows a way to calculate the time required to retransmit the first video frame after the bit rate is reduced.

[0113]

[0114] In formula (6):

[0115] T′ k This indicates the time required to retransmit the first video frame after the bitrate is reduced.

[0116] RTT AP This represents the round-trip time between the sending device and the network device;

[0117] FR represents the frame rate of the first video frame after the bitrate is reduced. In order to ensure that the quality of the video frame remains stable, the frame rate of the video frame will also be adaptively changed after the bitrate of the video stream is reduced.

[0118] β is a scaling factor (its value can be greater than 1, for example, set to 2.5) used to account for variations in frame size, encoding time, and transmission delay.

[0119] In one example, the aforementioned first time delay threshold can be greater than or equal to T′. k In this scenario, if the forwarding delay of the first video frame before the network device reduces the bitrate is greater than or equal to the time required to retransmit the first video frame after the bitrate reduction, the network device sends an indication message to the sending device to instruct the sending device to reduce the bitrate of the video stream starting from the first video frame.

[0120] In one example, the indication information includes the frame number of the first video frame, so that the transmitting device can determine from which video frame to begin reducing the bitrate of the video stream based on the frame number. For example, if the frame number of the first video frame is k, then the indication information carries the frame number k, and accordingly, the transmitting device can start reducing the bitrate of the video stream from the video frame with frame number k.

[0121] In another example, the indication information also includes activation information. Accordingly, the indication information is further used to instruct the sending device to include the activation information in the video frames it sends to the network device after the bitrate reduction. This allows the network device to distinguish whether the video frames it receives are those sent by the sending device before the bitrate reduction or those sent after the bitrate reduction. This application does not specifically limit the activation information; the activation information may be, for example, a timestamp, such as a timestamp greater than the current timestamp plus the aforementioned round-trip time.

[0122] As described above, in one example, the first video frame may be a video frame that has already arrived at the network device. In this case, if the forwarding delay is greater than or equal to a first delay threshold, the network device, in addition to sending an indication message to the sending device, may also discard the first video frame cached in the target queue. Furthermore, for other second video frames cached in the target queue with frame numbers greater than the first video frame's frame number, the network device also discards the second video frame. The second video frame mentioned here refers to the video frame sent to the network device by the sending device before reducing the video stream's bitrate. In other words, the network device actively discards video frames (e.g., the first and second video frames) in the target queue to avoid video frame congestion at the network device.

[0123] As described above, in another example, the first video frame may be a video frame that has not yet reached the network device. In this case, after executing S102, the network device may receive the first video frame sent by the sending device. In this case, after receiving the first video frame, the network device can further determine whether the first video frame was sent to the network device before the sending device reduced the bitrate of the video stream, or whether the sending device sent the video stream to the network device after reducing the bitrate of the video stream. If the first video frame was sent to the network device before the sending device reduced the bitrate of the video stream, the network device can directly discard the received first video frame. If the first video frame was sent to the network device after the sending device reduced the bitrate of the video stream, the network device can forward the received first video frame.

[0124] In a specific example, the network device can determine whether the received first video frame is a first video frame sent to the network device before the sending device reduced the video stream bitrate, or a first video stream sent to the network device after the sending device reduced the video stream bitrate, based on whether the first video frame includes the aforementioned activation information. Specifically, if the first video frame received by the network device includes activation information, it can be determined that the first video frame was sent to the network device after the sending device reduced the video stream bitrate. In this case, the network device can forward the received first video frame that includes the activation information. If the first video frame received by the network device does not include activation information, it can be determined that the first video frame was sent to the network device before the sending device reduced the video stream bitrate. In this case, the network device can discard the received first video frame that does not include the activation information.

[0125] In some examples, when the latency of the network device forwarding the first video frame is greater than or equal to a first latency threshold, the network device, in addition to sending indication information to the sending device, may also send other auxiliary information to the sending device. This auxiliary information allows the sending device to determine the reduced bitrate of the video stream based on the auxiliary information. The auxiliary information indicates the network device's ability to forward the video stream and / or the congestion status of the video stream at the network device. Since this auxiliary information is determined by the network device and sent to the sending device, it accurately characterizes the network device's ability to forward the video stream and / or the congestion status of the video stream at the network device. Accordingly, the sending device's determination of the reduced bitrate of the video stream based on this auxiliary information ensures that the determined bitrate matches the network device's ability to forward the video stream, thereby effectively reducing congestion at the network device.

[0126] In one example, considering the bandwidth (e.g., available bandwidth) that the network device uses to forward the video stream, its ability to forward the video stream can be characterized. Therefore, the aforementioned auxiliary information may include bandwidth information indicating the bandwidth the network device uses to forward the video stream. Wherein:

[0127] If the first video frame is a video frame that has already arrived at the network device, then the bandwidth information is used to indicate the network device's current capacity to forward the video stream.

[0128] If the first video frame is a video frame that has not yet arrived at the network device, then the bandwidth information is used to indicate the network device's ability to forward the video stream when the first video frame arrives at the network device. In this scenario, the bandwidth indicated by the bandwidth information can be predicted by the network device. The principle used to predict the bandwidth is described in the previous section on formulas (4) and (5), and will not be repeated here.

[0129] In one example, considering the length of the target queue, which can characterize the congestion of the video stream at the network device to some extent, the aforementioned auxiliary information may further include queue length information, which indicates the length of the target queue. Wherein:

[0130] If the first video frame is a video frame that has already arrived at the network device, then the queue length information is used to indicate the current length of the target queue.

[0131] If the first video frame is a video frame that has not yet arrived at the network device, then the queue length is used to indicate the length of the target queue when the first video frame arrives at the network device. In this scenario, the length of the target queue indicated by the queue length information can be predicted by the network device.

[0132] In one example, the network device can predict the length of the target queue based on the types of data packets in the target queue. The types of data packets in the target queue can include: forwarded and acknowledged data packets, forwarded but lost data packets, forwarded but unacknowledged data packets, queued but not yet forwarded data packets, and data packets sent by the sending device but not yet arrived at the network device. In some scenarios, "data packets" can also be referred to as "messages."

[0133] Forwarded and acknowledged data packets: will not introduce any delay.

[0134] Forwarded but lost packets: will be re-entered into the forwarding queue (e.g., the destination queue), increasing the queue length.

[0135] Forwarded but unacknowledged data packets: Some of these may be confirmed as lost, increasing the queue length associated with the packet loss rate Dr.

[0136] Packets that are queued but not yet forwarded: These packets directly contribute to the queue length.

[0137] Data packets sent by the sending device but not yet arrived at the network device: These data packets may continue to arrive at the network device and increase the queue length.

[0138] In one example, the queue lengths for the five types of data packets mentioned above are denoted as PL1, PL2, PL3, PL4, and PL5, respectively, where: PL1 represents the queue length for forwarded and acknowledged data packets; PL2 represents the queue length for forwarded but lost data packets; PL3 represents the queue length for forwarded but unacknowledged data packets; PL4 represents the queue length for queued but not yet forwarded data packets; and PL5 represents the queue length for data packets sent by the sending device but not yet arrived at the network device. Therefore, the length of the target queue can be calculated using the following formula (7).

[0139] QLen=PL2+Dr·PL3+PL4+PL5-RTT AP ·Bw formula (7)

[0140] In formula (7):

[0141] QLen represents the length of the target queue;

[0142] Dr represents the packet loss rate during which the network device forwards the video stream;

[0143] Bw represents the available bandwidth of the network device for forwarding the video stream; specifically, if the first video frame is a video frame that has already arrived at the network device, then Bw represents the available bandwidth of the network device for forwarding the video stream; if the first video frame is a video frame that has not yet arrived at the network device, then Bw represents the available bandwidth of the network device for forwarding the video stream when the first video frame arrives at the network device.

[0144] RTT AP This represents the round-trip time between the sending device and the network device.

[0145] S103: The sending device receives the indication information sent by the network device.

[0146] S104: In response to receiving the indication information, the transmitting device reduces the bitrate of the video stream starting from the first video frame.

[0147] S105: The transmitting device sends a video frame with a reduced bitrate to the network device, the video frame with a reduced bitrate including the first video frame.

[0148] After the network device sends the indication information to the sending device, the sending device can receive the indication information and, in response to receiving the indication information, reduce the bitrate of the video stream starting from the first video frame.

[0149] The transmitting device can first determine the reduced bitrate of the video stream, and then encode the video frames starting from the first video frame in the video stream using the reduced bitrate, and send the reduced bitrate video frames to the network device. The reduced bitrate video frames mentioned here refer to video frames encoded using the reduced bitrate. As described above, the first video frame in the reduced bitrate video frames is the first video frame, and the reduced bitrate video frames include not only the first video frame but also other video frames (e.g., the second video frame) with frame sequence numbers greater than the first video frame. The reduced bitrate video frames sent by the transmitting device include the aforementioned activation information to facilitate the network device's distinction between video frames before and after the bitrate reduction. Specifically, the activation information may be carried in the message included in the video frame.

[0150] In addition, for the messages included in the video frames after the bitrate reduction sent by the sending device, the messages may also include: the frame number of the video frame to which the message belongs, the total number of messages included in the video frame to which the message belongs, and the number of the message in all the messages included in the video frame to which the message belongs, etc., which will not be described in detail here.

[0151] Because the amount of data that needs to be transmitted in a video frame after the bitrate is reduced compared to the video frame before the bitrate reduction—for example, before the bitrate reduction, the first video frame included 100 packets, while after the bitrate reduction, the first video frame only includes 70 packets—the sending device can reduce the bitrate of the video stream from the first video frame and send the video frames with the reduced bitrate to the network device. This can alleviate the bandwidth pressure on the network device when forwarding the video stream, effectively avoid congestion at the network device, and thus ensure the transmission quality of the video stream in the network.

[0152] In one example, the transmitting device can determine the reduced bitrate of the video stream based on its own bitrate reduction strategy.

[0153] In another example, where the network device also sends auxiliary information to the sending device, the sending device can determine the reduced bitrate of the video stream based on the auxiliary information sent by the network device. As a specific example, the sending device can determine the reduced bitrate of the video stream based on queue length information and / or bandwidth information sent by the sending device. For example, the sending device can determine the reduced bitrate of the video stream according to the following formula (8).

[0154]

[0155] In formula (8):

[0156] CR indicates the reduced bitrate of the video stream;

[0157] Bw represents the bandwidth indicated by the aforementioned bandwidth information;

[0158] QLen represents the length of the target queue indicated by the aforementioned queue information;

[0159] t is the emptying time of the target queue, which can be an empirical value.

[0160] In one example, after receiving the indication information, the sending device, in addition to reducing the bitrate of the video stream, can also reduce the rate at which it sends the video stream to the network device, thereby further alleviating the bandwidth pressure on the network device for forwarding the video stream. In this scenario, in a specific implementation of S105, the sending device can send video frames with reduced bitrate to the network device based on the reduced first rate.

[0161] Next, we will introduce several possible methods for determining the first velocity.

[0162] In one example, the network device can periodically execute the aforementioned S101-S103, thereby instructing the sending device to dynamically adjust the bitrate of the video stream during the forwarding process. The queue length information received by the sending device in two consecutive receptions, and the time difference between the two consecutive receptions of the queue length information by the sending device, can determine the magnitude of the change in the target queue during the time between the two consecutive receptions of the instruction information by the sending device. The magnitude of the change in the target queue can also characterize the magnitude of the change in the rate at which the network device forwards the video stream. Therefore, in one example, the first rate can be determined based on the queue length information received by the sending device in two consecutive receptions, the time difference between the two consecutive receptions of the queue length information by the sending device, and the second rate before the rate decrease. As a specific example, it can be understood in conjunction with formula (9), which shows a formula for calculating the first rate.

[0163]

[0164] In formula (9):

[0165] SndRate represents the first rate;

[0166] SndRate′ represents the second rate;

[0167] QLen new and QLen old This indicates the queue length indicated by two consecutive queue length messages received by the sending device, where QLen new It is the queue length indicated by the queue length information received by the sending device for the (i+1)th time, QLen. old It is the queue length indicated by the queue length information received by the sending device for the i-th time, where i is an integer greater than 1.

[0168] ΔT represents the aforementioned time difference.

[0169] In another example, considering that in practical applications, if the rate at which the sending device transmits the video stream is reduced too much, it will also affect the user experience, for example, making the video unplayable. Therefore, in one example, a rate reduction threshold can be set to limit the sending device from reducing its own video stream rate indefinitely. In this scenario, the first rate can be determined based on the aforementioned second rate and the rate reduction threshold. This can be understood with reference to formula (10), which shows another formula for calculating the first rate.

[0170] SndRate=γ×SndRate′ Formula (10)

[0171] In formula (10):

[0172] SndRate represents the first rate;

[0173] SndRate′ represents the second rate;

[0174] γ represents the deceleration threshold.

[0175] In one example, assuming the video stream is not severely congested at the network device, the sending device can calculate the first rate using both formulas (9) and (10), and then determine the larger of the two as the final first rate. That is,

[0176] As an example, the sending device can determine whether severe congestion has occurred at the network device based on the aforementioned queue length information. For instance, the sending device can determine that severe congestion has occurred at the network device if the queue lengths indicated by the queue length information received multiple times consecutively are very long. Conversely, if the queue length indication information received by the sending device is not long, it can be determined that severe congestion has not occurred at the network device.

[0177] In another example, where severe congestion occurs at the network device in the video stream, the queue length information received by the sending device can be considered as the queue length forwarded by the network device within the aforementioned time difference. In this scenario, the sending device can determine the first rate based on the aforementioned time difference and the queue length information sent by the network device. This can be understood with reference to formula (11). Formula (11) shows another formula for calculating the first rate.

[0178]

[0179] In formula (11):

[0180] SndRate represents the first rate;

[0181] QLen represents the queue length indicated by the queue length information received by the sending device. For example, QLen can correspond to the aforementioned QLen. new ;

[0182] ΔT represents the aforementioned time difference.

[0183] In one example, in the event of severe congestion of the video stream at the network device, the transmitting device can calculate the first rate using both formula (11) and formula (10), and then determine the larger of the two as the final first rate. That is,

[0184] In one example, to mitigate queue length (referring to the length of the target queue) fluctuations caused by measurement errors and transient network conditions, the network device can perform multiple consecutive measurements to filter noise and provide a more accurate queue length. Additionally, before adjusting its own video stream transmission rate, the transmitting device applies a smoothing algorithm to process the queue length indicated by the received queue length information. This approach ensures more stable rate adjustment, preventing excessive fluctuations in the video stream transmission rate from affecting the user's video viewing experience. The smoothing algorithms mentioned here include, but are not limited to, the exponentially weighted moving average (EWMA) algorithm.

[0185] In one example, if the network device's estimated forwarding delay for the first video frame is less than the aforementioned first delay threshold, the network device may not drop the first video frame. Correspondingly, the network device does not need to send the aforementioned indication information to the sending device. In this scenario, the network device can send the current length of the target queue to the sending device so that the sending device can determine the congestion status of the video stream at the network device based on this queue length information. Alternatively, the sending device may perform other processing based on the current length of the target queue; this embodiment does not impose specific limitations on these processing methods.

[0186] Based on the video stream processing method provided in the above embodiments, this application also provides a corresponding video stream processing apparatus. The video stream processing apparatus provided in this application will be described below.

[0187] See Figure 3 The figure is a schematic diagram of the structure of a video stream processing device provided in an embodiment of this application.

[0188] Figure 3The video stream processing apparatus shown is applied to a network device and is used to execute the methods provided in the above embodiments that are executed by the network device, for example, to execute the above... Figure 2 The steps performed by the network device in the method shown.

[0189] like Figure 3 As shown: The device 300 includes a processing unit 301 and a sending unit 302.

[0190] The processing unit 301 is used to estimate the forwarding delay of the network device in forwarding the first video frame, which is a video frame sent by the sending device to the network device.

[0191] The sending unit 302 is configured to send indication information to the sending device if the forwarding delay is greater than or equal to a first delay threshold. The indication information is used to instruct the sending device to reduce the bitrate of the video stream starting from the first video frame, where the first video frame belongs to the video stream.

[0192] In one possible implementation, the indication information includes the frame number of the first video frame.

[0193] In one possible implementation, the indication information further includes activation information, which is also used to instruct the sending device to carry the activation information in the video frames it sends to the network device after reducing the bit rate.

[0194] In one possible implementation, the processing unit 301 is specifically configured to: estimate the forwarding delay of the network device for forwarding the first video frame based on the delay of the network device forwarding video frames queued before the first video frame and the delay of the network device forwarding the portion of the first video frame that has not yet been forwarded.

[0195] In one possible implementation, the delay of the network device forwarding video frames queued before the first video frame is determined based on a first bandwidth and the length of video frames queued before the first video frame in a target queue, wherein the first bandwidth is the bandwidth currently used by the network device to forward the video stream, and the target queue is the queue used by the network device to forward the video stream.

[0196] In one possible implementation, the delay of the network device forwarding the portion of the first video frame that has not yet been forwarded is determined based on a first bandwidth, a second bandwidth, the frame rate of the first video frame, and a first ratio. The first ratio is the ratio between the length of the first video frame that has not yet been forwarded by the network device and the total length of the first video frame. The first bandwidth is the bandwidth currently used by the network device to forward the video stream, and the second bandwidth is the bandwidth historically used by the network device to forward the video stream.

[0197] In one possible implementation, the processing unit 301 is specifically configured to: estimate the forwarding delay of the network device for forwarding the first video frame in response to the network device reducing the bandwidth used for forwarding the video stream from the second bandwidth to the first bandwidth.

[0198] In one possible implementation, the processing unit 301 is specifically configured to: in response to the network device reducing the bandwidth used for forwarding the video stream from the second bandwidth to the first bandwidth, determine the time required for the current video frames included in the target queue to complete forwarding based on the first bandwidth and the current length of the target queue, wherein the target queue is the queue used by the network device for forwarding the video stream; if the time is greater than or equal to a preset time, estimate the forwarding delay of the network device for forwarding the first video frame.

[0199] In one possible implementation, the first latency threshold is determined based on the latency of the first video frame after the bitrate reduction being forwarded by the network device, and the round-trip time between the network device and the sending device.

[0200] In one possible implementation, if the forwarding delay is greater than or equal to a first delay threshold, the sending unit 302 is further configured to: send queue length information and / or bandwidth information to the sending device, wherein the queue length information is used to indicate the length of a target queue, the target queue being a queue used by the network device to forward the video stream, and the bandwidth information is used to indicate the bandwidth used by the network device to forward the video stream, and the queue length information and / or the bandwidth information are used by the sending device to determine the reduced bitrate of the video stream.

[0201] In one possible implementation, the first video frame is a video frame that has already arrived at the network device, and the processing unit 301 is further configured to: discard the first video frame cached in the target queue used by the network device to forward the video stream.

[0202] In one possible implementation, the first video frame is a video frame that has not yet arrived at the network device, and the apparatus further includes: a receiving unit, configured to receive the first video frame sent by the sending device; the processing unit 301 is further configured to discard the received first video frame that does not include the validity information if the received first video frame does not include the validity information; or, the sending unit 302 is further configured to forward the received first video frame that includes the validity information if the first video frame includes the validity information.

[0203] See Figure 4 The figure is a schematic diagram of another video stream processing device provided in an embodiment of this application.

[0204] Figure 4 The video stream processing apparatus shown is applied to a transmitting device and is used to execute the methods provided in the above embodiments by the transmitting device, for example, to execute the above... Figure 2 The steps performed by the sending device in the method shown.

[0205] like Figure 4 As shown: The device 400 includes: a receiving unit 401, a processing unit 402, and a sending unit 403.

[0206] The receiving unit 401 is used to receive indication information sent by the network device. The indication information is used to instruct the sending device to reduce the bit rate of the video stream starting from the first video frame, where the first video frame belongs to the video stream.

[0207] The processing unit 402 is configured to reduce the bitrate of the video stream starting from the first video frame in response to receiving the indication information.

[0208] The sending unit 403 is used to send a video frame with a reduced bitrate to the network device, the video frame with a reduced bitrate including the first video frame.

[0209] In one possible implementation, the indication information includes the frame number of the first video frame.

[0210] In one possible implementation, the indication information further includes activation information, which is also used to: instruct the sending device to carry the activation information in the video frame sent to the network device after reducing the bit rate, wherein the activation information is included in the first video frame after the bit rate reduction.

[0211] In one possible implementation, the receiving unit 401 is further configured to: receive queue length information and / or bandwidth information sent by the network device, wherein the queue length information is used to indicate the length of a target queue, the target queue being a queue used by the network device to forward the video stream, the bandwidth information is used to indicate the bandwidth used by the network device to forward the video stream, and the queue length information and / or the bandwidth information are used by the sending device to determine the reduced bitrate of the video stream.

[0212] In one possible implementation, reducing the bitrate of the video stream starting from the first video frame includes: determining the bitrate of the video stream after reduction based on the queue length information and / or the bandwidth information; and reducing the bitrate of the video stream to the reduced bitrate starting from the first video frame.

[0213] In one possible implementation, the processing unit 402 is further configured to: reduce the rate at which the transmitting device sends the video stream in response to receiving the indication information; the sending unit 403 is specifically configured to: send video frames with reduced bitrate to the network device based on a first rate after the rate reduction.

[0214] In one possible implementation, the first rate is determined based on the queue length information received by the transmitting device in two consecutive transactions, the time difference between the two consecutive transactions of queue length information received by the transmitting device, and the second rate before the rate reduction; or, the first rate is determined based on the second rate and a rate reduction threshold; or, the first rate is determined based on the time difference and the queue length information sent by the network device.

[0215] For details on the specific implementation of each unit of the devices 300 and 400, please refer to the relevant description of the video stream processing method provided in this application above; the description will not be repeated here.

[0216] In this application, the hardware structure of the aforementioned video stream processing devices 300 and 400 can be as follows: Figure 5 The structure shown, Figure 5 This is a schematic diagram of the structure of a device provided in an embodiment of this application.

[0217] Please see Figure 5 As shown, device 500 includes: processor 510, communication interface 520, and memory 530. The number of processors 510 in device 500 may be one or more. Figure 5 Taking a processor as an example. In this embodiment, the processor 510, communication interface 520, and memory 530 can be connected via a bus system or other means, wherein, Figure 5Taking the connection between China and Israel via the 540 bus system as an example.

[0218] Processor 510 may be a central processing unit (CPU), a network processor (NP), or a combination of a CPU and an NP. Processor 510 may further include hardware chips. These hardware chips may be application-specific integrated circuits (ASICs), programmable logic devices (PLDs), or combinations thereof. The PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0219] Memory 530 may include volatile memory, such as random-access memory (RAM); memory 530 may also include non-volatile memory, such as flash memory, hard disk drive (HDD), or solid-state drive (SSD); memory 530 may also include combinations of the above types of memory. Figure 5 When the device 500 shown is a network device, the memory 530 can be used, for example, to store the aforementioned first latency threshold; when Figure 5 When the device 500 shown is a transmitting device, the memory 530 can be used, for example, to store the aforementioned queue length information and / or bandwidth information.

[0220] Optionally, the memory 530 stores an operating system and programs, executable modules, or data structures, or subsets thereof, or extended sets thereof. The programs may include various operation instructions for implementing various operations. The operating system may include various system programs for implementing various basic services and handling hardware-based tasks. The processor 510 can read the programs in the memory 530 to implement the video stream processing method provided in this embodiment.

[0221] The bus system 540 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus system 540 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 5 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0222] This application also provides a computer-readable storage medium, including instructions or a computer program, which, when run on a computer, causes the computer to perform the video stream processing method provided in the above embodiments. For example, it causes the computer to perform the method steps provided in the above embodiments that are executed by a network device (e.g., Figure 2 For example, causing a computer to execute the method steps performed by the sending device as shown in the above embodiments (e.g., Figure 2 (The steps performed by the sending device are shown).

[0223] This application also provides a computer program product comprising instructions or a computer program, which, when run on a computer, causes the computer to perform the video stream processing method provided in the above embodiments. For example, it causes the computer to perform the method steps performed by a network device as provided in the above embodiments; or, for example, it causes the computer to perform the method steps performed by a sending device as provided in the above embodiments.

[0224] This application also provides a video stream processing system, which includes the network device and transmitting device mentioned in the above embodiments. The network device is used to perform the operations performed by the network device provided in the above embodiments (e.g., Figure 2 The steps shown are performed by the network device. The sending device is used to perform the operations performed by the sending device as provided in the above embodiments (e.g., Figure 2 (The steps performed by the sending device are shown).

[0225] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0226] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0227] In the embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical business division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0228] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0229] Furthermore, the various business units in the embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software business unit.

[0230] If the integrated unit is implemented as a software business unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0231] Those skilled in the art will recognize that, in one or more of the examples above, the services described in this invention can be implemented using hardware, software, firmware, or any combination thereof. When implemented in software, these services can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein communication media include any medium that facilitates the transmission of computer programs from one place to another. Storage media can be any available medium accessible to general-purpose or special-purpose computers.

[0232] The above specific embodiments further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above are merely specific embodiments of the present invention.

[0233] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A method for processing a video stream, characterized by, Applied to network devices, the method includes: Estimate the forwarding delay of the network device for the first video frame, where the first video frame is a video frame sent from the sending device to the network device; If the forwarding delay is greater than or equal to the first delay threshold, an indication message is sent to the sending device. The indication message is used to instruct the sending device to reduce the bitrate of the video stream starting from the first video frame, where the first video frame belongs to the video stream.

2. The method of claim 1, wherein, The indication information includes the frame number of the first video frame.

3. The method according to claim 1 or 2, characterized in that, The indication information also includes activation information, which is further used to instruct the sending device to carry the activation information in the video frames it sends to the network device after reducing the bit rate.

4. The method according to any one of claims 1 to 3, characterized in that, The estimated forwarding delay of the network device for the first video frame includes: Based on the latency of the network device forwarding video frames queued before the first video frame and the latency of the network device forwarding the portion of the first video frame that has not yet been forwarded, the forwarding latency of the network device for the first video frame is estimated.

5. The method of claim 4, wherein, The latency for the network device to forward video frames queued before the first video frame is determined based on a first bandwidth and the length of the video frames queued before the first video frame in the target queue. The first bandwidth is the bandwidth currently used by the network device to forward the video stream, and the target queue is the queue used by the network device to forward the video stream.

6. The method according to claim 4 or 5, characterized in that, The delay of the network device forwarding the portion of the first video frame that has not yet been forwarded is determined based on a first bandwidth, a second bandwidth, the frame rate of the first video frame, and a first ratio. The first ratio is the ratio between the length of the first video frame that has not yet been forwarded by the network device and the total length of the first video frame. The first bandwidth is the bandwidth currently used by the network device to forward the video stream, and the second bandwidth is the bandwidth historically used by the network device to forward the video stream.

7. The method of claim 6, wherein, The estimated forwarding delay of the network device for the first video frame includes: In response to the network device reducing the bandwidth used for forwarding the video stream from the second bandwidth to the first bandwidth, the forwarding delay of the network device for forwarding the first video frame is estimated.

8. The method of claim 7, wherein, In response to the network device reducing the bandwidth used for forwarding the video stream from the second bandwidth to the first bandwidth, estimating the forwarding delay of the network device for forwarding the first video frame includes: In response to the network device reducing the bandwidth used for forwarding the video stream from the second bandwidth to the first bandwidth, the time required for the video frames currently included in the target queue to complete forwarding is determined based on the first bandwidth and the current length of the target queue, wherein the target queue is the queue used by the network device for forwarding the video stream; If the duration is greater than or equal to the preset duration, the forwarding delay of the network device in forwarding the first video frame is estimated.

9. The method according to any one of claims 1 to 8, characterized in that, The first latency threshold is determined based on the latency of the first video frame after the bitrate reduction being forwarded by the network device, and the round-trip time between the network device and the sending device.

10. The method according to any one of claims 1 to 9, characterized in that, If the forwarding delay is greater than or equal to the first delay threshold, the method further includes: The network device sends queue length information and / or bandwidth information to the sending device. The queue length information indicates the length of the target queue, which is the queue used by the network device to forward the video stream. The bandwidth information indicates the bandwidth used by the network device to forward the video stream. The queue length information and / or the bandwidth information are used by the transmitting device to determine the reduced bitrate of the video stream.

11. The method according to any one of claims 1 to 10, characterized in that, The first video frame is a video frame that has already arrived at the network device, and the method further includes: The first video frame cached in the target queue used by the network device to forward the video stream is discarded.

12. The method of claim 3, wherein, The first video frame is a video frame that has not yet arrived at the network device, and the method further includes: Receive the first video frame sent by the sending device; If the received first video frame does not include the activation information, the received first video frame that does not include the activation information is discarded; or, if the first video frame includes the activation information, the received first video frame that includes the activation information is forwarded.

13. A method for processing a video stream, the method comprising: Applied to a transmitting device, the method includes: The device receives an instruction message sent by a network device, the instruction message being used to instruct the sending device to reduce the bit rate of the video stream starting from the first video frame, the first video frame belonging to the video stream; In response to receiving the indication information, the bitrate of the video stream is reduced starting from the first video frame, and the bitrate-reduced video frames are sent to the network device, the bitrate-reduced video frames including the first video frame.

14. The method of claim 13, wherein, The indication information includes the frame number of the first video frame.

15. The method according to claim 13 or 14, characterized in that, The indication information also includes activation information, and the indication information is further used to: instruct the sending device to carry the activation information in the video frame sent to the network device after reducing the bit rate, and the first video frame after the bit rate reduction includes the activation information.

16. The method according to any one of claims 13-15, characterized in that, The method further includes: The sending device receives queue length information and / or bandwidth information sent by the network device. The queue length information is used to indicate the length of the target queue, which is the queue used by the network device to forward the video stream. The bandwidth information is used to indicate the bandwidth used by the network device to forward the video stream. The queue length information and / or the bandwidth information are used by the sending device to determine the reduced bitrate of the video stream.

17. The method of claim 16, wherein, The step of reducing the bitrate of the video stream starting from the first video frame includes: Based on the queue length information and / or the bandwidth information, determine the reduced bitrate of the video stream; Starting from the first video frame, the bitrate of the video stream is reduced to the reduced bitrate of the video stream.

18. The method according to any one of claims 13-17, characterized by, The method further includes: In response to receiving the indication information, the sending device reduces the rate at which it sends the video stream; Sending the video frames with reduced bitrate to the network device includes: Based on the first rate after the rate reduction, video frames with reduced bitrate are sent to the network device.

19. The method according to claim 18, characterized in that, The first rate is determined based on the queue length information received by the transmitting device in two consecutive receptions, the time difference between the two consecutive receptions of queue length information by the transmitting device, and the second rate before the rate reduction; or, The first rate is determined based on the second rate and the deceleration threshold; or, The first rate is determined based on the time difference and the queue length information sent by the network device.

20. A video stream processing apparatus, characterized by comprising: The device includes multiple functional modules that interact with each other to implement the method as described in any one of claims 1-19.

21. A video stream processing device, comprising a processor and a memory, the memory for storing program code, the processor for calling the program code in the memory to cause the communication device to perform the method as described in any one of claims 1-19.

22. A computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the method as described in any one of claims 1-19.

23. A computer program product, characterised in that, Includes program code that, when a computer runs the computer program product, causes the computer to perform the method as described in any one of claims 1-19.

24. A video stream processing system, characterized by The system includes: A network device that performs the method according to any one of claims 1-12, and a transmitting device that performs the method according to any one of claims 13-19.