Obtaining video frame state from rate control feedback

Abstract

An encoding system (110) receives a report from a decoding system (120). The report identifies a packet that includes a first video portion of the video stream and includes feedback indicating whether the decoding system (120) successfully receives the first video portion. The encoding system (110) configures the video encoder (115) to encode a second video portion of the video stream based on whether the feedback indicates that the first video portion is successfully received. Correspondingly, a decoding system (120) sends the report and decodes the second video portion according to the encoding.

Description

Technical Field

[0001] The present disclosure relates generally to video streaming within a network, and more particularly to techniques for enhanced video coding. Background Art

[0002] There are several cloud-based applications, such as gaming or augmented reality (AR), that benefit from generating frames to be displayed to the end user at a remote server (e.g., in a data center) rather than on the end device. Such applications can stream these frames from the server to the edge device. To reduce the network resources required for such streaming, video encoding technologies such as H.264 and H.265 (also known as High Efficiency Video Coding (HEVC)) have been applied. In addition to reducing network resources, these applications have traditionally set strict end-to-end latency requirements, which necessitates configuring the encoder to minimize the time required to encode and transmit the video stream. Summary of the Invention

[0003] The present disclosure generally relates to an encoding system that tracks the reception status of transmitted packets and determines whether frames carried by the packets are available at a decoding system. The encoding system configures a video encoder based on this information. A corresponding decoding system provides the reception status of packets transmitted by the encoding system and decodes the video according to the video encoder.

[0004] Embodiments of the present disclosure include a method implemented by an encoding system. The method includes receiving a report from a decoding system. The report identifies a packet comprising a first video portion of a video stream. The report includes feedback indicating whether the first video portion was successfully received by the decoding system. The method also includes configuring a video encoder to encode a second video portion of the video stream based on whether the feedback indicates that the first video portion was successfully received.

[0005] In some embodiments, configuring the video encoder based on the feedback comprises, in response to the feedback indicating that the first video portion was not successfully received, configuring the video encoder to encode the second video portion without reference to the first video portion. In some such embodiments, configuring the video encoder to encode the second video portion without reference to the first video portion comprises configuring the video encoder to designate a video slice comprising the first video portion as intra-coded. In other such embodiments, configuring the video encoder to encode the second video portion without reference to the first video portion comprises configuring the video encoder to encode the second video portion as an Instantaneous Decoding Refresh (IDR) frame. In some embodiments, configuring the video encoder to encode the second video portion as an IDR frame is in response to the feedback, together with at least one further feedback received from the decoding system, indicating that an entirety of a video frame comprising the first video portion was not successfully received.

[0006] In some embodiments, configuring the video encoder based on the feedback comprises, in response to the feedback indicating that the first video portion was successfully received, configuring the video encoder to encode the second video portion using a video frame comprising the first video portion as a reference frame. In some such embodiments, configuring the video encoder to encode the second video portion using the video frame as the reference frame is further in response to the feedback, together with at least one further feedback received from the decoding system, indicating that an entirety of the video frame was successfully received at the decoding system.

[0007] In some embodiments, the feedback indicates that the first video portion was successfully received by indicating that a video slice comprising the first video portion was successfully received at the decoding system.

[0008] In some embodiments, the method further comprises, in a data store, storing, for each packet of a plurality of packets carrying respective portions of the video stream, an entry associating a packet identifier identifying that packet with a frame identifier identifying a video frame comprising the respective portion and with decoder feedback indicating whether the respective portion is decodable by the decoding system. In some such embodiments, the method further comprises adding, to the data store based on the report, a further entry for that packet.

[0009] In some embodiments, the method further comprises determining a one-way network delay between a time of transmission of the packet from the encoding system and a time of receipt of the packet at the decoding system. The method further comprises interpreting the feedback as indicating whether the first video portion is decodable or not decodable based on whether the one-way network delay is less than a threshold, respectively.

[0010] Other embodiments include an encoding system comprising an interface circuit and a processing circuit communicatively connected to the interface circuit. The processing circuit is configured to receive a report from a decoding system via the interface circuit. The report identifies a packet comprising a first video portion of a video stream. The report includes feedback indicating whether the first video portion was successfully received by the decoding system. The processing circuit is further configured to configure a video encoder to encode a second video portion of the video stream based on whether the feedback indicates that the first video portion was successfully received.

[0011] In some embodiments, the processing circuit is further configured to perform any of the above methods.

[0012] Other embodiments include a computer program comprising instructions that, when executed on processing circuitry of an encoding system, cause the encoding system to perform any of the methods described above.

[0013] Still other embodiments include a carrier containing the aforementioned computer program, wherein the carrier is one of an electronic signal, an optical signal, a radio signal, or a computer-readable storage medium.

[0014] Embodiments of the present disclosure also include a method implemented by a decoding system. The method includes transmitting a report to an encoding system. The report identifies a packet comprising a first video portion of a video stream. The report includes feedback indicating whether the decoding system successfully received the first video portion. The method also includes receiving additional packets from the encoding system, the additional packets comprising a second video portion of the video stream, the second video portion being encoded with or without reference to the first video portion, respectively, based on whether the feedback indicates that the first video portion was successfully received. The method also includes decoding the second video portion based on how the second video portion was encoded.

[0015] In some embodiments, in response to the feedback indicating that the first video portion was not successfully received, the second video portion is encoded as an Instantaneous Decoding Refresh (IDR) frame. In some such embodiments, in response to the feedback and at least one further feedback transmitted to the encoding system (110) together indicating that the entirety of the video frame including the first video portion was not successfully received, the second video portion is encoded as an IDR frame.

[0016] In some embodiments, in response to the feedback indicating that the first video portion was successfully received, encoding the second video portion using the video frame including the first video portion as a reference frame. In some such embodiments, in response to the feedback and at least one further feedback transmitted to the encoding system together indicating that the entirety of the video frame was successfully received, encoding the second video portion using the video frame including the first video portion as a reference frame.

[0017] In some embodiments, the feedback indicates that the first video portion was successfully received by indicating that a video slice comprising the first video portion was successfully received at the decoding system.

[0018] Other embodiments include a decoding system comprising an interface circuit and a processing circuit communicatively connected to the interface circuit. The processing circuit is configured to transmit a report to an encoding system via the interface circuit. The report identifies a packet comprising a first video portion of a video stream. The report includes feedback indicating whether the decoding system successfully received the first video portion. The processing circuit is further configured to receive, from the encoding system via the interface circuit, additional packets comprising a second video portion of the video stream, the second video portion being encoded with or without reference to the first video portion, respectively, based on whether the feedback indicates that the first video portion was successfully received. The processing circuit is further configured to decode the second video portion based on how the second video portion was encoded.

[0019] In some embodiments, the processing circuit is further configured to perform any of the above methods implemented by the decoding system.

[0020] Other embodiments include a computer program comprising instructions that, when executed on processing circuitry of a decoding system, cause the decoding system to perform any of the above-described decoding system-implemented methods.

[0021] Still other embodiments include a carrier containing the aforementioned computer program, wherein the carrier is one of an electronic signal, an optical signal, a radio signal, or a computer-readable storage medium. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The aspects of the present disclosure are illustrated by way of example and not limitation in the accompanying figures in which like references indicate similar elements.

[0023] Figure 1A and 1B is a schematic block diagram illustrating a network example according to an embodiment of the present disclosure.

[0024] Figure 2is a flow chart illustrating an example method of encapsulating RTP packets according to one or more embodiments of the present disclosure.

[0025] Figure 3 is a flow chart illustrating an example method of tracking packet reception status according to one or more embodiments of the present disclosure.

[0026] Figure 4 is a flowchart illustrating an example method implemented by an encoding system according to one or more embodiments of the present disclosure.

[0027] Figure 5 is a flow chart illustrating an example method implemented by a decoding system according to one or more embodiments of the present disclosure.

[0028] Figure 6 is a block diagram schematically illustrating an example encoding system according to one or more embodiments of the present disclosure.

[0029] Figure 7 is a block diagram schematically illustrating an example decoding system according to one or more embodiments of the present disclosure. DETAILED DESCRIPTION

[0030] Traditional encoders may use information from previously coded frames to encode the current frame. Previously coded frames used in this way are often called reference pictures, and frames encoded using reference pictures are often called predicted frames (P frames). Frames that can initiate a decoding session do not reference other pictures. These frames are often called Instantaneous Decoding Refresh (IDR) frames.

[0031] Using prediction during video frame encoding results in significantly less output. A typical encoder configuration for cloud gaming or AR minimizes both encoding time and required capacity by encoding the video stream as a series of predicted frames, each of which references the previous frame after sending an IDR frame at the beginning. The disadvantage is that if a frame cannot be decoded correctly, the error impairs all subsequent frames that reference the failed frame. Such error propagation often results in visible artifacts over image sequences for long periods of time. The strict latency requirements on remote rendering and streaming solutions often preclude packet loss handling techniques, such as resending missing packets.

[0032] Various approaches can be used to overcome visible artifacts caused by packet loss. For example, the encoder can send IDR frames at regular intervals, regardless of any transmission errors between IDR frames. This limits the amount of error that can propagate. Additionally or alternatively, the encoder can receive an explicit error indication from the decoder and, in response, encode the next frame as an IDR frame. This is commonly referred to as forced IDR. Additionally or alternatively, the encoder can only use successfully decoded frames as reference frames. Successfully decoded reference frames can also be retained by the system and used as references for an unlimited period of time, making them a reliable resource. Such reference frames are commonly referred to as long-term reference frames.

[0033] The data network over which encoded frames are transmitted does not guarantee that all encoded frames will be delivered error-free and on time. For example, packets may be lost due to transmission errors between nodes in the data network. Packets may not arrive at the receiver in time, or even be lost if an intermediate node in the data network does not have sufficient capacity to forward the packet to the next node downstream. In such cases, a node may store the packet in an intermediate queue and, if the queue is full, simply discard the packet.

[0034] The Real-time Transport Protocol (RTP) was developed to deliver coded video frames (among other things) over IP-based networks for use with different encoder types. Because the size of a coded video frame can be larger than the maximum size of a single IP packet to be transmitted as a unit over the network, the standardized RTP encapsulation function splits the contents of the coded frame and encapsulates the resulting fragments into IP packets along with additional headers defined by the RTP protocol designed for the encoder type (e.g., H.265). The encoder provides the coded video frame along with metadata to the encapsulation process.

[0035] To control the data flow, an additional protocol, the Real-time Transport Control Protocol (RTCP), is defined. An RTCP session is created for each RTP session. Among other things, the receiver can provide decoding feedback to the transmitter using RTCP feedback messages. For example, a decoder can send a Picture Loss Indication (PLI) RTCP message to notify the encoder of the loss of coded video data belonging to one or more pictures. As a subsequent action, the encoder can encode the next frame as an IDR. The Reference Picture Selection Indication (RPSI) RTCP message enables the decoder to communicate the identifier of a correctly decoded picture or slice to the encoder. As a result, the encoder can adjust which frames are used as long-term reference frames.

[0036] The throughput of data networks that can be used to encode video streams can vary, especially in wireless networks. Streamers typically detect these throughput changes and adjust the size of the video stream to avoid network congestion, which can lead to increased network latency and packet loss. For this purpose, rate control algorithms are typically utilized. Such algorithms can run on the transmitting side, i.e., where encoding takes place. In some examples, the receiver side generates feedback to the transmitter running the rate control algorithm about RTP packet loss, congestion, and / or timing. In response, the rate control algorithm at the transmitter proposes a target bit rate for the encoded video stream that the encoder should follow. An example transmitter-side rate control solution is the Multimedia Self-Clocked Rate Adaptation (SCReAM).

[0037] While existing PLI and RPSI based indication techniques support the suppression of video artifacts caused by packet loss, these techniques have drawbacks as currently known in the art. For example, the forced IDR generation using PLI typically generates IDR frames that are several times larger than typical P-frames. Thus, when network congestion causes packet loss or unacceptable delay, increasing the throughput can easily lead to additional delay and packet loss.

[0038] As another example, indicating to the encoder via RTCP whether a frame was successfully or unsuccessfully decoded adds latency to the frame-to-frame encoding control loop. For example, when a certain frame cannot be decoded due to packet loss, certain parts of the frame can be missing. The decoder detects this after processing the frame, which can already be stored in a jitter buffer. Then, an RTCP message encoding an indication of the picture loss is generated, transmitted across the network, and processed by the transmitter. All these steps contribute to the reaction delay of the encoder. These delays accumulate and lead to long-time visible video artifacts or stalling if the failed frame is simply discarded.

[0039] Furthermore, these existing techniques typically require the encoder and decoder to use a synchronized set of frame IDs and provide APIs to use these frame IDs, e.g., during the reference frame configuration.

[0040] In view of the above, example embodiments of the present disclosure can improve upon known techniques by having an encoding system estimate whether all segments of a previously transmitted encoded frame are available at a decoding system based on per-packet feedback from the decoding system. The feedback indicates successful arrival of video stream packets. The encoding system can then respond accordingly, e.g., by using frames that were successfully received and decoded as references for subsequent frames that have not yet been encoded. The encoding system can also declare a certain frame to be unsuccessful and generate an IDR frame in response.

[0041] At least some of the embodiments presented herein speed up the frame decoding feedback loop at the expense of a small risk of false positive confirmations. By sending feedback immediately when all required RTP packets have arrived, a 10-40ms improvement is generally expected, depending on the decoding duration (which is typically less than 5ms) and the jitter buffer configuration (which is typically in the 10-40ms range).

[0042] A faster frame decoding feedback loop offers several advantages. For example, the encoder can react to errors earlier, making video artifacts visible for a shorter period of time. Furthermore, combined with reference frame positive validation techniques, frames can be used as references for a longer period of time without risking quality, which in turn enables improved frame quality or smaller frame sizes (through better compression).

[0043] Another advantage may be the ability to reduce the RTCP flow rate without compromising control loop latency. In radio environments, reducing the uplink data rate is important. For example, the current RTCP RPSI message is required to be sent immediately in a dedicated RTCP packet to minimize the latency of receiving feedback. This message has relatively little content and the encoding and packaging cost is significant. While the RTCP RPSI message can be piggybacked into the next rate control feedback message to reduce overhead, this delays the RPSI message until the next rate control feedback is sent. Therefore, the conventional approach increases the control loop latency. Note that this increase also applies to other encoding-related notification messages, such as picture loss indication.

[0044] Figure 1A An example network 10 is shown that includes an encoding system 110 and a decoding system 120, each of which is its own computing system including any number of computing devices or components thereof, as will be described in further detail below. The encoding system 110 and the decoding system 120 each include components that can be implemented by any combination of hardware and software, depending on the particular embodiment. Although for the purpose of clarity of explanation, Figure 1A The encoding system 110 and the decoding system 120 will be described in terms of specific components that perform specific functions. However, these components are merely one way to arrange the functionality described herein. It should be appreciated that in other embodiments, additional, fewer, or different components may be used. Generally speaking, in other embodiments, the functionality of any component may be attributed to the system 110 and 120 in which it is included, such as Figure 1A or as shown in 1B.

[0045] The encoding system 110 and the decoding system 120 communicate with each other over a networked medium (e.g., cable, radio frequency). Although not shown for clarity, there can be any number of intermediary devices (e.g., routers, switches, gateways, proxies) between the encoding system 110 and the decoding system 120 that carry messages between the systems.

[0046] For the purpose of explaining various concepts of the encoding and decoding systems 110, 120, FIG. 1 shows a number of different components within each system. In this example, the encoding system 110 encodes an unencoded video stream into an encoded video stream (e.g., using the video encoder 115), splits the encoded video stream into a series of RTP packets (e.g., using the RTP encapsulator 116), and transmits the RTP packets over a transport network (e.g., via the RTP transmitter 118), preferably such that the network is not overloaded. The decoding system 120 receives the RTP packets (e.g., at the RTP receiver controller 128), reconstructs the encoded video stream (e.g., at the RTP decapsulator 126), and decodes it (e.g., at the video decoder 125). To properly reconstruct the encoded video stream, other components can be included, such as a jitter buffer 126. The RTP transmitter 118 and the RTP receiver 128 each implement a rate control function, as will be further explained below.

[0047] The encoding system 110 also includes an encoder manager 112 and an RTP sequence mapping datastore 114 (sometimes referred to as an RTP-SEQ-MAP datastore). The datastore 114 stores associations between RTP packets and encoded video frames, as well as the reception status of RTP packets. The RTP encapsulator 116 writes entries to the datastore 114. More specifically, the datastore 114 includes entries that establish relationships between an encoded frame, an RTP packet that carries the frame's contents, and the reception status of the frame as known by the transmitter 118.

[0048] In light of the foregoing, an entry within the RTP sequence mapping can include a number of values. For example, an entry can include an encoder-specific identifier of an encoded video frame. This value can be opaque to other components described herein. In some embodiments, an entry can additionally or alternatively include an encoder-specific identifier of one or more portions (e.g., frame slices) of an encoded video frame.

[0049] An entry may additionally or alternatively include one or more RTP packet descriptors. Each descriptor may, for example, include an RTP sequence number of the RTP packet, whether the RTP packet has been received at the decoding system 120, and / or whether the RTP packet has been declared lost. Additionally or alternatively, each RTP packet descriptor may include the transmission time of the RTP packet, the one-way network delay of the RTP packet, and / or an identifier of which parts of the coded frame are carried in the RTP packet (e.g., a slice index).

[0050] An entry may additionally or alternatively include an indication of whether the entry is final (i.e., whether all contents of the coded video frame have been passed to the RTP encapsulator 116). This flag may indicate that no further RTP packets will be registered for this entry. Thus, an indication that all RTP packets are available for decoding means that the entire coded frame is available for decoding.

[0051] The encoder manager 112 receives RTP transmission feedback information from the RTP transmitter 118, reads and modifies the data storage 114, and configures the video encoder 115 accordingly.

[0052] It should be noted that other embodiments may use Figure 1A Other computing architectures that are different from the computing architecture depicted in . For example, Figure 1B The example depicts an encoding system 110 that includes an encoding device 111 and a cloud-based system 119. Figure 1B In , encoding device 111 includes video encoder 115, RTP transport 118, and RTP encapsulator 116, any or each of which may use encoder manager 112 and / or data storage device 114 provided as a cloud-based service by cloud-based system 119.

[0053] In another cloud-based example, the encoder manager 112 might be run as a dedicated process on the same physical hardware device as the device that streams the video to the decoding system 120. Additionally or alternatively, the data storage device 114 may be a shared medium between the encoder manager 112 and the components of the encoding device 111. Other centralized and / or distributed solutions may additionally or alternatively be employed without departing from the inventive aspects of the present disclosure.

[0054] Regardless of the specific arrangement of certain components of the encoding system 110, the RTP encapsulator 116 monitors the construction of RTP packets from the encoded video frame content and generates a series of RTP packets carrying the encoded video frame content. The video encoder 115 can deliver the content of the entire encoded video frame in a single call or in several calls. The RTP encapsulation process may include collecting the identifier, metadata, and RTP sequence number provided by the video encoder 115 and storing these pieces of information in the RTP sequence map data storage device 114. The metadata may include, for example, a presentation timestamp.

[0055] Figure 2 2 is a flow chart illustrating an example RTP encapsulation method 200 according to one or more embodiments of the present disclosure. Method 200 is implemented by encoding system 110, and in some embodiments, more specifically by RTP encapsulator 116. Method 200 includes receiving content that is a portion of an encoded video frame (block 210). The content may be received, for example, along with metadata and an encoder-specific frame identifier provided by video encoder 115. The content may be encoded, for example, as a series of Network Abstraction Layer (NAL) units.

[0056] The method 200 also includes storing the encoder-specific frame identifier and metadata in the RTP sequence map data storage 114 (block 220). When an entry indexed with the frame identifier does not exist, a new entry is created using the frame identifier.

[0057] Method 200 also includes generating an RTP packet for carrying a fragment of the video frame content (block 230). The fragment can be, for example, one or more small NAL units, or a portion of a large NAL unit. As part of this step, a unique sequence number is generated for the RTP packet and included in the corresponding RTP field.

[0058] The method 200 also includes storing the RTP sequence number of the generated RTP packet in an entry in the data storage 114 indexed by the encoder-specific frame identifier (block 240).

[0059] Method 200 also includes checking whether additional RTP packets are needed to encapsulate the contents of the encoded video frame portion (block 250). If so (block 250, "yes" path), the process generates one or more additional RTP packets as previously described (block 230).

[0060] If no additional RTP packet is needed to encapsulate the contents of the encoded video frame portion (block 250, "No" path), the method 200 further includes checking whether the last RTP packet generated is marked as the last (block 260). For example, if the RTP packet includes an RTP marker bit set to 0 (i.e., not 1) (block 260, "No" path), this can indicate that there are additional portions of the same video frame, in which case the method 200 ends (block 280).

[0061] However, if the entire frame has been encapsulated into RTP packets (block 260, "Yes" path), the method 200 further includes marking the entry for the encoded video frame in the RTP sequence mapping data store 114 as final (block 270), and the method 200 ends (block 280).

[0062] Figure 3 Another example method 300 implemented by the encoding system 110 is shown. In some embodiments, the method 300 is more particularly performed by the encoder manager 112. The method 300 includes obtaining a RTP packet delivery report (e.g., from a rate controller such as the RTP receiver 128 and / or the RTP transmitter 118) (block 310). The report includes a sequence number associated with the RTP packet and whether the packet was successfully received on the RTP receiver side (e.g., as opposed to having been deemed lost).

[0063] In some embodiments, the one-way delay and / or other characteristics of the RTP packet are obtained as observed by the rate controller. Such information can be obtained periodically from the rate controller and / or in response to changes in the information. For example, the encoding system 110 and / or the decoding system 120 can determine that a frame has arrived or has been lost, and can generate delivery information in response.

[0064] The contents of the RTP packet delivery report can be reviewed to determine whether the packet has arrived on time or has been lost. For example, the report can include an explicit indication of whether the packet has been received or lost. Additionally or alternatively, the report can include the one-way network delay of the packet, and use the delay as a basis for determining whether the packet has been lost.

[0065] The method 300 further includes checking whether there is an entry for the packet referenced in the report in the data store 114 (block 320). If there is no such entry (block 320, "No" path), the report is ignored, and the method 300 ends (block 350). For example, a report of a packet for which there is no entry in the data store 114 can indicate that the packet was not sent by the encoding system 110, or that some other error has occurred at the encoding system 110 or the decoding system 120.

[0066] If the packet corresponds to an entry in the data storage device 114, the method 300 further includes obtaining a descriptor for the packet from the appropriate entry (block 330). For example, to find the correct entry, the encoding system 110 may iterate through the RTP packet descriptors of different entries until a descriptor is found whose sequence number is equal to the sequence number of the RTP packet.

[0067] Method 300 also includes updating the entry corresponding to the packet in data storage 114 (block 340). For example, encoding system 110 may update the RTP packet descriptor corresponding to the RTP packet with a corresponding indication of whether the packet was received (and, if calculated, the one-way determined network delay). In particular, encoding system 110 may check whether the delivery report indicates that the RTP packet was received or lost and update data storage 114 accordingly. In one specific example, encoding system 110 uses the local RTP packet delivery time and the reception time provided in the report to calculate the one-way network delay of the RTP packet to determine whether the packet was successfully received. Alternatively, the value provided in the report may be used to determine whether the packet was successfully received. In some embodiments, updating the entry may also include updating the entry with the one-way network delay (if available / determined).

[0068] In some embodiments, when determining whether a packet has been successfully received, encoder manager 112 may additionally consider information about the components involved in the packet exchange, for example, so that corresponding entries in data store 114 are appropriately updated. For example, encoding manager 112 may be aware of the receiver-side processing components between RTP receiver 128 and RTP decoder 125, their actual configuration, and the one-way network delay of the RTP packet. Any of these factors (individually or in any combination) may be considered by encoding system 110. For example, packet jitter buffer 126 may discard packets that do not arrive before a deadline determined based on, for example, an RTP timestamp. Encoder manager 112 may estimate the reception deadline for the RTP packet, and if the reported reception time exceeds the estimated reception deadline, encoding system 110 may consider the packet lost and may update the corresponding RTP packet descriptor in data store 114 accordingly. After the appropriate entries have been updated, method 300 ends (block 350).

[0069] The process of updating an entry in data storage 114 may involve one or more checks or processes. For example, when an update to an entry is made, the entire entry may be reviewed to determine whether a determination can be made regarding the ability or inability of decoding system 120 to successfully decode the corresponding frame (e.g., the frame is considered ready for decoding because all portions have been successfully received, or the frame is considered undecodable because it has missing or undecodable portions).

[0070] Particular embodiments may follow one or more rules to make this determination. For example, the encoding system 110 may determine that the entire frame represented by the entry is decodable (e.g., ready for decoding) in response to all RTP packets indicated by the entry as corresponding to the frame being marked as available. In another example, the encoding system 110 may determine that the entire frame is undecodable in response to at least one RTP packet in the entry being marked as lost. In still another example, the encoding system 110 may determine that one or more slices are undecodable in response to identifying that one or more corresponding packets are lost. In other words, if there is at least one RTP packet marked as lost, the frame slice(s) carried in the lost RTP packet may be considered undecodable.

[0071] The encoding system 110 may configure the video encoder 115 based on the results of the above frame review. Specifically, the video encoder 115 may be configured to use a certain frame as a reference frame or to respectfully ignore the frame based on whether the decoding system 120 successfully received the frame.

[0072] For example, the encoder manager 112 may configure the video encoder 115 to use frames determined to be ready for decoding (eg, because they were successfully received) as reference frames.

[0073] In another example, the encoding system 110 configures the video encoder 115 with respect to a particular slice of a frame of interest. For example, in response to the encoding system 110 determining that one or more slices of a frame are undecodable (e.g., because the decoding system 120 has not received one or more packets corresponding to those slices), the encoder manager 112 may configure the video encoder 115 to encode one or more subsequent frames or slices without reference to the undecodable slices. In one specific example, the video encoder 115 may be configured to treat the undecodable slices as intra-coded (i.e., encoded without reference to other slices or frames).

[0074] In yet another example, encoding system 110 may configure video encoder 115 to encode a frame as an IDR frame in response to an entire previously transmitted frame being deemed undecodable, such as where a long-term reference frame scheme is not used for error handling.

[0075] It should also be noted that after using the entry from data store 114 to make a determination regarding whether a frame or slice is decodable and configuring video encoder 115 accordingly as described above, encoding system 110 may then delete the entry data store 114 in some embodiments.

[0076] In view of the above, the embodiments of the present disclosure include, for example, Figure 4. The method 400 is implemented by the encoding system 110 shown in FIG. The method 400 includes receiving a report from the decoding system 120 (block 410). The report identifies packets comprising a first video portion of a video stream. Furthermore, the report includes feedback indicating whether the decoding system 120 successfully received the first video portion. The method 400 also includes configuring the video encoder to encode a second video portion of the video stream based on whether the feedback indicates that the first video portion was successfully received (block 420).

[0077] Other embodiments include, for example, Figure 5 . Method 500 is implemented by the decoding system 120 shown in FIG. Method 500 includes transmitting a report to the encoding system 110 (block 510). The report identifies packets comprising a first video portion of a video stream. Furthermore, the report includes feedback indicating whether the decoding system 120 successfully received the first video portion. Method 500 also includes receiving additional packets from the encoding system, the additional packets comprising a second video portion of the video stream, the second video portion being encoded with or without reference to the first video portion, respectively, based on whether the feedback indicates that the first video portion was successfully received (block 520). Method 500 also includes decoding the second video portion based on how the second video portion was encoded (block 530).

[0078] The encoding system 110 may be, for example, Figure 6 The examples shown schematically are implemented. Figure 6 The encoding system 110 includes processing circuitry 610, memory circuitry 620, and interface circuitry 630. The processing circuitry 610 is communicatively coupled to the memory circuitry 620 and the interface circuitry 630, for example, via a bus 604. The processing circuitry 610 may include one or more microprocessors, microcontrollers, hardware circuits, discrete logic circuits, hardware registers, digital signal processors (DSPs), field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), or combinations thereof. For example, the processing circuitry 610 may be programmable hardware capable of executing software instructions stored in the memory circuitry 620, for example, as a machine-readable computer program 640. The memory circuit 620 of various embodiments may include any non-transitory machine-readable medium known or developable in the art, whether volatile or non-volatile, including but not limited to solid-state media (e.g., SRAM, DRAM, DDRAM, ROM, PROM, EPROM, flash memory, solid-state drives, etc.), removable storage devices (e.g., Secure Digital (SD) cards, miniSD cards, microSD cards, memory sticks, thumb drives, USB flash drives, ROM cartridges, universal media disks), fixed drives (e.g., magnetic hard drives), or the like (in whole or in any combination).

[0079] The interface circuit 630 may be a controller hub configured to control input and output (I / O) data paths of the encoding system 110. Such I / O data paths may include data paths for exchanging signals over a network. The interface circuit 630 may be implemented as a single physical component, or as multiple physical components arranged serially or separately, any of which may be communicatively coupled to any other component or may communicate with any other component via the processing circuit 610. For example, the interface circuit 630 may include a transmitter 632 configured to send wireless communication signals and a receiver 634 configured to receive wireless communication signals.

[0080] The encoding system 110 may be configured to perform the method 400 described above. In one example, the processing circuit 610 may be configured to receive a report from the decoding system 120 via the interface circuit 630. The report identifies packets comprising a first video portion of the video stream. In addition, the report includes feedback indicating whether the decoding system 120 successfully received the first video portion. The processing circuit 610 is further configured to configure the video encoder 115 to encode a second video portion of the video stream based on whether the feedback indicates that the first video portion was successfully received.

[0081] Still other embodiments include a control program 640 that includes instructions that, when executed on the processing circuit 610 of the encoding system 110 , cause the encoding system 110 to perform the method 400 described above.

[0082] Still other embodiments include a carrier containing the control program 640. The carrier is one of an electronic signal, an optical signal, a radio signal, or a computer-readable storage medium.

[0083] Correspondingly, the decoding system 120 may be as follows Figure 7 The examples shown schematically are implemented. Figure 7The decoding system 120 includes processing circuitry 710, memory circuitry 720, and interface circuitry 730. Processing circuitry 710 is communicatively coupled to memory circuitry 720 and interface circuitry 730, for example, via bus 704. Processing circuitry 710 may include one or more microprocessors, microcontrollers, hardware circuits, discrete logic circuits, hardware registers, digital signal processors (DSPs), field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), or combinations thereof. For example, processing circuitry 710 may be programmable hardware capable of executing software instructions stored in memory circuitry 720, for example, as a machine-readable computer program 740. The memory circuit 720 of various embodiments may include any non-transitory machine-readable medium known or developable in the art, whether volatile or non-volatile, including but not limited to solid-state media (e.g., SRAM, DRAM, DDRAM, ROM, PROM, EPROM, flash memory, solid-state drives, etc.), removable storage devices (e.g., Secure Digital (SD) cards, miniSD cards, microSD cards, memory sticks, thumb drives, USB flash drives, ROM cartridges, universal media disks), fixed drives (e.g., magnetic hard drives), or the like (in whole or in any combination).

[0084] The interface circuit 730 may be a controller hub configured to control input and output (I / O) data paths of the decoding system 120. Such I / O data paths may include data paths for exchanging signals over a network. The interface circuit 730 may be implemented as a single physical component, or as multiple physical components arranged serially or separately, any of which may be communicatively coupled to any other component or may communicate with any other component via the processing circuit 710. For example, the interface circuit 730 may include a transmitter 732 configured to send wireless communication signals and a receiver 734 configured to receive wireless communication signals.

[0085] The decoding system 120 may be configured to perform the method 500 described above. In one example, the processing circuit 710 is configured to transmit a report to the encoding system 110 via the interface circuit 730. The report identifies a packet comprising a first video portion of the video stream. In addition, the report includes feedback indicating whether the decoding system 120 successfully received the first video portion. The processing circuit 710 is further configured to receive, from the encoding system 110 via the interface circuit 730, additional packets comprising a second video portion of the video stream, the second video portion being encoded with or without reference to the first video portion, respectively, based on whether the feedback indicates that the first video portion was successfully received. The processing circuit 710 is further configured to decode the second video portion based on how the second video portion was encoded.

[0086] Still other embodiments include a control program 740 that includes instructions that, when executed on the processing circuit 710 of the decoding system 120 , cause the decoding system 120 to perform the method 500 described above.

[0087] Still other embodiments include a carrier containing the control program 740. The carrier is one of an electronic signal, an optical signal, a radio signal, or a computer-readable storage medium.

[0088] While the computing systems described herein (e.g., encoding system 110, decoding system 120) may include the hardware components shown, other embodiments may include computing systems having different components and / or combinations thereof. It is understood that these computing systems may include any suitable combination of hardware and / or software required to perform the tasks, features, functions, and methods disclosed herein. The determinations, calculations, acquisitions, or similar operations described herein may be performed by processing circuitry that processes information, for example, by converting the acquired information into other information, comparing the acquired information or the converted information with information stored in a database, and / or performing one or more operations based on the acquired information or the converted information and making determinations as a result of the processing. Furthermore, while components are depicted as individual blocks within larger blocks or nested within multiple blocks, in practice, the apparatus described herein may include multiple different physical components that constitute a single illustrated component, and functionality may be divided between separate components.

Claims

1. A method (400), implemented by a coding system (110), comprising: receiving (410) a report from a decoding system (120), wherein the report: identifying a packet comprising a first video portion of a video stream; and including feedback indicating whether the first video portion was successfully received by the decoding system (120); and A video encoder (115) is configured (420) to encode a second video portion of the video stream based on whether the feedback indicates that the first video portion was successfully received.

2. The method according to claim 1, wherein Configuring the video encoder (115) based on the feedback includes, in response to the feedback indicating that the first video portion was not successfully received, configuring the video encoder (115) to encode the second video portion without reference to the first video portion.

3. The method according to claim 2, wherein: Configuring the video encoder (115) to encode the second video portion without reference to the first video portion includes configuring the video encoder (115) to designate a video slice including the first video portion as intra-coded.

4. The method according to claim 2, wherein: Configuring the video encoder (115) to encode the second video portion without reference to the first video portion includes configuring the video encoder (115) to encode the second video portion as an Instantaneous Decoding Refresh (IDR) frame.

5. The method according to claim 4, wherein: Configuring the video encoder (115) to encode the second video portion as an IDR frame is responsive to the feedback and at least one further feedback received from the decoding system (120) together indicating that the entirety of the video frame including the first video portion was not successfully received.

6. The method of claim 1, wherein: Configuring the video encoder (115) based on the feedback includes, in response to the feedback indicating that the first video portion was successfully received, configuring the video encoder (115) to encode the second video portion using video frames including the first video portion as reference frames.

7. The method according to claim 6, wherein: Configuring the video encoder (115) to encode the second video portion using the video frame as the reference frame is further responsive to the feedback and at least one further feedback received from the decoding system (120) together indicating that the entirety of the video frame was successfully received at the decoding system (120).

8. The method according to any one of claims 1 to 7, wherein The feedback indicates that the first video portion was successfully received by indicating that a video slice comprising the first video portion was successfully received at the decoding system (120).

9. The method of any one of claims 1 to 8, further comprising storing, in a data storage device, for each of a plurality of packets carrying a respective portion of the video stream, an entry associating the following identifier with decoder feedback indicating whether the respective portion is decodable by the decoding system (120): a packet identifier identifying the packet; and A frame identifier identifies a video frame that includes the corresponding portion.

10. The method of claim 9, further comprising adding an additional entry for the group to the data store based on the report.

11. The method according to any one of claims 1 to 10, further comprising: determining a one-way network delay between a transmission time of the packet from the encoding system (110) and a reception time of the packet at the decoding system (120); The feedback is interpreted as indicating whether the first video portion is decodable or undecodable based on whether the one-way network delay is less than a threshold, respectively.

12. A coding system (110), comprising: An interface circuit (630) and a processing circuit (610) communicatively connected to the interface circuit (630), wherein the processing circuit (610) is configured to: receiving a report from a decoding system (120) via the interface circuit (610), wherein the report: identifying a packet comprising a first video portion of a video stream; and including feedback indicating whether the first video portion was successfully received by the decoding system (120); and A video encoder (115) is configured to encode a second video portion of the video stream based on whether the feedback indicates that the first video portion was successfully received.

13. The coding system of claim 12, wherein: The processing circuit (610) is further configured to perform the method according to any one of claims 2-11.

14. A computer program (640) comprising instructions which, when executed on a processing circuit (610) of a coding system (110), cause the coding system (110) to perform the method according to any one of claims 1-11.

15. A carrier containing the computer program (640) according to claim 14, wherein The carrier is one of an electronic signal, an optical signal, a radio signal or a computer-readable storage medium.

Images