Visual data processing method, device and medium

Abstract

Embodiments of the present disclosure provide a solution for visual data processing. A method for visual data processing is presented. The method includes performing a conversion between a bitstream of a current frame of visual data and at least one packet based on a real-time transport protocol (RTP), where the at least one packet includes a first indication indicating a difference between a decoding timestamp of the current frame and a presentation timestamp of the current frame.

Description

Technical Field

[0001] Embodiments of the present disclosure generally relate to computer technology, and more particularly, to visual data processing and transmission technology. Background Art

[0002] Nowadays, digital video capabilities are being applied to all aspects of people's lives. For video encoding / decoding, various types of video compression technologies have been proposed, such as MPEG-2, MPEG-4, ITU-T H.263, ITU-TH.264 / MPEG-4 Part 10 Advanced Video Codec (AVC), ITU-T H.265 High Efficiency Video Codec (HEVC) standard, and Versatile Video Coding (VVC) standard.

[0003] Real-time Transport Protocol (RTP) is a standardized packet format for delivering multimedia data over an Internet Protocol (IP) network, and is optimized for consistent delivery of real-time data. RTP is used for Internet telephony, IP voice, and video telecommunications. Since visual data codecs (e.g., visual data codecs based on AVC or HEVC) are widely used in RTP-based systems, it is desirable to adapt RTP-based systems to visual data codecs to support the latest visual data coding and decoding techniques. Summary of the invention

[0004] Embodiments of the present disclosure provide a solution for visual data processing.

[0005] In a first aspect, a method for visual data processing is provided. The method comprises: performing a conversion between a bit stream of a current frame of visual data and at least one packet based on a real-time transport protocol (RTP), wherein the at least one packet comprises a first indication indicating a difference between a decoding timestamp of the current frame and a presentation timestamp of the current frame.

[0006] Based on the method of the first aspect of the present disclosure, a first indication can be added to a packet based on RTP to convey the difference between the decoding timestamp of the current frame and the presentation timestamp of the current frame. With the help of the first indication, an RTP-based system can correctly decode visual data with bidirectional prediction frames (B frames). Compared to a conversion solution in which only intra-frame coded frames (I frames) and predicted frames (P frames) are supported by an RTP-based system, the proposed method can advantageously enable an RTP-based system to support B frames and thereby improve compatibility with the latest visual data coding and decoding technologies.

[0007] In a second aspect, a device for visual data processing is provided. The device includes a processor and a non-volatile memory having instructions thereon. The instructions, when executed by the processor, cause the processor to perform the method according to the first aspect of the present disclosure.

[0008] In a third aspect, a non-transitory computer-readable storage medium is provided, wherein the non-transitory computer-readable storage medium stores instructions for causing a processor to execute the method according to the first aspect of the present disclosure.

[0009] In a fourth aspect, another non-transitory computer-readable recording medium is provided. The non-transitory computer-readable recording medium stores a bitstream of a current visual data frame generated by a method performed by an apparatus for visual data processing. The method includes: performing conversion between the bitstream and at least one packet based on a real-time transport protocol (RTP), wherein the at least one packet includes a first indication indicating a difference between a decoding timestamp of the current frame and a presentation timestamp of the current frame.

[0010] In a fifth aspect, a method for storing a bitstream of a current frame of visual data is provided. The method comprises: performing conversion between the bitstream and at least one packet based on a real-time transport protocol (RTP); storing the bitstream in a non-transitory computer-readable recording medium, wherein the at least one packet comprises a first indication indicating a difference between a decoding timestamp of the current frame and a presentation timestamp of the current frame.

[0011] In a sixth aspect, another non-transitory computer-readable recording medium is provided. The non-transitory computer-readable recording medium stores at least one packet generated by a method performed by an apparatus for visual data processing. The method includes: based on a real-time transport protocol (RTP), performing conversion between a bit stream of a current frame of visual data and at least one packet, wherein the at least one packet includes a first indication indicating a difference between a decoding timestamp of the current frame and a presentation timestamp of the current frame.

[0012] In a seventh aspect, a method for storing at least one packet is provided. The method includes: performing conversion between a bit stream of a current frame of visual data and at least one packet based on a real-time transport protocol (RTP); storing at least one packet in a non-transitory computer-readable recording medium, wherein the at least one packet includes a first indication indicating a difference between a decoding timestamp of the current frame and a presentation timestamp of the current frame.

[0013] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The above and other objects, features and advantages of the exemplary embodiments of the present disclosure will become more apparent through the following detailed description with reference to the accompanying drawings. In the exemplary embodiments of the present disclosure, the same reference numerals generally refer to the same components.

[0015] Figure 1 A block diagram of an example RTP-based system according to some embodiments of the present disclosure is shown;

[0016] Figure 2 A block diagram of a first example video encoder according to some embodiments of the present disclosure is shown;

[0017] Figure 3 A block diagram of an example video decoder according to some embodiments of the present disclosure is shown;

[0018] Figure 4 An example of an RTP header extension is shown;

[0019] Figure 5 An example extension element is shown;

[0020] Figure 6 An example header extension is shown with three extension elements and some padding;

[0021] Figure 7 An example of a 2-byte header is shown;

[0022] Figure 8 An example of an extension element is shown;

[0023] Fig. 9 An example header extension is shown with three extension elements and some padding;

[0024] Fig.10 shows an extended format according to some embodiments of the present disclosure;

[0025] Fig.11 A schematic diagram showing an example environment in which techniques for visual data processing according to some embodiments of the present disclosure may be implemented;

[0026] Fig.12 shows a signaling diagram for visual data transmission according to some embodiments of the present disclosure;

[0027] Fig.13 A flowchart showing a method for visual data processing according to an embodiment of the present disclosure; and

[0028] Fig.14 A block diagram of a computing device is shown in which various embodiments of the present disclosure may be implemented.

[0029] Throughout the drawings, same or similar reference numbers generally refer to same or similar elements. DETAILED DESCRIPTION

[0030] The principle of the present disclosure will now be described with reference to some embodiments. It should be understood that these embodiments are described only for the purpose of illustrating and helping those skilled in the art to understand and implement the present disclosure, without implying any limitation on the scope of the present disclosure. In addition to the methods described below, the disclosure described herein can also be implemented in various ways.

[0031] In the following description and claims, unless defined otherwise, all scientific and technical terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present disclosure belongs.

[0032] References in this disclosure to "one embodiment," "an embodiment," "an example embodiment," etc. indicate that the described embodiment may include a particular feature, structure, or characteristic, but not every embodiment must include the particular feature, structure, or characteristic. Furthermore, these phrases do not necessarily refer to the same embodiment. Furthermore, when a particular feature, structure, or characteristic is described in conjunction with an example embodiment, whether or not explicitly described, it is considered to be within the knowledge of those skilled in the art to affect such feature, structure, or characteristic in relation to other embodiments.

[0033] It should be understood that although the terms "first" and "second" etc. can be used to describe various elements, these elements should not be limited to these terms. These terms are only used to distinguish one element from another element. For example, a first element can be referred to as a second element, and similarly, a second element can be referred to as a first element without departing from the scope of the exemplary embodiments. As used herein, the term "and / or" includes any and all combinations of one or more of the listed terms.

[0034] The terms used herein are only used for the purpose of describing specific embodiments and are not intended to limit the exemplary embodiments. As used herein, the singular forms "a", "an", and "the" are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the terms "including", "comprising", and / or "having" when used herein indicate the presence of the features, elements, and / or components, etc., but do not exclude the presence or addition of one or more other features, elements, components, and / or combinations thereof.

[0035] As used herein, the terms "must," "required," and "should" mean that the definitions are absolute requirements of the specification. The terms "shall not" and "should not" mean that the definitions are absolute prohibitions of the specification. The terms "should" and "recommended" mean that there may be legitimate reasons in particular circumstances to ignore a particular item, but the full implications must be understood and carefully weighed before choosing a different route. The terms "should not" and "not recommended" mean that there may be legitimate reasons in particular circumstances when a particular behavior is acceptable or even useful, but the full implications should be understood and the case carefully weighed before implementing any behavior described with that label. The terms "may" and "optionally" mean that an item is truly optional. Example Environment

[0036] Figure 1 1 is a block diagram illustrating an example RTP-based system 100 that can utilize the techniques of the present disclosure. It should be understood that the system 100 may also include additional blocks not shown, and / or the blocks shown may be omitted. For purposes of illustration, Figure 1 The description is made with reference to video processing. The techniques of the present disclosure may also be applied to any other suitable system or any other suitable visual data, such as a series of images, etc. The scope of the present disclosure is not limited in this respect.

[0037] like Figure 1 As shown, the system 100 includes a source device 110 (e.g., a transmitting device, which may be an eNB or a user equipment (UE)) and a destination device 120 (e.g., a receiving device, which may be a UE or an eNB), connected by a transmission channel 130 (wireless / wired channel). Figure 1 In the example of , the source device 110 may be associated with a first video communication device (e.g., an eNB) and include a video source 111, a video encoder 112, an RTP conversion unit (also referred to as an RTP unit) 113, a radio link protocol (RLP) queue 114, a MAC layer unit 115, and a physical (PHY) layer unit 116. The destination device 120 may be associated with another video communication device (e.g., a UE) and include a PHY layer unit 126, a MAC layer unit 125, an RLP queue 124, an RTP unit 123, a video decoder 122, and / or a video output unit 121. It should be noted that both the network device and the UE may be a sending device or a receiving device, and that the diagram is not intended to be a networking diagram, and thus, the channel is not intended to be limited to a wireless channel.

[0038] System 100 may provide bidirectional video transmission for video telephony, for example, via transmission channel 130. Thus, reciprocal encoding, decoding, and conversion units may generally be provided at opposite ends of channel 130. In some embodiments, source device 110 and destination device 120 may be embodied in a video communication device, such as a wireless mobile terminal equipped for video streaming, video telephony, or both. The mobile terminal may support VT according to a packet switching standard such as RTP.

[0039] For example, at the source device 110, the RTP unit 113 adds appropriate RTP header data to the video data received from the video encoder 112 and places this data in the RLP queue 114. In some examples, as described herein, the RTP unit 113 complies with a particular standard, such as "RFC 3550: RTP: A Transport Protocol for Real-Time Applications", H. Schulzrinne et al., July 2003, "RFC 5104: Codec Control Messages in the RTP Audio-Visual Provide with Feedback (AVPF)", S. Wenger et al., February 2008 (hereinafter referred to as RFC 5104), and / or other applicable standards for real-time or near real-time data transmission. The MAC layer unit 115 generates a MAC RLP packet from the contents of the RLP queue 114. The PHY layer unit 116 converts the MAC RLP packets into PHY layer packets for transmission on the channel 130 .

[0040] The PHY layer unit 126 and the MAC layer unit 125 of the destination device 120 operate in a reciprocal manner. The PHY layer unit 126 converts the PHY layer packet received from the channel 130 into a MAC RLP packet. The MAC layer unit 125 places the MAC RLP packet into the RLP queue 124. The RTP unit 123 strips the header information from the data in the RLP queue 124 and reassembles the video data to be delivered to the video decoder 122.

[0041] The system 100 can be designed to support one or more wireless communication technologies, such as code division multiple access (CDMA), frequency division multiple access (FDMA), time division multiple access (TDMA), or orthogonal frequency division multiplexing (OFDM) or other suitable wireless technologies. The above wireless communication technologies can be transmitted according to any of a variety of radio access technologies. For example, CDMA can be transmitted according to cdma2000 or wideband CDMA (WCDMA) standards. TDMA can be transmitted according to the global system for mobile communications (GSM) standard. The universal mobile telecommunications system (UMTS) standard allows GSM or WCDMA operation. Generally, for VT applications, the system 100 can be designed to support high data rate (HDR) technology.

[0042] The video encoder 112 generates encoded video data according to a video compression method such as MPEG-4, High Efficiency Video Codec (HEVC), or other video codec standards. Other video compression methods include International Telecommunication Union (ITU) H.263, ITU H.264, or MPEG-2 methods.

[0043] In operation, the RTP unit 113 obtains video packets from the video encoder 112. As previously mentioned, the RTP unit 113 adds appropriate header information to the video packets and inserts the resulting data into the RLP queue 114. The MAC layer unit 115 retrieves the data from the RLP queue 114 and forms MAC layer packets. Each MAC layer packet carries the RTP header information and the video packet data contained in the RLP queue 114.

[0044] In many cases, the MAC layer packets will carry header information and video packet data, depending on the contents of the RLP queue 114. The MAC layer packets may be configured according to the Radio Link Protocol (RLP) and may be referred to as MAC RLP packets. The PHY layer unit 116 converts the MAC RLP video packets into PHY layer packets for transmission on the channel 130.

[0045] Channel 130 carries PHY layer packets to destination device 120. Channel 130 can be any physical connection between source device 110 and destination device 120. For example, channel 130 can be a wired connection, such as a local or wide area wired network. Alternatively, as described herein, channel 130 can be a wireless connection, such as a cellular, satellite, or optical connection. Channel conditions can be a problem for both wired and wireless channels, but may be particularly relevant for mobile VT applications performed over wireless channel 16, where channel conditions may be affected by fading or congestion. Channel 130 can support a specific network link rate (e.g., a specific bandwidth) that can fluctuate depending on channel conditions. For example, channel 130 can be characterized by a reverse link (RL) having a throughput that varies depending on channel conditions.

[0046] Typically, the PHY layer unit 126 of the destination device 120 identifies the MAC layer packets from the PHY layer packets and reassembles the contents into MAC RLP packets. The MAC layer unit 125 then reassembles the contents of the MAC RLP packets to provide video packets for insertion into the RLP queue 124. The RTP unit 123 removes the accompanying header information and provides the video packets to the video decoder 122. The video decoder 122 decodes the video data frames to generate a video data stream for driving a display device.

[0047] Figure 2 is a block diagram showing an example of a video encoder 200 according to some embodiments of the present disclosure, which may be Figure 1 An example of a video encoder 112 in the system 100 is shown.

[0048] Video encoder 200 may be configured to implement any or all of the techniques of this disclosure. Figure 2 In the example of , video encoder 200 includes multiple functional components. The techniques described in this disclosure can be shared between the various components of video encoder 200. In some examples, a processor can be configured to perform any or all of the techniques described in this disclosure.

[0049] In some embodiments, the video encoder 200 may include a partitioning unit 201, a prediction unit 202, a residual generation unit 207, a transformation unit 208, a quantization unit 209, an inverse quantization unit 210, an inverse transformation unit 211, a reconstruction unit 212, a buffer 213 and an entropy coding unit 214, and the prediction unit 202 may include a mode selection unit 203, a motion estimation unit 204, a motion compensation unit 205 and an intra-frame prediction unit 206.

[0050] In other examples, the video encoder 200 may include more, fewer, or different functional components. In one example, the prediction unit 202 may include an intra-block copy (IBC) unit. The IBC unit may perform prediction in an IBC mode, where at least one reference picture is a picture where the current video block is located.

[0051] Furthermore, although some components (such as the motion estimation unit 204 and the motion compensation unit 205) may be integrated, for the purpose of explanation, these components are described in detail below. Figure 2 are shown separately in the example.

[0052] The partitioning unit 201 may partition a picture into one or more video blocks. The video encoder 200 and the video decoder 300 may support various video block sizes.

[0053] The mode selection unit 203 may select one of a plurality of coding modes (intra-frame coding or inter-frame coding), for example, based on the error result, and provide the generated intra-frame coded block or inter-frame coded block to the residual generation unit 207 to generate residual block data, and to the reconstruction unit 212 to reconstruct the coded block for use as a reference picture. In some examples, the mode selection unit 203 may select a combination of intra-frame and inter-frame prediction (CIIP) modes, where the prediction is based on an inter-frame prediction signal and an intra-frame prediction signal. In the case of inter-frame prediction, the mode selection unit 203 may also select a resolution for the motion vector (e.g., sub-pixel precision or integer pixel precision) for the block.

[0054] To perform inter-frame prediction on the current video block, the motion estimation unit 204 may generate motion information for the current video block by comparing the current video block with one or more reference frames from the buffer 213. The motion compensation unit 205 may determine a predicted video block for the current video block based on the motion information and decoded samples of pictures from the buffer 213 other than the picture associated with the current video block.

[0055] The motion estimation unit 204 and the motion compensation unit 205 may perform different operations on the current video block, for example, depending on whether the current video block is in an I slice, a P slice, or a B slice. As used herein, an "I slice" may refer to a portion of a picture consisting of macroblocks, all of which are based on macroblocks within the same picture. Furthermore, as used herein, in some aspects, a "P slice" and a "B slice" may refer to a portion of a picture consisting of macroblocks that are independent of macroblocks in the same picture.

[0056] In some examples, the motion estimation unit 204 may perform unidirectional prediction on the current video block, and the motion estimation unit 204 may search the reference pictures of list 0 or list 1 to find the reference video block for the current video block. The motion estimation unit 204 may then generate a reference index and a motion vector, the reference index indicating the reference picture in list 0 or list 1 containing the reference video block, and the motion vector indicating the spatial displacement between the current video block and the reference video block. The motion estimation unit 204 may output the reference index, the prediction direction indicator, and the motion vector as the motion information of the current video block. The motion compensation unit 205 may generate a predicted video block for the current video block based on the reference video block indicated by the motion information of the current video block.

[0057] Alternatively, in other examples, the motion estimation unit 204 may perform bidirectional prediction on the current video block. The motion estimation unit 204 may search the reference pictures in list 0 for a reference video block for the current video block, and may also search the reference pictures in list 1 for another reference video block for the current video block. The motion estimation unit 204 may then generate a plurality of reference indexes and a plurality of motion vectors, the plurality of reference indexes indicating a plurality of reference pictures in list 0 and list 1 containing a plurality of reference video blocks, and the plurality of motion vectors indicating a plurality of spatial displacements between the plurality of reference video blocks and the current video block. The motion estimation unit 204 may output the plurality of reference indexes and the plurality of motion vectors of the current video block as motion information of the current video block. The motion compensation unit 205 may generate a predicted video block for the current video block based on the plurality of reference video blocks indicated by the motion information of the current video block.

[0058] In some examples, motion estimation unit 204 may output a complete set of motion information for use in a decoding process by a decoder. Alternatively, in some embodiments, motion estimation unit 204 may signal motion information of a current video block with reference to motion information of another video block. For example, motion estimation unit 204 may determine that motion information of a current video block is sufficiently similar to motion information of a neighboring video block.

[0059] In one example, motion estimation unit 204 may indicate to video decoder 300 a value in a syntax structure associated with the current video block that indicates that the current video block has the same motion information as another video block.

[0060] In another example, the motion estimation unit 204 may identify another video block and a motion vector difference (MVD) in a syntax structure associated with the current video block. The motion vector difference indicates the difference between the motion vector of the current video block and the motion vector of the indicated video block. The video decoder 300 may use the motion vector of the indicated video block and the motion vector difference to determine the motion vector of the current video block.

[0061] As discussed above, the video encoder 200 may signal motion vectors in a predictive manner.Two examples of prediction signaling techniques that may be implemented by the video encoder 200 include Advanced Motion Vector Prediction (AMVP) and merge mode signaling.

[0062] The intra prediction unit 206 may perform intra prediction on the current video block. When the intra prediction unit 206 performs intra prediction on the current video block, the intra prediction unit 206 may generate prediction data for the current video block based on decoded samples of other video blocks in the same picture. The prediction data for the current video block may include a prediction video block and various syntax elements.

[0063] The residual generation unit 207 may generate residual data for the current video block by subtracting (e.g., indicated by a minus sign) the prediction video block(s) of the current video block from the current video block. The residual data of the current video block may include residual video blocks corresponding to different sample portions of samples in the current video block.

[0064] In other examples, such as in skip mode, there may be no residual data for the current video block, and the residual generation unit 207 may not perform a subtraction operation.

[0065] Transform processing unit 208 may generate one or more transform coefficient video blocks for a current video block by applying one or more transforms to the residual video block associated with the current video block.

[0066] After transform processing unit 208 generates a transform coefficient video block associated with the current video block, quantization unit 209 may quantize the transform coefficient video block associated with the current video block based on one or more quantization parameter (QP) values ​​associated with the current video block.

[0067] The inverse quantization unit 210 and the inverse transform unit 211 may apply inverse quantization and inverse transform to the transform coefficient video block, respectively, to reconstruct a residual video block from the transform coefficient video block. The reconstruction unit 212 may add the reconstructed residual video block to corresponding samples from one or more prediction video blocks generated by the prediction unit 202 to generate a reconstructed video block associated with the current video block for storage in the buffer 213.

[0068] After reconstruction unit 212 reconstructs the video block, a loop filtering operation may be performed to reduce video blocking artifacts in the video block.

[0069] The entropy coding unit 214 may receive data from other functional components of the video encoder 200. When the entropy coding unit 214 receives data, the entropy coding unit 214 may perform one or more entropy coding operations to generate entropy coded data and output a code stream including the entropy coded data.

[0070] Figure 3 is a block diagram showing an example of a video decoder 300 according to some embodiments of the present disclosure, which may be Figure 1 An example of a video decoder 122 in the system 100 is shown.

[0071] Video decoder 300 may be configured to perform any or all of the techniques of this disclosure. Figure 3In the example of , video decoder 300 includes multiple functional components. The techniques described in this disclosure can be shared between the various components of video decoder 300. In some examples, a processor can be configured to perform any or all of the techniques described in this disclosure.

[0072] exist Figure 3 In the example of , the video decoder 300 includes an entropy decoding unit 301, a motion compensation unit 302, an intra prediction unit 303, an inverse quantization unit 304, an inverse transform unit 305, and a reconstruction unit 306 and a buffer 307. In some examples, the video decoder 300 can perform a decoding process that is generally opposite to the encoding process described with respect to the video encoder 200.

[0073] The entropy decoding unit 301 can retrieve the encoded code stream. The encoded code stream may include entropy encoded video data (e.g., encoded video data blocks). The entropy decoding unit 301 can decode the entropy encoded video data, and the motion compensation unit 302 can determine motion information from the entropy decoded video data, the motion information including motion vectors, motion vector precision, reference picture list indexes, and other motion information. The motion compensation unit 302 can determine the information, for example, by performing AMVP and merge mode. AMVP is used, including deriving several most likely candidates based on data and reference pictures of neighboring PBs. The motion information typically includes horizontal and vertical motion vector displacement values, one or two reference picture indexes, and in the case of prediction areas in B stripes, also includes an identification of which reference picture list is associated with each index. As used herein, in some aspects, "merge mode" may refer to deriving motion information from blocks that are adjacent in space or time.

[0074] The motion compensation unit 302 may generate motion compensated blocks, possibly performing interpolation based on interpolation filters.Identifiers for the interpolation filters used with sub-pixel precision may be included in the syntax elements.

[0075] The motion compensation unit 302 may calculate interpolated values ​​for sub-integer pixels of a reference block using interpolation filters used by the video encoder 200 during encoding of the video block. The motion compensation unit 302 may determine the interpolation filters used by the video encoder 200 based on received syntax information, and the motion compensation unit 302 may use the interpolation filters to generate a prediction block.

[0076] The motion compensation unit 302 may use at least part of the syntax information to determine the size of blocks used to encode (multiple) frames and / or (multiple) slices of the encoded video sequence, partition information describing how each macroblock of a picture of the encoded video sequence is partitioned, a mode indicating how each partition is encoded, one or more reference frames (and reference frame lists) for each inter-frame coding block, and other information for decoding the encoded video sequence. As used herein, in some aspects, a "slice" may refer to a data structure that can be decoded independently of other slices of the same picture in terms of entropy coding, signal prediction, and residual signal reconstruction. A slice may be an entire picture, or it may be a region of a picture.

[0077] The intra prediction unit 303 may use, for example, an intra prediction mode received in a bitstream to form a prediction block from spatially neighboring blocks. The inverse quantization unit 304 inversely quantizes (i.e., dequantizes) the quantized video block coefficients provided in the bitstream and decoded by the entropy decoding unit 301. The inverse transform unit 305 applies an inverse transform.

[0078] The reconstruction unit 306 may obtain the decoded block, for example, by adding the residual block to the corresponding prediction block generated by the motion compensation unit 302 or the intra prediction unit 303. If necessary, a deblocking filter may also be applied to filter the decoded block in order to remove blocking artifacts. The decoded video block is then stored in a buffer 307, which provides reference blocks for subsequent motion compensation / intra prediction, and the buffer 307 also generates the decoded video for presentation on a display device.

[0079] Some exemplary embodiments of the present disclosure will be described in detail below. It should be noted that the section titles used in this document are for ease of understanding, rather than limiting the embodiments disclosed in the section to that section. In addition, although some embodiments are described with reference to general video codecs or other specific video codecs, the disclosed technology is also applicable to other video codec technologies. In addition, although some embodiments describe the video encoding steps in detail, it should be understood that the corresponding decoding steps of the unencoding will be implemented by the decoder. In addition, the term video processing includes video encoding or compression, video decoding or decompression, and video transcoding, in which video pixels are represented from one compression format to another compression format or at different compression bit rates. 1. Brief Overview This disclosure describes an RTP header extension for conveying decoding time information about a video when a bidirectional prediction frame is present. It adds the composition time (CTS) as a value so that the receiver can decode the video with the correct sequence. 2. Introduction As video codecs, H264 / HEVC is widely used in RTP-based systems. These codecs support I frames, B frames, and P frames. Most RTP systems do not support B frames, which are widely used in streaming systems. With the rapid deployment of real-time communication (RTC) in low-latency streaming scenarios, it is necessary to support bidirectional prediction frames in RTP-based systems. 2.1RTP Header Extension An extension mechanism is provided to allow individuals to implement new payload format-independent functionality that requires additional information to be carried in the RTP data packet header. The mechanism is designed so that header extensions can be ignored by other interoperators that have not yet been extended. Figure 4 An example of an RTP header extension is shown. Note that this header extension is intended for limited use only. Most of the potential uses of this mechanism would be better implemented in another way using the methods described in the previous sections. For example, the processing of profile-specific extensions to a fixed header is less expensive because it is not conditional or in a variable position. Additional information required for a specific payload format should not use this header extension, but should be carried in the payload portion of the packet. If the X bit in the RTP header is 1, the variable length header extension must be attached to the RTP header after the CSRC list (if present). The header extension contains a 16-bit length field, which counts the number of 32-bit words in the extension, excluding the four-byte extension header (therefore, zero is a valid length). Only a single extension can be attached to the RTP data header. In order to allow multiple interoperability implementations to independently perform each experiment with different header extensions, or to allow a specific implementation to experiment with more than one type of header extension, the first 16 bits of the header extension are open for distinguishing identifiers or parameters. These 16-bit formats will be defined by the profile specification under which the implementation operates. The RTP specification itself does not define any header extensions. 2.2 Header Extension Type Considerations Each extension element in the packet has a local identifier (ID) and a length. The local identifiers present in the stream must have been negotiated or defined out-of-band. There is no static allocation of local identifiers. Each different extension must have a unique ID. ID value 0 is reserved for padding and shall not be used as a local identifier. An extension element with an ID value equal to 0 MUST NOT have an associated Length field greater than 0. If such an extension element is encountered, its Length field MUST be ignored, processing of the entire extension MUST terminate at that point, and only extension elements that existed before the element with ID 0 and a Length field greater than 0 SHOULD be considered. There are two variants of extensions: 1-byte header and 2-byte header. Since it is expected that (a) the number of extensions in any given RTP session is small, and (b) the extension itself is small, the 1-byte header form is preferred and must be supported by all receivers. The stream must contain only 1-byte headers or only 2-byte headers, unless it is known that all receivers support mixing through SDP offer / answer negotiation or out-of-band knowledge. Each RTP packet with an RTP header extension after this specification will indicate whether it contains a 1-byte or 2-byte header extension through the use of the "defined by profile" field. The extension element type that does not match the header extension format, i.e. 1-byte or 2-byte, shall not be used in the RTP packet. When all extensions are small enough for the 1-byte header form, the sender should not use the 2-byte header form. The sender that intends to send the 2-byte form should negotiate the use of an ID greater than 14 if it wants to let the receiver know that the sender intends to use the 2-byte form-for example, if the RTP header extension is longer than 16 bytes. The sender can know that intermediate devices can add RTP header extensions; in this case, the sender should use the 2-byte form. The extended element sequence that may have filling forms the header extension defined in the RTP specification.As indicated by the RTP header extension length, there are many extended elements that will be suitable for the RTP header extension.Because this length is sent with signal in full 32 bit words, the filling byte is used to fill to 32 bit boundaries.The whole extension is parsed byte by byte to find each extended element (not needing alignment), and parses (1) at the end of the whole header extension, or (2) in the "only 1 byte header" situation, stops when encountering the identifier with a reserved value of 15--whichever occurs first. In both forms, padding bytes have the value 0 (zero). They may be placed between extension elements if needed for alignment, or after the last extension element if needed for padding. Padding bytes do not provide an element ID, nor a length field. When a padding byte is found, it is ignored, and the parser moves on to interpreting the next byte. Note that the 1-byte header form allows data lengths between 1 byte and 16 bytes by adding 1 to the length value being signaled (thus, a 0 in the length field indicates one byte of data as follows). This allows for the important case of a 16-byte payload. This addition is not performed for the 2-byte header, where the length field signals a data length between 0 bytes and 255 bytes. The use of RTP header extensions will reduce the efficiency of RTP header compression because unless the RTP header compression module is updated to recognize the extension header, the header extension will be sent in an uncompressed manner. 2.3 1-byte header In the extended 1-byte header form, the 16-bit value of the header extension required by the RTP specification, marked as "defined by profile" in the RTP specification, must have a fixed bit pattern of 0xBEDE (a pattern chosen for unimportant reasons). Figure 5 Example extension elements are shown. Figure 5 As shown, each extension element must begin with bytes containing an ID and a length. The 4-bit ID is the local identifier of this element in the range 1 to 14 inclusive. In the signalling part, this is referred to as the valid range. Local identifier value 15 is reserved for future extensions and shall not be used as an identifier. If an ID value of 15 is encountered, its length field MUST be ignored, processing of the entire extension MUST terminate at that point, and only extension elements that existed before the element with ID 15 SHOULD be considered. The 4-bit length is the number of data bytes for this header extension element that follow the 1-byte header, minus one. Thus, a value of zero (0) in this field indicates that one byte of data follows, and a value of 15 (the maximum value) indicates 16 bytes of element data. (This allows 16-byte values ​​to be carried, which is a common length for tags and identifiers, but loses the possibility of zero-length values, since these are typically padded. Figure 6 An example header extension is shown with three extension elements and some padding. 2.4 2-Byte Header In 2-byte header form, the RTP specification defines a 16-bit value for the header extension, which is marked as "defined by the profile" in the RTP specification, such as Figure 7 shown. Figure 7 An example of a 2-byte header is shown. The appbits field is application dependent and may be defined as 4 bits of any value or meaning; this subject is outside the scope of this specification. For signaling purposes, this field is treated as a special extension value assigned to the local identifier 256. If no extension is specified (256) by configuration or signaling for this local identifier value, then the appbits field SHOULD be set to all 0s (zeros) by the sender and MUST be ignored by the receiver. Figure 8 An example of an extension element is shown. Figure 8 As shown, each extension element begins with a byte containing an ID and a byte containing a length. The 8-bit ID is a local identifier for this element in the range 1-255. In the signaling section, the range 1-256 is referred to as the valid range, where values ​​1-255 refer to the extended element and the value 256 refers to the 4-bit appbits field (above). Note that there is one ID space for both the 1-byte form and the 2-byte form. This means that smaller values ​​(1-14) can be used in the 4-bit ID field in the 1-byte header format with the same meaning. The 8-bit length field is the length of the extended data in bytes, excluding the ID and length fields. A value of zero (0) indicates that there is no subsequent data. Fig. 9 An example header extension is shown with three extension elements and some padding. 2.5RTP header SDP signal design The indication of the presence of this extension, and the mapping of local identifiers used in the header extension to the larger namespace, MUST be performed out-of-band - for example as part of the SDP offer / answer. This subclause defines such signalling in SDP. Available mappings must use IDs in a valid range, and each ID in this range must be used only once per media stanza (or only once if the mapping is session level). Mappings that do not conform to these rules may be presented, for example, during SDP offer / answer negotiation as described in the next section, but need to be remapped to conforming values ​​before they can be applied. Each extension is named by a URI. This URI MUST be absolute; it identifies precisely the format and meaning of the extension. URIs that contain a domain name SHOULD also contain a month-date of the form mmyyyy. Definition of the element and assignment of the URI MUST have been authorized by the owner of the domain name on or very near that date. (This avoids problems of ownership when domain names change.) If a resource or document defines several extensions, the URI MUST identify the actual extension that is used, for example using a fragment or query identifier (the characters following a "#" or "?" in the URI). Rationale: The use of URIs provides a large, unallocated space and provides documentation about extensions. URIs do not have to be referenceable, to allow confidential or experimental use, or to cover situations where extensions continue to be used after the organization that defined them no longer exists. An extension URI with the same attributes MUST NOT appear more than once when applying to the same stream, i.e., in the declaration of a single stream either at the session level or at the media level. (Of course, the same extension MAY be used for several streams, and MAY appear with different <extensionattributes> for the same stream.) For extensions defined in RFCs, the URI used SHOULD be a URN starting with "urn:ietf:params:rtp-hdretxt:", followed by a registered descriptive name. The registration requirements are detailed in the IANA Notices. An example where "avg-example-metadata" is a hypothetical name for the header extension could be: urn:ietf:params:rtp-hdretxt:avg-example-metadata. Example names not from the IETF could be: http: / / example.com / 082005 / ext.htm#example-metadata. The mapping can be provided per media stream (in the media level part of the SDP, i.e. after the "m=" line) or globally for all streams (i.e. before the first "m=" line at the session level). The definition must be either all session level or all media level; mixing the two styles is not allowed. In addition, as described above, the ID used must be unique in each media part of the SDP or must be unique within the session declared by the session-level SDP. Each local identifier potentially used in a stream is mapped to an extension identified by a URI using attributes of the following form: a=extmap: <value> [" / ” <direction> ] <uri> <extensionattributes> in, <value>is the local identifier (ID) of this extension, and is an integer in the valid range (0 is reserved for padding in both forms, and 15 is reserved in the 1-byte header form, as described above); <direction>is one of "sendonly", "recvonly", "sendrecv", or "inactive" (without quotes) for the device being configured; <uri>is a URI as described above. In the related description the formal BNF grammar is presented. Example: a=extmap:1http: / / example.com / 082005 / ext.htm#ttime a=extmap:2 / sendrecv http: / / example.com / 082005 / ext.htm#xmeta short When SDP signaling is used for an RTP session, the "extmap" attribute is present to determine whether this type of header extension is used, rather than the magic number indicated above ("BEDE" or "100"). 3. Question The video stream contains many details, including timestamps, so the decoder knows how to process the content appropriately. The DTS (Decoding Timestamp) determines when a frame must be decoded, while the PTS (Presentation Timestamp) describes when a frame must be presented. This difference becomes important when using B-frames, which can be frames with references to past frames or frames with references to future frames. Assuming that a following frame will exist in the future, the decoder needs to decode that frame first in order to use it as a reference. Therefore, the decoder needs DTS when B-frames are present, while the RTP timestamp only reflects the presentation time (PTS). The disclosure specifies that the video RTP transmitter is allowed to deliver CTS (synthesis time) to the RTP extended header of the RTP receiver, and the CTS value is PTS minus DTS. Therefore, the RTP receiver obtains the DTS value by adding the CTS value through the RTP timestamp. This new header extension uses the universal mechanism for RTP header extension. The RTP transmitter only needs to add CTS to the first RTP packet when the video frame comprises several packets, which reduces overhead. 4. Examples the term *RTP: Real-time Transport Protocol *RTCP:RTP Control Protocol *RTCP RR: RTCP Receiver Report *RTCP SR: RTCP Sender Report *SDP: Session Description Protocol * Clock rate: The multiplier used to convert from a wall-clock value in seconds to an equivalent RTP timestamp value (without a fixed random offset). Note that various terms such as "clock frequency", "media clock rate", "timestamp unit", "timestamp frequency", and "RTP timestamp clock rate" may be used as synonyms for clock rate. *RTP sender: A logical network unit that sends RTP packets, sends RTCP SR packets, and receives RTCP reception report blocks. *RTP receiver: A logical network element that receives RTP packets, receives RTCP SR packets, and sends RTCP reception report blocks. *RTC: Real-time Communication *PTS: Video presentation timestamp *DTS: Video decoding timestamp *CTS: Video synthesis time 4.1RTP header extension format The generic RTP payload format follows the RTP header format and the generic RTP header extension, which can be encoded with a 1-byte header or a 2-byte header as described above. In the present disclosure, the 1-byte header format is used as an example. The following RTP header extensions are RECOMMENDED. IDs are assigned as per ([RFC8285]) and are in the format Fig.10 It is shown in Fig.10 An extended format according to some embodiments of the present disclosure is shown. Fig.10 In the example shown, ID represents the extended id, and CTS is equal to PTS minus DTS and divided by 90 (video clock rate). 4.2 Video RTP Sender The video sender here can be a video client or a middlebox for RTP switching. The video client can encode the video using B frames, and this RTP header extension should be added in the RTP packaging module. When the video frame contains multiple RTP packets, it is recommended to add it only in the first RTP packet, which will reduce the overhead. The middlebox can perform RTMP or other streaming video protocols to convert to RTP streams, and this header extension should be added when the streaming video contains B frames. 4.3 Video RTP Receiver The video RTP receiver here is the client that decodes the video. It should extract the CTS value when this extension exists and calculate the DTS value using the RTP timestamp (PTS) and CTS. DTS = PTS-CTS*90, where 90 is the video clock rate. The video receiver builds the frame and puts it into the jitter buffer. The decoder must decode the frame through the DTS sequence, and the video rendering module must render the decoded frame with the PTS sequence, which comes from the RTP timestamp. 4.4 Usage considerations In fact, when the receiver of the decoded video does not support B frames, in order to successfully decode the incoming video stream, it is recommended that the RTP middlebox discard b-frames when the video RTP sender contains B frames. The decoder at the endpoint should add a specific parameter (a=fmtp) in the SDP payload format indicating whether it supports video B-frame capabilities, and follow the offer / answer process described in ([RFC8285]). 4.5 Security Considerations The security considerations of the RTP specification ([RFC3550]) and the general mechanisms for the RTP header extension ([RFC8285]) apply. And the security considerations of all types of RTP intermediaries ([RFC7667])([RFC7201]) for both types of RTP apply to this header extension. The security considerations of SDP are described in the corresponding sections in ([RFC8866]). In the Secure Real-time Transport Protocol (SRTP) ([RFC3711]), the RTP header extension is authenticated but not encrypted. When this header extension is used, the CTS can therefore be seen frame by frame by an attacker who is passively observing the video stream. In scenarios where this becomes a problem, additional mechanisms may have to be used to protect the confidentiality of the header extension. The mechanism may be header extension encryption ([RFC6904]), or a lower-level security and authentication mechanism such as IPsec ([RFC4301]). 4.6 Session Description Protocol (SDP) Signaling The URI used to declare this header extension in the extmap attribute is "urn:ietf:params:rtp-hdretxt:CompositionTime". It does not contain any extension attributes. It follows the standard mechanism described in ([RFC8285]). An example attribute line in SDP is as follows: a=extmap:19uri:ietf:rtc:rtp-hdretxt:video:CompositionTime. The entire contents of the above standard documents (such as RFC8285, RFC3550, etc.) are incorporated herein by reference.

[0080] More details of the embodiments of the present disclosure are described below, involving visual data processing and transmission technology.

[0081] For discussion purposes, Fig.11 1 shows a schematic diagram of an example environment 1100 in which techniques for visual data processing according to some embodiments of the present disclosure may be implemented. Fig.11 As shown, environment 1100 includes a first device 1110 and a second device 1120. First device 1110 and second device 1120 may be communicatively coupled. As an example and not limitation, first device 1110 and second device 1120 may be coupled by a network, which may be the Internet.

[0082] In some embodiments, the first device 1110 may be an RTP sender, which may be Figure 1 An example of the RTP unit 113 in the system 100 is shown in FIG. In addition, the second device 1120 may be an RTP receiver, which may be Figure 1 100. As an example, the first device 1110 and / or the second device 1120 may comply with a particular standard, such as RFC3550, RFC5104, and / or other applicable standards for real-time or near real-time transmission of data.

[0083] Fig.12 A signaling diagram 1200 for visual data transmission according to some embodiments of the present disclosure is shown. The signaling diagram 1200 involves a first device 1110 and a second device 1120.

[0084] In operation, the first device 1110 encapsulates 1225 the bitstream of the current frame of visual data into at least one packet associated with the current frame. As an example and not limitation, the visual data can be a video, a series of images, a multimedia file, etc. In some embodiments, the current frame is a B frame. It should be understood that the current frame can also be any other suitable frame, such as an I frame or a P frame. The scope of the present disclosure is not limited in this regard.

[0085] At least one packet includes a first indication of a difference between a decoding timestamp of the current frame and a presentation timestamp of the current frame. At least one packet may also include any other suitable information related to the current frame. It should be understood that at least one packet may also include any other suitable information unrelated to the current frame, such as information related to another frame. The scope of the present disclosure is not limited in this respect.

[0086] In some embodiments, at least one packet may include multiple packets. In this case, the first indication is included in a first packet of the multiple packets. The first packet is generated before the remaining packets of the multiple packets. Thus, overhead can be reduced.

[0087] In some embodiments, the first indication can be implemented as a header extension. In one example, the header extension is a 1-byte header format, which has been described in detail in Section 2.3 above. Alternatively, the header extension has a 2-byte header format, which has been described in detail in Section 2.4 above.

[0088] In some embodiments, the value for the first indication may be determined based on a clock rate associated with the visual data and a difference between a decoding timestamp and a presentation timestamp. By way of example and not limitation, the value for the first indication may be determined as follows: Wherein CTS represents the value for the first indication, which is also referred to as the composition time in this article, PTS represents the decoding timestamp, DTS represents the presentation timestamp, and Clk represents the visual data clock rate. The value for the first indication and the presentation timestamp of the current frame can be encapsulated into at least one packet. It should be understood that the above explanations and / or examples are described only for the purpose of description. The scope of the present disclosure is not limited in this regard.

[0089] In certain embodiments, bitstream can be obtained from the encoder that encodes the current frame into the bitstream. Alternatively, based on another protocol different from RTP, bitstream can be extracted from at least one other grouping of the current frame. It should be understood that bitstream can also be obtained in any other suitable manner. The scope of the present disclosure is not limited in this respect.

[0090] In some embodiments, the first indication may be encrypted. Where the first indication is a header extension, the header extension is encrypted based on a header extension encryption mechanism. Alternatively, the header extension is encrypted by a lower level security and authentication mechanism. It should be understood that the first indication may be encrypted in any other suitable manner. The scope of the present disclosure is not limited in this respect.

[0091] The first device 1110 may send 1230 at least one packet to the second device 1120, for example during RTC or WebRTC. After receiving 1235 the at least one packet, the second device 1120 may extract 1240 the bitstream from the at least one packet. In some embodiments, the second device 1120 may extract the value for the first indication and the presentation timestamp of the current frame from the at least one packet. In addition, the second device 1120 may determine a decoding timestamp of the current frame based on the presentation timestamp, the clock rate associated with the visual data, and the value for the first indication.

[0092] By way of example and not limitation, the decode timestamp of the current frame may be determined as follows: DTS=PTS-CTS×Clk (2) Wherein DTS represents presentation time stamp, PTS represents decoding time stamp, CTS represents the value of the first indication, and Clk represents the visual data clock rate. It should be understood that the above explanations and / or examples are described for illustrative purposes only. The scope of the present disclosure is not limited in this regard.

[0093] In some embodiments, prior to encapsulating 1225 the bitstream into at least one packet, processing may be performed to declare support for the first indication. Fig.12 As shown, the first device 1110 may send 1205 a capability indication to the second device 1120. The capability indication is used to indicate that the first device 1110 supports the first indication. After receiving the capability indication 1210, the second device 1120 may send 1215 another capability indication to the first device 1110, which is used to indicate that the second device 1120 supports the first indication. In addition, after receiving 1220 another capability indication from the second device 1120, the first device 1110 encapsulates 1225 the bitstream into at least one packet. As an example and not a limitation, the capability indication is transmitted according to the session description protocol (SDP). An example attribute line in the SDP is as follows: a=extmap:19uri:ietf:rtc:rtp-hdretxt:video:CompositionTime. It should be understood that the above explanations and / or examples are described for descriptive purposes only. The scope of the present disclosure is not limited in this regard.

[0094] Fig.13 FIG. 1 is a flowchart of a method 1300 for visual data processing according to some embodiments of the present disclosure. For example, the method 1300 may be Fig.11 The embodiment is implemented at the first device 1110 or the second device 1120 shown in FIG.

[0095] At 1302, based on RTP, conversion is performed between the bitstream of the current frame of visual data and at least one packet. At least one packet includes a first indication of the difference between the decoding timestamp of the current frame and the presentation timestamp of the current frame. As an example and not limitation, the visual data can be a video, a series of images, a multimedia file, etc. In some embodiments, the current frame can be a bidirectional prediction frame (B frame). In some embodiments, conversion can include encapsulating the bitstream into at least one packet. Alternatively, conversion can include extracting the bitstream from at least one packet. In some embodiments, communication (such as RTC, WebRTC, etc.) can be performed based on at least one packet.

[0096] With the help of the first indication, the RTP-based system can correctly decode visual data with bidirectionally predicted frames (B frames). Compared with the conversion solution in which only intra-coded frames (I frames) and predicted frames (P frames) are supported by the RTP-based system, the proposed method can advantageously enable the RTP-based system to support B frames and thus improve the compatibility with the latest visual data coding and decoding technologies.

[0097] In some embodiments, the at least one packet may include a plurality of packets. The first indication may be included in a first packet of the plurality of packets. The first packet is generated before the remaining packets of the plurality of packets.

[0098] In some embodiments, the first indication may be encrypted. In some embodiments, the first indication may be a header extension. For example, the header extension may be a 1-byte header format or a 2-byte header format. The header extension may be encrypted based on a header extension encryption mechanism or a low-level security and authentication mechanism.

[0099] In the case where the method is implemented at the first device, based on the difference and the clock rate associated with the visual data, the value of the first indication can be determined. At 1302, the value of the first indication and the presentation timestamp of the current frame can be encapsulated into the at least one packet. In some embodiments, the bitstream can be obtained from the encoder encoded into the bitstream from the current frame. Alternatively, based on another protocol different from RTP, the bitstream can be extracted from at least one other packet of the current frame. The first device can transmit to the second device a second indication for indicating that the first device supports the first indication. In addition, the first device can receive from the second device a third indication for indicating that the second device supports the first indication.

[0100] In the case where the method is implemented at the second device, the value for the first indication and the presentation timestamp of the current frame can be extracted from the at least one packet. In addition, based on the presentation timestamp, the clock rate associated with the visual data, and the value for the first indication, the decoding timestamp of the current frame can be determined. In some embodiments, the second device can receive from the first device a second indication indicating that the first device supports the first indication. In addition, the second device can transmit to the first device a third indication indicating that the second device supports the first indication.

[0101] In some embodiments, at least one of the second indication or the third indication may be transmitted based on a Session Description Protocol (SDP).

[0102] According to another embodiment of the present disclosure, a non-transitory computer-readable recording medium is provided. The non-transitory computer-readable recording medium stores a bitstream of a current frame of visual data generated by a method performed by a device for visual data processing. In the method, conversion is performed between the bitstream and at least one packet based on RTP. At least one packet includes a first indication for indicating a difference between a decoding timestamp of the current frame and a presentation timestamp of the current frame.

[0103] According to some other embodiments of the present disclosure, a method for storing a bitstream of a current frame of visual data is provided. In the method, conversion is performed between the bitstream and at least one packet based on RTP. At least one packet includes a first indication for indicating the difference between a decoding timestamp of the current frame and a presentation timestamp of the current frame. In addition, the bitstream is stored in a non-transitory computer-readable recording medium.

[0104] According to some further embodiments of the present disclosure, a non-transitory computer-readable recording medium is provided. The non-transitory computer-readable recording medium stores at least one packet generated by a method performed by an apparatus for visual data processing. In the method, conversion is performed between a bit stream of a current frame of visual data and at least one packet based on RTP. At least one packet includes a first indication for indicating a difference between a decoding timestamp of the current frame and a presentation timestamp of the current frame.

[0105] According to some further embodiments of the present disclosure, a method for storing at least one packet is proposed. In the method, conversion is performed between a bit stream of a current frame of visual data and at least one packet based on RTP. At least one packet includes a first indication of a difference between a decoding timestamp of the current frame and a presentation timestamp of the current frame. In addition, at least one packet is stored in a non-transitory computer-readable recording medium.

[0106] Embodiments of the present disclosure may be described according to the following clauses, features of which may be combined in any reasonable way.

[0107] Clause 1. A method for visual data processing, comprising: based on a real-time transport protocol (RTP), performing a conversion between a bit stream of a current frame of visual data and at least one packet, wherein the at least one packet includes a first indication for indicating a difference between a decoding timestamp of the current frame and a presentation timestamp of the current frame.

[0108] Clause 2. The method of claim 1 , wherein the first indication is encrypted.

[0109] Clause 3. The method of any one of claims 1 to 2, wherein the at least one packet comprises a plurality of packets, and the first indication is included in a first packet of the plurality of packets, the first packet being generated before the remaining packets of the plurality of packets.

[0110] Clause 4. The method of any one of Claims 1 to 3, wherein the first indication is a header extension.

[0111] Clause 5. The method of claim 4, wherein the header extension is a 1-byte header format or a 2-byte header format.

[0112] Clause 6. The method of any one of claims 4 to 5, wherein the header extension is encrypted based on a header extension encryption mechanism or a low-level security and authentication mechanism.

[0113] Clause 7. The method of any one of claims 1 to 6, wherein the current frame is a bidirectionally predicted frame (B frame).

[0114] Clause 8. A method according to any one of claims 1 to 7, wherein the method is implemented on a first device, the method further comprising: determining a value for a first indication based on a difference and a clock rate associated with the visual data, and wherein performing the conversion comprises: encapsulating the value for the first indication and a presentation timestamp of the current frame into at least one packet.

[0115] Clause 9. The method of claim 8, further comprising obtaining a bitstream from an encoder that encodes the current frame into the bitstream.

[0116] Clause 10. The method of claim 8, further comprising extracting a bitstream from at least one additional packet of the current frame based on another protocol other than RTP.

[0117] Clause 11. The method of any one of claims 8 to 10, further comprising: transmitting to the second device a second indication indicating that the first device supports the first indication; and receiving from the second device a third indication indicating that the second device supports the first indication.

[0118] Clause 12. A method according to any one of claims 1 to 7, wherein the method is implemented at a second device, performing the conversion comprises: extracting a value for a first indication and a presentation timestamp of a current frame from at least one packet; and the method further comprises: determining a decoding timestamp of the current frame based on the presentation timestamp, a clock rate associated with the visual data, and the value for the first indication.

[0119] Clause 13. The method of claim 12, further comprising: receiving from the first device a second indication indicating that the first device supports the first indication; and transmitting to the first device a third indication indicating that the second device supports the first indication.

[0120] Clause 14. The method of claim 11 or 13, wherein at least one of the second indication or the third indication is transmitted based on a Session Description Protocol (SDP).

[0121] Clause 15. The method of any one of Claims 1 to 7, wherein converting comprises encapsulating the bitstream into at least one packet.

[0122] Clause 16. The method of any one of Claims 1 to 7, wherein converting comprises extracting a bit stream from at least one packet.

[0123] Clause 17. An apparatus for visual data processing, comprising a processor and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to perform a method according to any one of claims 1 to 16.

[0124] Clause 18. A non-transitory computer-readable storage medium storing instructions that cause a processor to perform the method according to any one of claims 1 to 16.

[0125] Item 19. A non-transitory computer-readable recording medium storing a bitstream of a current frame of visual data generated by a method executed by a device for visual data processing, wherein the method includes: performing conversion between the bitstream and at least one packet based on a real-time transport protocol (RTP), wherein the at least one packet includes a first indication for indicating a difference between a decoding timestamp of the current frame and a presentation timestamp of the current frame.

[0126] Clause 20. A method for storing a bitstream of a current frame of visual data, comprising: performing conversion between a bitstream and at least one packet based on a real-time transport protocol (RTP); storing the bitstream in a non-transitory computer-readable recording medium, wherein at least one packet includes a first indication of a difference between a decoding timestamp of the current frame and a presentation timestamp of the current frame.

[0127] Item 21. A non-transitory computer-readable recording medium storing at least one packet generated by a method performed by an apparatus for visual data processing, wherein the method comprises: based on a real-time transport protocol (RTP), performing a conversion between a bit stream of a current frame of visual data and at least one packet, wherein at least one packet comprises a first indication for indicating a difference between a decoding timestamp of the current frame and a presentation timestamp of the current frame.

[0128] Clause 22. A method for storing at least one packet, comprising: performing conversion between a bit stream of a current frame of visual data and at least one packet based on a real-time transport protocol (RTP), storing the at least one packet in a non-transitory computer-readable recording medium, wherein the at least one packet includes a first indication for indicating a difference between a decoding timestamp of the current frame and a presentation timestamp of the current frame. Example Device

[0129] Fig.14 A block diagram of a computing device 1400 in which various embodiments of the present disclosure may be implemented is shown. The computing device 1400 may be implemented as a source device 110 (or RTP unit 113) or a destination device 120 (or RTP unit 123), or may be included in a source device 110 (or RTP unit 113) or a destination device 120 (or RTP unit 123).

[0130] It should be understood that Fig.14 The computing device 1400 shown in FIG. 1 is for illustrative purposes only and is not intended to in any way imply any limitation on the functionality and scope of the embodiments of the present disclosure.

[0131] like Fig.14 As shown, computing device 1400 includes a general computing device 1400. Computing device 1400 may include at least one or more processors or processing units 1410, memory 1420, storage unit 1430, one or more communication units 1440, one or more input devices 1450, and one or more output devices 1460.

[0132] In some embodiments, the computing device 1400 can be implemented as any user terminal or server terminal with computing capabilities. The server terminal can be a server provided by a service provider, a large computing device, etc. The user terminal can be, for example, any type of mobile terminal, fixed terminal or portable terminal, including a mobile phone, a station, a unit, a device, a multimedia computer, a multimedia tablet computer, an Internet node, a communicator, a desktop computer, a laptop computer, a notebook computer, a netbook computer, a personal communication system (PCS) device, a personal navigation device, a personal digital assistant (PDA), an audio / video player, a digital camera / camcorder, a positioning device, a television receiver, a radio broadcast receiver, an electronic book device, a gaming device, or any combination thereof, and includes accessories and peripherals of these devices, or any combination thereof. It is conceivable that the computing device 1400 can support any type of interface to the user (such as a "wearable" circuit device, etc.).

[0133] The processing unit 1410 may be a physical processor or a virtual processor and may implement various processes based on a program stored in the memory 1420. In a multi-processor system, multiple processing units execute computer executable instructions in parallel to improve the parallel processing capability of the computing device 1400. The processing unit 1410 may also be referred to as a central processing unit (CPU), a microprocessor, a controller, or a microcontroller.

[0134] The computing device 1400 typically includes various computer storage media. Such media can be any media accessible by the computing device 1400, including but not limited to volatile media and non-volatile media, or removable media and non-removable media. The memory 1420 can be a volatile memory (e.g., a register, a cache, a random access memory (RAM)), a non-volatile memory (such as a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM) or flash memory) or any combination thereof. The storage unit 1430 can be any removable or non-removable medium, and can include machine-readable media, such as a memory, a flash drive, a disk, or other media that can be used to store information and / or data and can be accessed in the computing device 1400.

[0135] The computing device 1400 may also include additional removable / non-removable storage media, volatile / non-volatile storage media. Fig.14 Although not shown in the figure, a magnetic disk drive for reading from and / or writing to a removable nonvolatile magnetic disk and an optical disk drive for reading from and / or writing to a removable nonvolatile optical disk may be provided. In this case, each drive may be connected to the bus (not shown) via one or more data medium interfaces.

[0136] The communication unit 1440 communicates with another computing device via a communication medium. In addition, the functionality of the components in the computing device 1400 can be implemented by a single computing cluster or multiple computing machines that can communicate via a communication connection. Therefore, the computing device 1400 can operate in a networked environment using logical connections to one or more other servers, networked personal computers (PCs), or other general network nodes.

[0137] Input device 1450 may be one or more of various input devices, such as a mouse, keyboard, trackball, voice input device, etc. Output device 1460 may be one or more of various output devices, such as a display, speaker, printer, etc. With the aid of communication unit 1440, computing device 1400 may also communicate with one or more external devices (not shown), such as storage devices and display devices, and computing device 1400 may also communicate with one or more devices that enable a user to interact with computing device 1400, or any device that enables computing device 1400 to communicate with one or more other computing devices (e.g., a network card, a modem, etc.), if necessary. Such communication may be performed via an input / output (I / O) interface (not shown).

[0138] In some embodiments, some or all components of the computing device 1400 may also be arranged in a cloud computing architecture rather than being integrated in a single device. In a cloud computing architecture, components may be provided remotely and work together to implement the functions described in the present disclosure. In some embodiments, cloud computing provides computing, software, data access and storage services, which will not require the end user to know the physical location or configuration of the system or hardware that provides these services. In various embodiments, cloud computing provides services via a wide area network (e.g., the Internet) using a suitable protocol. For example, a cloud computing provider provides applications via a wide area network, which can be accessed through a web browser or any other computing component. The software or components of the cloud computing architecture and the corresponding data may be stored on a remote server. The computing resources in a cloud computing environment may be merged or distributed at the location of a remote data center. Cloud computing infrastructures may provide services through a shared data center, although they appear as a single access point for users. Therefore, cloud computing architectures may be used to provide components and functions described herein from a service provider at a remote location. Alternatively, they may be provided by a conventional server, or installed directly or otherwise on a client device.

[0139] In an embodiment of the present disclosure, computing device 1400 may be used to implement video processing. Memory 1420 may include one or more visual data processing modules 1425 having one or more program instructions. These modules can be accessed and executed by processing unit 1410 to perform the functions of various embodiments described herein.

[0140] Although the present disclosure has been specifically shown and described with reference to the preferred embodiments of the present disclosure, it will be understood by those skilled in the art that various changes can be made in form and detail without departing from the spirit and scope of the present application as defined by the appended claims. These changes are intended to be encompassed by the scope of the present application. Therefore, the foregoing description of the embodiments of the present application is not intended to be limiting.< / uri> < / direction> < / value> < / extensionattributes> < / uri> < / direction> < / value>

Claims

1. A method for visual data processing, comprising: performing conversion between a bit stream of a current frame of visual data and at least one packet based on a real-time transport protocol (RTP), Wherein the at least one packet comprises a first indication for indicating a difference between a decoding timestamp of the current frame and a presentation timestamp of the current frame. The method of claim 1 , wherein the first indication is encrypted.

3. The method according to any one of claims 1 to 2, wherein the at least one packet comprises a plurality of packets, and the first indication is included in a first packet of the plurality of packets, the first packet being generated before the remaining packets of the plurality of packets.

4. The method according to any one of claims 1 to 3, wherein the first indication is a header extension. The method according to claim 4 , wherein the header extension is a 1-byte header format or a 2-byte header format.

6. The method according to any one of claims 4 to 5, wherein the header extension is encrypted based on a header extension encryption mechanism or a low-level security and authentication mechanism.

7. The method according to any one of claims 1 to 6, wherein the current frame is a bidirectionally predicted frame (B frame).

8. The method according to any one of claims 1 to 7, wherein the method is implemented on a first device, and the method further comprises: determining a value for the first indication based on the difference and a clock rate associated with the visual data, and Wherein performing the conversion comprises: The value for the first indication and the presentation timestamp of the current frame are encapsulated into the at least one packet.

9. The method according to claim 8, further comprising: The bitstream is obtained from an encoder that encodes the current frame into the bitstream.

10. The method according to claim 8, further comprising: The bitstream is extracted from at least one other packet of the current frame based on another protocol different from the RTP.

11. The method according to any one of claims 8 to 10, further comprising: Transmitting to a second device a second indication indicating that the first device supports the first indication; as well as A third indication is received from the second device indicating that the second device supports the first indication.

12. The method according to any one of claims 1 to 7, wherein the method is implemented at a second device, performing the conversion comprises: extracting from the at least one packet a value for the first indication and the presentation timestamp of the current frame; and The method further comprises: The decoding timestamp of the current frame is determined based on the presentation timestamp, a clock rate associated with the visual data, and the value for the first indication.

13. The method according to claim 12, further comprising: receiving, from a first device, a second indication indicating that the first device supports the first indication; as well as A third indication is transmitted to the first device, indicating that the second device supports the first indication.

14. The method according to claim 11 or 13, wherein at least one of the second indication or the third indication is transmitted based on a Session Description Protocol (SDP).

15. The method according to any one of claims 1 to 7, wherein the converting comprises encapsulating the bitstream into the at least one packet.

16. The method according to any one of claims 1 to 7, wherein the converting comprises extracting the bit stream from the at least one packet.

17. An apparatus for visual data processing, comprising a processor and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to perform the method according to any one of claims 1 to 16.

18. A non-transitory computer-readable storage medium storing instructions, the instructions causing a processor to execute the method according to any one of claims 1 to 16.

19. A non-transitory computer-readable recording medium storing a bitstream of a current frame of visual data generated by a method performed by an apparatus for visual data processing, wherein the method comprises: performing conversion between the bit stream and at least one packet based on a real-time transport protocol (RTP), Wherein the at least one packet comprises a first indication for indicating a difference between a decoding timestamp of the current frame and a presentation timestamp of the current frame.

20. A method for storing a bitstream of a current frame of visual data, comprising: performing conversion between the bitstream and at least one packet based on a real-time transport protocol (RTP); storing the bitstream in a non-transitory computer-readable recording medium, Wherein the at least one packet comprises a first indication for indicating a difference between a decoding timestamp of the current frame and a presentation timestamp of the current frame.

21. A non-transitory computer-readable recording medium storing at least one group generated by a method performed by an apparatus for visual data processing, wherein the method comprises: performing conversion between a bit stream of a current frame of visual data and the at least one packet based on a real-time transport protocol (RTP), Wherein the at least one packet comprises a first indication for indicating a difference between a decoding timestamp of the current frame and a presentation timestamp of the current frame.

22. A method for storing at least one packet, comprising: performing conversion between a bit stream of a current frame of visual data and the at least one packet based on a real-time transport protocol (RTP), storing the at least one packet in a non-transitory computer-readable recording medium, Wherein the at least one packet comprises a first indication for indicating a difference between a decoding timestamp of the current frame and a presentation timestamp of the current frame.

Images