An RTP encapsulation and decapsulation method, device and equipment for video slice data and a readable storage medium

Abstract

The application discloses an RTP encapsulation and decapsulation method, device and equipment of video slice data and a readable storage medium. In the process of encapsulating video code stream data, a video load data type identification value corresponding to a video slice data type is written in a video load data type field. Then, the slice data is encapsulated as a video elementary code stream, so that a slice data RTP data packet with the video load data type identification value is obtained. The slice data RTP data packet is subjected to decapsulation processing, so that the video slice data type is determined based on the video load data type identification value, that is, the video slice data type is determined as any one of an I frame slice, an RL frame slice, a P frame slice and a B frame slice, and the problem that RTP does not support encapsulation and decapsulation of video slice data is solved.

Description

Technical Field

[0001] This application relates to the field of communication technology, and in particular to a method, apparatus, device, and readable storage medium for RTP encapsulation and decapsulation of video clip data. Background Technology

[0002] Ultra-high-definition video live streaming and real-time communication applications demand efficient and reliable video transmission. Encoding and decoding of video streams are therefore crucial technologies for video transmission. In the encoding and decoding of video streams, the hierarchical structure of the bitstream contains lower layers within higher layers. The coding layers, from highest to lowest, are sequence, image, patch, and coding unit. For example, the AVS video coding standard introduced a patch (a group of adjacent largest coding units arranged in raster scan order) partitioning mechanism, allowing a frame to be divided into multiple independent regions for encoding; these independent regions are called frame patches. A patch is a rectangular region in an image, containing the portion of several largest coding units within the image; patches should not overlap. See the patch structure below. Figure 8 .

[0003] The RTP payload of AVS video is encapsulated and transmitted in the form of AVS video metastream. An AVS video's RTP payload can contain one or more AVS video metastreams, or a fragment of an AVS video metastream. An AVS video metastream is a video coded bitstream segment, composed of coded data between two or more adjacent start codes (including the first start code). The data type of the AVS video metastream is identified by the start code value, such as sequence header, user data, extended data, and intra / inter-frame prediction images. Specifically, an AVS video metastream of the intra / inter-frame prediction image type consists of consecutive bytes from the start code (inclusive) to the next intra / inter-frame prediction image start code, sequence start code, sequence end code, or video edit code. The start code of the AVS metastream is a specific set of bit strings. In bitstreams conforming to T / AI 109.2-2021, these bit strings should not appear under any circumstances other than the start code. The start code consists of a start code prefix and a start code value. All start codes should be byte-aligned.

[0004] An AVS video RTP packet consists of two parts: an RTP header and an AVS video payload, such as... Figure 7As shown. IETF RFC3550 defines the structure of the RTP header. According to the RTP packet format of AVS video, the fields in the RTP header should be set as follows: (1) Marker bit (M): This field is 1 bit, which specifies the boundary of the video frame. The marker bit of the last RTP packet of the video frame is set to 1, and the marker bits of other RTP packets are all set to 0; (2) Payload Type (PT): This field is 7 bits. According to the definition of IETF RFC 3551, the upper layer application should dynamically allocate a PT value in each RTP session (such as in SDP) and maintain the mapping relationship between the PT value and the payload format. The range of PT values ​​should follow the IETF RFC 3551 specification (e.g., 97-127); (3) Timestamp: This field is 32 bits. The RTP timestamp is set to the sampling timestamp of the encoded content. The clock used is 90000Hz, that is, the time unit of the RTP timestamp is 1 / 90000s. The initial value of the timestamp should be random, and the timestamps of different RTP packets of the same frame image should be consistent. For payloads that do not have time attributes, such as sequence headers, user extended data, and image headers, their timestamps should be consistent with the timestamps of the image data that immediately follows; (4) Sequence number: This field is 16 bits and represents the transmission sequence number of the RTP data packet. The sequence number is incremented by 1 for each RTP data packet sent. The receiver can use it to detect data packet loss and recover the data packet sequence. The initial value of the sequence number should be random; (4) Other fields in the RTP header (V, P, X, CC, etc.) should comply with the provisions of IETF RFC 3550.

[0005] Currently, RTP transmission technology does not support processing slice-level data and cannot meet the needs of actual video transmission. Therefore, how to effectively solve problems related to video transmission is a technical issue that urgently needs to be addressed by those skilled in the art. Summary of the Invention

[0006] The purpose of this application is to provide a method, apparatus, device, and readable storage medium for RTP encapsulation and decapsulation of video data, so as to identify the data type of the data segment when encapsulating the data packet and identify the data segment type when decapsulating the data packet.

[0007] To solve the above-mentioned technical problems, this application provides the following technical solution: A method for encapsulating video clip data using RTP, comprising: Determine the video segment data type of the video stream data; wherein, the video segment data type is the type corresponding to the video segment data; Write the video payload data type identifier value corresponding to the video slice data type into the video payload data type field; The video segment data is encapsulated as a video meta-stream to obtain video segment data RTP packets.

[0008] Write the video payload data type identifier value corresponding to the video slice data type in the video payload data type field, including: According to the video payload data type table, write the video payload data type identifier value corresponding to the video slice data type into the video payload data type field; The video payload data type table includes video slice data type information and its corresponding video payload data type identifier value; the video slice data type is I-frame slice, RL-frame slice, P-frame slice, or B-frame slice; the video payload data type field is located in the payload header of the RTP data packet and is used to indicate the data type of the RTP data packet payload.

[0009] The video fragment bitstream consists of consecutive bytes from the start code of the video frame (inclusive) to the start code of the next video frame; the data in the video fragment bitstream may include the image header immediately preceding the fragment, the video extension data after the image header, and user data.

[0010] The video meta stream types are I-frame fragment stream type, RL-frame fragment stream type, P-frame fragment stream type, and B-frame fragment stream type.

[0011] The meta stream of the I-frame fragment stream type consists of consecutive bytes starting from the fragment start code (inclusive) of the I-frame and ending before the next fragment start code of the I-frame; the data in the I-frame fragment stream type may optionally include the image header of the I-frame immediately preceding the fragment, as well as the video extension data and user data following the image header.

[0012] The meta stream of the RL frame fragment stream type consists of consecutive bytes starting from the fragment start code (inclusive) of the RL frame and ending before the next fragment start code of the RL frame; the data in the meta stream of the RL frame fragment stream type may optionally include the image header of the RL frame immediately preceding the fragment and its video extension data and user data after the image header.

[0013] The meta stream of the P-frame fragment stream type consists of consecutive bytes starting from the fragment start code (inclusive) of the P-frame and ending before the next fragment start code of the P-frame; the data in the meta stream of the P-frame fragment stream type may optionally include the image header of the P-frame immediately preceding the fragment, as well as the video extension data and user data after the image header.

[0014] The meta-stream of the B-frame fragment bitstream type consists of consecutive bytes starting from the start code of the B-frame (inclusive) and ending before the start code of the next B-frame fragment bitstream; the data in the meta-stream of the B-frame fragment bitstream type may optionally include the image header of the B-frame immediately preceding the fragment, as well as the video extension data and user data following the image header.

[0015] The video segment data is encapsulated as a video meta-stream to obtain segment data RTP packets, including: Based on the size of the video meta stream, the RTP load structure type is determined and the video meta stream is encapsulated according to the RTP load structure type to obtain fragmented RTP data packets; wherein, the RTP load structure type is a single load, a fragmented load, or an aggregated load.

[0016] The video segment data is encapsulated as a video meta-stream to obtain segment data RTP packets, including: If the video metastream does not exceed the size of the maximum transmission unit, the video segment data is encapsulated into a single payload type RTP data packet; Alternatively, if the video metastream exceeds the size of the maximum transmission unit, the video fragment data is encapsulated into fragmented payload type RTP data packets; Alternatively, if the size of consecutive video meta streams of the same type does not exceed the size of the maximum transmission unit, the video slice data is encapsulated into an RTP data packet of aggregate payload type.

[0017] A method for RTP decapsulation of video clip data, comprising: Retrieve the video payload data type field from the received RTP data packet; Determine the video payload data type identifier value based on the video payload data type field; The video segment data type of the RTP data packet is determined based on the video payload data type identifier value; the meta stream of the RTP data packet is determined to be a video meta stream based on the video segment data type, and the decapsulated video segment data is obtained based on the video meta stream.

[0018] The video segment data type of the RTP data packet is determined based on the video payload data type identifier value, including: The video segment data type of the RTP data packet is determined based on the video payload data type table and the video payload data type identifier value; the video payload data type table includes video segment data type information and its corresponding video payload data type identifier value.

[0019] The video clip data type is I-frame clip, RL-frame clip, P-frame clip, or B-frame clip.

[0020] The types of video meta streams include I-frame fragment stream type, RL-frame fragment stream type, P-frame fragment stream type, and B-frame fragment stream type.

[0021] The meta stream of the I-frame fragment stream type consists of consecutive bytes starting from the fragment start code (inclusive) of the I-frame and ending before the next fragment start code of the I-frame; the data in the meta stream of the I-frame fragment stream type may optionally include the image header of the I-frame immediately preceding the fragment, as well as the video extension data and user data after the image header.

[0022] The meta stream of the RL frame fragment stream type consists of consecutive bytes starting from the fragment start code (inclusive) of the RL frame and ending before the next fragment start code of the RL frame; the data in the meta stream of the RL frame fragment stream type may optionally include the image header of the RL frame immediately preceding the fragment, as well as the video extension data and user data after the image header.

[0023] The meta stream of the P-frame fragment stream type consists of consecutive bytes starting from the fragment start code (inclusive) of the P-frame and ending before the next fragment start code of the P-frame; the data in the meta stream of the P-frame fragment stream type may optionally include the image header of the P-frame immediately preceding the fragment, as well as the video extension data and user data after the image header.

[0024] The meta stream of the B-frame fragment stream type consists of consecutive bytes starting from the start code of the B-frame (inclusive) and ending before the next start code of the B-frame; the data in the meta stream of the B-frame fragment stream type may optionally include the image header of the B-frame immediately preceding the fragment, as well as the video extension data and user data after the image header.

[0025] The decapsulated video segment data is obtained from the video meta-stream, including: The RTP payload structure type is determined based on the video generic payload header of the RTP data packet, and the video meta-stream is decapsulated according to the RTP payload structure type to obtain video fragment data; the RTP payload structure type is a single payload, a fragmented payload, or an aggregated payload.

[0026] The decapsulated video segment data is obtained from the video meta-stream, including: If the RTP data packet is a single-payload RTP payload structure type, then the decapsulated video fragment data is obtained from the video meta-stream of the RTP data packet. Alternatively, if the RTP data packet is a fragmented RTP payload structure type, then the decapsulated video fragment data is obtained from the video meta stream of multiple RTP data packets. Alternatively, if the RTP data packet is an RTP payload structure type with aggregated payload, then multiple decapsulated video fragments are obtained from the video meta-stream of the RTP data packet.

[0027] The video payload data type field is located in the video payload header of the RTP packet and is used to indicate the data type of the RTP packet payload.

[0028] The size of the video segment corresponding to the video meta-stream is smaller than the size of the video image in which the video segment is located.

[0029] An RTP encapsulation device for video clip data, comprising: The type determination module is used to determine the video segment data type of the video stream data; wherein, the video segment data type is the type corresponding to the video segment data; The type identification module is used to write the video load data type identification value corresponding to the video slice data type into the video load data type field; The encapsulation processing module is used to encapsulate the video slice data as a video meta-stream to obtain slice data RTP data packets.

[0030] An RTP decapsulation and decapsulation device for video clip data, comprising: The field acquisition module is used to obtain the video payload data type field from the received RTP data packets; The identifier value acquisition module is used to determine the video payload data type identifier value based on the video payload data type field. The type identification module is used to determine the video segment data type of the RTP data packet based on the video payload data type identifier value; The decapsulation module is used to determine that the meta-stream of the RTP data packet is a video meta-stream based on the video segment data type, and to obtain the decapsulated video segment data based on the video meta-stream.

[0031] An electronic device, comprising: Memory, used to store computer programs; A processor is configured to implement the steps of the RTP encapsulation method for the video clip data when executing the computer program, or to implement the steps of the RTP decapsulation method for the video clip data when executing the computer program.

[0032] A readable storage medium storing a computer program, wherein when executed by a processor, the computer program implements the steps of the RTP encapsulation method for the video clip data described above, or, when executed by a processor, the computer program implements the steps of the RTP decapsulation method for the video clip data described above.

[0033] The video segment data RTP encapsulation method provided in this application embodiment is used to determine the video segment data type of the video stream data; wherein, the video segment data type is the type corresponding to the video segment data; the video payload data type identifier value corresponding to the video segment data type is written into the video payload data type field; the video segment data is encapsulated as a video meta stream to obtain the video segment data RTP data packet.

[0034] During the encapsulation of video stream data, a video payload data type identifier value corresponding to the video slice data type is written into the video payload data type field. Then, the video slice data is encapsulated as a video metastream to obtain slice data RTP packets with the video payload data type identifier value. Subsequent processing of the slice data RTP packets allows for the identification of the video slice data type based on this video payload data type identifier, specifically identifying which frame type (I-frame, RL-frame, P-frame, or B-frame) the slice corresponds to, overcoming the limitation of insufficiently detailed payload type identification when processing slice-level data.

[0035] The RTP decapsulation method for video segment data provided in this application involves obtaining the video payload data type field from the received RTP data packet; determining the video payload data type identifier value based on the video payload data type field; determining the video segment data type of the RTP data packet based on the video payload data type identifier value; determining the meta-stream of the RTP data packet as a video meta-stream based on the video segment data type; and obtaining the decapsulated video segment data based on the video meta-stream.

[0036] During the decapsulation of RTP data packets, the video payload data type identifier value written to the video payload data type field during the encapsulation phase can be obtained from the RTP data packet. Based on this identifier value, the video segment data type of the RTP data packet can be determined, specifically which frame type (I-frame, RL-frame, P-frame, or B-frame) the video segment corresponds to. Then, based on the video segment data type, the RTP data packet's meta-stream is determined to be a video meta-stream. Furthermore, based on the video meta-stream, the decapsulated video segment data corresponding to the definitively identified I-frame, RL-frame, P-frame, or B-frame is obtained, thus resolving the issue of RTP's lack of support for segment-level data encapsulation and decapsulation.

[0037] Accordingly, embodiments of this application also provide an RTP encapsulation device for video clip data, an RTP decapsulation device for video clip data, an electronic device, and a computer-readable storage medium corresponding to the above-described RTP encapsulation method and RTP decapsulation method for video clip data, which have the above-described technical effects and will not be repeated here. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1This is a flowchart illustrating the implementation of an RTP encapsulation method for video clip data in an embodiment of this application. Figure 2 This is a flowchart illustrating the implementation of an RTP decapsulation method for video clip data in an embodiment of this application. Figure 3 This is a schematic diagram of the structure of an RTP encapsulation device for video clip data in an embodiment of this application; Figure 4 This is a schematic diagram of the structure of an RTP decapsulation device for video clip data in an embodiment of this application; Figure 5 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application; Figure 6 This is a schematic diagram of the specific structure of an electronic device according to an embodiment of this application; Figure 7 This is a schematic diagram of an RTP data packet for video. Figure 8 This is a schematic diagram of a sheet structure; Figure 9 This is a diagram showing the meta-code stream structure of the I-frame fragment meta-code stream type corresponding to the first fragment within an I-frame in this embodiment of the application. Figure 10 This is a diagram showing the meta-code stream structure of the I-frame fragment meta-code stream type corresponding to the non-first fragment within the I-frame in this embodiment of the application. Figure 11 This is a syntax structure diagram of a single payload header in an embodiment of this application; Figure 12 This is a syntax structure diagram of the fragmented payload header in the embodiments of this application; Figure 13 This is a syntax structure diagram of the aggregate payload header in the embodiments of this application; Figure 14 This is an example diagram of the RTP load structure of the meta-bit stream corresponding to a single video segment data in a single load mode in this application embodiment; Figure 15 This is an example diagram of the RTP load structure of the metadata stream fragment corresponding to a single video fragment in the fragmented load mode of this application embodiment; Figure 16 This is an example diagram of an RTP load structure composed of multiple meta-streams corresponding to multiple video clips under the aggregated load mode in this application embodiment. Detailed Implementation

[0040] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0041] The existing RTP standard has a defect that the payload type does not support video slices: the RTP encapsulation scheme does not support slice data types in the payload header, which makes it difficult for the receiving end to directly know whether the current payload is an I-frame slice, RL-frame slice, P-frame slice or B-frame slice through the RTP header without parsing the start code inside the video meta-stream.

[0042] To address the aforementioned issues, this application proposes writing a video payload data type identifier corresponding to the video slice data type in the video payload data type field during the encapsulation process, thereby enabling the RTP header of the slice data RTP packet to carry information about whether the current payload is an I-frame slice, RL-frame slice, P-frame slice, or B-frame slice.

[0043] For details, please refer to Figure 1 , Figure 1 This is a flowchart of an RTP encapsulation method for video clip data according to an embodiment of this application. The method includes the following steps.

[0044] S101. Determine the video segment data type of the video stream data.

[0045] Among them, the video clip data type is the type corresponding to the video clip data, which is I-frame clip, RL-frame clip, P-frame clip or B-frame clip.

[0046] A frame refers to an image that makes up a video. A video has multiple frames, and frame data types include I-frames, RL frames, P-frames, and B-frames. Among them, I-frames (intra-frames) are entirely coded intra-frames, without referencing any other preceding or following frames, and can be decoded independently; they are generally used for random access points. P-frames (forward-predicting frames) are primarily based on forward inter-frame prediction, referencing only already encoded preceding frames. B-frames (bi-directional prediction frames) support bi-directional prediction, referencing both preceding and following frames. RL frames (low-latency frames, such as those in AVS) are a special subclass of P-frames; they only perform forward prediction, do not depend on future frames, and have latency constraints.

[0047] A video slice, obtained by dividing a video frame into slices, is a rectangular region within an image frame, containing the portion of the image containing several maximum coding units (MCUs). Slices should not overlap. Depending on the encoding method, such as intra-frame coding (encoding only the pixels of the current frame without referencing other frames) and inter-frame coding (encoding with reference to previously encoded frames), different video slice data types can be obtained, such as I-frame slices, RL-frame slices, P-frame slices, and B-frame slices.

[0048] Among them, the I-frame slice (Intra Patch) only performs intra-frame prediction and does not refer to any other frame. It can be decoded independently and used for random access points.

[0049] P-frames (Predictive Patch) are unidirectionally referenced, meaning they only refer to the previously coded frames and use forward motion compensation for prediction.

[0050] B-frames (Bidirectional Predictive Patch) are bidirectional references, meaning they reference both the preceding frame and the following coded frame.

[0051] RL frames (Low Delay Patch), such as those available in AVS, are dedicated P-class frames for low latency. They only predict forward and do not depend on future frames, reducing encoding wait time.

[0052] Once the video stream data is obtained, the data type of the video segments can be directly determined. For example, I-frames can be segmented to obtain I-frame segments.

[0053] S102. Write the video payload data type identifier value corresponding to the video clip data type in the video payload data type field.

[0054] The video payload data type field, PDT (payload_data_type), is an unsigned integer that indicates the data type of the payload and is located in the video payload header of the RTP packet.

[0055] In this embodiment, to effectively identify the specific frame type corresponding to the payload data of the data packet, the video payload data type identifier value corresponding to the video frame data type can be directly written into the video payload data type field. Specifically, the bits and values ​​corresponding to different video frame data types in the PDT can be predefined, and these bits and values ​​correspond to the video payload data type identifier.

[0056] In one specific embodiment of this application, writing a video payload data type identifier value corresponding to the video clip data type into the video payload data type field includes: According to the video payload data type table, write the video payload data type identifier value corresponding to the video slice data type in the video payload data type field; the video slice data type is I-frame slice, RL-frame slice, P-frame slice, or B-frame slice; the video payload data type field is located in the payload header of the RTP packet and is used to indicate the data type of the RTP packet payload.

[0057] The data types of video payloads are shown in Table 1.

[0058] Table 1. Example of Video Payload Data Type Definitions

[0059] In other words, referring to Table 1, when the video clip data type is I-frame, the PDT value is 9; when the video clip data type is RL-frame, the PDT value is 10; when the video clip data type is P-frame, the PDT value is 11; and when the video clip data type is B-frame, the PDT value is 12.

[0060] It's important to note that in practical applications, the payload data type value can also be other values ​​for different video slice data types, as long as the encoding and decoding are labeled and identified using the same lookup table. For example, the payload data type value for an I-frame slice can be 12, while a payload data type value of 10 represents a B-frame slice. Furthermore, the PDT field can be further expanded, such as by adding more slice data subtype identifiers to the PDT table to accommodate slice data transmission of newly added frame types.

[0061] The video fragment bitstream consists of consecutive bytes starting from (inclusive) the start code of a video frame and ending before the start code of the next video frame. The data within the video fragment bitstream may optionally include the immediately preceding image header, subsequent video extension data, and user data. Specifically, each fragment bitstream can consist of consecutive bytes starting from (inclusive) the start code of a video frame and ending before the start code of the next video frame. For different frame fragments belonging to the same frame, the data within the fragment bitstream corresponding to the first frame fragment includes the immediately preceding image header, subsequent video extension data, and user data; other frame fragments do not have an immediately preceding image header, nor subsequent video extension data or user data, therefore other frame fragments do not include an image header, nor subsequent video extension data or user data.

[0062] Among them, the types of video meta streams are I-frame fragment stream type, RL-frame fragment stream type, P-frame fragment stream type, and B-frame fragment stream type.

[0063] The meta-stream of the I-frame fragment stream type consists of consecutive bytes from the start code (inclusive) of the I-frame to the start code of the next I-frame. The data in the I-frame fragment stream type may optionally include the image header of the immediately preceding I-frame, its subsequent video extension data, and user data. In other words, the meta-stream of the I-frame fragment stream type consists of consecutive bytes from the start code (inclusive) of the I-frame to the start code of the next I-frame, and may also optionally include the image header of the immediately preceding I-frame, its subsequent video extension data, and user data. For example, if it is the first fragment within an I-frame, the meta-stream of the I-frame fragment stream type needs to include the image header of the I-frame before the start code, its subsequent video extension data, user data, and consecutive bytes from the start code (inclusive) of the I-frame to the start code of the next I-frame, such as... Figure 9 As shown; if it is not the first fragment in an I-frame, then the metadata stream of the I-frame fragment metadata stream type starts from the fragment start code (inclusive) of the I-frame and continues to the consecutive bytes before the fragment start code of the next I-frame. The consecutive bytes from the fragment start code (inclusive) to the fragment start code of the next I-frame correspond to the single I-frame fragment data and fragment end code, as shown. Figure 10 As shown.

[0064] The meta-stream of the RL frame fragment stream type consists of consecutive bytes from the start code (inclusive) of the RL frame to the start code of the next RL frame. The data in the meta-stream of the RL frame fragment stream type may optionally include the image header of the immediately preceding RL frame, its subsequent video extension data, and user data. That is, the meta-stream of the RL frame fragment stream type consists of consecutive bytes from the start code (inclusive) of the RL frame to the start code of the next RL frame, and may also optionally include the image header of the immediately preceding RL frame, its subsequent video extension data, and user data. For example, if it is the first fragment in an RL frame, the meta-stream of the RL frame fragment stream type needs to include the image header of the immediately preceding RL frame, its subsequent video extension data and user data, and consecutive bytes from the start code (inclusive) of the RL frame to the start code of the next RL frame; if it is not the first fragment in an RL frame, the meta-stream of the RL frame fragment stream type consists of consecutive bytes from the start code (inclusive) of the RL frame to the start code of the next RL frame.

[0065] The P-frame fragment stream type's metadata stream consists of consecutive bytes starting from (inclusive) the start code of the P-frame and ending before the start code of the next P-frame. The data in the P-frame fragment stream type's metadata stream may optionally include the image header of the immediately preceding P-frame, its subsequent video extension data, and user data. That is, the P-frame fragment stream type's metadata stream consists of consecutive bytes starting from (inclusive) the start code of the P-frame and ending before the start code of the next P-frame, and may also optionally include the image header of the immediately preceding P-frame, its subsequent video extension data, and user data. For example, if it is the first fragment within a P-frame, the P-frame fragment stream type's metadata stream includes the image header of the immediately preceding P-frame, its subsequent video extension data, user data, and consecutive bytes starting from (inclusive) the start code of the P-frame and ending before the start code of the next P-frame; if it is not the first fragment within a P-frame, the P-frame fragment stream type's metadata stream consists of consecutive bytes starting from (inclusive) the start code of the P-frame and ending before the start code of the next P-frame.

[0066] The B-frame fragment stream type's meta-stream consists of consecutive bytes starting from (inclusive) the start code of the B-frame and ending before the start code of the next B-frame. The data in the B-frame fragment stream type's meta-stream may optionally include the image header of the immediately preceding B-frame, its subsequent video extension data, and user data. That is, the B-frame fragment stream type's meta-stream consists of consecutive bytes starting from (inclusive) the start code of the B-frame and ending before the start code of the next B-frame, and may also optionally include the image header of the immediately preceding B-frame, its subsequent video extension data, and user data. For example, if it is the first fragment within a B-frame, the B-frame meta-stream type's meta-stream consists of the image header of the immediately preceding B-frame, its subsequent video extension data, user data, and consecutive bytes starting from (inclusive) the start code of the B-frame and ending before the start code of the next B-frame; if it is not the first fragment within a B-frame, the B-frame meta-stream type's meta-stream consists of consecutive bytes starting from (inclusive) the start code of the B-frame and ending before the start code of the next B-frame.

[0067] Furthermore, for the specific structures of RL frames, P frames, and B frames, please refer to [reference needed]. Figure 9 and Figure 10 When referring to the specific details, simply replace the frame type in the image. For example, the first frame of a P-frame can be compared to... Figure 9 Simply replace "I-frame" with "P-frame". Correspondingly, the non-first frames within the P-frame can be compared... Figure 10 Simply replace the words "I frame" with "P frame"; they will not be drawn one by one here.

[0068] S103. Encapsulate the video slice data as a video meta-stream to obtain slice data RTP data packets.

[0069] After writing the video payload data type identifier corresponding to the video slice data type into the video payload data type field, the video slice data can be encapsulated as a video meta stream to obtain slice data RTP packets with slice-level corresponding data payload types.

[0070] Furthermore, in practical implementation, the video segment size can be adaptively adjusted, such as dynamically adjusting the video segment size based on network bandwidth and decoding performance, optimizing the balance between packet loss resistance and decoding efficiency. Redundancy strategies can also be combined, such as adding lightweight redundancy information to the segment data encapsulation, further improving the reliability of segment data transmission in weak network environments.

[0071] Using the method provided in the embodiments of this application, the video segment data type of the video stream data is determined; wherein, the video segment data type is the type corresponding to the video segment data; the video payload data type identifier value corresponding to the video segment data type is written into the video payload data type field; the video segment data is encapsulated as a video metastream to obtain video segment data RTP data packets.

[0072] During the encapsulation of video stream data, a video payload data type identifier value corresponding to the video slice data type is written into the video payload data type field. Then, the video slice data is encapsulated as a video metastream to obtain slice data RTP packets with the video payload data type identifier value. Subsequent processing of the slice data RTP packets allows for the identification of the video slice data type based on this video payload data type identifier, specifically identifying which frame type (I-frame, RL-frame, P-frame, or B-frame) the slice corresponds to, overcoming the limitation of insufficiently detailed payload type identification when processing slice-level data.

[0073] It should be noted that, based on the above embodiments, the embodiments of this application also provide corresponding improvement schemes. In the preferred / improved embodiments, the same or corresponding steps as in the above embodiments can be referred to each other, and the corresponding beneficial effects can also be referred to each other; however, these will not be elaborated upon in the preferred / improved embodiments herein.

[0074] In one specific embodiment of this application, video slice data is encapsulated as a video meta-stream to obtain slice data RTP packets, including: Based on the size of the video meta stream, the RTP payload structure type is determined and the video meta stream is encapsulated according to the RTP payload structure type to obtain fragmented RTP data packets; wherein, the RTP payload structure type is a single payload, a fragmented payload, or an aggregated payload.

[0075] Specifically, RTP transmission requires differentiated processing of frame segments with varying data sizes. It necessitates consideration of IP network transmission limitations and the Maximum Transmission Unit (MTU), necessitating different encapsulation types for video segment data that is larger or significantly smaller than the MTU.

[0076] To address the aforementioned issues, in this embodiment, the RTP payload structure type can be determined based on the size of the video meta stream, and the video meta stream can be encapsulated according to the RTP payload structure type to obtain fragmented RTP data packets; wherein, the RTP payload structure type is a single payload, fragmented payload, or aggregated payload structure type.

[0077] Among them, a single load includes a general video load header and a single load header (such as...). Figure 11 (as shown) and a video meta-stream; in the fragmented payload, there is a general video payload header, a fragmented payload header (as shown) and a video meta-stream; Figure 12 (as shown) and a video meta-stream fragment; in the aggregated payload, the video payload includes a general payload header, multiple aggregated payload headers (such as... Figure 13 (as shown) and the corresponding multiple video metastreams.

[0078] Specifically, based on the size of the video metastream, the RTP payload structure type is determined and the video metastream is encapsulated according to the RTP payload structure type to obtain fragmented RTP data packets, including: If the video metastream does not exceed the maximum transmission unit size, the video segment data is encapsulated into RTP packets of a single payload type, such as... Figure 14 As shown; Alternatively, if the video metastream exceeds the maximum transmission unit size, the video fragment data is encapsulated into fragmented payload type RTP packets, such as... Figure 15 As shown; Alternatively, if the size of consecutive video metadata streams of the same type does not exceed the maximum transmission unit size, the video slice data is encapsulated into RTP packets of aggregate payload type, such as... Figure 16 As shown, provided the size of the aggregate packet meets the MTU limit of the IP layer, consecutive video fragment streams of the same type are allowed to be encapsulated into one aggregate packet.

[0079] In other words, when encapsulating video metastreams, the size of the video metastream can be compared with the size of the maximum transmission unit (MTU) to select the appropriate payload encapsulation method. If the video metastream does not exceed the MTU size, video fragments can be encapsulated into RTP packets of a single payload type. However, if the size of consecutive video metastreams of the same type does not exceed the MTU size, these video fragments can be encapsulated into RTP packets of an aggregate payload type. If the video metastream exceeds the MTU size, the video metastream needs to be divided into multiple fragments, and each fragment of the metastream needs to be encapsulated separately, i.e., the video fragment data is encapsulated into RTP packets of a fragmented payload type.

[0080] As can be seen, the method of the present invention can be adapted to a variety of transmission scenarios. Through three load structures—single, fragmented, and aggregated—it can perfectly adapt to different resolutions and network MTU limitations, thereby improving the utilization rate of data carried by RTP data packets.

[0081] Furthermore, in practical applications, a dynamic encapsulation strategy for different frame types can be implemented. This involves enabling full encapsulation for I-frames / RL-frames and selectively encapsulating P / B-frames based on network packet loss rate and service priority, thereby balancing bandwidth and packet loss resistance.

[0082] Corresponding to the above embodiment of the RTP encapsulation method for video clip data, this application embodiment also provides an RTP decapsulation method for video clip data. The RTP decapsulation method for video clip data described below can be referred to in conjunction with the RTP encapsulation method for video clip data described above.

[0083] Please refer to Figure 2 The RTP decapsulation method for video clip data includes the following steps.

[0084] S201. Obtain the video payload data type field from the received RTP data packet.

[0085] The RTP data packet can be generated using, for example... Figure 1 The RTP encapsulation method shown above encapsulates video stream data into a data packet. As mentioned above, the PDT (Video Payload Data Type Field) of this data packet contains a video payload data type identifier value. Therefore, the video payload data type field can be directly obtained from the received RTP data packet.

[0086] S202. Determine the video payload data type identifier value based on the video payload data type field.

[0087] The video payload data type identifier is a unique identifier value that is written into the PDT during RTP encapsulation.

[0088] In one specific embodiment of this application, the video payload data type field is located in the video payload header of the RTP data packet, used to indicate the data type of the RTP data packet payload. The video payload header includes a PDT (Packet Data Type), which indicates the data type of the RTP data packet payload. In practical applications, this can also enhance fault tolerance; that is, explicit fragment type identification helps the receiving end to adopt more accurate recovery strategies when packets are lost.

[0089] S203. Determine the video segment data type of the RTP data packet based on the video payload data type identifier value.

[0090] In one specific embodiment of this application, determining the video fragment data type of an RTP data packet based on a video payload data type identifier value includes: The video segment data type of the RTP data packet is determined based on the video payload data type table and the video payload data type identifier value; the video payload data type table includes video segment data type information and its corresponding video payload data type identifier value.

[0091] The video slice data type can be I-frame slice, RL-frame slice, P-frame slice, or B-frame slice.

[0092] Once the video payload data type identifier value is obtained, the video segment data type of the RTP data packet can be determined based on the mapping relationship between the video payload data type identifier value and the video segment data type. That is, it can be determined which type of segment the payload data in the RTP data packet belongs to: I-frame, RL-frame, P-frame, or B-frame.

[0093] The video payload data type table can be found in Table 1. Table 1 shows that it includes the types of video metastreams, and these types are the data types of the video slice payload metastreams, specifically I-frame slices, RL-frame slices, P-frame slices, and B-frame slices. In other words, once the video payload data type identifier is determined, the video slice data type of the RTP data packet can be identified, specifically which of the I-frame, RL-frame, P-frame, or B-frame slices corresponds to the metastream.

[0094] Among them, the types of video meta streams include I-frame fragment stream type, RL-frame fragment stream type, P-frame fragment stream type, and B-frame fragment stream type.

[0095] The meta stream of the I-frame fragment stream type consists of consecutive bytes from the start of the I-frame (inclusive) to the start of the next I-frame fragment; the data in the meta stream of the I-frame fragment stream type may include the image header of the I-frame immediately preceding the fragment, as well as the video extension data and user data after the image header.

[0096] The meta stream of the RL frame fragment bitstream type consists of consecutive bytes from the start of the RL frame (inclusive) to the start of the next RL frame bitstream; the data in the meta stream of the RL frame fragment bitstream type may include the image header of the RL frame immediately preceding the fragment, as well as the video extension data and user data after the image header.

[0097] The P-frame fragment bitstream type of meta bitstream consists of consecutive bytes starting from the start code of the P-frame (inclusive) and ending before the start code of the next P-frame. The data in the P-frame fragment bitstream type of meta bitstream may include the image header of the P-frame immediately preceding the fragment, as well as the video extension data and user data after the image header.

[0098] The meta stream of the B-frame fragment stream type consists of consecutive bytes from the start code of the B-frame (inclusive) to the start code of the next B-frame. The data in the meta stream of the B-frame fragment stream type may include the image header of the B-frame immediately preceding the slice and its video extension data and user data after the image header.

[0099] S204. Determine the RTP data packet's meta stream as a video meta stream based on the video segment data type, and obtain the decapsulated video segment data based on the video meta stream.

[0100] Based on the video fragment data type, the RTP data packet's metastream can be determined to be a video metastream. Then, the decapsulated video fragment data can be obtained from the video metastream. This video fragment data is the decapsulated frame fragment data.

[0101] In this context, the size of the video segment corresponding to the video metastream is smaller than the size of the video image containing the video segment. Since a video segment is obtained by dividing a video image into segments, the size of the video segment corresponding to this video metastream will be smaller than the size of the video image containing the video segment.

[0102] In one specific embodiment of this application, obtaining decapsulated video segment data based on the video meta-stream includes: The RTP payload structure type is determined based on the video generic payload header of the RTP data packet, and the video metadata stream is decapsulated according to the RTP payload structure type to obtain video fragment data; the RTP payload structure type is a single payload, fragmented payload, or aggregated payload.

[0103] In other words, when decapsulating RTP packets, the RTP payload structure type needs to be determined based on the video general payload header of the RTP packets. This RTP payload structure type can be a single payload, a fragmented payload, or an aggregated payload. Then, the video metadata stream is decapsulated according to the decapsulation method corresponding to the corresponding RTP payload structure type to obtain the video fragment data.

[0104] The decapsulated video segment data obtained from the video meta-stream includes: If the RTP data packet is a single-payload RTP payload structure type, then the decapsulated video fragment data is obtained from the video meta-stream of the RTP data packet. Alternatively, if the RTP packets are of the fragmented RTP payload structure type, then the decapsulated video fragment data is obtained from the video meta stream of multiple RTP packets. Alternatively, if the RTP packet is an RTP payload structure type with aggregated payload, then multiple decapsulated video segments are obtained from the video metastream of the RTP packet.

[0105] In other words, when the RTP packet has a single payload, a single video segment can be decapsulated from one RTP packet. If the RTP packet has a fragmented payload structure, a single decapsulated video segment needs to be obtained from the video metastream of multiple RTP packets. If the RTP packet has an aggregated payload structure, multiple decapsulated video segments can be obtained from the video metastream of one RTP packet.

[0106] As can be seen, the decapsulation process can be quickly parsed and efficiently processed. That is, the receiving end does not need to parse the complex internal syntax of the video meta-stream, but can know the segment type only through the RTP payload header, which facilitates fast packet loss processing or QoE scheduling.

[0107] The RTP decapsulation method for video segment data provided in this application involves obtaining the video payload data type field from the received RTP data packet; determining the video payload data type identifier value based on the video payload data type field; determining the video segment data type of the RTP data packet based on the video payload data type identifier value; determining the meta-stream of the RTP data packet as a video meta-stream based on the video segment data type; and obtaining the decapsulated video segment data based on the video meta-stream.

[0108] During the decapsulation and recapsulation of RTP data packets, the video payload data type identifier value written to the video payload data type field during the encapsulation phase can be obtained from the RTP data packet. Based on this identifier value, the video segment data type of the RTP data packet can be determined, specifically which frame type (I-frame, RL-frame, P-frame, or B-frame) the video segment corresponds to. Then, based on the video segment data type, the RTP data packet's meta-stream is determined to be a video meta-stream. Furthermore, based on the video meta-stream, the decapsulated video segment data corresponding to the definitively identified I-frame, RL-frame, P-frame, or B-frame is obtained. This solves the problem that the existing RTP standard does not support the encapsulation and decapsulation of video segment data.

[0109] As can be seen, the method provided in this application, based on the slice data types of I-frames, RL-frames, P-frames, and B-frames, matches and writes the corresponding Video Payload Data Type (PDT) identifier, improves the PDT table to include video fragment bitstream types, and realizes differentiated RTP encapsulation of slice data of different frame types; at the same time, it supports three payload structures: single, fragmented, and aggregated. The receiving end accurately identifies the slice data type through the PDT identifier and completes decapsulation, fully releasing the independence of patch encoding, and solving the problems of no slice type identifier, coarse encapsulation granularity, and weak packet loss resistance in the prior art.

[0110] Corresponding to the above method embodiments, this application also provides an RTP encapsulation device for video clip data. The RTP encapsulation device for video clip data described below and the RTP encapsulation method for video clip data described above can be referred to each other.

[0111] See Figure 3 As shown, the device includes the following modules: The type determination module 101 is used to determine the video segment data type of the video stream data; wherein, the video segment data type is the type corresponding to the video segment data; The type identifier module 102 is used to write the video load data type identifier value corresponding to the video slice data type into the video load data type field; The encapsulation processing module 103 is used to encapsulate the video slice data as a video meta-stream to obtain video slice data RTP data packets.

[0112] Using the apparatus provided in the embodiments of this application, the video segment data type of the video stream data is determined; wherein, the video segment data type is the type corresponding to the video segment data; the video payload data type identifier value corresponding to the video segment data type is written into the video payload data type field; the video segment data is encapsulated as a video meta-stream to obtain segment data RTP data packets.

[0113] During the encapsulation of video stream data, a video payload data type identifier value corresponding to the video slice data type is written into the video payload data type field. Then, the video slice data is encapsulated as a video metastream, resulting in slice data RTP packets with the video payload data type identifier value. Subsequent processing of these slice data RTP packets allows for the identification of the video slice data type based on this video payload data type identifier, specifically determining which frame type (I-frame, RL-frame, P-frame, or B-frame) corresponds to the slice. This solves the problem of existing RTP standards not supporting the encapsulation and decapsulation of video slice data.

[0114] In one specific embodiment of this application, the type identification module is specifically used to write a video load data type identifier value corresponding to the video slice data type in the video load data type field according to the video load data type table; wherein, the video load data type table includes video slice data type information and its corresponding video load data type identifier value; the video slice data type is I-frame slice, RL-frame slice, P-frame slice or B-frame slice; the video load data type field is located in the payload header of the RTP data packet and is used to indicate the data type of the RTP data packet payload.

[0115] In one specific embodiment of this application, the video fragment bitstream consists of consecutive bytes from the start code (inclusive) of the video frame to the start code of the next video frame; the data in the video fragment bitstream may optionally include the image header immediately preceding the fragment, the video extension data after the image header, and user data.

[0116] In one specific embodiment of this application, the video meta stream type is I-frame fragment stream type, RL-frame fragment stream type, P-frame fragment stream type, and B-frame fragment stream type.

[0117] In one specific embodiment of this application, the meta stream of the I-frame fragment stream type comprises consecutive bytes from the start code of the I-frame (inclusive) to the start code of the next I-frame; the data in the I-frame fragment stream type may optionally include the image header of the I-frame immediately preceding the fragment and its video extension data and user data after the image header.

[0118] In one specific embodiment of this application, the meta-stream of the RL frame fragment bitstream type comprises consecutive bytes from the start of the RL frame (inclusive) to the start of the next RL frame; the data in the meta-stream of the RL frame fragment bitstream type may optionally include the image header of the RL frame immediately preceding the fragment and its subsequent video extension data and user data.

[0119] In one specific embodiment of this application, the meta-bitstream of the P-frame fragment bitstream type comprises consecutive bytes from the start code of the P-frame (inclusive) to the start code of the next P-frame; the data in the meta-bitstream of the P-frame fragment bitstream type may optionally include the image header of the P-frame immediately preceding the fragment and its video extension data and user data after the image header.

[0120] In one specific embodiment of this application, the meta-bitstream of the B-frame fragment bitstream type comprises consecutive bytes from the start code of the B-frame (inclusive) to the start code of the next B-frame; the data in the meta-bitstream of the B-frame fragment bitstream type may optionally include the image header of the B-frame immediately preceding the slice and its video extension data and user data after the image header.

[0121] In one specific embodiment of this application, the encapsulation processing module is specifically used to determine the RTP load structure type according to the size of the video meta stream and encapsulate the video meta stream according to the RTP load structure type to obtain fragmented RTP data packets; wherein, the RTP load structure type is a single load, a fragmented load, or an aggregated load.

[0122] In one specific embodiment of this application, the encapsulation processing module is specifically used to encapsulate video slice data into RTP packets of a single payload type if the video meta stream does not exceed the size of the maximum transmission unit; or, if the video meta stream exceeds the size of the maximum transmission unit, encapsulate video slice data into RTP packets of a fragmented payload type; or, if the size of consecutive video meta streams of the same type does not exceed the size of the maximum transmission unit, encapsulate video slice data into RTP packets of an aggregated payload type.

[0123] Corresponding to the above method embodiments, this application also provides an RTP decapsulation device for video clip data. The RTP decapsulation device for video clip data described below can be referred to in correspondence with the RTP decapsulation method for video clip data described above.

[0124] See Figure 4 As shown, the device includes the following modules: Field acquisition module 201 is used to obtain video payload data type fields from received RTP data packets; The identifier value acquisition module 202 is used to determine the video payload data type identifier value based on the video payload data type field. The type identification module 203 is used to determine the video fragment data type of the RTP data packet based on the video payload data type identifier value; The decapsulation module 204 is used to determine that the meta stream of the RTP data packet is a video meta stream based on the video segment data type, and to obtain the decapsulated video segment data based on the video meta stream.

[0125] Using the apparatus provided in the embodiments of this application, a video payload data type field is obtained from the received RTP data packet; a video payload data type identifier value is determined based on the video payload data type field; the video segment data type of the RTP data packet is determined based on the video payload data type identifier value; the meta-stream of the RTP data packet is determined to be a video meta-stream based on the video segment data type, and the decapsulated video segment data is obtained based on the video meta-stream.

[0126] During the decapsulation of RTP data packets, the video payload data type identifier value written to the video payload data type field during the encapsulation phase can be obtained from the RTP data packet. Based on this identifier value, the video segment data type of the RTP data packet can be determined, specifically which frame type (I-frame, RL-frame, P-frame, or B-frame) the video segment corresponds to. Then, based on the video segment data type, the RTP data packet's meta-stream is determined to be a video meta-stream. Furthermore, based on the video meta-stream, the decapsulated video segment data, which clearly identifies the frame type (I-frame, RL-frame, P-frame, or B-frame), is obtained, overcoming the limitation of insufficient payload type identification when processing segment-level data.

[0127] In one specific embodiment of this application, the type identification module is specifically used to determine the video segment data type of the RTP data packet based on the video payload data type table and the video payload data type identifier value; the video payload data type table includes video segment data type information and its corresponding video payload data type identifier value.

[0128] In one specific embodiment of this application, the video slice data type is I-frame slice, RL-frame slice, P-frame slice, or B-frame slice.

[0129] In one specific embodiment of this application, the video meta stream type includes I-frame fragment stream type, RL-frame fragment stream type, P-frame fragment stream type, and B-frame fragment stream type.

[0130] In one specific embodiment of this application, the meta-bitstream of the I-frame fragment bitstream type comprises consecutive bytes from the start code of the I-frame (inclusive) to the start code of the next I-frame; the data in the meta-bitstream of the I-frame fragment bitstream type may optionally include the image header of the I-frame immediately preceding the fragment and its video extension data and user data after the image header.

[0131] In one specific embodiment of this application, the meta-stream of the RL frame fragment bitstream type comprises consecutive bytes from the start of the RL frame (inclusive) to the start of the next RL frame; the data in the meta-stream of the RL frame fragment bitstream type may optionally include the image header of the RL frame immediately preceding the fragment and its subsequent video extension data and user data.

[0132] In one specific embodiment of this application, the meta-bitstream of the P-frame fragment bitstream type comprises consecutive bytes from the start code of the P-frame (inclusive) to the start code of the next P-frame; the data in the meta-bitstream of the P-frame fragment bitstream type may optionally include the image header of the P-frame immediately preceding the fragment and its video extension data and user data after the image header.

[0133] In one specific embodiment of this application, the meta stream of the B-frame fragment stream type comprises consecutive bytes from the start code of the B-frame (inclusive) to the start code of the next B-frame; the data in the meta stream of the B-frame fragment stream type may optionally include the image header of the B-frame immediately preceding the slice and its video extension data and user data after the image header.

[0134] In one specific embodiment of this application, the decapsulation module is specifically used to determine the RTP payload structure type based on the video general payload header of the RTP data packet, and decapsulate the video meta-stream according to the RTP payload structure type to obtain video fragment data; the RTP payload structure type is a single payload, fragmented payload, or aggregated payload.

[0135] In one specific embodiment of this application, the decapsulation module is specifically used to obtain decapsulated video fragment data from the video meta-stream of the RTP data packet if the RTP data packet is a single-payload RTP payload structure type; or, if the RTP data packet is a fragmented payload RTP payload structure type, obtain decapsulated video fragment data from the video meta-stream of multiple RTP data packets; or, if the RTP data packet is an aggregated payload RTP payload structure type, obtain multiple decapsulated video fragment data from the video meta-stream of the RTP data packet.

[0136] In one specific embodiment of this application, the video payload data type field is located in the video payload header of the RTP data packet and is used to indicate the data type of the RTP data packet payload.

[0137] In one specific embodiment of this application, the size of the video slice corresponding to the video meta-stream is smaller than the size of the video image in which the video slice is located.

[0138] Corresponding to the above method embodiments, this application also provides an electronic device. The electronic device described below can be referred to in conjunction with the RTP encapsulation method and the RTP decapsulation method for video clip data described above.

[0139] See Figure 5 As shown, the electronic device includes: Memory 332 is used to store computer programs; The processor 322 is configured to implement the steps of the RTP encapsulation method for video segment data in the above method embodiments when executing a computer program, or to implement the steps of the RTP decapsulation method for video segment data in the above method embodiments when executing a computer program.

[0140] For details, please refer to Figure 6 , Figure 6This is a schematic diagram of the specific structure of an electronic device provided in this embodiment. The electronic device can vary significantly due to differences in configuration or performance. It may include one or more central processing units (CPUs) (e.g., one or more processors) and a memory 332. The memory 332 stores one or more computer programs 342 or data 344. The memory 332 can be temporary or permanent storage. The program stored in the memory 332 may include one or more modules (not shown in the diagram), each module may include a series of instruction operations on the data processing device. Furthermore, the processor 322 may be configured to communicate with the memory 332 and execute the series of instruction operations stored in the memory 332 on the electronic device 301.

[0141] Electronic device 301 may also include one or more power supplies 326, one or more wired or wireless network interfaces 350, one or more input / output interfaces 358, and / or one or more operating systems 341.

[0142] The steps in the RTP encapsulation method for video data described above can be implemented by the structure of an electronic device, and the steps in the RTP decapsulation method for video data described above can be implemented by the structure of an electronic device.

[0143] Corresponding to the above method embodiments, this application also provides a readable storage medium. The readable storage medium described below can be referred to in conjunction with the RTP encapsulation method and the RTP decapsulation method for video clip data described above.

[0144] A readable storage medium storing a computer program, wherein when the computer program is executed by a processor, it implements the steps of the RTP encapsulation method for video segment data in the above method embodiments, or, when the computer program is executed by a processor, it implements the steps of the RTP decapsulation method for video segment data in the above method embodiments.

[0145] The readable storage medium can specifically be a USB flash drive, external hard drive, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, or any other readable storage medium capable of storing program code.

[0146] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0147] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0148] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0149] Finally, it should be noted that in this document, relationships such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "include," "contain," or any other variations are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus.

[0150] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for RTP encapsulation of video clip data, characterized in that, include: Determine the video segment data type of the video stream data; wherein, the video segment data type is the type corresponding to the video segment data; Write the video payload data type identifier value corresponding to the video slice data type into the video payload data type field; The video segment data is encapsulated as a video meta-stream to obtain video segment data RTP packets.

2. The method according to claim 1, characterized in that, Write the video payload data type identifier value corresponding to the video slice data type in the video payload data type field, including: According to the video payload data type table, write the video payload data type identifier value corresponding to the video slice data type into the video payload data type field; The video payload data type table includes video slice data type information and its corresponding video payload data type identifier value; the video slice data type is I-frame slice, RL-frame slice, P-frame slice, or B-frame slice; the video payload data type field is located in the payload header of the RTP data packet and is used to indicate the data type of the RTP data packet payload.

3. The method according to claim 1, characterized in that, The video fragment bitstream consists of consecutive bytes from the start code of the video frame (inclusive) to the start code of the next video frame; the data in the video fragment bitstream may include the image header immediately preceding the fragment, the video extension data after the image header, and user data.

4. The method according to claim 1, characterized in that, The video meta stream types are I-frame fragment stream type, RL-frame fragment stream type, P-frame fragment stream type, and B-frame fragment stream type.

5. The method according to claim 4, characterized in that, The meta stream of the I-frame fragment stream type consists of consecutive bytes starting from the fragment start code (inclusive) of the I-frame and ending before the next fragment start code of the I-frame; the data in the I-frame fragment stream type may optionally include the image header of the I-frame immediately preceding the fragment, as well as the video extension data and user data following the image header.

6. The method according to claim 4, characterized in that, The meta stream of the RL frame fragment stream type consists of consecutive bytes starting from the fragment start code (inclusive) of the RL frame and ending before the next fragment start code of the RL frame; the data in the meta stream of the RL frame fragment stream type may optionally include the image header of the RL frame immediately preceding the fragment and its video extension data and user data after the image header.

7. The method according to claim 4, characterized in that, The meta stream of the P-frame fragment stream type consists of consecutive bytes starting from the fragment start code (inclusive) of the P-frame and ending before the next fragment start code of the P-frame; the data in the meta stream of the P-frame fragment stream type may optionally include the image header of the P-frame immediately preceding the fragment, as well as the video extension data and user data after the image header.

8. The method according to claim 4, characterized in that, The meta-stream of the B-frame fragment bitstream type consists of consecutive bytes starting from the start code of the B-frame (inclusive) and ending before the start code of the next B-frame fragment bitstream; the data in the meta-stream of the B-frame fragment bitstream type may optionally include the image header of the B-frame immediately preceding the fragment, as well as the video extension data and user data following the image header.

9. The method according to any one of claims 1 to 8, characterized in that, The video segment data is encapsulated as a video meta-stream to obtain segment data RTP packets, including: Based on the size of the video meta stream, the RTP load structure type is determined and the video meta stream is encapsulated according to the RTP load structure type to obtain fragmented RTP data packets; wherein, the RTP load structure type is a single load, a fragmented load, or an aggregated load.

10. The method according to any one of claims 1 to 8, characterized in that, The video segment data is encapsulated as a video meta-stream to obtain segment data RTP packets, including: If the video metastream does not exceed the size of the maximum transmission unit, the video segment data is encapsulated into a single payload type RTP data packet; Alternatively, if the video metastream exceeds the size of the maximum transmission unit, the video fragment data is encapsulated into fragmented payload type RTP data packets; Alternatively, if the size of consecutive video meta streams of the same type does not exceed the size of the maximum transmission unit, the video slice data is encapsulated into an RTP data packet of aggregate payload type.

11. A method for RTP decapsulation of video clip data, characterized in that, include: Retrieve the video payload data type field from the received RTP data packet; Determine the video payload data type identifier value based on the video payload data type field; The video segment data type of the RTP data packet is determined based on the video payload data type identifier value; Based on the video segment data type, the meta stream of the RTP data packet is determined to be a video meta stream, and the decapsulated video segment data is obtained based on the video meta stream.

12. The method according to claim 11, characterized in that, The video segment data type of the RTP data packet is determined based on the video payload data type identifier value, including: The video segment data type of the RTP data packet is determined based on the video payload data type table and the video payload data type identifier value; the video payload data type table includes video segment data type information and its corresponding video payload data type identifier value.

13. The method according to claim 11, characterized in that, The video clip data type is I-frame clip, RL-frame clip, P-frame clip, or B-frame clip.

14. The method according to claim 12, characterized in that, The types of video meta streams include I-frame fragment stream type, RL-frame fragment stream type, P-frame fragment stream type, and B-frame fragment stream type.

15. The method according to claim 14, characterized in that, The meta stream of the I-frame fragment stream type consists of consecutive bytes starting from the fragment start code (inclusive) of the I-frame and ending before the next fragment start code of the I-frame; the data in the meta stream of the I-frame fragment stream type may optionally include the image header of the I-frame immediately preceding the fragment, as well as the video extension data and user data after the image header.

16. The method according to claim 14, characterized in that, The meta stream of the RL frame fragment stream type consists of consecutive bytes starting from the fragment start code (inclusive) of the RL frame and ending before the next fragment start code of the RL frame; the data in the meta stream of the RL frame fragment stream type may optionally include the image header of the RL frame immediately preceding the fragment, as well as the video extension data and user data after the image header.

17. The method according to claim 14, characterized in that, The meta stream of the P-frame fragment stream type consists of consecutive bytes starting from the fragment start code (inclusive) of the P-frame and ending before the next fragment start code of the P-frame; the data in the meta stream of the P-frame fragment stream type may optionally include the image header of the P-frame immediately preceding the fragment, as well as the video extension data and user data after the image header.

18. The method according to claim 14, characterized in that, The meta stream of the B-frame fragment stream type consists of consecutive bytes starting from the start code of the B-frame (inclusive) and ending before the next start code of the B-frame; the data in the meta stream of the B-frame fragment stream type may optionally include the image header of the B-frame immediately preceding the fragment, as well as the video extension data and user data after the image header.

19. The method according to claim 11, characterized in that, The decapsulated video segment data is obtained from the video meta-stream, including: The RTP payload structure type is determined based on the video generic payload header of the RTP data packet, and the video meta-stream is decapsulated according to the RTP payload structure type to obtain video fragment data; the RTP payload structure type is a single payload, a fragmented payload, or an aggregated payload.

20. The method according to claim 11, characterized in that, The decapsulated video segment data is obtained from the video meta-stream, including: If the RTP data packet is a single-payload RTP payload structure type, then the decapsulated video fragment data is obtained from the video meta-stream of the RTP data packet. Alternatively, if the RTP data packet is a fragmented RTP payload structure type, then the decapsulated video fragment data is obtained from the video meta stream of multiple RTP data packets. Alternatively, if the RTP data packet is an RTP payload structure type with aggregated payload, then multiple decapsulated video fragments are obtained from the video meta-stream of the RTP data packet.

21. The method according to claim 11, characterized in that: The video payload data type field is located in the video payload header of the RTP packet and is used to indicate the data type of the RTP packet payload.

22. The method according to any one of claims 11 to 21, characterized in that, The size of the video segment corresponding to the video meta-stream is smaller than the size of the video image in which the video segment is located.

23. An RTP encapsulation device for video clip data, characterized in that, include: The type determination module is used to determine the video segment data type of the video stream data; wherein, the video segment data type is the type corresponding to the video segment data; The type identification module is used to write the video load data type identification value corresponding to the video slice data type into the video load data type field; The encapsulation processing module is used to encapsulate the video segment data as a video meta-stream to obtain RTP data packets of the video segment data.

24. An RTP decapsulation and repackaging device for video clip data, characterized in that, include: The field acquisition module is used to obtain the video payload data type field from the received RTP data packets; The identifier value acquisition module is used to determine the video payload data type identifier value based on the video payload data type field. The type identification module is used to determine the video segment data type of the RTP data packet based on the video payload data type identifier value; The decapsulation module is used to determine that the meta-stream of the RTP data packet is a video meta-stream based on the video segment data type, and to obtain the decapsulated video segment data based on the video meta-stream.

25. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the RTP encapsulation method for video data as described in any one of claims 1 to 10 when executing the computer program, or to implement the steps of the RTP decapsulation method for video data as described in any one of claims 11 to 22 when executing the computer program.

26. A readable storage medium, characterized in that, The readable storage medium stores a computer program that, when executed by a processor, implements the steps of the RTP encapsulation method for video data as described in any one of claims 1 to 10, or, when executed by a processor, implements the steps of the RTP decapsulation method for video data as described in any one of claims 11 to 22.

Images