Multi-channel code stream encapsulation method and video data processing method

By extracting data units with the same timestamp in the video code stream and transmitting them in the form of an RTP packet, the screen out-synchronization problem caused by asynchronous transmission of multiple code streams is solved, and video synchronization is realized in the CDN environment, which is suitable for CDN servers.

CN115883886BActive Publication Date: 2024-11-05GUANGZHOU AVA ELECTRONICS TECH CO LTD
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN202211077280.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-09-05
Publication Date
2024-11-05
Estimated Expiration
2042-09-05

AI Technical Summary

Technical Problem

In scenarios such as video conferencing, asynchronous transmission of multiple code streams leads to screen out-synchronization problems, and the existing video synchronization methods are not applicable to content distribution network (CDN) technology.

Method used

By extracting data units with the same timestamp in the multiplexed video code stream, they are composed of an RTP packet according to the RTP protocol, ensuring that data units with the same timestamp are transmitted simultaneously, and appropriate processing is performed on the receiving end to maintain the synchronization of the video stream.

Benefits of technology

The synchronous transmission of multiple video code streams is realized, avoiding the phenomenon of screen out of synchronization. This method complies with the RTP and RTMP protocol standards and is suitable for CDN servers, solving the problem of video synchronization in the CDN environment.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115883886B_ABST
    Figure CN115883886B_ABST
Patent Text Reader

Abstract

The present invention discloses a multi-channel code stream encapsulation method, a video data processing method and a video transmission system. The multi-channel code stream encapsulation method comprises the following steps: obtaining multiple channels of video code streams to be encapsulated; wherein the GOP lengths of the multiple channels of video code streams to be encapsulated are the same; extracting data units of video frames with the same timestamp from each channel of video code streams to be encapsulated; and encapsulating multiple data units into an RTP packet according to the RTP protocol to obtain a frame of video data. The present invention enables each data unit with the same timestamp to be transmitted synchronously, avoiding the phenomenon of asynchronous transmission of each channel of video due to the separate transmission of multiple code streams, and the method complies with the RTP and RTMP protocol standards, is applicable to CDN servers, and solves the problem of video synchronization using a general CDN server.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of video network transmission, and more specifically, to a multi-channel code stream encapsulation method, a video data processing method and a video transmission system. Background Art

[0002] In the encapsulated bitstream, each frame of the data packet generally includes a timestamp placed in the packet header. When playing, the bitstream is played in the order of the timestamps, so that each frame of the bitstream is played in sequence. However, in scenarios such as video conferencing, if multiple images need to be played, multiple bitstreams need to be played synchronously. Currently, multi-image transmission generally uses multiple streams to transmit separately. Various conditions such as poor network conditions will cause delays in multi-stream transmission. When the playback end receives multiple images for playback, it is easy to cause the images to be out of sync. In addition, with the development of CDN (Content Delivery Network) technology, CDN technology is also widely used in live video broadcasting. However, CDN has its own data requirements and specifications, and current video synchronization methods cannot be used in CDN. Summary of the invention

[0003] In order to solve the applicability problem of CDN, the present invention proposes a video synchronization method applicable to general CDN technology. Specifically, a multi-channel code stream encapsulation method, a video data processing method and a video transmission system are provided. The technical solution adopted by the present invention is as follows.

[0004] In a first aspect, the present invention provides a method for encapsulating multiple code streams, comprising the steps of:

[0005] Acquire multiple video code streams to be packaged; wherein the GOP lengths of the multiple video code streams to be packaged are the same;

[0006] Extracting data units of video frames with the same timestamp from each channel of video code stream to be encapsulated;

[0007] According to the RTP protocol, multiple data units are grouped into an RTP packet for encapsulation to obtain a frame of video data.

[0008] In one implementation, the data unit of the video frame is a NAL unit.

[0009] In one embodiment, before obtaining multiple video streams to be packaged, the method further includes the following steps:

[0010] Encode each video stream to obtain the video code stream to be encapsulated;

[0011] In the encoding process, the GOP lengths of the encoded video code streams are made the same, and the video frame types of the video code streams to be encapsulated at the same timestamp are the same.

[0012] In one embodiment, after obtaining multiple video streams to be packaged, the method further includes the following steps:

[0013] Force I-frame alignment is performed on the video code stream to be encapsulated.

[0014] In one embodiment, the steps are further included:

[0015] Mark information is added to the extracted data unit, where the mark information is used to record the total number of video streams and the sequence number of the current video stream.

[0016] In a second aspect, the present invention provides a method for processing video data, comprising the steps of:

[0017] Acquire a transport stream TS data packet to be processed; wherein the transport stream TS data packet is generated based on the encapsulation method of any of the aforementioned embodiments;

[0018] Initialize multiple caches, where the number of caches corresponds to the total number of video channels during encapsulation;

[0019] The video frame data unit data in the transport stream TS data packet is extracted, and the extracted video frame data unit data is placed in each buffer respectively according to the requirements of the TS stream format.

[0020] In one embodiment, the process of extracting the video frame data unit data from the transport stream TS data packet and placing the extracted video frame data unit data in each cache according to the requirements of the TS stream format includes the steps of:

[0021] When the transport stream TS data packet is a complete data packet start packet and contains a sequence parameter set SPS, a picture parameter set PPS and video frame data unit data, the sequence parameter set SPS and the picture parameter set PPS are taken out and encapsulated into a first TS data packet;

[0022] The first TS data packet is copied to the multiple caches.

[0023] In one embodiment, the process of extracting the video frame data unit data from the transport stream TS data packet and placing the extracted video frame data unit data in each cache according to the requirements of the TS stream format further includes the steps of:

[0024] The video frame data unit data of each video stream is respectively taken out, and each taken out video frame data unit data is respectively encapsulated to form multiple second TS data packets,

[0025] Each second TS data packet is stored in the corresponding cache.

[0026] In one embodiment, when the I frames in the transport stream TS data packets are aligned, the method further comprises the steps of:

[0027] The non-video data in the transport stream TS data packet is copied to each buffer respectively.

[0028] In a third aspect, the present invention provides a video transmission system, comprising: a transmitting end and a receiving end;

[0029] The sending end is used to push the video data encapsulated by the encapsulation method of any of the above embodiments to the public network CDN via RTMP;

[0030] The receiving end is used to obtain streaming media data converted from the video data of the sending end from the public network CDN, and process the streaming media data using the processing method of any of the above-mentioned embodiments.

[0031] In a fourth aspect, the present invention provides a multi-channel code stream encapsulation device, comprising:

[0032] An acquisition module, used for acquiring multiple channels of video code streams to be encapsulated; wherein the GOP lengths of the multiple channels of video code streams to be encapsulated are the same;

[0033] An extraction module, used for extracting data units of video frames with the same timestamp from each channel of video code stream to be encapsulated;

[0034] The encapsulation module is used to encapsulate multiple data units into an RTP packet according to the RTP protocol to obtain a frame of video data.

[0035] In a fifth aspect, the present invention provides a video data processing device, comprising:

[0036] An acquisition module, used for acquiring a transport stream TS data packet to be processed; wherein the transport stream TS data packet is generated based on the aforementioned encapsulation device;

[0037] An initialization module, used to initialize multiple caches, wherein the number of caches corresponds to the total number of video channels during encapsulation;

[0038] The writing module is used to extract the video frame data unit data in the transport stream TS data packet, and put the extracted video frame data unit data into each cache according to the requirements of the TS stream format.

[0039] In a sixth aspect, the present invention provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method of any of the above-mentioned embodiments when executing the program.

[0040] In a seventh aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, characterized in that when the program is executed by a processor, the method of any of the above-mentioned embodiments is implemented.

[0041] In the present invention, data units of video frames with the same timestamp in the video code stream to be encapsulated are grouped into an RTP packet for encapsulation to obtain a frame of video data, so that each data unit with the same timestamp can be transmitted synchronously, avoiding the phenomenon of asynchronous transmission of each video channel caused by the separate transmission of multiple code streams. In addition, the method complies with the RTP and RTMP protocol standards, is applicable to CDN servers, and solves the video synchronization problem using a general CDN server. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a flow chart of the first embodiment of the present invention.

[0043] Figure 2 It is a flowchart of the second embodiment of the present invention.

[0044] Figure 3 It is a schematic diagram of the overall structure of the fourth embodiment of the present invention.

[0045] Figure 4 It is a schematic diagram of the overall structure of the fifth embodiment of the present invention. DETAILED DESCRIPTION

[0046] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0047] It should be noted that the terms "first\second\..." involved in the embodiments of the present invention are only used to distinguish similar objects, and do not represent a specific order for the objects. It is understandable that the specific order or sequence of "first\second\..." can be interchanged where permitted. It should be understood that the objects distinguished by "first\second\..." can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein.

[0048] Embodiment 1

[0049] See also Figure 1 , Figure 1The flowchart of a multi-channel code stream encapsulation method provided in the first embodiment of the present invention includes step S110, step S120 and step S130. It should be noted that step S110, step S120 and step S130 are only reference numerals for clarifying the embodiment and the appendix. Figure 1 The corresponding relationship does not limit the order of the steps in this embodiment.

[0050] Step S110, obtaining multiple channels of video code streams to be packaged; wherein the GOP lengths of the multiple channels of video code streams to be packaged are the same.

[0051] Step S120, extracting data units of video frames with the same timestamp from each channel of video code stream to be encapsulated;

[0052] Step S130: According to the RTP protocol, multiple data units are grouped into an RTP packet for encapsulation to obtain a frame of video data.

[0053] Take three-channel video as an example, and obtain three channels of video streams to be encapsulated. Generally speaking, the three channels of video streams are all H.264 streams. Specifically, the GOP lengths of the three channels of video streams to be encapsulated are the same. This is because the encoding parameters of the three channels of video are the same by default, that is, the three channels of video streams share the same SPS and PPS, and only one SPS and PPS are needed during RTP encapsulation and transmission. When the GOP is different, it means that the encoding parameters of the three channels are different. At this time, three different SPS and PPS need to be carried, while in standard RTP transmission, usually only one SPS and PPS are carried, and some CDNs will parse the SPS and PPS. It is not appropriate to make arbitrary superposition modifications to the SPS and PPS. Therefore, this situation will not be analyzed for the time being, and the default GOP is consistent.

[0054] For each frame of each video stream to be encapsulated, extract the data unit of the video frame with the same timestamp. Generally speaking, preferably, the data unit in the H.264 stream is the NAL unit. Extract the NAL unit of the video frame with the same timestamp in each stream. According to the RTP protocol, the extracted NAL units are combined and packaged. One of the RTP packet modes is the combined packet mode, which can be composed of multiple NAL units to form an RTP packet. There are 4 combinations: STAP-A, STAP-B, MTAP16 and MTAP24. This method uses this packet mode to package the NAL units of the three video streams to obtain a spliced ​​composite video frame. When pushing the RTMP stream, the composite frame can be used as a frame of data for RTP packet transmission.

[0055] Since the data units of the video frames with the same timestamp are packed into the same video frame, the transmission process of each video source can be ensured to be synchronized, and there will be no phenomenon of asynchronous playback caused by some streams being transmitted faster and some streams being transmitted slower when multiple streams are transmitted separately. In addition, this method complies with the RTP and RTMP protocol standards, and the CDN can identify the RTP packet as a single complete frame of data, which is beneficial for the web page to receive the streaming media data for splitting and processing.

[0056] In this method, data units of video frames with the same timestamp in the video code stream to be encapsulated are grouped into an RTP packet for encapsulation to obtain a frame of video data, so that each data unit with the same timestamp can be transmitted synchronously, avoiding the phenomenon of asynchronous transmission of each video channel caused by the separate transmission of multiple code streams. In addition, this method complies with the RTP and RTMP protocol standards, is applicable to CDN servers, and solves the video synchronization problem using general CDN servers.

[0057] In one embodiment, before obtaining multiple video streams to be packaged, the method further includes: step S101 and step S102;

[0058] Step S101, obtaining multiple video streams;

[0059] Step S102, encoding each video stream to obtain a video code stream to be packaged;

[0060] In the encoding process, the GOP lengths of the encoded video code streams are made the same, and the video frame types of the video code streams to be encapsulated at the same timestamp are the same.

[0061] When combining packets, if the video frames at the same timestamp are of the same type, that is, they are all I frames or P frames, then the SPS (sequence parameter set) and PPS (picture parameter set) data of each video stream are the same, and only one SPS or PPS needs to be stored for RTP packet streaming.

[0062] Therefore, in this embodiment, if the received video stream is not encoded, attention is paid to aligning the video frame types of each video during the encoding process so that the video frame types at the same timestamp are the same.

[0063] In one embodiment, after obtaining multiple video streams to be packaged, the method further includes: step S111;

[0064] Step S111, forcibly aligning the video code stream to be packaged by I frame.

[0065] As mentioned above, there are certain advantages to having the same video frame type at the same timestamp. In this embodiment, if the received data is encoded, forced I frame alignment is required to make the video frame type of the video stream to be encapsulated at the same timestamp the same.

[0066] In one implementation, the multi-channel code stream encapsulation method further includes: step S121.

[0067] Step S121, adding marking information to the extracted data unit, wherein the marking information is used to record the total number of video streams and the sequence number of the current video stream.

[0068] For each frame of each channel, two bytes (called tag fields) are added between the NAL header and the frame data based on the standard H.264 frame data. The first byte's first 4 bits indicate the total number of channels of the composite frame data, and the last 4 bits indicate which channel the current frame belongs to. The second byte is a reserved field that can be used for subsequent expansion. Each modified frame is called a multi-channel tag frame.

[0069] Marking the video frames facilitates the subsequent restoration process after receiving the video stream.

[0070] It should be noted here that some new CDNs have the ability to further optimize video streams through encoding and decoding, and will decode and re-encode the video source or perform other operations. The current practice of adding fields after the start code to identify the total number of channels and the current number of channels will affect the H.264 code stream decoding. Therefore, it is not appropriate to add padding fields in such new CDNs.

[0071] Embodiment 2

[0072] Corresponding to the method of the first embodiment, the present invention further provides a method for processing video data, which is used to restore video data obtained from a public network CDN, and the video data is packaged by the encapsulation method of the first embodiment.

[0073] See also Figure 2 , Figure 2 The flowchart of a multi-channel code stream encapsulation method provided in the second embodiment of the present invention includes step S210, step S220 and step S230. It should be noted that step S210, step S220 and step S230 are only reference numerals for clarifying the embodiment and the appendix. Figure 1 The corresponding relationship does not limit the order of the steps in this embodiment.

[0074] Step S210, obtaining a transport stream TS data packet to be processed; wherein the transport stream TS data packet is generated based on the encapsulation method described in the first embodiment;

[0075] Step S220, initializing multiple caches, wherein the number of caches corresponds to the total number of video channels during encapsulation;

[0076] Step S230, extracting the video frame data unit data in the transport stream TS data packet, and putting the extracted video frame data unit data into each buffer respectively according to the requirements of the TS stream format.

[0077] Take the case where the CDN plays the media stream as HLS stream. The receiving end, such as the web page end JS playback control module, obtains the media stream TS file content merged by Example 1 from the CDN. After the WASM stream splitting module receives the incoming media stream TS file content, it analyzes each received transport stream TS data packet in turn and initializes the cache. Each cache corresponds to one video stream. If there are three videos in Example 1, three caches are initialized.

[0078] In the transport stream TS data packet, there is data shared by the three video streams, as well as video frame data unit data of each video stream. According to the requirements of the TS stream format, the video frame data unit data of each video stream is placed in the corresponding cache.

[0079] In one implementation, the process of step S230 includes: step S231 and step S232.

[0080] Step S231, when the transport stream TS data packet is a complete data packet start packet and contains a sequence parameter set SPS, a picture parameter set PPS and video frame data unit data, the sequence parameter set SPS and the picture parameter set PPS are taken out and encapsulated into a first TS data packet;

[0081] Step S232, copying the first TS data packet to the multiple caches.

[0082] When the TS data packet is video data, when the video data TS data packet is a complete data packet start packet (the start field is marked as 1) and contains SPS, PPS and I frame data, the SPS and PPS data are first taken out and encapsulated into a TS data packet that does not contain I frame data, the empty position is filled with padding bits, and the TS data packet is copied to three caches.

[0083] In Example 1, there are implementation methods that align I frames, and there are implementation methods that do not align I frames. When I frames are not aligned, usually not every video channel contains I frame data after the SPS and PPS data packets. For the video channel that is I frame data, operate as mentioned above. For the video channel that is not an I frame, it is necessary to cache after receiving the SPS and PPS data. When receiving the I frame, use the cached SPS and PPS data to fill the SPS and PPS data first to ensure that the video channel has SPS and PPS. That is, compared with the case of I frame alignment, when I frames are not aligned, caching is required before the received SPS and PPS can be stored in the cache of each channel.

[0084] What is expressed in this implementation is the process of extracting SPS and PPS data, encapsulating them and copying them to each cache respectively, which includes both the situation where no caching is required and the situation where caching is required.

[0085] Similarly, when the next TS packet is identified to contain the next data unit data, the current data unit data in the TS packet is taken out and encapsulated into a new TS packet, the empty position is filled with padding, and the TS packet is stored in the corresponding cache, and so on. After processing the incoming TS file content, the total number of channels and multiple caches are simultaneously output to the web page JS layer for playback.

[0086] In one implementation, the process of step S230 further includes: step S233 and step S234.

[0087] Step S233, respectively extract the video frame data unit data of each video stream, and respectively encapsulate each extracted video frame data unit data to form multiple second TS data packets.

[0088] Step S234, storing each second TS data packet in a corresponding buffer.

[0089] When I frame alignment is performed, the original SPS and PPS data packets usually contain I frame data of each video stream. In this case, the I frame data is taken out and encapsulated into a TS data packet containing only the I frame data unit data (including the start code and frame data content) of the original data packet, and stored in the cache of the corresponding channel. When the TS data packet is a complete data packet start packet, all TS data packets until the next complete data packet start packet appear are the data content of a composite frame, that is, a frame contains three channels of multi-stream video data with the same timestamp and the same type. The composite frame contains three H.264 NAL start codes of I frames or P frames. The first channel of video data is from the first start code to the second start code, the second channel of video data is from the second start code to the third start code, and the third channel of video data is from the third start code to the next complete data packet start packet. The three channels of video data can be split by saving them in the cache in sequence.

[0090] When I frames are not aligned, they cannot be directly split to generate TS packets like the I frame alignment scheme. Instead, the TS data needs to be cached as described above. Each channel starts to form the second TS data packet and puts it into the corresponding cache after receiving the I frame. The cache is not ended and the TS packet is generated until the next I frame is received (the data of the next I frame is not cached in this TS packet, but is used as the beginning of the next TS packet).

[0091] When the video data TS data packet is a complete data packet start packet and does not contain SPS and PPS, the TS data packet is a P frame content. The P frame data processing method is basically the same as the I frame data, except that SPS and PPS are no longer processed. After processing the incoming TS file content, the total number of channels and multiple caches are simultaneously output to the web page JS layer for playback.

[0092] In one implementation, encapsulation is performed in the I-frame alignment manner in the first embodiment, further comprising step S240.

[0093] Step S240: copy the non-video data in the transport stream TS data packets to each buffer respectively.

[0094] When I frames are aligned, the non-video data of each split data packet is the same, and the shared data is copied to each buffer respectively.

[0095] The splitting method of the present invention does not involve the encoding and decoding of video data. Only the TS data packets with the start code appearing at the beginning and in the middle need to be modified, and the rest can be directly copied and cached, and three-way video stream data can be quickly split and output. Because the three-way video stream data are output and played at the same time, the synchronization effect of three video images can be achieved.

[0096] The playback end can set the playback layout according to the total number of channels. For example, if the total number of channels is 3 and 3 TS files are received, the three-screen video will be played normally. If the total number of channels is 3 and 2 TS files are received, the data of one of the channels is abnormal and one of the three screens is in a buffered state (this situation basically does not occur). If the total number of channels is 2 and 2 TS files are received, the playback screen is changed to a two-screen video.

[0097] Embodiment 3

[0098] The present invention also provides a video transmission system, which includes: a sending end and a receiving end; wherein the sending end is used to push the video data encapsulated by the encapsulation method described in Example 1 to a public network CDN via RTMP; the receiving end is used to obtain streaming media data converted from the video data of the sending end from the public network CDN, and use the processing method described in Example 2 to process the streaming media data.

[0099] The video transmission system obtains a frame of video data by encapsulating the data units of the video frames with the same timestamp in the video code stream to be encapsulated into an RTP packet, so that the data units with the same timestamp can be transmitted synchronously, avoiding the phenomenon of asynchronous transmission of each video channel caused by the separate transmission of multiple code streams. In addition, the method complies with the RTP and RTMP protocol standards, is applicable to CDN servers, and solves the video synchronization problem using general CDN servers.

[0100] Embodiment 4

[0101] Corresponding to the method of embodiment 1, Figure 3 As shown, the present invention further provides a multi-channel code stream encapsulation device 4, including: an acquisition module 410, an extraction module 420 and an adjustment module 430.

[0102] The acquisition module 410 is used to acquire multiple video streams to be packaged; wherein the GOP lengths of the multiple video streams to be packaged are the same;

[0103] An extraction module 420 is used to extract data units of video frames with the same timestamp from each channel of video code stream to be encapsulated;

[0104] The encapsulation module 430 is used to encapsulate multiple data units into an RTP packet according to the RTP protocol to obtain a frame of video data.

[0105] In one implementation, the data unit of the video frame is a NAL unit.

[0106] In one embodiment, the multi-channel code stream encapsulation device further includes: an encoding module;

[0107] The encoding module is used to encode each video stream to obtain a video code stream to be encapsulated; wherein, during the encoding process, the GOP lengths of the encoded video code streams are made the same, and the video frame types of the video code streams to be encapsulated are the same at the same timestamp.

[0108] In one embodiment, the multi-channel code stream encapsulation device further includes: an alignment module;

[0109] The alignment module is used to perform forced I-frame alignment on the video code stream to be encapsulated.

[0110] In one embodiment, the multi-channel code stream encapsulation device further includes: a marking module;

[0111] The marking module is used to add marking information to the extracted data unit, and the marking information is used to record the total number of video streams and the sequence number of the current video stream.

[0112] In the device, data units of video frames with the same timestamp in the video code stream to be encapsulated are grouped into an RTP packet for encapsulation to obtain a frame of video data, so that each data unit with the same timestamp can be transmitted synchronously, avoiding the phenomenon of asynchronous transmission of each video channel caused by the separate transmission of multiple code streams. In addition, the method complies with the RTP and RTMP protocol standards, is applicable to CDN servers, and solves the video synchronization problem using a general CDN server.

[0113] Embodiment 5

[0114] Corresponding to the method of the second embodiment, as Figure 4 As shown, the present invention further provides a video data processing device 5, comprising: an acquisition module 510, an initialization module 520 and a writing module 530.

[0115] The acquisition module 510 is used to acquire a transport stream TS data packet to be processed; wherein the transport stream TS data packet is generated based on the encapsulation device described in the fourth embodiment;

[0116] Initialization module 520, used to initialize multiple caches, wherein the number of caches corresponds to the total number of video channels during encapsulation;

[0117] The writing module 530 is used to extract the video frame data unit data in the transport stream TS data packet, and put the extracted video frame data unit data into each buffer according to the requirements of the TS stream format.

[0118] In one embodiment, the writing module performs the process of extracting the video frame data unit data from the transport stream TS data packet and placing the extracted video frame data unit data in each cache according to the requirements of the TS stream format, including the steps of:

[0119] When the transport stream TS data packet is a complete data packet start packet and contains a sequence parameter set SPS, a picture parameter set PPS and video frame data unit data, the sequence parameter set SPS and the picture parameter set PPS are taken out and encapsulated into a first TS data packet;

[0120] The first TS data packet is copied to the multiple caches.

[0121] In one embodiment, the writing module, when executing the process of extracting the video frame data unit data from the transport stream TS data packet and placing the extracted video frame data unit data in each cache according to the requirements of the TS stream format, further includes the steps of:

[0122] The video frame data unit data of each video stream is respectively taken out, and each taken out video frame data unit data is respectively encapsulated to form multiple second TS data packets,

[0123] Each second TS data packet is stored in the corresponding cache.

[0124] In one implementation, when the I frames in the transport stream TS data packets are aligned, the writing module is further configured to copy the non-video data in the transport stream TS data packets to each cache respectively.

[0125] Embodiment 6

[0126] An embodiment of the present invention further provides a storage medium on which computer instructions are stored. When the instructions are executed by a processor, the multi-channel code stream encapsulation method or video data processing method of any of the above embodiments is implemented.

[0127] Those skilled in the art can understand that: all or part of the steps of implementing the above method embodiments can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiments; and the aforementioned storage medium includes: mobile storage devices, random access memories (RAM), read-only memories (ROM), magnetic disks or optical disks, etc., various media that can store program codes.

[0128] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention can be essentially or partly reflected in the form of a software product that contributes to the relevant technology. The computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, terminal, or network device, etc.) to execute all or part of the methods of each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, RAM, ROM, magnetic disks or optical disks.

[0129] Corresponding to the above-mentioned computer storage medium, in one embodiment, a computer device is also provided, which includes a memory, an encoder, and a computer program stored in the memory and executable on the encoder, wherein when the encoder executes the program, any one of the multi-channel code stream encapsulation methods or video data processing methods in the above-mentioned embodiments is implemented.

[0130] The above-mentioned computer device obtains a frame of video data by encapsulating the data units of the video frames with the same timestamp in the video code stream to be encapsulated into an RTP packet, so that the data units with the same timestamp can be transmitted synchronously, avoiding the phenomenon of asynchronous video channels caused by the separate transmission of multiple code streams. In addition, this method complies with the RTP and RTMP protocol standards, is applicable to CDN servers, and solves the video synchronization problem using general CDN servers.

[0131] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0132] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. A multi-channel code stream encapsulation method, characterized in that: Includes steps: Acquire multiple video streams to be packaged; wherein the GOP lengths of the multiple video streams to be packaged are the same, that is, the multiple video streams to be packaged share the same sequence parameter set SPS and picture parameter set PPS; Extracting data units of video frames with the same timestamp from each channel of video code stream to be encapsulated; According to the RTP protocol, multiple data units are grouped into an RTP packet for encapsulation to obtain a frame of video data.

2. The multi-channel code stream encapsulation method according to claim 1, characterized in that: The data unit of the video frame is a NAL unit.

3. The multi-channel code stream encapsulation method according to claim 1, characterized in that: Before obtaining multiple channels of video code streams to be packaged, the method further includes the following steps: Encode each video stream to obtain the video code stream to be encapsulated; In the encoding process, the GOP lengths of the encoded video code streams are made the same, and the video frame types of the video code streams to be encapsulated at the same timestamp are the same.

4. The multi-channel code stream encapsulation method according to claim 1, characterized in that: After obtaining multiple channels of video code streams to be packaged, the method further includes the following steps: Force I-frame alignment is performed on the video code stream to be encapsulated.

5. The multi-channel code stream encapsulation method according to any one of claims 1 to 4, characterized in that: Also includes the steps: Mark information is added to the extracted data unit, where the mark information is used to record the total number of video streams and the sequence number of the current video stream.

6. A method for processing video data, characterized in that: Includes steps: Acquire a transport stream TS data packet to be processed; wherein the transport stream TS data packet is generated based on the encapsulation method according to any one of claims 1 to 5; Initialize multiple caches, where the number of caches corresponds to the total number of video channels during encapsulation; The video frame data unit data in the transport stream TS data packet is extracted, and the extracted video frame data unit data is placed in each buffer respectively according to the requirements of the TS stream format.

7. The method for processing video data according to claim 6, characterized in that: The process of extracting the video frame data unit data in the transport stream TS data packet and placing the extracted video frame data unit data in each cache according to the requirements of the TS stream format includes the steps of: When the transport stream TS data packet is a complete data packet start packet and contains a sequence parameter set SPS, a picture parameter set PPS and video frame data unit data, the sequence parameter set SPS and the picture parameter set PPS are taken out and encapsulated into a first TS data packet; The first TS data packet is copied to the multiple caches.

8. The method for processing video data according to claim 7, characterized in that: The process of extracting the video frame data unit data from the transport stream TS data packet and placing the extracted video frame data unit data in each cache respectively according to the requirements of the TS stream format also includes the steps of: The video frame data unit data of each video stream is respectively taken out, and each taken out video frame data unit data is respectively encapsulated to form multiple second TS data packets, Each second TS data packet is stored in the corresponding cache.

9. The method for processing video data according to any one of claims 6 to 8, characterized in that: When the I frames in the transport stream TS data packets are aligned, the method further comprises the steps of: The non-video data in the transport stream TS data packet is copied to each buffer respectively.

10. A video transmission system, characterized in that: include: The sender and receiver; The sending end is used to push the video data encapsulated by the encapsulation method according to any one of claims 1 to 5 to the public network CDN via RTMP; The receiving end is used to obtain streaming media data converted from the video data of the sending end from the public network CDN, and process the streaming media data using the processing method described in any one of claims 6-8.

11. A multi-channel code stream encapsulation device, characterized in that: include: An acquisition module is used to acquire multiple video streams to be packaged; wherein the GOP lengths of the multiple video streams to be packaged are the same, that is, the multiple video streams to be packaged share the same sequence parameter set SPS and picture parameter set PPS; An extraction module, used for extracting data units of video frames with the same timestamp from each channel of video code stream to be encapsulated; The encapsulation module is used to encapsulate multiple data units into an RTP packet according to the RTP protocol to obtain a frame of video data.

12. A video data processing device, characterized in that: include: An acquisition module, used for acquiring a transport stream TS data packet to be processed; wherein the transport stream TS data packet is generated based on the encapsulation device according to claim 11; An initialization module, used to initialize multiple caches, wherein the number of caches corresponds to the total number of video channels during encapsulation; The writing module is used to extract the video frame data unit data in the transport stream TS data packet, and put the extracted video frame data unit data into each cache according to the requirements of the TS stream format.

13. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 9 is implemented.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Citation Information

Patent Citations

  • Method and device for generating timestamp of multimedia data

    CN113573119A

  • Distributed ultra-high-definition video synchronous processing method

    CN113873345A

  • Picture synchronization method, coding method, video playing equipment and video coding equipment

    CN114079813A

  • Media stream downloading method and device, equipment and storage medium

    CN114697712A

  • Apparatus for transmitting broadcast signal, apparatus for receiving broadcast signal, method for transmitting broadcast signal and method for receiving broadcast signal

    US20160234355A1