RTP multicast video and auxiliary data multiplexing method and system

Abstract

The application provides an RTP multicast video and auxiliary data multiplexing method and system, which is applied to a sending end and comprises the following steps: generating video data blocks and auxiliary data blocks based on video frames and auxiliary data to be sent; the auxiliary data blocks are spliced from auxiliary data headers and auxiliary data, the auxiliary data headers comprise data type identifiers, data lengths and time stamps; determining a plurality of target data blocks to be encapsulated into the same RTP data packet; determining the starting offset of each target data block in the RTP payload, generating a block index header based on the starting offset; splicing the block index header and the plurality of target data blocks in the order of the starting offset to form the RTP payload; adding an RTP standard header to the RTP payload to generate an RTP data packet, and sending the RTP data packet through a multicast network. The application reduces the system complexity, improves the bandwidth utilization and analysis efficiency.

Description

Technical Field

[0001] This invention relates to the field of real-time streaming media transmission protocols and network communication technology, specifically to a method and system for multiplexing RTP multicast video and auxiliary data. Background Technology

[0002] In real-time streaming scenarios (such as online education, interactive live streaming, and remote conferencing), the RTP / RTCP protocol is the mainstream standard. However, existing RTP payload designs are mainly designed for single media encoded data streams and cannot efficiently carry video and various auxiliary data (such as text messages, control commands, status updates, etc.) in the same RTP multicast stream. Therefore, the conventional approach is to establish separate multicast groups or unicast connections for video and auxiliary data, transmit them separately, and then synchronize them.

[0003] This multi-stream scheme has the following technical drawbacks: High system complexity: The sending and receiving ends need to maintain multiple independent RTP session states simultaneously (multiple SSRCs, multiple UDP ports, multiple sequence number spaces), increasing the complexity and resource consumption of connection management, NAT traversal, and firewall configuration. Significant bandwidth waste: Each independent stream needs to repeatedly add RTP / UDP / IP protocol headers (at least 40 bytes / packet), and in multicast scenarios, the same content needs to be distributed in multiple multicast groups, resulting in low network bandwidth utilization. Low parsing efficiency: After receiving an RTP packet, the receiving end cannot directly know whether the packet contains auxiliary data or what types of auxiliary data it contains; it often needs to traverse the payload for probing parsing, increasing processing latency.

[0004] Therefore, there is an urgent need for a method and system for reusing RTP multicast video and auxiliary data, which can perform structured reuse of video and auxiliary data to reduce system complexity, reduce bandwidth overhead, and support deterministic, low-latency payload parsing. Summary of the Invention

[0005] In view of this, it is necessary to provide a method and system for multiplexing RTP multicast video and auxiliary data to solve the technical problems of high system complexity, low bandwidth utilization and low parsing efficiency in the existing technology.

[0006] To address the aforementioned technical problems, in a first aspect, the present invention provides a method for multiplexing RTP multicast video and auxiliary data, applied at the transmitting end, the method comprising: Video data blocks and auxiliary data blocks are generated based on the video frames to be sent and auxiliary data, respectively; the auxiliary data block is formed by concatenating the auxiliary data with the auxiliary data header, and the auxiliary data header includes a data type identifier, data length and timestamp; Multiple target data blocks to be encapsulated into the same RTP data packet are identified, and the target data blocks include video data blocks and auxiliary data blocks; Determine the starting offset of each target data block in the RTP payload, and generate a block index header based on the starting offset; the block index header includes the total payload length, the number of the plurality of target data blocks, and the block type and starting offset corresponding to each target data block; The block index header and the plurality of target data blocks are concatenated in order of the starting offset to form the RTP payload; An RTP standard header is added to the RTP payload to generate an RTP data packet, which is then sent via a multicast network.

[0007] In one possible implementation, the auxiliary data header further includes a priority field and a time-to-live field; the priority field is used to indicate the processing priority of the receiving end, and the time-to-live field is used to indicate the effective lifespan of the auxiliary data block from the time of transmission.

[0008] In one possible implementation, the block index header also includes a version number and a flag bit for each of the target data blocks; the flag bit is used to indicate whether the auxiliary data block is incrementally updated.

[0009] In one possible implementation, determining the multiple target data blocks to be encapsulated into the same RTP packet includes: The plurality of target data blocks are determined based on the maximum transmission unit of the current network path and the priority order of the video data blocks and the auxiliary data blocks; The total length of the RTP data packet generated after encapsulation is less than or equal to the maximum transmission unit.

[0010] Secondly, the present invention also provides a method for multiplexing RTP multicast video and auxiliary data, applied at a receiving end, the method comprising: Receive RTP data packets, parse the RTP standard header of the RTP data packets, and determine the RTP payload and payload start position of the RTP data packets; Parse the block index header in the RTP payload to obtain the number of target data blocks in the RTP data packet, the total length of the payload, the block type of each target data block, and the starting offset of each target data block in the RTP payload; The starting position of each target data block is determined based on the starting offset, and the length of each target data block is determined by the offset difference between two adjacent starting offsets and the total load length. Based on the block type, video data blocks are sent to the video decoding pipeline, and auxiliary data blocks are sent to the corresponding message processing engine for parallel processing.

[0011] In one possible implementation, the method further includes: Parse the RTP standard header of the RTP data packet to determine the timestamp and sequence number of the RTP data packet; The RTP data packets are lost and reordered according to the sequence number; A unified time base is established based on the timestamp, and the time base is used to realize the synchronous processing of the video data block and the auxiliary data block.

[0012] In one possible implementation, the method further includes: Parse the priority field in the auxiliary data block and process the auxiliary data block in descending order of priority; Parse the sending timestamp and lifetime fields in the auxiliary data block. If the current time exceeds the sum of the sending timestamp and the lifetime, discard the auxiliary data block.

[0013] In one possible implementation, the method further includes: When parsing the block index header fails or a corrupted index header is detected, a preset synchronization marker is scanned starting from the RTP payload start position to locate the start position of each target data block. The boundaries of each target data block are determined based on the length field or video encoding start code in the auxiliary data header, thereby re-dividing each target data block, and the index list is reconstructed based on the division results.

[0014] Thirdly, the present invention also provides an RTP multicast video and auxiliary data multiplexing transmitter system, comprising: The data block construction module is used to generate video data blocks and auxiliary data blocks based on the video frames to be sent and auxiliary data, respectively; the auxiliary data block is formed by concatenating the auxiliary data with the auxiliary data header, and the auxiliary data header includes a data type identifier, data length and timestamp; An index generation module is used to determine multiple target data blocks to be encapsulated into the same RTP data packet; and to determine the starting offset of each target data block in the RTP payload, and to generate a block index header based on the starting offset; the block index header includes the total payload length, the number of the multiple target data blocks, and the block type and starting offset corresponding to each target data block; The payload assembly and transmission module is used to concatenate the block index header and the plurality of target data blocks in the order of the starting offset to form an RTP payload; add an RTP standard header to the RTP payload to generate an RTP data packet; and transmit the RTP data packet through a multicast network.

[0015] Fourthly, the present invention also provides an RTP multicast video and auxiliary data multiplexing receiver system, comprising: The RTP receiving and parsing module is used to receive RTP data packets, parse the RTP standard header of the RTP data packets, and determine the RTP payload and payload start position of the RTP data packets. The index parsing and positioning module is used to parse the block index header in the RTP payload to obtain the number of target data blocks in the RTP data packet, the total length of the payload, the block type of each target data block, and the starting offset of each target data block in the RTP payload; determine the starting position of each target data block according to the starting offset, and determine the length of each target data block by the offset difference between two adjacent starting offsets; The concurrent distribution module is used to send video data blocks to the video decoding pipeline and auxiliary data blocks to the corresponding message processing engine for parallel processing, based on the block type.

[0016] The beneficial effects of this invention are as follows: The RTP multicast video and auxiliary data multiplexing method provided by this invention, applied to the sending end, determines and encapsulates video data blocks and auxiliary data blocks into the same RTP data packet, enabling all data blocks to share the RTP standard header. The shared RTP standard header means that both the sending and receiving ends only need to maintain one RTP session state, significantly reducing the maintenance complexity of the protocol stack and the system complexity. Simultaneously, the shared RTP standard header eliminates the overhead of repeatedly adding protocol headers for each independent stream in a multi-stream scheme. In multicast scenarios, a single multicast stream can distribute all data, eliminating the need to transmit multiple copies in multiple multicast groups, effectively saving network bandwidth and improving bandwidth utilization.

[0017] Furthermore, by setting the total payload length, the number of multiple target data blocks, and the index header of the block type and starting offset block corresponding to each target data block, the present invention enables the receiving end to obtain the complete layout of the entire RTP payload at once. During parsing, each target data block is directly located based on the starting offset, and the block length is calculated, achieving direct location with O(1) complexity. This avoids the inefficient operation of byte-by-byte scanning or trial parsing, reduces the receiving end's processing latency and CPU overhead, and improves parsing efficiency. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a schematic flowchart of an embodiment of the RTP multicast video and auxiliary data multiplexing method applied to the transmitting end provided by the present invention; Figure 2 This is a schematic flowchart of an embodiment of the RTP multicast video and auxiliary data multiplexing method applied to the receiving end provided by the present invention; Figure 3 A schematic diagram of an embodiment of the RTP multicast video and auxiliary data multiplexing transmitter system provided by the present invention; Figure 4 This is a schematic diagram of an embodiment of the RTP multicast video and auxiliary data multiplexing receiving system provided by the present invention. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0021] It should be understood that the illustrative drawings are not drawn to scale. The flowcharts used in this invention illustrate operations implemented according to some embodiments of the invention. It should be understood that the operations in the flowcharts may be implemented out of order, and steps without logical contextual relationships may be reversed or performed simultaneously. Furthermore, those skilled in the art, guided by the content of this invention, may add one or more other operations to the flowcharts, or remove one or more operations from the flowcharts. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor systems and / or microcontroller systems.

[0022] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0023] This invention provides a method and system for multiplexing RTP multicast video and auxiliary data, which will be described below.

[0024] Figure 1 This is a schematic flowchart of an embodiment of the RTP multicast video and auxiliary data multiplexing method provided by the present invention, applied to the transmitting end, as shown below. Figure 1 As shown, the RTP multicast video and auxiliary data multiplexing methods include: S101. Generate video data blocks and auxiliary data blocks based on the video frames to be sent and auxiliary data respectively; the auxiliary data block is composed of an auxiliary data header and auxiliary data, and the auxiliary data header includes data type identifier, data length and timestamp.

[0025] The video data block generation process involves directly using the encoded data of the video frame to be sent as the content of the video data block without adding any additional header information, in order to maintain compatibility with existing video coding standards.

[0026] The process of generating auxiliary data blocks is as follows: add auxiliary data headers to the auxiliary data, and then concatenate the auxiliary data headers with the auxiliary data content in sequence to form a complete auxiliary data block.

[0027] Auxiliary data includes, but is not limited to, various types such as text messages, control commands, status updates, log information, voting results, page-turning commands, and PTZ control signaling.

[0028] The structure of the auxiliary data header is defined as follows: Data type identifier (1 byte): Used to distinguish different types of auxiliary data, with a value range of 0~255. For example, 0x01 represents chat text, 0x02 represents page turning instructions, and 0x03 represents PTZ control, etc. The receiving end distributes data blocks to the corresponding processing engine based on this field.

[0029] Data length (2 bytes): Used to indicate the length of auxiliary data content.

[0030] Timestamp (4 bytes): Indicates the generation time of the auxiliary data block, based on the local system clock or the main video source clock, with an accuracy of up to microseconds, used by the receiving end to achieve fine synchronization control.

[0031] In other words, video data blocks have no additional encapsulation overhead, while auxiliary data blocks achieve self-description capabilities through auxiliary data headers, laying the foundation for subsequent block index reuse and zero-traversal parsing.

[0032] It should be noted that the timestamp is uniformly taken from the time reference of the primary video source. The primary video source refers to the continuous video stream that is selected as the time reference source in the real-time communication session, such as the speaker's image captured by the camera, the screen sharing stream, or the core output stream of the encoder.

[0033] S102. Determine multiple target data blocks to be encapsulated into the same RTP data packet. The target data blocks include video data blocks and auxiliary data blocks.

[0034] S103. Determine the starting offset of each target data block in the RTP payload, and generate a block index header based on the starting offset. The block index header includes the total payload length, the number of multiple target data blocks, and the block type and starting offset of each target data block.

[0035] Specifically, the structure of the block index header is defined as follows: The block index header is located at the beginning of the RTP payload and is arranged in a 4-byte alignment. Its overall structure consists of two parts: a fixed header and a variable-length list of index entries.

[0036] The fixed header (4 bytes in total) contains the following fields: Version number (1 byte): Used to identify the version of this payload format, facilitating protocol evolution and backward compatibility.

[0037] Number of target data blocks (1 byte): Indicates the total number of target data blocks.

[0038] Total payload length (2 bytes): Indicates the total length of the entire RTP payload. This field is used by the receiver to verify payload integrity and calculate the boundary of the last target data block.

[0039] The index entry list (each entry is a fixed 4 bytes) contains N entries, where N equals the number of target data blocks. The structure of each index entry is as follows: Block Type (1 byte): Identifies the content type of the target data block. For example, 0x01 indicates a video data block, 0x02 indicates a text message, 0x03 indicates a control command, and 0x04 indicates a status update, etc. The receiving end distributes the data block to the corresponding processing engine based on this field.

[0040] Starting offset (2 bytes): Represents the offset (in bytes) of the target data block's starting position in the RTP payload relative to the payload's starting byte. The starting offset is counted from 0 and increments strictly. Using this offset, the receiver can directly jump to the data block's starting position, achieving O(1) positioning.

[0041] The flag bit (1 byte) is the incremental update flag, used to indicate whether the current auxiliary data block is incrementally updated data, that is, whether it is an incremental update compared to a previously sent auxiliary data block with the same data type identifier (i.e., carrying only the difference, not the complete data). This preceding data block can be located within the same RTP packet or in an earlier sent RTP packet. When the receiving end parses and detects that the incremental update flag is set, it needs to merge or replace the content of the current data block with the corresponding preceding complete data block in the buffer to reconstruct the complete data. For video data blocks or auxiliary data types appearing for the first time, this flag bit should always be set to 0 (indicating complete data). This mechanism effectively reduces the transmission of redundant data, further saving bandwidth.

[0042] S104. Concatenate the block index header with multiple target data blocks in order of their starting offsets to form the RTP payload.

[0043] The specific formation process of the RTP payload is as follows: First, a block index header is written to the very beginning of the payload buffer. The block index header has a fixed length of 4 + 4 × N bytes, where N is the number of target data blocks within the RTP packet. The list of index entries in the block index header is written strictly according to the order in which the target data blocks are arranged in the payload: the first index entry corresponds to the first target data block, the second index entry corresponds to the second target data block, and so on. The starting offset recorded in each index entry is equal to the byte offset of the actual starting position of the target data block in the payload relative to the starting position of the payload buffer.

[0044] Then, starting from buffer offset 4 + 4×N, the contents of each target data block are written sequentially: the first target data block is written immediately after the block index header, the second target data block is written immediately after the first target data block, and so on. The starting position of each target data block is exactly equal to the starting offset declared in its corresponding index entry, thus ensuring a precise correspondence between the index entry and the target data block. For example, if the block index header length is 40 bytes, then the starting offset of the first target data block is 40, and its contents are written starting from the 41st byte of the buffer; the starting offset of the second target data block is 40 plus the length of the first target data block, and so on, forming the complete RTP payload.

[0045] S105. Add an RTP standard header to the RTP payload, generate an RTP data packet, and send the RTP data packet through a multicast network.

[0046] The RTP standard header includes an RTP timestamp, which identifies the unified time base of the entire RTP data packet. The RTP timestamp is also taken from the primary video source. The timestamp in the auxiliary header has higher precision (e.g., microsecond level) and is used to assist in fine-grained timing control, relative sorting, or expiration determination within data blocks. The two do not need to be equal in value. The receiving end can perform macroscopic synchronization based on the RTP timestamp and then combine it with the timestamp in the auxiliary header to achieve precise microscopic processing.

[0047] The RTP multicast video and auxiliary data multiplexing method in this embodiment of the invention can be implemented in any device based on this method, such as online education servers, interactive live streaming platform nodes, video conferencing terminals, edge computing gateways, cloud transcoding servers, and other electronic devices. Specifically, the RTP multicast video and auxiliary data multiplexing method is stored in the aforementioned device as a pre-compiled program or software library. When video streams and auxiliary data (such as page-turning commands, text chat, and control signaling) need to be sent synchronously in real time, the program is invoked.

[0048] Compared with existing technologies, the RTP multicast video and auxiliary data multiplexing method provided in this invention, applied at the sending end, determines and encapsulates video data blocks and auxiliary data blocks into the same RTP data packet, enabling all data blocks to share the RTP standard header. The shared RTP standard header means that both the sending and receiving ends only need to maintain one RTP session state, significantly reducing the maintenance complexity of the protocol stack and the system complexity. Simultaneously, the shared RTP standard header eliminates the overhead of repeatedly adding protocol headers for each independent stream in a multi-stream scheme. In multicast scenarios, a single multicast stream can distribute all data, eliminating the need to transmit multiple copies in multiple multicast groups, effectively saving network bandwidth and improving bandwidth utilization.

[0049] Furthermore, this embodiment of the invention, by setting the total payload length, the number of multiple target data blocks, and the index header of the block type and starting offset block corresponding to each target data block, enables the receiving end to obtain the complete layout of the entire RTP payload at once. During parsing, each target data block is directly located based on the starting offset, and the block length is calculated, achieving direct location with O(1) complexity. This avoids the inefficient operation of byte-by-byte scanning or trial parsing, reduces the receiving end's processing latency and CPU overhead, and improves parsing efficiency.

[0050] When transmitting multiple types of auxiliary data (such as control commands, chat messages, status updates, etc.) in the same RTP multicast stream, the receiving end cannot distinguish the processing priority of different data blocks. This can cause data with high real-time requirements (such as page turning commands and device control signaling) to be delayed due to queuing delays. At the same time, if expired auxiliary data (such as invalid voting results and outdated location coordinates) caused by network delays or retransmissions is used by upper-layer applications, it may cause incorrect operations or displays.

[0051] To address the aforementioned technical problems, in some embodiments of the present invention, the auxiliary data header further includes a priority field and a time-to-live field; the priority field is used to indicate the processing priority of the receiving end, and the time-to-live field is used to indicate the effective lifespan of the auxiliary data block from the time of transmission.

[0052] Specifically, the priority field is 1 byte. In cases of network congestion or high receiver load, selective data discarding can be implemented based on priority: lower priority data is discarded first to ensure the reliability of critical data transmission. For example, in online education, the teacher's "page turning command" has higher priority than "chat messages"; in remote conferencing, the "chairperson's speaking right control signaling" has higher priority than "participant status update".

[0053] The Time-to-Live (TTL) field is 1 byte. The valid TTL indicates how many time units the target data block is valid for from the time it was sent (the time unit can be milliseconds, seconds, or RTP timestamps, depending on system agreement). For example, in interactive live streaming, if the "voting result" does not arrive within 2 seconds, it is meaningless and can be discarded. In remote control scenarios, expired "pan-tilt rotation commands" should not be executed to avoid malfunctions.

[0054] This invention improves the real-time performance and reliability of the system by setting a priority field and a time-to-live field in the auxiliary data header. This is achieved by allowing the receiving end to dynamically adjust the processing order based on the priority, ensuring that critical data is delivered to the application layer first, and automatically discarding expired data that exceeds the valid time limit based on the time-to-live.

[0055] When packaging video data blocks with multiple auxiliary data blocks into a single RTP packet, uncontrolled processing can result in the final RTP packet exceeding the network path's Maximum Transmission Unit (MTU), triggering IP layer fragmentation. Fragmentation reduces transmission reliability and increases reassembly latency at the receiver, which is extremely detrimental to real-time scenarios such as online education and interactive live streaming.

[0056] To solve this technical problem, in some embodiments of the present invention, step S102 includes: Based on the maximum transmission unit of the current network path and the priority order of video data blocks and auxiliary data blocks, multiple target data blocks are determined; Among them, the total length of the generated RTP data packet after encapsulation is less than or equal to the maximum transmission unit.

[0057] It should be noted that: the video data block is used as the target data block that must be placed; on the premise that the remaining capacity allows, in the order of the auxiliary data block priorities from high to low, each auxiliary data block is successively tried to be added to the RTP data packet. Before each addition, it is checked whether the currently accumulated payload length (block index header length + total length of the selected data blocks + length of the auxiliary data block to be added) exceeds the maximum payload capacity. If it does not exceed, it is added; otherwise, the auxiliary data block is skipped and the next auxiliary data block with a lower priority is continued to be tried until all the auxiliary data blocks to be sent are traversed or the remaining capacity is not enough to accommodate any next data block. Finally, all the successfully added video data blocks and auxiliary data blocks together constitute multiple target data blocks.

[0058] In the embodiment of the present invention, by setting the total length of the generated RTP data packet after encapsulation to be less than or equal to the maximum transmission unit, IP layer fragmentation is avoided, and the real-time performance and transmission efficiency of the multicast stream are ensured.

[0059] On the other hand, the embodiment of the present invention also provides an RTP multicast video and auxiliary data multiplexing method, which is applied to the receiving end, as Figure 2 shown, the RTP multicast video and auxiliary data multiplexing method applied to the receiving end includes: S201. Receive an RTP data packet, parse the RTP standard header of the RTP data packet, and determine the RTP payload of the RTP data packet and the payload start position; S202. Parse the block index header in the RTP payload to obtain the number of target data blocks in the RTP data packet, the total payload length, the block type of each target data block, and the start offset of each target data block in the RTP payload; S203. Determine the start position of each target data block according to the start offset, and determine the length of each target data block by the offset difference between two adjacent start offsets and the total payload length.

[0060] Specifically, the total number of target data blocks is N. For the i-th target data block (i starts from 0), its length calculation method is: if i < N - 1, then the length = the (i + 1)-th offset - the i-th offset; if i = N - 1, then the length = L - the i-th offset. L is the total payload length.

[0061] In other words, the receiving end can know the positions and boundaries of all target blocks without traversing the content of the target data blocks, realizing zero-traversal deterministic parsing.

[0062] S204. According to the block type, send the video data block to the video decoding pipeline and send the auxiliary data block to the corresponding message processing engine for parallel processing.

[0063] In multicast transmission, RTP data packets may arrive out of order or be lost due to network congestion or routing changes. At the same time, the video stream and the auxiliary data stream need to be aligned under the same time reference.

[0064] Therefore, in some embodiments of the present invention, the RTP multicast video and auxiliary data multiplexing method further includes: Parse the RTP standard header of the RTP data packet to determine the timestamp and sequence number of the RTP data packet; Perform packet loss detection and reordering on the RTP data packet according to the sequence number; Establish a unified time reference according to the timestamp, and the time reference is used to implement the synchronous processing of the video data block and the auxiliary data block.

[0065] Among them, the specific process of packet loss detection and reordering is as follows: When the receiving end initializes the RTP session, it maintains a variable expected_seq, and the initial value is the sequence number of the first RTP packet received. Every time an RTP packet is received, extract its sequence number seq and compare it with expected_seq: if seq == expected_seq, it means that the packet arrives in order, and it is directly sent to the subsequent processing module (such as index parsing and concurrent distribution), and expected_seq is incremented by 1; if seq>expected_seq, it is determined that packet loss occurs within the range from expected_seq to seq-1. The receiving end can record the missing sequence numbers in the loss list, and at the same time temporarily store the current packet in the reordering buffer, and optionally send a RTCP NACK message to request retransmission; if seq<expected_seq, it means that the packet is a late-arriving packet that arrived out of order before (usually due to network jitter or after retransmission), then search and fill in the missing sequence numbers from the buffer. When there are data packets with consecutive sequence numbers in the buffer, take them out in order and send them to the subsequent module, and update expected_seq to the next expected sequence number. Through this mechanism, the receiving end can recover the sequence disorder caused by IP network out-of-order or retransmission, and ensure that the video block and the auxiliary data block are processed in the original order of the sending end.

[0066] Establishing a unified time base means that the receiving end uses the sending end's RTP timestamp as a reference to align its local system clock with it, so that timestamps parsed from different RTP packets can be mapped to a unified playback timeline. Specifically, when the receiving end receives the first RTP packet, it records its RTP timestamp ts0 and local reception time t0, and calculates the mapping relationship between its local clock and the sending end's clock. For each subsequent RTP packet, its RTP timestamp ts is converted to the local playback time. After the conversion, all data blocks (video blocks and auxiliary blocks) share the same playback time coordinate.

[0067] The specific implementation of synchronization is as follows: After the video data block is sent to the video decoding pipeline, the decoded video frames are rendered according to the playback time converted above. Although auxiliary data blocks (such as page-turning instructions and text messages) may carry higher-precision internal timestamps, the RTP timestamp is still used as the standard for macro-synchronization. When the receiving end parses the auxiliary data block, it obtains the RTP timestamp of the block, maps it to the same playback timeline, and then puts the auxiliary data block into the scheduling queue for execution or display at a specified time point or time interval. For example, when a page-turning instruction auxiliary data block is received, if the playback time mapped from the timestamp of its RTP packet happens to correspond to the display time of a key video frame, the receiving end will execute the page-turning action while the video frame is being rendered, thereby achieving sub-millisecond-level precise alignment between auxiliary information and the video frame. If the auxiliary data block contains a more refined time offset field, the receiving end can fine-tune it based on the RTP timestamp benchmark to meet higher-precision synchronization requirements.

[0068] A single RTP data packet may contain multiple auxiliary data blocks (such as high-priority control commands and low-priority log messages). If these are processed indiscriminately, urgent commands may be placed at the end of the queue, introducing additional delays. At the same time, auxiliary data that has lost its timeliness due to network latency (such as expired voting results or invalid PTZ commands) may cause erroneous operations if sent to the application layer.

[0069] To avoid this technical problem, in some embodiments of the present invention, the RTP multicast video and auxiliary data multiplexing method further includes: Parse the priority field in the auxiliary data block and process the auxiliary data blocks in descending order of priority; Parse the sending timestamp and lifetime fields in the auxiliary data block. If the current time exceeds the sum of the sending timestamp and the lifetime, discard the auxiliary data block.

[0070] This invention improves the system's real-time response capability and data validity by parsing the priority field in the auxiliary data block at the receiving end and scheduling the data according to priority from high to low; and by parsing the time-to-live field and comparing it with the current time, automatically discarding expired data that exceeds the valid duration.

[0071] In actual network transmission, the block index header of RTP packets may be corrupted due to bit errors, packet loss, or malicious attacks, preventing the receiving end from obtaining critical information such as the number of data blocks and the starting offset. Without fault-tolerance measures, the receiving end will be unable to correctly delineate the boundaries of each data block in the payload and will have to discard the entire RTP packet, causing video stuttering and loss of auxiliary data, thus reducing the robustness of the system.

[0072] To address the aforementioned technical problems, in some embodiments of the present invention, the RTP multicast video and auxiliary data multiplexing method further includes: When parsing the block index header fails or a corrupted index header is detected, a preset synchronization marker is scanned starting from the RTP payload start position to locate the start position of each target data block. The boundaries of each target data block are determined based on the length field or video encoding start code in the auxiliary data header, thereby re-dividing each target data block, and the index list is reconstructed based on the division results.

[0073] The synchronization marker is used to identify the potential starting position of a target data block. Once a synchronization marker is found, the receiver assumes that position is the starting point of a target data block.

[0074] Specifically, the boundary determination process for auxiliary data blocks is as follows: When the receiving end scans a preset synchronization marker (e.g., a fixed two-byte sequence 0x7E7E), it uses the end position of this marker as the starting point of the auxiliary data block. Following the synchronization marker is the auxiliary data header, whose structure is fixed and includes a data type identifier (1 byte), data length (2 bytes), timestamp (4 bytes), and optional priority and time-to-live fields. The receiving end parses the data length field (in bytes, indicating the size of subsequent auxiliary data content) from the auxiliary data header. Therefore, the end position of the auxiliary data block = start position + synchronization marker length + auxiliary data header length + data length. Based on this, the receiving end accurately divides the complete auxiliary data block and records its starting offset, block type, and length for index reconstruction.

[0075] The process of determining the boundaries of video data blocks: Video data blocks do not use auxiliary headers; their boundaries are identified using the start codes of the video encoding format itself. After the receiving end scans for a synchronization marker (e.g., 0x7E7E), it starts searching for the start code of the video encoding from after that marker. Once the start code is found, the receiving end uses the starting position of that start code as the starting point of the video data block. Then, it parses the NAL unit header after the start code or directly searches for the position of the next start code. The byte range between the two start codes is the boundary of the current video data block. After determining the boundary, the offset and block type are also recorded for index reconstruction.

[0076] This invention introduces an index self-recovery mechanism based on synchronization markers, enabling the receiving end to scan preset synchronization markers from the payload start position when the index header is damaged. It then combines the length field in the auxiliary data header or the video encoding start code to redetermine the boundary of each data block, reconstruct the index list, and thus restore the normal parsing and distribution of data blocks. This significantly improves fault tolerance and data integrity in harsh network environments.

[0077] In summary, the RTP multicast video and auxiliary data multiplexing method proposed in this embodiment of the invention firstly achieves zero-traversal deterministic parsing at the receiving end through the block index header and starting offset mechanism, reducing the data block location time complexity from O(n) to O(1), significantly improving parsing performance; secondly, video and auxiliary data share the same multicast stream and RTP / UDP / IP header, saving 35%~50% bandwidth compared to multi-stream solutions, and adaptive encapsulation based on network MTU avoids IP fragmentation, maximizing bandwidth utilization; thirdly, through scalable data type identifiers (supporting 256 types), version number fields, and byte... The system employs an order-independent data structure, enabling plug-and-play extensions to new data formats, backward compatibility with protocols, and cross-platform compatibility. Fourth, the introduction of an index self-recovery mechanism allows the index to be rebuilt even if the index header is damaged, enhancing the system's fault tolerance and robustness. Fifth, all data blocks share a unified RTP time base accurate to microseconds, supplemented by high-precision timestamps within the data blocks and receiver clock drift compensation, achieving sub-millisecond-level precise synchronization. Sixth, priority-aware scheduling (supporting 16 priority levels), automatic cleanup of expired messages based on time-to-live, and sequence number-based deduplication and order-preserving transmission provide industrial-grade message processing capabilities. In summary, this invention, in real-time streaming media transmission scenarios, balances high performance, low overhead, strong synchronization, high fault tolerance, and flexible scalability, significantly outperforming existing multi-stream discrete transmission schemes.

[0078] On the other hand, embodiments of the present invention also provide an RTP multicast video and auxiliary data multiplexing transmitter system, such as... Figure 3 As shown, the RTP multicast video and auxiliary data multiplexing transmitter system 300 includes: The data block construction module 301 is used to generate video data blocks and auxiliary data blocks based on the video frame to be sent and auxiliary data, respectively. The auxiliary data block is composed of an auxiliary data header and auxiliary data. The auxiliary data header includes a data type identifier, data length and timestamp. The index generation module 302 is used to determine multiple target data blocks to be encapsulated into the same RTP data packet; and to determine the starting offset of each target data block in the RTP payload, and to generate a block index header based on the starting offset; the block index header includes the total payload length, the number of multiple target data blocks, and the block type and starting offset of each target data block; The payload assembly and transmission module 303 is used to concatenate the block index header and multiple target data blocks in the order of the starting offset to form an RTP payload; add an RTP standard header to the RTP payload to generate RTP data packets; and send the RTP data packets through a multicast network.

[0079] The RTP multicast video and auxiliary data multiplexing transmitter system 300 provided in the above embodiments can implement the technical solutions described in the above embodiments of the RTP multicast video and auxiliary data multiplexing method applied to the transmitter. The specific implementation principles of each module or unit can be found in the corresponding content in the above embodiments of the RTP multicast video and auxiliary data multiplexing method applied to the transmitter, and will not be repeated here.

[0080] On the other hand, embodiments of the present invention also provide an RTP multicast video and auxiliary data multiplexing receiver system; such as Figure 4 As shown, the RTP multicast video and auxiliary data multiplexing receiver system 400 includes: The RTP receiving and parsing module 401 is used to receive RTP data packets, parse the RTP standard header of the RTP data packets, and determine the RTP payload and payload start position of the RTP data packets. The index parsing and positioning module 402 is used to parse the block index header in the RTP payload to obtain the number of target data blocks in the RTP data packet, the total length of the payload, the block type of each target data block, and the starting offset of each target data block in the RTP payload; determine the starting position of each target data block based on the starting offset, and determine the length of each target data block by the offset difference between two adjacent starting offsets. The concurrent distribution module 403 is used to send video data blocks to the video decoding pipeline and auxiliary data blocks to the corresponding message processing engine for parallel processing, based on the block type.

[0081] The RTP multicast video and auxiliary data multiplexing receiver system 400 provided in the above embodiments can implement the technical solutions described in the above embodiments of the RTP multicast video and auxiliary data multiplexing method applied to the receiver. The specific implementation principles of each module or unit can be found in the corresponding content in the above embodiments of the RTP multicast video and auxiliary data multiplexing method applied to the receiver, and will not be repeated here.

[0082] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware (such as a processor, controller, etc.), and the computer program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.

[0083] The present invention provides a detailed description of an RTP multicast video and auxiliary data multiplexing method and system. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, those skilled in the art will recognize that there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for multiplexing RTP multicast video and auxiliary data, applied at the transmitting end, characterized in that, The method includes: Video data blocks and auxiliary data blocks are generated based on the video frames to be sent and auxiliary data, respectively; the auxiliary data block is formed by concatenating the auxiliary data with the auxiliary data header, and the auxiliary data header includes a data type identifier, data length and timestamp; Multiple target data blocks to be encapsulated into the same RTP data packet are identified, and the target data blocks include video data blocks and auxiliary data blocks; Determine the starting offset of each target data block in the RTP payload, and generate a block index header based on the starting offset; the block index header includes the total payload length, the number of the plurality of target data blocks, and the block type and starting offset corresponding to each target data block; The block index header and the plurality of target data blocks are concatenated in order of the starting offset to form the RTP payload; An RTP standard header is added to the RTP payload to generate an RTP data packet, which is then sent via a multicast network.

2. The RTP multicast video and auxiliary data multiplexing method according to claim 1, characterized in that, The auxiliary data header also includes a priority field and a time-to-live field; the priority field is used to indicate the processing priority of the receiving end, and the time-to-live field is used to indicate the effective lifespan of the auxiliary data block from the time of transmission.

3. The RTP multicast video and auxiliary data multiplexing method according to claim 1, characterized in that, The block index header also includes a version number and a flag bit for each target data block; the flag bit is used to indicate whether the auxiliary data block is incrementally updated.

4. The RTP multicast video and auxiliary data multiplexing method according to claim 1, characterized in that, The determination of multiple target data blocks to be encapsulated into the same RTP data packet includes: The plurality of target data blocks are determined based on the maximum transmission unit of the current network path and the priority order of the video data blocks and the auxiliary data blocks; The total length of the RTP data packet generated after encapsulation is less than or equal to the maximum transmission unit.

5. A method for multiplexing RTP multicast video and auxiliary data, applied at a receiving end, characterized in that, The method includes: Receive RTP data packets, parse the RTP standard header of the RTP data packets, and determine the RTP payload and payload start position of the RTP data packets; Parse the block index header in the RTP payload to obtain the number of target data blocks in the RTP data packet, the total length of the payload, the block type of each target data block, and the starting offset of each target data block in the RTP payload; The starting position of each target data block is determined based on the starting offset, and the length of each target data block is determined by the offset difference between two adjacent starting offsets and the total load length. Based on the block type, video data blocks are sent to the video decoding pipeline, and auxiliary data blocks are sent to the corresponding message processing engine for parallel processing.

6. The RTP multicast video and auxiliary data multiplexing method according to claim 5, characterized in that, The method further includes: Parse the RTP standard header of the RTP data packet to determine the timestamp and sequence number of the RTP data packet; The RTP data packets are lost and reordered according to the sequence number; A unified time base is established based on the timestamp, and the time base is used to realize the synchronous processing of the video data block and the auxiliary data block.

7. The RTP multicast video and auxiliary data multiplexing method according to claim 5, characterized in that, The method further includes: Parse the priority field in the auxiliary data block and process the auxiliary data block in descending order of priority; Parse the sending timestamp and lifetime fields in the auxiliary data block. If the current time exceeds the sum of the sending timestamp and the lifetime, discard the auxiliary data block.

8. The RTP multicast video and auxiliary data multiplexing method according to claim 5, characterized in that, The method further includes: When parsing the block index header fails or a corrupted index header is detected, a preset synchronization marker is scanned starting from the RTP payload start position to locate the start position of each target data block. The boundaries of each target data block are determined based on the length field or video encoding start code in the auxiliary data header, thereby re-dividing each target data block, and the index list is reconstructed based on the division results.

9. A system for transmitting RTP multicast video and auxiliary data multiplexing, characterized in that, include: The data block construction module is used to generate video data blocks and auxiliary data blocks based on the video frames to be sent and auxiliary data, respectively. The auxiliary data block is formed by concatenating the auxiliary data header and the auxiliary data. The auxiliary data header includes a data type identifier, data length, and timestamp. An index generation module is used to determine multiple target data blocks to be encapsulated into the same RTP data packet; and to determine the starting offset of each target data block in the RTP payload, and to generate a block index header based on the starting offset; the block index header includes the total payload length, the number of the multiple target data blocks, and the block type and starting offset corresponding to each target data block; The payload assembly and transmission module is used to concatenate the block index header and the plurality of target data blocks in the order of the starting offset to form an RTP payload. An RTP standard header is added to the RTP payload to generate an RTP data packet, which is then sent via a multicast network.

10. An RTP multicast video and auxiliary data multiplexing receiver system, characterized in that, include: The RTP receiving and parsing module is used to receive RTP data packets, parse the RTP standard header of the RTP data packets, and determine the RTP payload and payload start position of the RTP data packets. The index parsing and positioning module is used to parse the block index header in the RTP payload to obtain the number of target data blocks in the RTP data packet, the total length of the payload, the block type of each target data block, and the starting offset of each target data block in the RTP payload; determine the starting position of each target data block according to the starting offset, and determine the length of each target data block by the offset difference between two adjacent starting offsets; The concurrent distribution module is used to send video data blocks to the video decoding pipeline and auxiliary data blocks to the corresponding message processing engine for parallel processing, based on the block type.

Images