Multi-path media stream sending method and device, electronic equipment and medium

Abstract

The invention discloses a multi-path media stream sending method and device, electronic equipment and a medium, and relates to the technical field of computers, according to the multi-path media stream sending method and device, copying of an original data stream, message header rewriting and multi-port synchronous sending are completed on a network card side, a timestamp is generated on a hardware level, and hardware unloading of multi-path media stream sending is achieved. Generation, packaging and synchronous sending of all copy streams are independently completed by a network card, and a central processing unit only needs to process one path of original data stream, so that CPU resources are effectively released. And meanwhile, the same timestamp is uniformly set for all streams at the same moment, so that the time deviation caused by software asynchronous processing is eliminated, the absolute consistency of the multi-path media streams in time is ensured, and seamless and smooth fault switching conditions are provided for a receiving end. Besides, a hardware processing path is extremely short, unpredictable time delay caused by operating system scheduling and context switching is avoided, sending time delay is remarkably reduced, and the stability of the system is improved.

Description

A method, apparatus, electronic device and medium for transmitting multipath media streams Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, apparatus, electronic device and medium for transmitting multipath media streams. Background Technology

[0002] In real-world critical business applications (such as live broadcasts and sports event broadcasts), a multi-path disaster recovery backup mechanism is essential for achieving high-reliability transmission. This involves simultaneously transmitting identical media data from the same source video stream through multiple independent network paths. When network jitter, packet loss, or interruption occurs on the primary transmission path, the receiving end can immediately and seamlessly switch to the backup path stream, thus ensuring broadcast stability and continuity. Currently, a typical method for implementing this multi-path backup stream requires the central processing unit (CPU) to perform multiple (the same number of paths) completely independent decoding, rendering, and frame processing on the same source video file or bitstream. For each stream to be transmitted, the CPU needs to independently execute the encapsulation process of the SMPTE ST 2110 protocol stack. After adding hardware timestamps to the encapsulated multiple independent data streams, they are sent through different network interface cards (NICs), resulting in extremely high CPU resource consumption and waste. Furthermore, the hardware timestamps on different streams may deviate, potentially causing image jitter, tearing, or interruption during switching. Summary of the Invention

[0003] This application provides a multipath media stream transmission method, apparatus, electronic device, and medium to at least solve the problems in the related art where multipath backup streams lead to extremely high central processing unit (CPU) resource consumption and waste, and where hardware timestamps on different streams may also deviate, potentially causing screen jitter, tearing, or interruption during switching.

[0004] This application provides a multipath media stream transmission method, comprising: in response to a network interface card (NIC) receiving an original data stream, obtaining a preset number of backup streams; the NIC copying the original data stream into the number of backup streams according to the number of backup streams; obtaining the original packet header information corresponding to the original data stream in the backup streams, and modifying the original packet header information in the backup streams according to target path address information; controlling the synchronous transmission of the original data stream and the backup streams through the NIC, generating timestamps at the time of transmission of the original data stream and the backup streams; setting the timestamps on the original data stream and the backup streams, and sending them into the transmission queues of their respective ports, and transmitting them via the physical network.

[0005] This application also provides a multipath media stream transmission device, the device comprising: a stream identifier, configured to obtain a preset number of backup streams in response to a network interface card (NIC) receiving an original data stream; a stream replicator, configured to have the NIC replicate the original data stream into the number of backup streams according to the number of backup streams; a packet rewriter, configured to obtain the original packet header information corresponding to the original data stream in the replicated streams and modify the original packet header information in the replicated streams according to target path address information; a hardware timestamp synchronizer, configured to control the synchronous transmission of the original data stream and the backup streams through the NIC and generate timestamps at the time of transmission of the original data stream and the backup streams; and a multi-port transmission control module, configured to set the timestamps on the original data streams and the backup streams and send them into the transmission queues of their respective corresponding ports for transmission via the physical network.

[0006] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of any of the above-described multipath media stream transmission methods: in response to a network interface card (NIC) receiving an original data stream, obtaining a preset number of backup streams, the NIC copying the original data stream into the number of backup streams according to the number of backup streams; obtaining the original packet header information corresponding to the original data stream in the copied streams, modifying the original packet header information in the copied streams according to the target path address information; controlling the synchronous transmission of the original data stream and the backup streams through the NIC, generating a timestamp at the moment of transmission of the original data stream and the backup streams; setting the timestamps on the original data stream and the backup streams, and sending them into the transmission queues of their respective corresponding ports, and transmitting them via the physical network.

[0007] This application also provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of any of the above-described multipath media stream transmission methods: in response to a network interface card (NIC) receiving an original data stream, obtaining a preset number of backup streams, the NIC copying the original data stream into the number of backup streams according to the number of backup streams; obtaining the original packet header information corresponding to the original data stream in the copy streams, modifying the original packet header information in the copy streams according to the target path address information; controlling the synchronous transmission of the original data stream and the backup streams through the NIC, generating timestamps at the time of transmission of the original data stream and the backup streams; setting the timestamps on the original data stream and the backup streams, and sending them into the transmission queues of their respective ports, and transmitting them via the physical network.

[0008] This application achieves a complete offload of the multi-path media stream transmission process from software to hardware by directly performing the copying of the original data stream, header rewriting, and multi-port synchronous transmission on the network interface card (NIC) side, and uniformly generating timestamps at the hardware level. Since the generation, encapsulation modification, and synchronous transmission of the copied stream are all handled by the NIC, the central processing unit (CPU) only needs to process one original data stream. This prevents the system CPU utilization from increasing with the number of backup streams, significantly freeing up CPU resources and multiplying the overall system throughput. Simultaneously, atomically setting the same timestamp for the original data stream and all backup streams at the same transmission moment completely eliminates time deviations caused by asynchronous software processing, ensuring absolute consistency of media streams across all paths in time. This provides a foundation for seamless and smooth failover at the receiving end. Furthermore, the extremely short hardware processing path avoids unpredictable delays caused by operating system kernel scheduling and context switching, resulting in lower and more stable transmission latency. Attached Figure Description

[0009] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 is a flowchart of a multipath backup stream method in the related art; Figure 2 is an application environment diagram of a multipath media stream transmission method in one embodiment of this application; Figure 3 is a flowchart of a multipath media stream transmission method in one embodiment of this application; Figure 4 is a structural block diagram of a multipath media stream transmission device in one embodiment of this application; Figure 5 is an internal structure diagram of a computer device in one embodiment of this application. Detailed Implementation

[0011] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0012] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0013] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0014] The SMPTE ST 2110 standard, developed by the Society of Motion Picture and Television Engineers (SMPTE), has become the core standard for IP-based audio and video transmission in professional live production and broadcast television, and is completely replacing traditional baseband signal transmission methods (such as SDI). This standard separates video, audio, and ancillary data into independent IP streams and relies on high-precision clock synchronization (PTP) and precise timing transmission mechanisms to ensure low end-to-end latency and lip-sync.

[0015] The multi-path backup stream method of the related technology is shown in Figure 1. Its workflow is as follows: Multiple independent decoding and processing: The central processing unit (CPU) must perform multiple (the same number of paths) completely independent decoding, rendering and frame processing on the same source video file or bitstream.

[0016] Multiple independent encapsulations: For each stream that needs to be sent, the CPU needs to independently execute the encapsulation process of the SMPTE ST 2110 protocol stack, including generating RTP packets, filling payloads, and generating precise timing information for each packet.

[0017] Multiple transmissions and stamping: Multiple encapsulated independent data streams are transmitted through different network interface cards (NICs). The NIC adds a hardware timestamp based on the Precision Time Protocol (PTP) to each packet before sending it out.

[0018] The multipath backup streaming technology has the following significant drawbacks: extremely high CPU resource consumption and waste: completely repetitive decoding and encapsulation calculations consume a large number of CPU cycles, resulting in low processing efficiency. The more streams there are, the more severe the resource waste becomes, becoming a major bottleneck for system performance and scalability.

[0019] Timestamp consistency is difficult to guarantee absolutely: Because the processing of multiple streams in the CPU is independent and asynchronous, the processing latency they experience may have minute differences at the microsecond level. This means that even data packets originating from the same frame may have discrepancies in their hardware timestamps on different streams. Such discrepancies are fatal to receivers that require precise frame-level switching (such as SDI gateways or IP video switchers), potentially causing image jitter, tearing, or interruption during the switching process.

[0020] High system complexity and cost: It requires a powerful CPU to handle the encoding / decoding and encapsulation of multiple streams, which increases hardware cost and system complexity.

[0021] Therefore, there is an urgent need for a multi-path media streaming solution that can completely solve the above problems and achieve low overhead and high consistency.

[0022] The multipath media stream transmission method provided in this application aims to offload the copying and forwarding of multipath backup streams from the CPU to the smart network interface card (NIC) or DPU (Data Processing Unit) hardware, thereby greatly reducing the CPU load and ensuring absolute timestamp consistency among multiple backup streams. The multipath media stream transmission method can be applied to the application environment shown in Figure 2.

[0023] Source stream receiving module: Configured on the host side, it is used to receive or generate a raw audio and video media stream (hereinafter referred to as the source stream) conforming to the SMPTE ST 2110 standard. This module completes the decoding, RTP encapsulation and initial timing scheduling of the source stream, generating a standard IP packet sequence with a predetermined transmission timestamp.

[0024] Flow rule configuration and management module: Configured on the host side, it is used to receive user instructions, define the number of backup flows to be generated (N), and the target network path attributes of each backup flow. The attributes include destination MAC address, destination IP address, destination UDP port number, VLAN tag, etc.

[0025] Hardware offloading engine: Integrated into the smart network interface card (NIC) or DPU hardware, this hardware offloading engine is used to implement multipath media stream transmission methods. Specifically, it includes: Flow identifier: Used to identify data packets from the source stream receiving module that need to be copied and forwarded. Identification can be based on a specific 5-tuple (a packet 5-tuple includes source IP, destination IP, source port, destination port, and transport layer protocol type. It is a unique identifier for network traffic), VLAN ID, or DSCP value, etc.

[0026] Stream replicator: Used to copy a single packet matched by a stream identifier N-1 times at the hardware level, thereby generating N identical packet copies (including the original packet).

[0027] Header rewriter: Used to rewrite the L2 / L3 / L4 headers of N copy packets (including the original packet) in parallel. Based on the rules issued by the flow rule configuration and management module, it assigns each copy a corresponding source MAC address, source IP address, source port number, destination MAC address, destination IP address, and destination port number as needed, and can modify some or all of the headers as required.

[0028] Hardware timestamp synchronizer: Tightly coupled to the precision clock (PTP Clock) on the network interface card (NIC). At the instant a packet leaves the NIC's hardware transmission queue, it simultaneously and atomically stamps all N packet copies with the exact same hardware transmission timestamp.

[0029] Multi-port transmission control module: Used to send out the processed N streams through their respective designated physical ports.

[0030] As shown in Figure 3, an embodiment of this application provides a multipath media stream transmission method, including the following steps: Step S1, in response to the network card receiving the original data stream, a preset number of backup streams is obtained, and the network card copies the original data stream into the number of backup streams according to the number of backup streams; Step S2, the original packet header information corresponding to the original data stream in the copy stream is obtained, and the original packet header information in the copy stream is modified according to the target path address information; Step S3, the original data stream and backup stream are sent synchronously through the network card, and a timestamp is generated at the time of sending the original data stream and backup stream; Step S4, after setting the timestamp on the original data stream and backup stream, they are sent into the transmission queue of their respective ports and transmitted via the physical network.

[0031] This system achieves complete offloading of the multi-path media stream transmission process from software to hardware by directly copying the original data stream, rewriting the packet header, and synchronously sending it across multiple ports at the network interface card (NIC) level, and uniformly generating timestamps at the hardware level. Since the generation, encapsulation, modification, and synchronous transmission of the copied stream are all handled by the NIC, the CPU only needs to process one original data stream. This prevents the system's CPU utilization from increasing with the number of backup streams, significantly freeing up CPU resources and multiplying the overall system throughput. Simultaneously, atomically setting the same timestamp for both the original data stream and all backup streams at the same transmission moment completely eliminates time deviations caused by asynchronous software processing, ensuring absolute consistency of media streams across all paths in terms of time. This provides a foundation for seamless and smooth failover at the receiving end. Furthermore, the extremely short hardware processing path avoids unpredictable delays caused by operating system kernel scheduling and context switching, resulting in lower and more stable transmission latency.

[0032] In this embodiment, the original packet header information in the replication stream is modified according to the target path address information, including: obtaining the destination Media Access Control (MAC) address, destination IP address and destination port number of each replication stream as the target path address information; and replacing the target path address information in the original packet header information of each replication stream with the target path address information corresponding to each replication stream.

[0033] This technology enables the same original data stream to be simultaneously sent to different receivers or different entry points of the same receiver via different network paths by independently configuring and replacing the destination MAC address, destination IP address, and destination port number for each replicated stream, while maintaining content consistency. This solution directly rewrites the packet header at the network interface card (NIC) level, avoiding the processing overhead and additional latency caused by software-based packet-by-packet modifications. This allows for highly flexible path configuration capabilities while maintaining extremely low latency, effectively supporting primary / backup path deployment and rapid failover in complex network environments, and providing a reliable data foundation for subsequent dynamic selection and disaster recovery based on path quality.

[0034] In this embodiment, obtaining the target path address information of each replication stream includes: receiving a user instruction and parsing the preset target path address information of each replication stream in the user instruction; or, obtaining the network transmission speed of multiple ports connected to the network card, determining the performance score of the port based on the network transmission speed, selecting multiple target ports among the multiple ports connected to the network card based on the performance score, and determining the target path address information of the replication stream sent by the target port.

[0035] By providing two target path address acquisition methods—user configuration and adaptive network interface card (NIC) port performance—multi-path media stream transmission can meet the manual planning needs under fixed network topologies while automatically selecting the optimal transmission path based on real-time port performance. When calculating performance scores based on port network transmission speeds and selecting target ports accordingly, the system can avoid poor-performing or congested physical ports without manual intervention, thereby improving overall transmission efficiency and reliability. This scheme enhances the intelligence and environmental adaptability of multi-path transmission, enabling the system to maintain stable, low-latency media transmission even under complex network conditions.

[0036] Understandably, performance scores are determined based on network transmission speed among the multiple ports connected to the network card, and the port with the best performance is selected based on the performance score to realize data transmission, thereby improving transmission efficiency and reliability.

[0037] In this embodiment, the method further includes: receiving the camera's raw bitstream; decoding, rendering, and frame processing the camera's raw bitstream using a central processing unit to form a source video file; obtaining the source Media Access Control (MAC) address, destination Media Access Control (MAC) address, source IP address, destination IP address, source port number, destination port number, and transport layer protocol type as the raw header information of the source video file; encapsulating the source video file according to the network protocol for real-time data transmission; and performing packet rate control to form a raw data stream with the raw header information.

[0038] This method integrates multipath media stream transmission with video capture, decoding, rendering, and real-time encapsulation processes, clarifying the source and encapsulation method of the original data stream. This creates a complete closed loop in the media link, from the video source to multipath network transmission. By performing only necessary video processing and real-time protocol encapsulation on the central processing unit (CPU) side, while delegating subsequent stream copying and transmission tasks to the network interface card (NIC) hardware, the flexibility of video encoding and rendering is ensured, and the maintenance of multipath transmission status at the application layer is avoided. This significantly reduces system software complexity and processing load while maintaining real-time performance and image quality.

[0039] In this embodiment, after setting timestamps on the original data stream and the backup stream, they are sent to the corresponding port's transmission queue. When sent via the physical network, the method further includes: real-time monitoring of the queue information of each physical network, identifying the sub-packet queue position corresponding to the original data stream or backup stream in the queue information; determining the predicted latency of the physical network receiving the original data stream or backup stream based on the total number of packets in the queue information and the sub-packet queue position; comparing the predicted latency of the physical network currently playing the video with the predicted latency of the backup physical network, and determining whether the predicted latency of the physical network currently playing the video will remain less than or equal to at least one within a first time period in the future. The predicted latency of the backup physical network; in response to the prediction latency of the physical network currently used for video playback being less than or equal to the predicted latency of at least one backup physical network within a first time period in the future, the physical network currently used for video playback is switched to the backup physical network, and continuous video playback is performed after aligning the timestamp of the current user's video playback with the timestamp of the original data stream or backup stream in the backup physical network; in response to the prediction latency of the physical network currently used for video playback not being less than or equal to the predicted latency of at least one backup physical network within a first time period in the future, continuous video playback is performed using the physical network currently used for video playback.

[0040] By real-time monitoring of the transmission queue status of each physical network and predictive latency calculation, the system can anticipate congestion and latency changes on different paths at the sending end and dynamically compare the transmission performance of the primary and backup networks within a future time window. When the backup network is detected to have an advantage in predicted latency, the system can smoothly switch the playback network based on the synchronized timestamps, thus avoiding stuttering or interruptions caused by network degradation; while when the current network still has an advantage, the original path is maintained to avoid unnecessary switching. This scheme fully utilizes the advantage of multi-path timestamp consistency to achieve a more intelligent, stable, and seamless network switching mechanism.

[0041] In this embodiment, the predicted latency for the physical network to receive the original data stream or backup stream is determined based on the total number of packets in the queue information and the position of the sub-packet queue. This includes: real-time monitoring of the transmission queue information of each physical network path, collecting key performance indicators, including network latency, bandwidth utilization, queue length, and packet loss rate; modeling network load and path performance based on multiple historical transmission data to determine a latency prediction algorithm, which is used to predict the latency fluctuation of each network path in the future; and calculating the predicted latency of each physical path at future times based on the key performance indicators and historical transmission data using the latency prediction algorithm.

[0042] Specifically, when network load increases, the system will assess in real time whether the path will lead to excessive latency, thereby affecting the playback quality of the video stream. Based on the path latency prediction results, the system will adjust the transmission path of the video stream, prioritizing paths with lower latency and stable bandwidth for data transmission, ensuring maximum transmission efficiency and stability when network conditions change.

[0043] In this system, real-time monitoring and historical data analysis of each sub-packet are incorporated into the transmission queue of each network path. The load status of each path is evaluated using historical latency information, and the latency judgment model is dynamically adjusted based on a prediction algorithm. When the load of a physical network is too high and the latency exceeds expectations, the activation of a backup network path is automatically triggered. Timestamp alignment is maintained during network switching to ensure continuous playback of the video stream and seamless user experience.

[0044] After receiving the latency prediction results, the system automatically selects the optimal path based on the predicted latency and the transmission quality of the network path. During path selection, the system comprehensively considers network performance indicators such as latency, bandwidth utilization, and packet loss rate to ensure that the path with the lowest latency and optimal bandwidth is chosen for data transmission.

[0045] In this embodiment, the predicted latency of the physical network where the current user is playing video is compared with the predicted latency of the backup physical network, including: obtaining the timestamp of the latest playback sub-message of the physical network where the current user is playing video as a first timestamp; obtaining the timestamp of the latest loading sub-message of each backup physical network as a second timestamp; in response to the first timestamp being earlier than the second timestamp, it is determined that the latest playback sub-message of the physical network where the current user is playing video is earlier than the latest loading sub-message of the backup physical network, and the predicted latency of the physical network where the current user is playing video is less than or equal to the predicted latency of the backup physical network; in response to the second timestamp being earlier than the first timestamp, it is determined that the latest loading sub-message of the backup physical network is earlier than the latest playback sub-message of the physical network where the current user is playing video is playing video, and the predicted latency of the physical network where the current user is playing video is greater than the predicted latency of the backup physical network.

[0046] This method utilizes the time stamp relationship between the latest sub-messages in the primary and backup networks as the basis for predicting latency comparisons, simplifying the complex network status assessment into a consistency judgment based on a time axis. Since the media streams on each path are guaranteed to have completely consistent timestamps at the sending end by hardware, this method is highly accurate and real-time, directly reflecting the actual progress of different paths at the receiving end. This eliminates the need for complex statistical models in determining network quality, reducing implementation complexity and improving the reliability of handover decisions.

[0047] In this embodiment, the method further includes: setting the transmission priority of the original data stream or backup stream in the physical network where the current user is playing the video to be the highest, and setting the transmission priority of the original data stream or backup stream in the backup physical network to be adjusted based on transmission efficiency; in response to the first timestamp being earlier than the second timestamp, deleting sub-packets in the backup physical network that are earlier than the first timestamp so that the sub-packets in the backup physical network that are at the first timestamp can be loaded, thereby clearing sub-packets in the sub-packet queue in the backup physical network that are earlier than the first timestamp to reduce the length of the sub-packet queue and improve the loading rate of the backup physical network.

[0048] This solution, building upon the multi-path transmission and handover mechanism, introduces a timestamp-based sub-packet queue cleanup and transmission priority adjustment strategy, enabling the backup network to quickly catch up with the primary network's playback progress. By deleting redundant sub-packets in the backup network that have timestamps earlier than the current playback time, its queue length is significantly shortened and loading rate is improved, ensuring that the backup network has sufficient fresh data for seamless handover when needed. This technical solution further reduces buffering pressure and latency jitter during handover, improving the stability and response speed of multi-path disaster recovery playback.

[0049] When implementing the multipath media streaming method, please refer to Figure 2. For example, in a 4K ultra-high-definition broadcast of a large-scale sports event, it is necessary to send one main camera signal to the remote production center simultaneously through three network paths.

[0050] Initial setup: The source receiving module (such as an ST 2110 transmitting software based on SOFTWARE) runs on the CPU, decodes the camera's raw bitstream, and encapsulates it into an ST 2110 video stream with a destination address of (MAC_A, IP_A, Port_A).

[0051] Rule Configuration: Through the management interface, the engineer instructs the system to create two backup flows for this traffic. The destination address of backup flow 1 is (MAC_B, IP_B, Port_B), and the destination address of backup flow 2 is (MAC_C, IP_C, Port_C). This configuration is then distributed to the network card driver.

[0052] Hardware offloading: The network card driver programs these rules into the hardware flow table of the smart network card.

[0053] Message processing: Source flow messages generated by the host enter the network interface card.

[0054] The flow identifier matches the packet according to preset rules.

[0055] The stream replicator instantly creates two additional copy of the packet, at which point the network card has a total of three identical packets.

[0056] The header rewriter operates in parallel: it does not modify the original packet; it rewrites the header of replica 1 to (MAC_B, IP_B, Port_B); and it rewrites the header of replica 2 to (MAC_C, IP_C, Port_C).

[0057] The three messages are temporarily stored. At the precise scheduled transmission time, the hardware timestamp synchronizer simultaneously stamps the three messages with the exact same transmission timestamp TX_timestamp within one hardware clock cycle.

[0058] The three messages were sent out through three different physical ports or queues.

[0059] Disaster recovery switching: The receiving end receives three streams. Since the timestamps are completely consistent, when the main path (stream 1) network fails, the switching device can immediately and seamlessly switch to the corresponding data packets of the backup path (copying stream 2 or copying stream 3). Because the timestamp sequences of all packets are perfectly matched, the continuity of playback is guaranteed.

[0060] This application perfectly solves the efficiency and consistency problems of redundant backup in high-performance media transmission through a collaborative design of hardware offloading and atomic stamping.

[0061] The beneficial effects of this technical solution include: Extreme performance enhancement and CPU resource release: High-consumption stream replication and encapsulation work is completely offloaded to hardware, requiring the CPU to process only one source stream, decoupling CPU utilization from the number of backup streams and improving system efficiency several times over (proportional to the number of paths N); Absolute timestamp consistency: By atomically and simultaneously applying the same timestamp to all replicas at the hardware level, time deviations caused by asynchronous software processing are completely eliminated, ensuring complete consistency between primary and backup streams in the time dimension, achieving seamless and smooth failover at the receiving end; Reduced system latency and jitter: The extremely short hardware processing path avoids unpredictable latency caused by operating system kernel scheduling and context switching, resulting in lower and more stable transmission latency; High scalability: The solution's performance depends on the capabilities of the smart network interface card / DPU, not the host CPU. With network interface card support, dozens or even hundreds of backup paths can be easily added, greatly improving the overall scalability of the system; Simplified software stack: Applications do not need to maintain independent states and processing logic for each stream, significantly simplifying software design and implementation.

[0062] In the aforementioned multipath media stream transmission method, the copying of the original data stream, header rewriting, and multi-port synchronous transmission are directly completed at the network interface card (NIC) side, and timestamps are uniformly generated at the hardware level. This achieves a complete offloading of the multipath media stream transmission process from software to hardware. Since the generation, encapsulation modification, and synchronous transmission of the copied stream are all completed by the NIC, the central processing unit (CPU) only needs to process one original data stream. This prevents the system CPU utilization from increasing with the number of backup streams, significantly freeing up CPU resources and multiplying the overall system throughput. Simultaneously, atomically setting the same timestamp for the original data stream and all backup streams at the same transmission moment completely eliminates the time deviation caused by asynchronous software processing, ensuring absolute consistency of the media streams across all paths in the time dimension. This provides a foundation for seamless and smooth failover at the receiving end. Furthermore, the extremely short hardware processing path avoids unpredictable delays caused by operating system kernel scheduling and context switching, resulting in lower and more stable transmission latency.

[0063] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0064] In one embodiment, as shown in FIG4, a multipath media stream sending device 10 is provided, including: a stream identifier 1, a stream replicator 2, a message rewriter 3, a hardware timestamp synchronizer 4, a multi-port sending control module 5, a source stream receiving module 6, and a transmission control module 7.

[0065] Stream identifier 1 is used to obtain the preset number of backup streams in response to the network card receiving the original data stream; stream replicator 2 is used by the network card to replicate the original data stream into the number of backup streams according to the number of backup streams.

[0066] The message rewriter 3 is used to obtain the original message header information of the corresponding original data stream in the copy stream, and modify the original message header information in the copy stream according to the target path address information.

[0067] The hardware timestamp synchronizer 4 is used to control the synchronous transmission of the original data stream and the backup stream through the network card, and generates a timestamp at the moment of transmission of the original data stream and the backup stream.

[0068] The multi-port transmission control module 5 is used to set timestamps on the original data stream and the backup stream and send them into the transmission queue of their respective ports, and then transmit them via the physical network.

[0069] By incorporating functional units such as flow identification, flow replication, packet rewriting, timestamp synchronization, and multi-port transmission into the network interface card (NIC) or its associated hardware, multi-path transmission capabilities can be independently implemented as a hardware device. This device architecture clearly defines the responsibilities of each functional module, facilitating integration and expansion on smart NIC or DPU platforms. This allows for support of large-scale, multi-path media streaming without relying on host CPU performance, further enhancing the system's performance ceiling and engineering feasibility.

[0070] In this embodiment, the original packet header information in the replication stream is modified according to the target path address information, including: obtaining the destination Media Access Control (MAC) address, destination IP address and destination port number of each replication stream as the target path address information; and replacing the target path address information in the original packet header information of each replication stream with the target path address information corresponding to each replication stream.

[0071] In this embodiment, obtaining the target path address information of each replication stream includes: receiving a user instruction and parsing the preset target path address information of each replication stream in the user instruction; or, obtaining the network transmission speed of multiple ports connected to the network card, determining the performance score of the port based on the network transmission speed, selecting multiple target ports among the multiple ports connected to the network card based on the performance score, and determining the target path address information of the replication stream sent by the target port.

[0072] In this embodiment, the source stream receiving module 6 is used to: receive the camera's raw bitstream; decode, render, and process the camera's raw bitstream using a central processing unit to form a source video file; obtain the source Media Access Control (MAC) address, destination Media Access Control (MAC) address, source IP address, destination IP address, source port number, destination port number, and transport layer protocol type as the raw header information of the source video file; encapsulate the source video file according to the network protocol for real-time data transmission, and perform packet rate control to form a raw data stream with the raw header information.

[0073] In this embodiment, after setting timestamps on the original data stream and the backup stream, they are sent to the corresponding port's transmission queue. When sent via the physical network, the method further includes: real-time monitoring of the queue information of each physical network, identifying the sub-packet queue position corresponding to the original data stream or backup stream in the queue information; determining the predicted latency of the physical network receiving the original data stream or backup stream based on the total number of packets in the queue information and the sub-packet queue position; comparing the predicted latency of the physical network currently playing the video with the predicted latency of the backup physical network, and determining whether the predicted latency of the physical network currently playing the video will remain less than or equal to at least one within a first time period in the future. The predicted latency of the backup physical network; in response to the prediction latency of the physical network currently used for video playback being less than or equal to the predicted latency of at least one backup physical network within a first time period in the future, the physical network currently used for video playback is switched to the backup physical network, and continuous video playback is performed after aligning the timestamp of the current user's video playback with the timestamp of the original data stream or backup stream in the backup physical network; in response to the prediction latency of the physical network currently used for video playback not being less than or equal to the predicted latency of at least one backup physical network within a first time period in the future, continuous video playback is performed using the physical network currently used for video playback.

[0074] In this embodiment, the predicted latency of the physical network where the current user is playing video is compared with the predicted latency of the backup physical network, including: obtaining the timestamp of the latest playback sub-message of the physical network where the current user is playing video as a first timestamp; obtaining the timestamp of the latest loading sub-message of each backup physical network as a second timestamp; in response to the first timestamp being earlier than the second timestamp, it is determined that the latest playback sub-message of the physical network where the current user is playing video is earlier than the latest loading sub-message of the backup physical network, and the predicted latency of the physical network where the current user is playing video is less than or equal to the predicted latency of the backup physical network; in response to the second timestamp being earlier than the first timestamp, it is determined that the latest loading sub-message of the backup physical network is earlier than the latest playback sub-message of the physical network where the current user is playing video is playing video, and the predicted latency of the physical network where the current user is playing video is greater than the predicted latency of the backup physical network.

[0075] In this embodiment, the transmission control module 7 is used to: set the transmission priority of the original data stream or backup stream in the physical network where the current user is playing the video to be the highest, and set the transmission priority of the original data stream or backup stream in the backup physical network to be adjusted based on transmission efficiency; in response to the first timestamp being earlier than the second timestamp, delete the sub-packets in the backup physical network that are earlier than the first timestamp so that the sub-packets in the backup physical network that are at the first timestamp can be loaded.

[0076] In the aforementioned multipath media stream transmission device, the copying of the original data stream, header rewriting, and multi-port synchronous transmission are completed directly at the network interface card (NIC) side, and timestamps are uniformly generated at the hardware level. This achieves a complete offloading of the multipath media stream transmission process from software to hardware. Since the generation, encapsulation modification, and synchronous transmission of the copied stream are all completed by the NIC, the central processing unit (CPU) only needs to process one original data stream. This prevents the system CPU utilization from increasing with the number of backup streams, significantly freeing up CPU resources and multiplying the overall system throughput. Simultaneously, by atomically setting the same timestamp for the original data stream and all backup streams at the same transmission moment, the time deviation caused by asynchronous software processing is completely eliminated, ensuring absolute consistency of the media streams across all paths in the time dimension. This provides a foundation for seamless and smooth failover at the receiving end. Furthermore, the extremely short hardware processing path avoids unpredictable delays caused by operating system kernel scheduling and context switching, resulting in lower and more stable transmission latency.

[0077] For a description of the features in the embodiments corresponding to the multipath media stream transmission device, please refer to the relevant descriptions in the embodiments corresponding to the multipath media stream transmission method, which will not be repeated here.

[0078] Embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above embodiments of the multipath media streaming method.

[0079] In one embodiment, the electronic device may be a server, and its internal structure may be as shown in Figure 5. The electronic device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database stores multipath media stream transmission data. The network interface communicates with external terminals via a network connection. When the computer program is executed by the processor, it implements a multipath media stream transmission method.

[0080] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above embodiments of the multipath media stream transmission method: in response to the network card receiving the original data stream, obtaining a preset number of backup streams, and the network card copying the original data stream into the number of backup streams according to the number of backup streams; obtaining the original packet header information corresponding to the original data stream in the copy streams, and modifying the original packet header information in the copy streams according to the target path address information; controlling the original data stream and backup streams to be transmitted synchronously through the network card, and generating timestamps at the time of transmitting the original data stream and backup streams; setting timestamps on the original data stream and backup streams and sending them into the transmission queues of their respective ports, and transmitting them via the physical network.

[0081] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0082] Embodiments of this application also provide a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the steps in any of the above-described multipath media stream transmission method embodiments: in response to the network card receiving the original data stream, obtaining a preset number of backup streams, and the network card copies the original data stream into the number of backup streams according to the number of backup streams; obtaining the original packet header information corresponding to the original data stream in the copy streams, and modifying the original packet header information in the copy streams according to the target path address information; controlling the original data stream and backup streams to be transmitted synchronously through the network card, and generating timestamps at the time of transmitting the original data stream and backup streams; setting timestamps on the original data stream and backup streams and sending them into the transmission queues of their respective ports, and transmitting them via the physical network.

[0083] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the steps in any of the above-described multipath media stream transmission method embodiments: in response to the network card receiving the original data stream, obtaining a preset number of backup streams, and the network card copying the original data stream into the number of backup streams according to the number of backup streams; obtaining the original packet header information corresponding to the original data stream in the copy streams, and modifying the original packet header information in the copy streams according to the target path address information; controlling the synchronous transmission of the original data stream and backup streams through the network card, and generating timestamps at the time of transmission of the original data stream and backup streams; setting timestamps on the original data stream and backup streams and sending them into the transmission queues of their respective ports, and transmitting them via the physical network.

[0084] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0085] The foregoing has provided a detailed description of a multipath media stream transmission method, apparatus, electronic device, and medium provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only intended to aid in understanding the method and core ideas of this application. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of this application.

Claims

1. A method for sending multipath media streams, characterized in that, include: In response to the network interface card receiving the original data stream, the network interface card obtains a preset number of backup streams and copies the original data stream into the number of backup streams according to the number of backup streams. Obtain the original packet header information corresponding to the original data stream in the replicated stream, modify the original packet header information in the replicated stream according to the target path address information; control the synchronous transmission of the original data stream and the backup stream through the network card, and generate a timestamp at the moment of transmission of the original data stream and the backup stream; After setting the timestamp on the original data stream and the backup data stream, they are sent into the sending queue of their respective ports and then transmitted via the physical network.

2. The multipath media stream transmission method according to claim 1, characterized in that, The step of modifying the original packet header information in the replicated stream according to the target path address information includes: obtaining the destination media access control address, destination IP address and destination port number of each replicated stream as the target path address information; and replacing the target path address information in the original packet header information of each replicated stream with the target path address information corresponding to each replicated stream.

3. The multipath media stream transmission method according to claim 2, characterized in that, The step of obtaining the target path address information of each replication stream includes: receiving a user instruction and parsing the preset target path address information of each replication stream in the user instruction; or, obtaining the network transmission speed of multiple ports connected to the network card, determining the performance score of the port based on the network transmission speed, selecting multiple target ports among the multiple ports connected to the network card based on the performance score, and determining the target path address information of the replication stream corresponding to the target port.

4. The multipath media stream transmission method according to claim 1, characterized in that, The method further includes: receiving the camera's raw bitstream; decoding, rendering, and frame processing the camera's raw bitstream using a central processing unit to form a source video file; obtaining the source media access control address, destination media access control address, source IP address, destination IP address, source port number, destination port number, and transport layer protocol type as the raw header information of the source video file; encapsulating the source video file according to the network protocol for real-time data transmission, and performing packet rate control to form a raw data stream with the raw header information.

5. The multipath media stream transmission method according to claim 1, characterized in that, When the original data stream and the backup data stream are sent to their respective corresponding port's transmission queue after the timestamps are set on them, and then transmitted via the physical network, the method further includes: real-time monitoring of the queue information of each physical network, identifying the sub-packet queue position corresponding to the original data stream or the backup stream in the queue information; determining the predicted latency of the physical network receiving the original data stream or the backup stream based on the total number of packets in the queue information and the sub-packet queue position; comparing the predicted latency of the physical network currently playing the video with the predicted latency of the backup physical network, and determining whether the predicted latency of the physical network currently playing the video will remain less than a certain value within a first time period in the future. The predicted latency of the physical network used by the current user to play the video is equal to the predicted latency of at least one backup physical network. In response to maintaining the predicted latency of the physical network used by the current user to play the video being less than or equal to the predicted latency of at least one backup physical network for a first time period in the future, the physical network used by the current user to play the video is switched to the backup physical network, and continuous video playback is performed after aligning the timestamp of the current user playing the video with the timestamp of the original data stream or the backup stream in the backup physical network. In response to not maintaining the predicted latency of the physical network used by the current user to play the video being less than or equal to the predicted latency of at least one backup physical network for a first time period in the future, continuous video playback is performed using the physical network used by the current user to play the video.

6. The multipath media stream transmission method according to claim 5, characterized in that, The step of comparing the predicted latency of the physical network currently playing the video with the predicted latency of the backup physical network includes: obtaining the timestamp of the latest playback sub-message of the physical network currently playing the video as a first timestamp; obtaining the timestamp of the latest loading sub-message of each of the backup physical networks as a second timestamp; in response to the first timestamp being earlier than the second timestamp, determining that the latest playback sub-message of the physical network currently playing the video is earlier than the latest loading sub-message of the backup physical network, and the predicted latency of the physical network currently playing the video is less than or equal to the predicted latency of the backup physical network; in response to the second timestamp being earlier than the first timestamp, determining that the latest loading sub-message of the backup physical network is earlier than the latest playback sub-message of the physical network currently playing the video, and the predicted latency of the physical network currently playing the video is greater than the predicted latency of the backup physical network.

7. The multipath media stream transmission method according to claim 6, characterized in that, The method further includes: setting the transmission priority of the original data stream or the backup stream in the physical network where the current user is playing the video to be the highest, and setting the transmission priority of the original data stream or the backup stream in the backup physical network to be adjusted based on transmission efficiency; in response to the first timestamp being earlier than the second timestamp, deleting the sub-packets in the backup physical network that are earlier than the first timestamp so that the sub-packets in the backup physical network that are at the first timestamp can be loaded.

8. A multipath media stream transmission device, characterized in that, The device includes: a flow identifier, used to obtain a preset number of backup flows in response to the network interface card (NIC) receiving the original data flow; a flow replicator, used by the NIC to replicate the original data flow into the number of backup flows according to the number of backup flows; a packet rewriter, used to obtain the original packet header information corresponding to the original data flow in the replicated flows, and modify the original packet header information in the replicated flows according to the target path address information; a hardware timestamp synchronizer, used to control the synchronous transmission of the original data flow and the backup flows through the NIC, and generate timestamps at the time of transmission of the original data flow and the backup flows; and a multi-port transmission control module, used to set the timestamps on the original data flow and the backup flows and send them into the transmission queues of their respective ports, and transmit them via the physical network.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the multipath media stream transmission method as described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the multipath media stream transmission method as described in any one of claims 1 to 7.

Images