A multimedia control system and method based on time-sensitive network

Abstract

This invention provides a multimedia control system and method based on time-sensitive networking (TSN). The system includes a network interface module, a traffic shaping and scheduling module, a stream classification module, a bandwidth dynamic allocation module, and an embedded hardware platform. The network interface module supports the IEEE 802.1 Qav protocol for accessing time-sensitive networks. The stream classification module classifies audio streams, video streams, signal streams, and data streams and maps them to corresponding priority queues. The traffic shaping and scheduling module implements traffic shaping based on a credit value mechanism. The bandwidth dynamic allocation module dynamically adjusts queue bandwidth parameters through a closed-loop control algorithm. The embedded hardware platform adapts to resource-constrained scenarios, achieving low-latency, high-synchronization, deterministic transmission of multiple network streams. This invention solves the problems of existing systems lacking hardware-level support for the Qav protocol, insufficient dynamic bandwidth scheduling capabilities, and missing multi-stream classification management, thus improving the reliability of multimedia communication operating in parallel with other data communications.

Description

Technical Field

[0001] This invention relates to the field of industrial Ethernet communication and real-time multimedia transmission technology, specifically to a multimedia control system and method based on time-sensitive networking. Background Technology

[0002] With the rapid development of industrial control, in-vehicle entertainment, video conferencing, virtual reality (VR), and smart cockpits, the requirements for real-time transmission of multimedia data in various systems are increasing. Traditional Ethernet has inherent defects in audio and video transmission, such as unpredictable latency, large jitter, and poor synchronization, making it difficult to meet the application requirements of low latency and high synchronization accuracy.

[0003] Time-Sensitive Networking (TSN), as a subset of protocols extending the IEEE 802.1 standard, aims to achieve deterministic communication over Ethernet. The IEEE 802.1 Qav protocol defines a Credit-Based Shaper (CBS) mechanism, which maintains a credit counter for each priority traffic flow and dynamically controls the transmission rate of various traffic types. This ensures smooth transmission of high-priority network flows with low latency, providing a technical foundation for real-time multimedia transmission.

[0004] However, existing multimedia control systems still have many shortcomings: on the one hand, most systems are based on ordinary Ethernet controllers or the AVB protocol, lacking hardware-level support for the IEEE 802.1Qav protocol, and thus unable to achieve precise dynamic bandwidth scheduling; on the other hand, existing systems do not perform targeted classification and management of different types of network streams such as audio streams, video streams, signal streams, and data streams, leading to easy interference when multiple streams are transmitted in parallel; furthermore, TSN control is rarely used on embedded platforms (such as the STM32 series), and due to system resource constraints, it is difficult to balance transmission performance and real-time requirements. Therefore, there is an urgent need for a high-precision, low-latency multimedia control system adapted to embedded platforms and based on the IEEE 802.1Qav protocol to address the deficiencies of existing technologies. Summary of the Invention

[0005] This invention aims to provide a multimedia control system and method based on time-sensitive networking, which solves the technical problems of existing systems such as lack of hardware-level support for the IEEE 802.1Qav protocol, insufficient dynamic bandwidth scheduling capability, lack of multi-stream classification management, and poor adaptability to embedded platforms. It realizes deterministic transmission and synchronous control of multiple network streams and improves the reliability of parallel communication in the system.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solution:

[0007] In a first aspect, the present invention provides a multimedia control system based on a time-sensitive network, comprising:

[0008] The network interface module is used to access time-sensitive networks and supports the IEEE 802.1Qav protocol.

[0009] The stream classification module is used to identify different types of network streams output by upper-layer applications and map the network streams to corresponding queues according to preset priorities.

[0010] The traffic shaping and scheduling module, based on the credit value mechanism of the IEEE 802.1Qav protocol, performs traffic shaping on the queue to ensure low-latency transmission of high-priority network flows.

[0011] The bandwidth dynamic allocation module is used to monitor link utilization and traffic demand in real time and dynamically adjust the bandwidth allocation parameters of the queue.

[0012] An embedded hardware platform runs an embedded system and driver that supports time-sensitive networking, providing a hardware operating environment for the network interface module, flow classification module, traffic shaping and scheduling module, and bandwidth dynamic allocation module, enabling deterministic transmission and synchronous control of multiple network flows.

[0013] As a further improvement to the technical solution of the present invention, the network interface module is a gigabit Ethernet interface and is compatible with a subset of the Time-Sensitive Networking (TSN) protocol, enabling it to establish a stable communication connection with an external time-sensitive network.

[0014] As a further improvement to the technical solution of this invention, the traffic shaping and scheduling module is a credit-based shaping and scheduling module (CBS), which operates at the Linux kernel driver layer through the network scheduler (TCqdisc). The module is implemented with configuration parameters including idleslope, hicredit, and locredit. It dynamically controls the sending rate of each queue by maintaining a credit counter.

[0015] As a further improvement to the technical solution of the present invention, the network stream types identified by the stream classification module include audio streams, video streams, signal streams and data streams, and the preset priority order is: audio stream > video stream > signal stream > data stream, and different types of network streams are mapped to independent Qav queues respectively.

[0016] As a further improvement to the technical solution of the present invention, the bandwidth dynamic allocation module adopts a closed-loop control algorithm. The closed-loop control algorithm is based on proportional-integral (PI) control logic. By calculating the deviation between the real-time utilization rate of the queue and the target utilization rate, the bandwidth allocation parameters are dynamically adjusted to keep the queue utilization rate stable within the target range.

[0017] As a further improvement to the technical solution of the present invention, the embedded hardware platform is an STM32MP257 processor, which integrates a dual-core ARM Cortex-A35 and a real-time coprocessor core Cortex-M33, and the embedded system is an embedded Linux system that supports the TSN protocol.

[0018] As a further improvement to the technical solution of the present invention, the control law of the closed-loop control algorithm is as follows:

[0019] ,

[0020] in, Assign parameters to the current bandwidth of the i-th queue, e i (t) represents the deviation between the real-time utilization rate and the target utilization rate of the i-th queue. For proportional gain, This is the integral gain.

[0021] As a further improvement to the technical solution of the present invention, the update cycle of the bandwidth dynamic allocation module is 10ms-200ms, and the target utilization rate ranges from 0.6 to 0.9.

[0022] As a further improvement to the technical solution of the present invention, the Ethernet switch (ETHSW) of the embedded hardware platform has at least 8 Tx hardware queues and the Gigabit MAC controller (GMAC) has at least 4 Tx hardware queues. The Tx hardware queues are interconnected through an internal AXI bus and traffic shaping is achieved by hardware scheduling logic.

[0023] A second aspect of the present invention provides a multimedia transmission method based on a time-sensitive network, applied to the aforementioned multimedia control system, comprising the following steps:

[0024] S1: Establish a connection with an external time-sensitive network through the network interface module and configure the network parameters of the embedded hardware platform;

[0025] S2: Identify the network stream type output by the upper-layer application through the stream classification module, and map the network stream to the corresponding Qav queue according to the preset priority;

[0026] S3: Initialize the credit value parameters of each queue through the traffic shaping scheduling module, and implement traffic shaping based on the IEEE802.1Qav protocol;

[0027] S4: The real-time utilization rate of each queue is periodically collected through the bandwidth dynamic allocation module, and the bandwidth allocation parameters of the queue are dynamically adjusted based on the closed-loop control algorithm.

[0028] S5: Through the hardware scheduling logic of the embedded hardware platform, parallel transmission and synchronous control of multiple network streams are achieved, ensuring determinism and low jitter in transmission.

[0029] The technical solution of the present invention has the following advantages over the prior art:

[0030] This invention establishes a stable communication link through a network interface module supporting the IEEE 802.1 Qav protocol. It combines a stream classification module for accurate classification and priority mapping of audio, video, signal, and data streams, with a traffic shaping and scheduling module based on a credit value mechanism to achieve low-latency transmission of high-priority streams. Furthermore, a bandwidth dynamic allocation module driven by a closed-loop control algorithm adapts to traffic demands in real time and optimizes link utilization. Leveraging the hardware-level scheduling and low resource consumption characteristics of the STM32MP257 embedded hardware platform, it effectively overcomes the shortcomings of existing systems, such as lack of Qav protocol hardware support, multi-stream transmission interference, insufficient dynamic bandwidth scheduling, and poor adaptability to embedded scenarios. It achieves determinism, low jitter, and high synchronization in parallel transmission of multiple network streams, significantly improving the reliability of concurrent multimedia communication and other data communication. It is suitable for various scenarios such as vehicle central control, industrial HMI, and remote video monitoring, possessing strong practicality and promotional value. Attached Figure Description

[0031] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0032] Figure 1 This is a block diagram of the module structure of the multimedia control system in the embodiments of this application;

[0033] Figure 2 This is a diagram showing the internal network flow topology of the system in this application embodiment;

[0034] Figure 3 This is a block diagram of bandwidth queue allocation in an embodiment of this application;

[0035] Figure 4 This is a flowchart of the multimedia transmission method in the embodiments of this application;

[0036] Figure 5 This is an overall framework diagram of a multimedia control system based on time-sensitive networking in an embodiment of this application. Detailed Implementation

[0037] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0038] The present invention will be further described in detail below with reference to the accompanying drawings.

[0039] Reference Figure 5 In a first aspect, the present invention provides a multimedia control system based on time-sensitive networking, comprising:

[0040] The network interface module is used to access time-sensitive networks and supports the IEEE 802.1Qav protocol.

[0041] The stream classification module is used to identify different types of network streams output by upper-layer applications and map the network streams to corresponding queues according to preset priorities.

[0042] The traffic shaping and scheduling module, based on the credit value mechanism of the IEEE 802.1Qav protocol, performs traffic shaping on the queue to ensure low-latency transmission of high-priority network flows.

[0043] The bandwidth dynamic allocation module is used to monitor link utilization and traffic demand in real time and dynamically adjust the bandwidth allocation parameters of the queue.

[0044] An embedded hardware platform runs an embedded system and driver that supports time-sensitive networking, providing a hardware operating environment for the network interface module, flow classification module, traffic shaping and scheduling module, and bandwidth dynamic allocation module, enabling deterministic transmission and synchronous control of multiple network flows.

[0045] It should be noted that the network interface module connects to an external time-sensitive network and establishes a communication link by supporting the IEEE 802.1Qav protocol; the flow classification module identifies different types of network flows output by upper-layer applications, such as audio streams, video streams, signal streams, and data streams, and maps each type of network flow to its corresponding independent queue according to preset priority rules; the traffic shaping and scheduling module maintains a credit counter for each queue based on the credit value mechanism of the IEEE 802.1Qav protocol, dynamically controls the transmission rate of each queue, and realizes traffic shaping; the bandwidth dynamic allocation module monitors the link utilization and the traffic demand of each queue in real time, and adjusts the queue bandwidth allocation parameters through a closed-loop control algorithm; the embedded hardware platform runs an embedded system and driver that supports time-sensitive networks, providing hardware support for each module, and realizes parallel transmission and synchronous control of multiple network flows through hardware-level scheduling.

[0046] This invention, through a multi-module collaborative design, addresses the core issues of existing systems, including lack of hardware-level support for the IEEE 802.1Qav protocol, insufficient dynamic bandwidth scheduling capabilities, missing multi-stream classification management, and poor adaptability to embedded platforms. It achieves deterministic transmission of multiple network streams, ensuring low-latency transmission of high-priority streams while avoiding interference between different types of network streams. This improves the reliability of concurrent multimedia communication with other data communication, broadens the application scope of time-sensitive networking technology in embedded scenarios, and is suitable for multiple fields such as vehicle central control systems and industrial human-machine interfaces.

[0047] In some embodiments, the network interface module is a Gigabit Ethernet interface and is compatible with a subset of Time-Sensitive Networking (TSN) protocols, enabling it to establish stable communication connections with external TSN networks. It should be noted that the network interface module uses a Gigabit Ethernet interface as the physical communication carrier, is compatible with a subset of TSN protocols, and through hardware-level protocol adaptation and interface design, ensures a stable, high-speed bidirectional communication connection with external TSN networks, providing sufficient bandwidth and protocol support for the transmission of various network streams. The Gigabit Ethernet interface design guarantees high-bandwidth transmission requirements, meeting the bandwidth needs of large-volume data transmissions such as multimedia streams and video streams; its compatibility with a subset of TSN protocols and the IEEE 802.1Qav protocol ensures seamless access to TSN environments, avoiding communication interruptions or transmission instability caused by protocol incompatibility, and laying a reliable link foundation for subsequent functions such as traffic shaping and dynamic bandwidth allocation.

[0048] In some embodiments, the traffic shaping scheduling module is a credit-based shaping scheduling module (CBS), which operates at the Linux kernel driver layer via the network scheduler (TCqdisc). The module is implemented with configuration parameters including idleslope, hicredit, and locredit. It dynamically controls the sending rate of each queue by maintaining a credit counter.

[0049] It should be noted that the traffic shaping and scheduling module adopts the credit shaping and scheduling module, which integrates the network scheduler at the Linux kernel driver layer. This module maintains a credit counter for each priority queue by configuring three core parameters: idleslope, hicredit, and locredit. When a queue has data to send, transmission is allowed if the credit value is greater than 0, and the credit value decreases at the sendslope rate during transmission. When a queue is idle, the credit value increases at the idleslope rate until it reaches the hicredit limit. This dynamic adjustment mechanism achieves traffic shaping. The kernel driver layer implementation reduces the data interaction latency between user space and kernel space, improving the real-time performance of traffic scheduling. By precisely configuring the three core parameters and the dynamic credit value adjustment mechanism, the transmission rate of each queue can be strictly controlled, ensuring low-latency, low-jitter transmission for high-priority queues, while avoiding the starvation problem caused by low-priority queues being occupied by bandwidth for a long time, thus achieving orderly transmission of various network flows.

[0050] In some embodiments, the network stream types identified by the stream classification module include audio streams, video streams, signal streams, and data streams, and the preset priority order is: audio stream > video stream > signal stream > data stream, with different types of network streams mapped to independent Qav queues.

[0051] It should be noted that the flow classification module accurately identifies audio, video, signal, and data streams by parsing the network stream's identifier information, transmission requirements, and service type. It predefines the priority order as audio stream > video stream > signal stream > data stream, and maps different types of network streams to independent Qav queues based on this order, achieving physical isolation and classified management of network streams of different service types. This classification mapping mechanism avoids interference problems when different types of network streams are transmitted in the same queue, ensuring that high-priority audio and video streams, and other network streams with high real-time requirements, receive bandwidth resources first. The independent queue design allows subsequent traffic shaping and bandwidth allocation to be precisely implemented according to the characteristics of different types of network streams, improving the system's adaptability to diverse network streams and the accuracy of transmission control.

[0052] In some embodiments, the bandwidth dynamic allocation module employs a closed-loop control algorithm. The closed-loop control algorithm is based on proportional-integral (PI) control logic. By calculating the deviation between the real-time queue utilization rate and the target utilization rate, the bandwidth allocation parameters are dynamically adjusted to stabilize the queue utilization rate within the target range.

[0053] It should be noted that the bandwidth dynamic allocation module employs a closed-loop control algorithm based on proportional-integral (PI) control. It periodically collects statistical data such as byte counts and idle time for each queue to calculate the real-time queue utilization. The real-time utilization is compared with a preset target utilization to obtain the deviation value. The response speed is adjusted using proportional gain, and steady-state error is eliminated using integral gain. Based on the deviation value, the bandwidth allocation parameters for each queue are dynamically adjusted to ensure that the utilization of each queue remains stable within the target range. The closed-loop control algorithm achieves adaptive adjustment of bandwidth allocation, enabling real-time response to dynamic changes in traffic demand and avoiding bandwidth waste or insufficiency caused by fixed bandwidth allocation. The PI control logic ensures the stability and accuracy of bandwidth adjustment, responding quickly to traffic fluctuations while avoiding transmission jitter caused by excessive parameter adjustments, significantly improving link utilization and network transmission stability.

[0054] In some embodiments, the embedded hardware platform is an STM32MP257 processor, which integrates a dual-core ARM Cortex-A35 and a real-time coprocessor core Cortex-M33, and the embedded system is an embedded Linux system that supports the TSN protocol.

[0055] It should be noted that the embedded hardware platform uses the STM32MP257 processor, whose integrated dual-core ARM Cortex-A35 provides high-performance computing capabilities to meet the needs of multi-module collaborative operation and data processing; the real-time coprocessing core Cortex-M33 is responsible for scheduling tasks with high real-time requirements, ensuring real-time response to operations such as traffic shaping and bandwidth adjustment; the platform runs an embedded Linux system that supports the TSN protocol, and customized drivers enable the adaptation and collaborative operation of each module with the hardware platform. The dual-core architecture of the STM32MP257 processor achieves a balance between high-performance computing and real-time processing, meeting the operational requirements of resource-constrained embedded scenarios; the embedded Linux system supporting the TSN protocol and customized drivers ensure the stable implementation of time-sensitive network-related functions, reduce system resource consumption, solve the problem that traditional embedded platforms struggle to balance TSN control performance and real-time performance, and improve the system's adaptability and operating efficiency in embedded scenarios.

[0056] In some embodiments, the control law of the closed-loop control algorithm is as follows:

[0057] ,

[0058] in, Assign parameters to the current bandwidth of the i-th queue. Let be the deviation between the real-time utilization rate and the target utilization rate of the i-th queue. For proportional gain, The integral gain is used to make bandwidth adjustment predictable and accurate. The synergistic effect of proportional gain and integral gain ensures a rapid response to traffic changes while effectively eliminating steady-state errors, ensuring that the utilization rate of each queue remains stable within the target range. The algorithm has simple logic, is easy to implement on embedded platforms, does not require a large amount of system resources, and balances adjustment accuracy and operating efficiency, further improving the reliability and practicality of dynamic bandwidth allocation.

[0059] In some embodiments, the update cycle of the bandwidth dynamic allocation module is 10ms-200ms, and the target utilization rate ranges from 0.6 to 0.9. It should be noted that the bandwidth dynamic allocation module sets the update cycle within the 10ms-200ms range to select an appropriate update frequency based on the traffic fluctuation characteristics of the actual application scenario; setting the target utilization rate within a reasonable range of 0.6-0.9 balances link utilization and transmission stability. The module collects queue statistics data according to the set update cycle, calculates the deviation value, and adjusts the bandwidth allocation parameters to ensure that the queue utilization rate remains stable within the target utilization rate range. A reasonable update cycle range ensures timely response to traffic changes while avoiding increased system resource consumption due to excessively high update frequencies; the target utilization rate range balances full utilization of link resources with transmission stability, preventing transmission congestion due to excessive utilization or bandwidth waste due to excessively low utilization; this design enables the bandwidth dynamic allocation module to adapt to the traffic characteristics of different scenarios, improving the system's versatility and flexibility.

[0060] In some embodiments, the embedded hardware platform’s Ethernet switch (ETHSW) has at least 8 Tx hardware queues and the Gigabit MAC controller (GMAC) has at least 4 Tx hardware queues. The Tx hardware queues are interconnected through an internal AXI bus and traffic shaping is taken over by hardware scheduling logic.

[0061] It should be noted that the embedded hardware platform's Ethernet switch has at least 8 Tx hardware queues, and the Gigabit MAC controller has at least 4 Tx hardware queues. The two are interconnected through an internal AXI bus. The GMAC's Tx queues are managed by ETHSW's traffic scheduling logic, which achieves automatic queue scheduling through a hardware mapping table. This eliminates the need for the CPU to participate in queue forwarding and scheduling, thus reducing CPU load.

[0062] The design of multiple hardware queues provides sufficient hardware resources to support the classified transmission of different types of network streams, meeting the needs of parallel transmission of multiple network streams. The design of internal AXI bus interconnection and hardware scheduling logic takeover realizes hardware-level acceleration of queue scheduling, significantly reduces CPU load, solves the problem of limited resources on embedded platforms, and improves the real-time performance and efficiency of queue scheduling, providing hardware guarantee for low-latency transmission of multiple network streams.

[0063] A second aspect of the present invention provides a multimedia transmission method based on a time-sensitive network, applied to the aforementioned multimedia control system, comprising the following steps:

[0064] S1: Establish a connection with an external time-sensitive network through the network interface module and configure the network parameters of the embedded hardware platform;

[0065] S2: Identify the network stream type output by the upper-layer application through the stream classification module, and map the network stream to the corresponding Qav queue according to the preset priority;

[0066] S3: Initialize the credit value parameters of each queue through the traffic shaping scheduling module, and implement traffic shaping based on the IEEE802.1Qav protocol;

[0067] S4: The real-time utilization rate of each queue is periodically collected through the bandwidth dynamic allocation module, and the bandwidth allocation parameters of the queue are dynamically adjusted based on the closed-loop control algorithm.

[0068] S5: Through the hardware scheduling logic of the embedded hardware platform, parallel transmission and synchronous control of multiple network streams are achieved, ensuring determinism and low jitter in transmission.

[0069] In specific implementation, step S1 establishes a connection with an external time-sensitive network through the network interface module, configures the network parameters of the embedded hardware platform, and completes the communication link initialization; step S2 identifies network flow types through the flow classification module and maps various network flows to corresponding Qav queues according to preset priorities; step S3 initializes the credit value parameters of each queue through the traffic shaping and scheduling module and starts the traffic shaping mechanism based on the IEEE 802.1 Qav protocol; step S4 periodically collects the real-time utilization rate of the queues through the bandwidth dynamic allocation module, adjusts the bandwidth allocation parameters based on the closed-loop control algorithm, and updates the hardware configuration; step S5 realizes the parallel transmission and synchronous control of multiple network flows through the hardware scheduling logic of the embedded hardware platform, and schedules according to queue priority and credit value status when bandwidth contention occurs.

[0070] The method of this invention ensures the coordinated implementation of various system functions through orderly step-by-step execution, and guarantees the stability and reliability of network transmission by controlling the entire process from link initialization to traffic transmission. The step design is tailored to the characteristics of the hardware platform and protocol requirements, and realizes the organic combination of traffic classification, shaping, and dynamic bandwidth allocation. This not only ensures low-latency transmission of high-priority streams, but also improves link utilization, effectively solving problems such as unpredictable latency and multi-stream interference in traditional transmission methods. It is suitable for various scenarios with high real-time requirements for multimedia transmission.

[0071] To provide a clearer understanding of the invention, the invention is further described below:

[0072] Reference Figure 5 A multimedia control system based on time-sensitive networking includes a network interface module, a stream classification module, a traffic shaping and scheduling module, a bandwidth dynamic allocation module, and an embedded hardware platform. These modules work together to achieve low-latency, high-synchronization transmission of multiple network streams.

[0073] Network interface module: It adopts a gigabit Ethernet interface for accessing external time-sensitive networks and supports the IEEE 802.1Qav protocol and a subset of the TSN protocol. It can establish a stable and high-speed communication connection with external networks and provide physical link support for multimedia data transmission.

[0074] Stream Classification Module: Its core function is to identify different types of network streams output by upper-layer applications, specifically including audio streams, video streams, signal streams, and data streams. This module predefines priority rules, with the priority order being audio stream > video stream > signal stream > data stream. By parsing the identification information and transmission requirements of the network streams, it maps different types of network streams to independent Qav queues, achieving multi-stream classification management and avoiding transmission interference.

[0075] Traffic shaping scheduling module: Employs the Credit Shaping Scheduler (CBS) at the Linux kernel driver layer via the network scheduler (TCqdisc). Module Implementation. This module is based on the IEEE 802.1 Qav protocol and maintains a credit counter for each Qav queue, configuring three core parameters: idlelope (credit growth rate during idle time), hicredit (credit cap), and locredit (credit cap). When there is data to send in the queue, if the credit value is greater than 0, sending is allowed. During sending, the credit value decreases at sendslope (credit decrease rate during sending). When the queue is idle, the credit value increases at idlelope until it reaches the hicredit cap. This mechanism achieves traffic shaping, ensuring low-latency transmission for high-priority queues.

[0076] The bandwidth dynamic allocation module employs a proportional-integral (PI) closed-loop control algorithm to monitor the link utilization and traffic demand of each queue in real time. This module periodically collects statistical data such as byte count and idle time for each queue to calculate the real-time queue utilization. and the preset target utilization rate (Values ​​ranging from 0.6 to 0.9) were compared to obtain the deviation. The idleslope parameter is adjusted based on the deviation value using PI control logic. The control law is as follows:

[0077] ,

[0078] in, This is the proportional gain (which affects the response speed). To improve the integral gain (eliminating steady-state error), the update period is set to 10ms-200ms to achieve adaptive dynamic allocation of bandwidth and improve link utilization.

[0079] Embedded hardware platform: Utilizing the STM32MP257 processor, which integrates a dual-core ARM Cortex-A35 and a real-time coprocessor core Cortex-M33, combining high-performance computing and real-time processing capabilities. The platform runs an embedded Linux system supporting the TSN protocol and dedicated drivers. Its Ethernet switch (ETHSW) has at least 8 Tx hardware queues, and its Gigabit MAC controller (GMAC) has at least 4 Tx hardware queues. Both are interconnected via an internal AXI bus, with hardware scheduling logic managing queue scheduling without consuming CPU resources, effectively reducing system resource consumption and adapting to resource-constrained embedded scenarios.

[0080] A multimedia transmission method based on time-sensitive networking, applied to the aforementioned multimedia control system, comprises the following steps:

[0081] S1: Network Interface Configuration. Establish a connection with an external time-sensitive network through the network interface module. Access the STM32MP257 device terminal and configure the sw0ep interface IP address (e.g., ifconfig sw0ep 192.168.0.10 netmask255.255.255.0) to complete the network parameter initialization.

[0082] S2: Network Stream Classification and Mapping. The stream classification module parses the network stream data output by the upper-layer application in real time, identifies the network stream type (audio stream, video stream, signal stream, data stream), and maps each type of network stream to its corresponding Qav queue according to preset priority rules. Audio streams are mapped to the highest priority queue, and data streams are mapped to the lowest priority queue.

[0083] S3: Traffic Shaping Initialization. Configure the CBS parameters of each QAV queue through the traffic shaping scheduling module, including idleslope, hicredit, locredit, and sendslope. For example, configure idleslope=500000 (corresponding to 500Mbps), hicredit=300, locredit=-1200, and sendslope=-800000 for the audio stream queue, initialize the credit counters of each queue, and start the traffic shaping mechanism.

[0084] S4: Dynamic Bandwidth Adjustment. The bandwidth dynamic allocation module reads the byte count and idle time of each queue from the ETHSW statistics register according to a preset update cycle (e.g., 100ms) to calculate the real-time queue utilization. Calculate deviation based on PI control algorithm It also dynamically adjusts the idleslope parameter and calls the tcqdisc interface to update the hardware configuration, ensuring that the queue utilization remains stable within the target range.

[0085] S5: Multi-stream parallel transmission. The embedded hardware platform's ETHSW and GMAC work together through an internal bus. The hardware scheduling logic schedules data transmission for each queue based on queue priority and credit value status, enabling parallel transmission of audio, video, signal, and data streams. This ensures low-latency, low-jitter transmission of high-priority streams while preventing long-term starvation of low-priority streams, thus improving the overall communication reliability of the system.

[0086] To better understand the technical solution of the present invention, the technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0087] A multimedia control system based on the IEEE 802.1Qav protocol includes:

[0088] The TSN network interface module is used to connect to external time-sensitive networks via a Gigabit Ethernet interface and supports the IEEE 802.1Qav protocol.

[0089] The Credit-Based Shaper (CBS) module is used to calculate the credit value of each priority queue according to the Qav protocol, so as to achieve bandwidth-controllable traffic shaping.

[0090] The multimedia stream classification module is used to identify different types of network streams (audio, video, signal, data) from upper-layer applications and map them to the corresponding Qav queues according to their priority.

[0091] The bandwidth dynamic allocation module is used to dynamically adjust the idleslope parameter of each queue based on link utilization and real-time traffic demand.

[0092] The embedded hardware platform uses an STM32MP257 processor, which integrates a dual-core ARM Cortex-A35 and a real-time coprocessor core Cortex-M33, and runs an embedded Linux system and drivers that support TSN.

[0093] The credit shaping scheduling module is implemented in the kernel driver layer, through the Linux network scheduler (TCqdisc). This module configures parameters for a specified network interface card (NIC) queue: idleslope, hicredit, and locredit.

[0094] The bandwidth dynamic allocation module employs a closed-loop control algorithm, periodically updating the idlelope value based on the actual queue utilization to achieve bandwidth self-adaptation. The closed-loop control algorithm is based on proportional-integral (PI) control, stabilizing the utilization of each queue at a target value. The control variables are as follows:

[0095] Control Laws:

[0096] By calculating the deviation And adjust the idleslope using a PI controller:

[0097] ,

[0098] in, This refers to the proportional gain (response speed). For integral gain (to eliminate steady-state error)

[0099] Algorithm pseudocode implementation:

[0100] / / Update cycle: 100ms

[0101] #define UPDATE_PERIOD_MS 100

[0102] #define KP 0.3f

[0103] #define KI 0.05f

[0104] #define U_TARGET 0.8f

[0105]

[0106] {

[0107] float error = U_TARGET - q->utilization; / / Bandwidth usage deviation of the current queue

[0108] ;

[0109] ;

[0110] / / Limit the range [minimum, maximum]

[0111] if (q->idleslope < q->min_idleslope) q->idleslope = q->min_idleslope;

[0112] if (q->idleslope > q->max_idleslope) q->idleslope = q->max_idleslope;

[0113] / / Call the kernel CBS interface to update hardware parameters

[0114] tc_update_cbs(q->id, q->idleslope);

[0115] }

[0116] Every 100ms, read the byte count and idle time of each Tx queue from the statistics register of ETHSW, and calculate... .

[0117] ETHSW provides hardware statistics registers for the number of bytes, frames sent, and idle clock cycles per Tx Queue; the MPU (A55 core) runs a bandwidth scheduling daemon that periodically reads queue utilization, executes a closed-loop control algorithm, and calls `tc qdisc change dev end1 parent 100:x cbs idleslope`.<new_value> Offload 1 is dynamically updated; the ETHSW hardware CBS module automatically loads new idleslopes, which take effect immediately at the hardware level without interrupting the data flow.

[0118] The multimedia stream classification module performs stream sorting at the receiving end based on priority and stream ID to achieve differentiated transmission.

[0119] This example uses the STM32MP257 development board, which features a Gigabit Ethernet interface supporting Time-Sensitive Networking (TSN) and the IEEE-802.1 Qav protocol. It allows flow reservation for network traffic queues, enhancing traffic shaping through this feature. This example demonstrates how to enhance traffic shaping using an Ethernet switch within a Time-Sensitive Networking (TSN) IEEE-802.1 Qav system, using a multimedia network system as an example. The system block diagram is shown below. Figure 1 As shown.

[0120] The STM32MP257's Ethernet Switch (Ethernet Switch Peripheral) connects to the Ethernet GMAC internal peripheral. The GMAC IP has four Tx hardware queues, and the ETHSW has eight Tx HW queues. These connections are hardware-managed and designed to handle different types of traffic with varying priorities and requirements. Queues are automatically scheduled via a hardware mapping table, implementing IEEE 802.1 Qav traffic shaping. This part of the traffic is automatically forwarded by hardware, without consuming CPU load, and supports the classification of multimedia streams, control streams, and data streams. The internal network topology is as follows: Figure 2 As shown.

[0121] The following example illustrates four reserved bandwidths on an STM32MP257 switch, using four queues (Q1~Q4), each with customizable traffic bandwidth. The network bandwidth queue diagram of this invention is shown below. Figure 3 As shown. The following is the configuration of the four queues:

[0122] Queue 1 (Network Data Stream): No bandwidth limit, uses spare bandwidth.

[0123] Queue 2 (Network Signal Flow): Reserves 10% (100Mbps) of the maximum bandwidth as network signal flow.

[0124] Queue 3 (Network Video Stream): Reserve 30% (300Mbps) of the maximum bandwidth as network video stream.

[0125] Queue 4 (Network Media Stream): Reserve 50% (500Mbps) of the maximum bandwidth as network media stream.

[0126] Queue 4 is the highest priority queue, and queue 1 is the lowest priority queue.

[0127] In the above example, an STM32MP257 is needed as a gigabit Ethernet interface (1000Mbps) on a network switch to achieve traffic shaping.

[0128] Network media streams can be used in video conferencing equipment to play video and audio, with the highest priority and a maximum bandwidth of 500 Mbps;

[0129] Network video streaming can be used to monitor security equipment and play real-time video, with a maximum bandwidth of 300Mbps.

[0130] Network signal streams can be used for remote control of devices to issue and receive commands, with a maximum bandwidth of 100Mbps.

[0131] Network data streams can be used for network packet sending and log uploading, etc. They have low real-time requirements but high usage frequency, the lowest priority, and no bandwidth limit (maximum 1000Mbps). When other queues use bandwidth simultaneously, the maximum bandwidth of the network data stream is 100Mbps, i.e., total bandwidth 1000Mbps - network media stream bandwidth 500Mbps - network video stream bandwidth 300Mbps - network signal stream bandwidth 100Mbps; however, when other queues are not using bandwidth, this queue can reach a maximum bandwidth of 1000Mbps (theoretical value).

[0132] The specific software configuration will be described below.

[0133] Access the STM32MP257 device terminal and configure the sw0ep interface IP address:

[0134] ifconfig sw0ep 192.168.0.10 netmask 255.255.255.0

[0135] Configure multi-queue priority scheduling function:

[0136] tc qdisc add dev end1 clsact

[0137] tc qdisc add dev end1 handle 100: parent root mqprio \

[0138] num_tc 4 \

[0139] map 0 0 1 1 2 2 3 3 3 3 3 3 3 3 3 3 \

[0140]

[0141] These two commands configure multi-queue priority scheduling (mqprio) and flow control hooks (clsact) for the end1 network interface. This is commonly used in TSN (Time-Sensitive Network) configurations, adding a multi-queue priority scheduling (mqprio) root queue rule to the device's end1 Ethernet interface. After execution, the network interface's queue structure is as follows.

[0142]

[0143] The bandwidth allocated to each queue is configured as follows: Queue 2 is allocated 100Mbps, Queue 3 is allocated 300Mbps, and Queue 4 is allocated 500Mbps. Since the total bandwidth is 1000Mbps, it is known that when other queues are using their bandwidth, Queue 1 will receive 100Mbps. The configuration is as follows:

[0144] tc qdisc replace dev end1 parent 100:2 cbs \

[0145] idleslope 100000 \

[0146] sendslope -800000 \

[0147] hicredit 300 \

[0148] locredit -1200 \

[0149] offload 1

[0150] tc qdisc replace dev end1 parent 100:3 cbs \

[0151] idleslope 300000 \

[0152] sendslope -800000 \

[0153] hicredit 300 \

[0154] locredit -1200 \

[0155] offload 1

[0156] tc qdisc replace dev end1 parent 100:4 cbs \

[0157] idleslope 500000 \

[0158] sendslope -800000 \

[0159] hicredit 300 \

[0160] locredit -1200 \

[0161] offload 1

[0162] These three commands configure CBS (Credit-Based Shaper) for different queues in end1, implementing IEEE 802.1Qav bandwidth shaping scheduling—one of the key mechanisms of TSN. On the STM32MP257:

[0163] The GMAC (Gigabit MAC Controller) has 4 Tx hardware queues;

[0164] An ETHSW (Ethernet Switch) can have a maximum of 8 Tx hardware queues internally;

[0165] The two are interconnected via an internal AXI bus, and the GMAC Tx queue can be taken over by the traffic scheduling logic of ETHSW;

[0166] The driver calls the hardware CBS offload function (offload1) through the Linux tc tool (tc qdisc).

[0167] `parent` 100:2 / 100:3 / 100:4 refers to the parent handles of the queues created in the previous `mqprio` configuration. The corresponding relationships are as follows:

[0168] 100:2 → Corresponds to Traffic Class 2 (Queue 2)

[0169] 100:3 → Corresponds to Traffic Class 3 (Queue 3)

[0170] 100:4 → Corresponds to Traffic Class 4 (Queue 4)

[0171] CBS parameter explanation, taking the command set `parent 100:4` as an example:

[0172] CBS working principle (IEEE 802.1Qav):

[0173] Each Traffic Class (TC) maintains a credit counter. When there is data available to send in the queue:

[0174] If credit > 0, then sending is allowed;

[0175] If credit < 0, then pause sending;

[0176] Credit change pattern:

[0177] When sending: credit decreases with sendslope;

[0178] During idle periods: credit increases with idle slope;

[0179] By setting idleslope, hicredit, and locredit, you can ensure that:

[0180] All types of traffic receive a defined bandwidth;

[0181] Prevent low-priority individuals from starving in the long term;

[0182] It meets latency and bandwidth constraints.

[0183] Resource allocation logic for the three CBS queues:

[0184] The total bandwidth allocation is approximately: 100 + 300 + 500 = 900 Mbps

[0185] The remaining approximately 100 Mbps is used as idle / data channel bandwidth for queue 1 to prevent congestion.

[0186] The relevant program flowchart is as follows Figure 4 As shown.

[0187] The technical solutions provided by the embodiments of the present invention have the following beneficial effects:

[0188] Outstanding deterministic transmission capability: Based on the IEEE 802.1Qav protocol, the credit shaping mechanism achieves low-latency (microsecond-level) and low-jitter transmission of multiple network streams through hardware-level traffic shaping and dynamic bandwidth scheduling, solving the problem of uncertainty in traditional Ethernet transmission and meeting the needs of real-time multimedia transmission.

[0189] Precise multi-stream classification management: The stream classification module classifies and maps audio streams, video streams, signal streams, and data streams. Combined with a priority scheduling mechanism, it avoids transmission interference between different types of network streams and ensures the transmission priority of critical business streams (such as audio streams and signal streams).

[0190] High bandwidth utilization: The PI-based closed-loop control algorithm dynamically adjusts the bandwidth parameters of each queue, keeping the link utilization stable within the target range, avoiding bandwidth waste, and preventing a single queue from over-consuming bandwidth, thereby improving the overall link resource utilization.

[0191] Strong adaptability to embedded platforms: Optimized for the STM32MP257 platform, it utilizes hardware queues and scheduling logic to reduce CPU load, solve the problem of limited resources on embedded platforms, balance system performance and real-time performance, and broaden the application scope of TSN technology in embedded scenarios.

[0192] It has a wide range of applications: applicable to multiple scenarios such as vehicle central control, industrial human-machine interface (HMI), remote video monitoring, smart gateway, and video conferencing equipment, with strong versatility and practicality, and broad market application prospects.

[0193] The technical solutions provided by the embodiments of the present invention have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of the embodiments of the present invention. The descriptions of the embodiments above are only for helping to understand the principles of the embodiments of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the embodiments of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A multimedia control system based on time-sensitive networking, characterized in that, include: The network interface module is used to access time-sensitive networks and supports the IEEE 802.1Qav protocol. The stream classification module is used to identify different types of network streams output by upper-layer applications and map the network streams to corresponding queues according to preset priorities. The traffic shaping and scheduling module, based on the credit value mechanism of the IEEE 802.1Qav protocol, performs traffic shaping on the queue to ensure low-latency transmission of high-priority network flows. The bandwidth dynamic allocation module is used to monitor link utilization and traffic demand in real time and dynamically adjust the bandwidth allocation parameters of the queue. An embedded hardware platform runs an embedded system and driver that supports time-sensitive networking, providing a hardware operating environment for the network interface module, flow classification module, traffic shaping and scheduling module, and bandwidth dynamic allocation module, enabling deterministic transmission and synchronous control of multiple network flows.

2. The multimedia control system based on time-sensitive networks according to claim 1, characterized in that: The network interface module is a gigabit Ethernet interface and is compatible with a subset of time-sensitive networking protocols, enabling it to establish stable communication connections with external time-sensitive networks.

3. The multimedia control system based on time-sensitive networks according to claim 1, characterized in that: The traffic shaping and scheduling module is a credit shaping and scheduling module, which operates at the Linux kernel driver layer through the network scheduler. The module is implemented with configuration parameters including idleslope, hicredit, and locredit. It dynamically controls the sending rate of each queue by maintaining a credit counter.

4. The multimedia control system based on time-sensitive networks according to claim 1, characterized in that: The network stream types identified by the stream classification module include audio streams, video streams, signal streams, and data streams, and the preset priority order is: audio stream > video stream > signal stream > data stream. Different types of network streams are mapped to independent Qav queues.

5. The multimedia control system based on time-sensitive networks according to claim 1, characterized in that: The bandwidth dynamic allocation module adopts a closed-loop control algorithm, which is based on proportional-integral control logic. By calculating the deviation between the real-time queue utilization rate and the target utilization rate, the bandwidth allocation parameters are dynamically adjusted to keep the queue utilization rate stable within the target range.

6. The multimedia control system based on time-sensitive networks according to claim 1, characterized in that: The embedded hardware platform is an STM32MP257 processor, which integrates a dual-core ARM Cortex-A35 and a real-time coprocessor core Cortex-M33. The embedded system is an embedded Linux system that supports the TSN protocol.

7. The multimedia control system based on time-sensitive networks according to claim 5, characterized in that: The control law of the closed-loop control algorithm is as follows: ， in, Assign parameters to the current bandwidth of the i-th queue. Let be the deviation between the real-time utilization rate and the target utilization rate of the i-th queue. For proportional gain, This is the integral gain.

8. The multimedia control system based on time-sensitive networks according to claim 5, characterized in that: The update cycle of the bandwidth dynamic allocation module is 10ms-200ms, and the target utilization rate ranges from 0.6 to 0.

9.

9. The multimedia control system based on time-sensitive networks according to claim 1, characterized in that: The embedded hardware platform's Ethernet switch has at least 8 Tx hardware queues, and the gigabit MAC controller has at least 4 Tx hardware queues. The Tx hardware queues are interconnected through an internal AXI bus, and traffic shaping is achieved by hardware scheduling logic.

10. A multimedia transmission method based on time-sensitive networking, characterized in that, The multimedia control system according to any one of claims 1-9 includes the following steps: S1: Establish a connection with an external time-sensitive network through the network interface module and configure the network parameters of the embedded hardware platform; S2: Identify the network stream type output by the upper-layer application through the stream classification module, and map the network stream to the corresponding Qav queue according to the preset priority; S3: Initialize the credit value parameters of each queue through the traffic shaping scheduling module, and implement traffic shaping based on the IEEE802.1Qav protocol; S4: The real-time utilization rate of each queue is periodically collected through the bandwidth dynamic allocation module, and the bandwidth allocation parameters of the queue are dynamically adjusted based on the closed-loop control algorithm. S5: Through the hardware scheduling logic of the embedded hardware platform, parallel transmission and synchronous control of multiple network streams are achieved, ensuring determinism and low jitter in transmission.

Images