RTSP video stream AI processing method and system based on tsn network

Abstract

The invention discloses an RTSP video stream AI processing method and system based on a tsn network, which realize end-to-end deterministic transmission and processing, guarantee low-delay and low-jitter transmission of a video stream from a source to an AI unit through a TSN, meet the industrial-grade real-time requirement and perform efficient edge AI processing. The problem that a traditional CPU is low in efficiency or a GPU is high in power consumption is solved, real-time analysis of the high energy efficiency ratio and high system integration are achieved, network transmission, video decoding, AI reasoning and result visualization are integrated into a single embedded platform, and hardware complexity and cost are reduced.

Description

Technical Field

[0001] This invention relates to the field of video processing, and in particular to an AI processing method and system for RTSP video streams based on TSN networks. Background Technology

[0002] With the rapid development of Industry 4.0, intelligent transportation, telemedicine, and other fields, higher demands are being placed on the remote intelligent analysis and control of real-time video streams. Currently, such systems mainly rely on traditional Ethernet or general-purpose networks to transmit video streams and use AI models for behavior recognition or event detection. A common implementation method is to pull video streams from network cameras (IPCs) or network video recorders (NVRs) through protocols such as RTSP or RTMP, decode them, and then send them to the AI ​​model for processing.

[0003] Traditional video AI processing solutions: Currently, video-based AI analysis (such as security monitoring and industrial quality inspection) typically employs two architectures:

[0004] Centralized computing architecture: This architecture transmits all video streams from cameras located in various locations to a central server or cloud platform for centralized AI processing. This architecture has high network bandwidth requirements, and suffers from large transmission latency and unpredictable jitter, making it unsuitable for scenarios with high real-time requirements, such as industrial control.

[0005] Traditional edge computing architecture: Deploying industrial control computers / servers using general-purpose CPUs or GPUs near the device. CPUs are inefficient at processing AI models and struggle to perform real-time analysis of multiple high-definition videos; while GPUs offer strong computing power, they are power-hungry, expensive, and lack industrial-grade real-time network interfaces.

[0006] Network transmission bottleneck: Standard Ethernet uses a "best-effort" transmission mechanism, which cannot guarantee data transmission latency and jitter. When video streams share the network with control and management streams, network congestion can lead to video stuttering and delayed AI analysis results, making the entire system lack "determinism".

[0007] Time synchronization issues: In scenarios where multiple cameras work together (such as multi-angle positioning and tracking), time asynchrony between cameras can lead to difficulties in data fusion and make it hard to build an accurate panoramic view.

[0008] Therefore, the existing technology has the following inherent defects:

[0009] Transmission uncertainties: Traditional Ethernet or ordinary IP networks suffer from issues such as packet contention, latency jitter, and packet loss. When the network load is high, video streams may experience stuttering, screen tearing, and increased latency, severely impacting the accuracy and real-time performance of backend AI analysis. For example, in autonomous driving scenarios, delays in keyframes may prevent vehicles from recognizing obstacles in a timely manner.

[0010] The computation and transmission are disconnected: the video streaming network and AI computing resources are independent systems. AI computing nodes are unaware of network status, and the network cannot provide differentiated services based on the criticality of AI tasks. When the system needs to process multiple video streams with different priorities (such as routine inspections and intrusion alarms in security monitoring), resources cannot be guaranteed for high-priority tasks.

[0011] Inefficient resource utilization: The system typically uses a fixed video bitrate and AI model for computation. When network conditions are good, it fails to utilize high bandwidth to transmit higher quality video to improve analysis accuracy; when the network is congested, it cannot adaptively reduce the bitrate or switch to a lightweight model to ensure the continuity of analysis.

[0012] In summary, there is a need for an AI processing method and system for RTSP video streams based on TSN networks to address the shortcomings of existing technologies. Summary of the Invention

[0013] To address the shortcomings of existing technologies, this invention provides an AI processing method and system for RTSP video streams based on TSN networks, aiming to solve the aforementioned problems.

[0014] To achieve the above objectives, the present invention provides the following technical solution: an AI processing method for RTSP video streams based on a TSN network, comprising the following steps:

[0015] Step S1: Establish a time base. On an embedded processor platform containing a native TSN Ethernet controller and a neural network processing unit (NPU), synchronize time with the switches in the TSN network using the General Precision Time Protocol (gPTP) to establish a unified time base.

[0016] Step S2: Reserve transmission time slots, register the RTSP video stream to be processed with the centralized network configuration node of the TSN network, the centralized network configuration node calculates the required bandwidth according to the registration information, and after the admission control is passed, generates a global gating list and a global gating list scheduling table based on the time-aware shaper algorithm, and sends the global gating list scheduling table and the stream filtering rules to the TSN switch and the TSN Ethernet controller of the embedded processor platform;

[0017] Step S3: Tagging and forwarding data packets. The embedded processor platform starts the RTSP client, actively connects to the video source to establish a session, so that the RTP data packets sent by the video source are tagged with VLAN priority according to the flow filtering rules as soon as they enter the TSN network, and are forwarded by the TSN switch in a non-contention manner within a dedicated time window in strict accordance with the global gating list scheduling table.

[0018] Step S4: Generate image data. Reassemble and hardware-accelerated decode the received RTP video data packets to obtain the original video frames. Perform scaling and normalization preprocessing on the original video frames to generate image data that meets the NPU input requirements.

[0019] Step S5: Analyze the image data, send the image data to the NPU built into the embedded processor platform, execute the loaded AI model to perform inference calculations, obtain the original inference results, perform confidence filtering and non-maximum suppression (NMS) post-processing on the original inference results to generate structured AI analysis results, and fuse the AI ​​analysis results with the original video frames to form video frames with analysis results.

[0020] Optionally, step S1 is implemented in the following manner:

[0021] An embedded processor with a native TSN Ethernet controller and a neural network processing unit (NPU) is selected. The embedded processor is used as an end node to run a time synchronization protocol. The system measures and calculates the link transmission delay hop by hop through message pairs, and incorporates the dwell time of intermediate nodes such as switches into the compensation. The slave node continuously monitors the frequency deviation between the master clock and the slave clock, and dynamically adjusts the local clock frequency.

[0022] Optionally, step S2 is implemented in the following manner:

[0023] Step A1: Terminal Node Registration and Stream Requirement Declaration. After the system starts, the TSN configuration agent module actively initiates a registration request to the centralized network configurator in the network through the interface. The request carries the complete QoS requirement information of the RTSP video stream to be processed.

[0024] Step A2: Resource calculation and admission control. The centralized network configurator receives the video stream request, calculates the bandwidth required for the video stream, queries the entire network topology to find the optimal path from source to destination, traverses all switches and links on the path, checks whether the bandwidth and time slot resources are sufficient, and makes a decision to grant admission or return a rejection to the terminal node.

[0025] Step A3: Generate and distribute scheduling and filtering rules. After admission is approved, the centralized network configurator plans a dedicated time window for the video stream on each hop of the transmission path, generates a global gating list containing periodic time slots and queue on / off states, generates stream identification rules, and distributes the global gating list and stream filtering rules to the TSN switches and the TSN controller of the embedded processor platform on the path through the interface. The switches write the rules into the forwarding engine, and the processor applies them to the egress queue scheduling.

[0026] Optionally, the bandwidth required for the video stream is calculated in step A2 in the following way:

[0027] B_required=(F_max*8) / T+Overhead_TSN,

[0028] In the formula, B_required is the bandwidth required for the stream, F_max is the maximum number of bytes in a video stream I-frame, T is the frame period, and Overhead_TSN is the header overhead that includes Ethernet, VLAN, IP, UDP, RTP, and TSN serialization.

[0029] Optionally, step S3 is implemented in the following manner:

[0030] Step B1: Start the RTSP client and establish a session. The application calls the lightweight RTSP client library, initiates a TCP connection to the video source with the specified IP and port, and sends OPTIONS, DESCRIBE, SETUP and PLAY requests in sequence to complete the session negotiation and successfully establish a complete RTSP session including SDP media description and RTP transport port.

[0031] Step B2: The video source sends an RTP stream and enters the TSN network. After receiving the PLAY command, the video source begins to encapsulate the encoded video frames into RTP data packets via UDP and sends them out from its network interface into the TSN network.

[0032] Step B3: Identify, mark, and schedule the forwarding of RTP flows. The ingress node identifies and marks the flow with a high-priority VLAN tag according to the flow filtering rules issued by the centralized network configurator. At the egress port of each hop switch, the time-aware shaper module forwards the flow within a dedicated, contention-free time window according to the global gating list scheduling table.

[0033] Optionally, step S4 is implemented in the following manner:

[0034] Step C1: Receive RTP packets from the network interface, sort and buffer out-of-order packets according to the sequence number and timestamp in the RTP header, identify fragmented units or aggregated packets in the RTP payload, cache multiple RTP fragments belonging to the same Network Abstraction Layer Unit (NALU) for fragmented units and reassemble them into a complete NALU; for a single NALU or aggregated packet, directly extract the payload.

[0035] Step C2: The reconstructed complete video frame data is sent to the dedicated video codec hardware unit of the embedded platform. The decoder parameters are configured through the standard API and the data is submitted. The decoder uses its internal hard-wired logic to efficiently complete the decoding.

[0036] Step C3: Image preprocessing. The application processor core or dedicated image signal processor performs preprocessing on the decoded raw video frames to generate a preprocessed image data block that conforms to the NPU tensor input specification.

[0037] Optionally, step S5 is implemented in the following manner:

[0038] Step D1: On the embedded processor platform, after loading the quantized model and initializing the runtime environment through the NPU inference SDK, the preprocessed image data is sent to the NPU to perform hardware-accelerated forward inference, and finally the original inference result tensor is obtained.

[0039] Step D2: Filter the predicted bounding boxes based on the product of object confidence and class probability. Perform non-maximum suppression (NMS) based on cross-union ratio on the retained high-scoring boxes to remove overlapping and redundant detection results, and output a set of structured target detection boxes.

[0040] Step D3: Use a computer vision library to overlay the AI ​​analysis results onto the original decoded video frames to generate a visual video frame that combines scene information and intelligent analysis content.

[0041] Optionally, the method further includes uploading the structured AI analysis results in the form of MQTT messages through a second TSN Ethernet interface of the embedded processor platform, and re-encapsulating the video frames with the analysis results into RTSP video streams for distribution. The first TSN Ethernet interface is dedicated to receiving RTSP video streams, and the second TSN Ethernet interface is dedicated to system control, result uploading, and network management.

[0042] An RTSP video stream AI processing system based on a TSN network, employing the aforementioned RTSP video stream AI processing method based on a TSN network, includes a time synchronization and reference establishment module, a TSN resource scheduling and admission control module, an RTSP session and RTP stream access module, an RTP reconstruction and hardware decoding module, an image preprocessing module, an NPU AI inference and post-processing module, a result fusion and visualization module, and a dual-port isolated communication module.

[0043] The time synchronization and reference establishment module is used to synchronize time with TSN switches via the gPTP protocol, compensate for link delay and dwell time, and achieve microsecond-level time alignment across the entire network.

[0044] The TSN resource scheduling and admission control module is used by the centralized network configurator to complete flow registration, bandwidth calculation, path planning, generate a global gating list and flow filtering rules, and distribute them to switches and terminals.

[0045] The RTSP session and RTP stream access module starts a lightweight RTSP client, completes the interaction, and establishes a session; after the video source sends out the RTP stream, the ingress switch adds a VLAN priority tag according to the stream filtering rules.

[0046] The RTP reassembly and hardware decoding module is used to receive RTP packets, reorder them by sequence number / timestamp, reassemble fragmented NALUs, and send them to dedicated video encoding and decoding hardware.

[0047] The image preprocessing module performs scaling, normalization, and other operations on the original frame to generate data blocks that conform to the NPU tensor input format;

[0048] The NPU AI inference and post-processing module is used to load quantized models, perform forward inference on the NPU, and perform confidence filtering and non-maximum suppression on the output.

[0049] The results fusion and visualization module is used to overlay structured AI results onto the original video frames to generate labeled visualizations.

[0050] The dual-port isolated communication module uses two independent TSN Ethernet interfaces. One interface is dedicated to receiving RTSP video streams, and the other interface is used to upload MQTT analysis results, system management, and control commands.

[0051] The beneficial effects of this invention are:

[0052] 1. In this invention, the method achieves end-to-end deterministic transmission and processing, and ensures low-latency and low-jitter transmission of video stream from source to AI unit through TSN, meeting industrial-grade real-time requirements and efficient edge AI processing: it uses the built-in NPU of the embedded platform for inference, avoiding the problems of low efficiency of traditional CPU or high power consumption of GPU, and achieving high energy efficiency in real-time analysis. The system is highly integrated, integrating network transmission, video decoding, AI inference, and result visualization into a single embedded platform, reducing hardware complexity and cost.

[0053] 2. In this invention, end-to-end determinism is achieved. From the moment the camera video data enters the TSN network to the moment the AI ​​video stream is output, the entire process is protected by the TSN, realizing predictable and extremely low end-to-end latency, meeting the needs of industrial real-time applications. It is highly efficient and low-power, making full use of the STM32MP257's 1.35 TOPS NPU for inference, with performance far exceeding that of pure CPU solutions, while power consumption and cost are far lower than those of GPU solutions, making it very suitable for edge deployment.

[0054] 3. In this invention, the high degree of system integration integrates video acquisition, AI processing, and streaming media services into a single chip, simplifying system design and reducing hardware costs and size. Precise collaborative analysis capabilities and gPTP-based full-network synchronization enable frame-level time alignment of multiple video sources, providing a technical foundation for cross-camera target tracking and 3D positioning. Flexible deployment options allow the system to function as an independent edge AI analysis node or as an intelligent processing unit within the entire TSN network, offering a flexible and easily expandable architecture. Attached Figure Description

[0055] Figure 1 This is a schematic diagram of a method flow of the present invention.

[0056] Figure 2 This is a schematic diagram of a video processing flow according to the present invention.

[0057] Figure 3 This is a schematic diagram of a hardware platform architecture according to the present invention.

[0058] Figure 4 This is a schematic diagram of a Runjian system architecture according to the present invention. Detailed Implementation

[0059] To more clearly illustrate the technical solutions in the embodiments of the invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0060] like Figures 1 to 2 As shown, an AI processing method for RTSP video streams based on TSN networks includes the following:

[0061] System initialization and TSN stream registration configuration: This step is performed when the system starts up and aims to establish a deterministic network transmission channel.

[0062] TSN Network Synchronization: The TSN switches in the system and the STM32MP257 platform with integrated TSN MAC achieve precise time synchronization through the gPTP (IEEE 802.1AS) protocol, establishing a unified sub-microsecond time reference for all devices.

[0063] Video stream resource reservation: The user or upper layer application sends a request to the TSN network controller (CNC) to declare the RTSP video stream information to be transmitted (such as source / destination IP, port, maximum frame rate, maximum frame size).

[0064] TSN Stream Parameter Calculation and Scheduling Table Generation:

[0065] The CNC calculates the bandwidth resources required for the video stream based on the request.

[0066] Formula: B_required = (F_max * 8) / T + Overhead_TSN

[0067] B_required: Bandwidth required for the stream (bps)

[0068] F_max: Maximum number of bytes per I-frame in the video stream, taking into account encoding format and resolution.

[0069] T: Frame period (seconds), e.g., 30fps corresponds to T=1 / 30.

[0070] Overhead_TSN: Includes header overhead (bits) for Ethernet, VLAN, IP, UDP, RTP, and TSN serialization.

[0071] The CNC performs admission control; if the available bandwidth of the path, B_available, is greater than or equal to B_required, then the flow is accepted.

[0072] The CNC uses a Time-Aware Shaper (TAS, 802.1Qbv) algorithm to generate a Globally Scheduled Gated List (GCL) for all accepted critical streams (including this video stream). This list defines the schedule for each port of the switch to open / close a specific priority queue within a specific time window.

[0073] Configuration Distribution: The CNC distributes the GCL scheduling table and flow filtering rules (such as those based on VLAN PCP or destination MAC) to the TSN switch and the STM32MP257 TSN Ethernet controller via protocols such as NETCONF / YANG. At this point, the network has reserved a dedicated, periodic transmission time window for the specified RTSP flow.

[0074] The RTSP video stream is pulled and transmitted via TSN, marking the beginning of continuous real-time video processing.

[0075] RTSP Session Establishment: The RTSP client running on the application core (Cortex-A35) of the STM32MP257 initiates a connection to the video source, establishes a session through standard RTSP commands such as DESCRIBE, SETUP, and PLAY, and negotiates transmission parameters (such as RTP over UDP).

[0076] Tagged video data reception: When RTP video packets sent by the video source enter the TSN network, the edge switch or the source itself assigns a specific VLAN priority tag (PCP) to them according to the S1 configuration. This tag corresponds to a high-priority queue (such as queue 7) in the TSN scheduling.

[0077] Deterministic network forwarding: TSN switches strictly adhere to the GCL (Good Cloud Computing) scheduling table. Within the time window allocated to the video stream, the switch opens the door to its high-priority queue, allowing video packets to pass without contention; in other time windows, this door is closed to ensure the video stream is not blocked by other background traffic. This achieves bounded low latency and extremely low jitter transmission.

[0078] Deterministic reception: The STM32MP257's TSN Ethernet controller receives video packets within the expected time window and the driver directly feeds them into a designated buffer in memory in preparation for decoding.

[0079] Video decoding and frame preprocessing are performed on the Cortex-A35 core of the STM32MP257.

[0080] RTP Reassembly and Decoding: The application calls libraries such as FFmpeg or GStreamer to reassemble and correct the received RTP packets, and uses hardware-accelerated decoders (such as built-in GPUs or video codec units) to decode the H.264 / H.265 bitstream into the original YUV or RGB frame images in real time.

[0081] Frame buffer management: Decoded frames are placed into a multi-stage pipelined buffer. This buffer is designed to decouple the network receive / decode rate from the NPU inference rate, while ensuring that the NPU can obtain the latest or complete frames for processing.

[0082] AI model preprocessing: Retrieve a frame from the buffer and perform the preprocessing operations required by the model:

[0083] Scaling: Scales the image to the NPU model input size (e.g., 640x640).

[0084] Normalization: Normalizes pixel values.

[0085] Formula: I_norm = (I - mean) / std

[0086] I: Input image pixel matrix.

[0087] mean, std: The mean and standard deviation used during model training.

[0088] Color space conversion (if needed): YUV to RGB.

[0089] The processed image data is prepared to be transferred to the NPU via DMA or shared memory.

[0090] This step is the core AI processing stage of NPU-based AI model inference, executed by the 1.35 TOPSNPU of the STM32MP257.

[0091] Model Loading and Scheduling: Upon system startup, neural network models optimized and converted using the STM32Cube.AI toolchain (such as YOLOv5-tiny for defect detection) are loaded into the NPU's dedicated memory. Applications schedule inference tasks via the NPU-driven API.

[0092] High-efficiency inference computation: The NPU performs forward propagation computations of convolution, pooling, activation, and other layers of the model with extremely high energy efficiency. This is the most computationally intensive part of the process, and the parallel architecture of the NPU is optimized for this purpose.

[0093] Inference result acquisition: After inference is completed, the NPU outputs the original result tensor (such as bounding box coordinates, class confidence, class label) and notifies the A35 core through interruption or polling.

[0094] Post-processing, feedback, and flow control are all performed on the Cortex-A35 core, forming a closed-loop processing mechanism.

[0095] Post-processing of results:

[0096] Confidence filtering: Discard predicted boxes with a confidence level lower than the preset threshold θ_conf (e.g., 0.5).

[0097] Non-maximum suppression (NMS): solves the problem of the same target being detected by multiple bounding boxes.

[0098] Key calculation (Intersection over Union (IoU)):

[0099] Formula: IoU(A, B) = Area(A ∩ B) / Area(A ∪ B)

[0100] A, B: Two predicted bounding boxes.

[0101] ∩: Area of ​​intersection.

[0102] ∪: Area of ​​the union.

[0103] NMS algorithm: Select the box with the highest confidence, remove all other boxes with IoU > θ_nms (e.g., 0.45), and iterate until all boxes have been processed.

[0104] Results fusion and output: The filtered structured results (such as [x, y, w, h, class_id,confidence]) are fused with the original video frames (bounding boxes and labels are drawn) to generate video frames with analysis results.

[0105] Dual TSN Ethernet service offloading design:

[0106] TSN Interface 1: Dedicated to RTSP video streaming transmission, configured with strict QoS guarantees.

[0107] The specific configuration is as follows:

[0108] tc qdisc add dev end0 handle 100 parent root mqprio num_tc 2 map 0 00 0 1 0 0 0 queues 1@0 1@1 hw 0

[0109] Functionality: Creates an MQPRIO queuing rule as the first-level classifier for traffic.

[0110] Key Parameter Explanation:

[0111] num_tc 2: Two traffic categories were created.

[0112] map 0 0 0 0 1 0 0 0: Maps the IEEE 802.1p priorities (8 in total, 0-7) of network packets to traffic categories. Here, priorities 0-3 are mapped to TC0, and priority 4 is mapped to TC1.

[0113] queues 1@0 1@1: Allocate one dedicated queue for TC0 and TC1 respectively.

[0114] hw 0: Hardware offloading is not enabled; classification and scheduling are handled by CPU software.

[0115] tc qdisc replace dev end0 parent 100:2 cbs idleslope 1000000sendslope -9000000 hicredit 150 locredit -1350 offload 1

[0116] Function: A credit shaper is configured on the queue corresponding to TC1, and the shaping work is offloaded to the network card hardware for execution.

[0117] Key Parameter Explanation:

[0118] `idleslope 1000000`: This is a key parameter. It represents the long-term average sending rate (1000 Mbps ≈ 1 Gbps) allowed when there is data available to send in the queue. This sets the guaranteed bandwidth for this type of traffic.

[0119] Offload 1: This is the core of achieving low latency and precise control. It offloads complex CBS calculations and packet scheduling to the network card's dedicated hardware, greatly reducing the CPU load and ensuring accurate timing.

[0120] tc qdisc add dev end0 parent root cbs idleslope 98688 sendslope -901312 hicredit 153 locredit -1389

[0121] Function: Another CBS shaper is configured under the root queue.

[0122] Deeper meaning: This is typically used for hierarchical shaping. The root CBS can apply a final total rate limit to all aggregated traffic coming from MQPRIO.

[0123] TSN Interface 2: Used for system control, AI result uploading, and network management.

[0124] Dual network port redundancy design improves system reliability.

[0125] like Figure 3 and Figure 4 As shown, an RTSP video stream AI processing system based on a TSN network, employing the aforementioned RTSP video stream AI processing method based on a TSN network, includes the following:

[0126] Time-Sensitive Networking (TSN): A set of IEEE 802.1 protocols based on standard Ethernet, which provide bounded low-latency, low-jitter, and zero-congestion-loss transmission guarantees for critical data streams through mechanisms such as time synchronization (gPTP, 802.1AS), flow scheduling (TAS, 802.1Qbv), and frame preemption (802.1Qbu). It is a core network technology in fields such as industrial automation and automotive.

[0127] The STM32MP25 processor: A high-performance microprocessor from STMicroelectronics (ST), integrating: dual-core Arm® Cortex®-A35 cores for running complex applications (such as the Linux operating system); a 1.35 TOPS NPU (Neural Processing Unit) designed for edge AI inference, offering significantly higher energy efficiency than CPUs and GPUs; and a native TSN Ethernet controller that directly supports gPTP and other TSN protocols, providing the hardware foundation for building deterministic networks.

[0128] A three-tier architecture consisting of a TSN network transport layer, an STM32MP257 core processing layer, and an AI inference optimization layer is constructed. The TSN network prioritizes and schedules RTSP video streams, ensuring low-latency and low-jitter transmission of video data to the STM32MP257 processor. The dual-core Cortex-A35 of the STM32MP257 handles RTSP protocol parsing and video decoding, while its built-in 1.35 TOPS NPU executes AI inference tasks. Model optimization is achieved through the STM32Cube.AI toolchain, ultimately enabling real-time reception, efficient decoding, and rapid AI analysis of RTSP video streams.

[0129] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions or improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for AI processing of RTSP video streams based on TSN networks, characterized in that, Includes the following steps: Step S1: Establish a time base. On an embedded processor platform containing a native TSN Ethernet controller and a neural network processing unit (NPU), synchronize time with the switches in the TSN network using the General Precision Time Protocol (gPTP) to establish a unified time base. Step S2: Reserve transmission time slots, register the RTSP video stream to be processed with the centralized network configuration node of the TSN network, the centralized network configuration node calculates the required bandwidth according to the registration information, and after the admission control is passed, generates a global gating list and a global gating list scheduling table based on the time-aware shaper algorithm, and sends the global gating list scheduling table and the stream filtering rules to the TSN switch and the TSN Ethernet controller of the embedded processor platform; Step S3: Tagging and forwarding data packets. The embedded processor platform starts the RTSP client, actively connects to the video source to establish a session, so that the RTP data packets sent by the video source are tagged with VLAN priority according to the flow filtering rules as soon as they enter the TSN network, and are forwarded by the TSN switch in a non-contention manner within a dedicated time window in strict accordance with the global gating list scheduling table. Step S4: Generate image data. Reassemble and hardware-accelerated decode the received RTP video data packets to obtain the original video frames. Perform scaling and normalization preprocessing on the original video frames to generate image data that meets the NPU input requirements. Step S5: Analyze the image data, send the image data to the NPU built into the embedded processor platform, execute the loaded AI model to perform inference calculations, obtain the original inference results, perform confidence filtering and non-maximum suppression (NMS) post-processing on the original inference results to generate structured AI analysis results, and fuse the AI ​​analysis results with the original video frames to form video frames with analysis results.

2. The AI ​​processing method for RTSP video streams based on TSN networks according to claim 1, characterized in that, Step S1 is implemented in the following manner: An embedded processor with a native TSN Ethernet controller and a neural network processing unit (NPU) is selected. The embedded processor is used as an end node to run a time synchronization protocol. The system measures and calculates the link transmission delay hop by hop through message pairs, and incorporates the dwell time of intermediate nodes such as switches into the compensation. The slave node continuously monitors the frequency deviation between the master clock and the slave clock, and dynamically adjusts the local clock frequency.

3. The AI ​​processing method for RTSP video streams based on TSN networks according to claim 2, characterized in that, Step S2 is implemented in the following manner: Step A1: Terminal Node Registration and Stream Requirement Declaration. After the system starts, the TSN configuration agent module actively initiates a registration request to the centralized network configurator in the network through the interface. The request carries the complete QoS requirement information of the RTSP video stream to be processed. Step A2: Resource calculation and admission control. The centralized network configurator receives the video stream request, calculates the bandwidth required for the video stream, queries the entire network topology to find the optimal path from source to destination, traverses all switches and links on the path, checks whether the bandwidth and time slot resources are sufficient, and makes a decision to grant admission or return a rejection to the terminal node. Step A3: Generate and distribute scheduling and filtering rules. After admission is approved, the centralized network configurator plans a dedicated time window for the video stream on each hop of the transmission path, generates a global gating list containing periodic time slots and queue on / off states, generates stream identification rules, and distributes the global gating list and stream filtering rules to the TSN switches and the TSN controller of the embedded processor platform on the path through the interface. The switches write the rules into the forwarding engine, and the processor applies them to the egress queue scheduling.

4. The AI ​​processing method for RTSP video streams based on TSN networks according to claim 3, characterized in that, In step A2, the bandwidth required for the video stream is calculated using the following method: B_required=(F_max*8) / T+Overhead_TSN, In the formula, B_required is the bandwidth required for the stream, F_max is the maximum number of bytes in a video stream I-frame, T is the frame period, and Overhead_TSN is the header overhead that includes Ethernet, VLAN, IP, UDP, RTP, and TSN serialization.

5. The AI ​​processing method for RTSP video streams based on TSN networks according to claim 1, characterized in that, Step S3 is implemented in the following manner: Step B1: Start the RTSP client and establish a session. The application calls the lightweight RTSP client library, initiates a TCP connection to the video source with the specified IP and port, and sends OPTIONS, DESCRIBE, SETUP and PLAY requests in sequence to complete the session negotiation and successfully establish a complete RTSP session including SDP media description and RTP transport port. Step B2: The video source sends an RTP stream and enters the TSN network. After receiving the PLAY command, the video source begins to encapsulate the encoded video frames into RTP data packets via UDP and sends them out from its network interface into the TSN network. Step B3: Identify, mark, and schedule the forwarding of RTP flows. The ingress node identifies and marks the flow with a high-priority VLAN tag according to the flow filtering rules issued by the centralized network configurator. At the egress port of each hop switch, the time-aware shaper module forwards the flow within a dedicated, contention-free time window according to the global gating list scheduling table.

6. The AI ​​processing method for RTSP video streams based on TSN networks according to claim 1, characterized in that, Step S4 is implemented in the following manner: Step C1: Receive RTP packets from the network interface, sort and buffer out-of-order packets according to the sequence number and timestamp in the RTP header, identify fragmented units or aggregated packets in the RTP payload, buffer multiple RTP fragments belonging to the same Network Abstraction Layer Unit (NALU) and reassemble them into a complete NALU; for a single NALU or aggregated packet, directly extract the payload. Step C2: The reconstructed complete video frame data is sent to the dedicated video codec hardware unit of the embedded platform. The decoder parameters are configured through the standard API and the data is submitted. The decoder uses its internal hard-wired logic to efficiently complete the decoding. Step C3: Image preprocessing. The application processor core or dedicated image signal processor performs preprocessing on the decoded raw video frames to generate a preprocessed image data block that conforms to the NPU tensor input specification.

7. The AI ​​processing method for RTSP video streams based on TSN networks according to claim 1, characterized in that, Step S5 is implemented in the following manner: Step D1: On the embedded processor platform, after loading the quantized model and initializing the runtime environment through the NPU inference SDK, the preprocessed image data is sent to the NPU to perform hardware-accelerated forward inference, and finally the original inference result tensor is obtained. Step D2: Filter the predicted bounding boxes based on the product of object confidence and class probability. Perform non-maximum suppression (NMS) based on cross-union ratio on the retained high-scoring boxes to remove overlapping and redundant detection results, and output a set of structured target detection boxes. Step D3: Use a computer vision library to overlay the AI ​​analysis results onto the original decoded video frames to generate a visual video frame that combines scene information and intelligent analysis content.

8. The AI ​​processing method for RTSP video streams based on TSN networks according to claim 1, characterized in that, The method also includes uploading structured AI analysis results in the form of MQTT messages through a second TSN Ethernet interface of the embedded processor platform, and re-encapsulating video frames with analysis results into RTSP video streams for distribution. The first TSN Ethernet interface is dedicated to receiving RTSP video streams, and the second TSN Ethernet interface is dedicated to system control, result uploading, and network management.

9. An RTSP video stream AI processing system based on a TSN network, employing the RTSP video stream AI processing method based on a TSN network as described in any one of claims 1-8, characterized in that, It includes a time synchronization and baseline establishment module, a TSN resource scheduling and admission control module, an RTSP session and RTP stream access module, an RTP reconstruction and hardware decoding module, an image preprocessing module, an NPU AI inference and post-processing module, a result fusion and visualization module, and a dual-port isolated communication module. The time synchronization and reference establishment module is used to synchronize time with TSN switches via the gPTP protocol, compensate for link delay and dwell time, and achieve microsecond-level time alignment across the entire network. The TSN resource scheduling and admission control module is used by the centralized network configurator to complete flow registration, bandwidth calculation, path planning, generate a global gating list and flow filtering rules, and distribute them to switches and terminals. The RTSP session and RTP stream access module starts a lightweight RTSP client, completes the interaction, and establishes a session; after the video source sends out the RTP stream, the ingress switch adds a VLAN priority tag according to the stream filtering rules. The RTP reassembly and hardware decoding module is used to receive RTP packets, reorder them by sequence number / timestamp, reassemble fragmented NALUs, and send them to dedicated video encoding and decoding hardware. The image preprocessing module performs scaling, normalization, and other operations on the original frame to generate data blocks that conform to the NPU tensor input format; The NPU AI inference and post-processing module is used to load quantized models, perform forward inference on the NPU, and perform confidence filtering and non-maximum suppression on the output. The results fusion and visualization module is used to overlay structured AI results onto the original video frames to generate labeled visualizations. The dual-port isolated communication module uses two independent TSN Ethernet interfaces. One interface is dedicated to receiving RTSP video streams, and the other interface is used to upload MQTT analysis results, system management, and control commands.

Images