RTSP video streaming playback and recording method based on ad hoc networks

Abstract

This application relates to a method for playing and recording RTSP video streams based on a self-organizing network. The method includes: constructing a layered multi-instance system architecture comprising a self-organizing network environment layer, a main control layer, a playback unit layer, and a playback unit internal architecture layer. For each playback unit, the main control thread initializes the aforementioned processes and threads and establishes communication connections. The streaming process uses FFmpeg to detect the network, obtain stream parameters, establish connections, stream decoding, and cache corresponding frames in a buffer. The rendering process uses OpenCV to extract frames, process them, and render them, while an event listening thread forwards interactive events. Upon receiving a recording command, the recording process extracts frames, processes them, encodes them, and saves them to a file. During video processing, a dynamic buffer adjustment mechanism adjusts the rendering rate based on queue depth and the target queue depth based on network jitter. When the self-organizing network state changes, an adaptive network strategy ensures stable playback and recording. This method enables low-latency, high-stability, and multi-functional processing of RTSP video streams in a self-organizing network environment.

Description

Technical Field

[0001] This application relates to the field of video stream processing technology, and in particular to a method for playing and recording RTSP video streams based on ad hoc networks. Background Technology

[0002] Wireless ad hoc networks are a network architecture that does not rely on fixed infrastructure and adopts distributed management. Their nodes possess self-discovery, self-organization, and self-repair capabilities, playing an irreplaceable role in scenarios such as emergency rescue and field exploration. In these scenarios, it is typically necessary to collect and monitor video data from multiple camera nodes within the ad hoc network in real time. RTSP (Real-Time Streaming Protocol), as an application-layer protocol specifically designed to control streaming media transmission, becomes the core protocol connecting camera nodes and terminal devices.

[0003] In ad hoc network applications, it is often necessary to monitor multiple camera videos simultaneously, and to perform real-time playback, recording, and screenshotting. Due to the unique characteristics of ad hoc networks—limited bandwidth, dynamic changes in network topology caused by the movement of ad hoc network devices, and frequent route switching—link bandwidth and stability are affected by topology changes and route switching. Common packet loss and reconnection phenomena make it difficult to simultaneously achieve low-latency playback and smooth video playback. Traditional linear caching strategies are prone to stuttering with small buffers and excessive latency with large buffers. Traditional video streaming solutions face many challenges: high latency, frequent stuttering, conflicts between recording and playback functions, and poor multi-channel concurrent processing capabilities. Mainstream players such as VLC and FFPlay generally set a fixed buffer of 2-5 seconds to ensure smoothness, resulting in video latency far exceeding the requirements of real-time monitoring scenarios (which typically require latency ≤1 second), and failing to meet the time-sensitive applications such as emergency command and real-time reconnaissance. Traditional solutions employ fixed buffering strategies, which cannot cope with the dynamic network fluctuations of self-organizing networks. When routes switch or bandwidth suddenly drops, they are prone to buffer exhaustion leading to stuttering, or buffer backlog leading to accumulated latency. Existing solutions mostly adopt a single-process architecture, integrating streaming, decoding, rendering, and recording functions into the same process, such as the OpenCV direct streaming solution and the VLC player solution. This leads to mutual interference between functions: when recording is started, an additional RTSP stream needs to be pulled, occupying double the bandwidth; when multiple streams are running concurrently, thread blocking is prone to occur, reducing system processing capacity. Players such as VLC and FFPlay do not support frame-by-frame overlay of custom graphics and cannot integrate advanced functions such as visual recognition and status monitoring; commercial player SDKs are often bound to specific hardware, resulting in poor compatibility and high difficulty in secondary development. When processing high-resolution video, traditional solutions can reach CPU utilization of over 80%, and on ordinary hardware, they can only process 4-6 1080P video streams simultaneously, which cannot meet the concurrent monitoring needs of multiple nodes in self-organizing networks; moreover, insufficient support for efficient encoding formats such as H.265 further limits bandwidth utilization. Summary of the Invention

[0004] Therefore, it is necessary to provide a method for playing and recording RTSP video streams based on self-organizing networks, which can achieve low latency, high stability, and multi-functional processing of RTSP video streams in a self-organizing network environment, in order to address the above-mentioned technical problems.

[0005] A method for playing and recording RTSP video streams based on ad hoc networks, the method comprising: A layered, multi-instance system architecture is constructed, consisting of, from top to bottom, a self-organizing network environment layer, a main control layer, a playback unit layer, and a playback unit internal architecture layer. The self-organizing network environment layer contains multiple camera nodes, each providing real-time video streams via the RTSP protocol. The main control layer includes a playback unit manager, used to manage and coordinate multiple playback unit instances within the playback unit layer. The playback unit layer adopts a multi-instance parallel architecture, with each playback unit processing one RTSP video stream independently. The playback unit internal architecture layer includes a streaming process, a frame buffer queue, a rendering process, a recording process, and an event listening thread. For each playback unit, the main control thread is started, and the streaming process, rendering process, recording process and event listening thread are initialized through the main control thread, and communication connections between the processes are established. The process begins by pulling the video stream. Using the FFmpeg framework, network connectivity is checked against the target camera node. Upon successful detection, the metadata parameters of the RTSP video stream are obtained, the pulling parameters are configured, and an RTSP connection is established. The video stream data is continuously pulled and decoded, and the decoded video frames are stored in a frame buffer queue. If recording is enabled, a copy of the decoded video frames is stored in a dedicated recording frame buffer queue. The rendering process then begins, using the OpenCV framework to retrieve video frames from the frame buffer queue. After image format conversion and custom graphic overlay processing, the video frames are rendered to the specified display window. Simultaneously, an event listening thread is started to listen for user interaction events and forward them to the corresponding processing modules. When a recording instruction is received, the recording process is started, video frames are obtained from the recording frame buffer queue, the video frames are format converted and image overlay processed, encoded into the specified format and written to the video file; During video stream processing, a dynamic cache adjustment mechanism is adopted to monitor the queue depth of the frame buffer queue in real time, adjust the rendering rate according to the queue depth range, and monitor network jitter to dynamically adjust the target queue depth according to network jitter. When the network status of the self-organizing network changes, an adaptive network strategy is activated, including detecting network connectivity by ping, performing automatic reconnection when a connection is abnormal, and performing primary and secondary stream switching operations based on network bandwidth or stream connection status, so as to ensure the stability of video stream playback and recording.

[0006] The aforementioned method for playing and recording RTSP video streams based on ad hoc networks addresses the fundamental issues of functional coupling and insufficient concurrency in traditional solutions by constructing a layered multi-instance architecture. The ad hoc network environment layer directly connects to multiple camera nodes, ensuring the supply of the original data source for the RTSP video streams. The main control layer uses a playback unit manager to uniformly schedule multiple playback units, avoiding resource overload and global coordination chaos. The core playback unit layer adopts a multi-instance parallel and independent design, with each unit corresponding to one video stream. This process-level isolation completely solves the problem of easy blocking in multi-channel concurrency in traditional single-process architectures. Even if a single video stream fails, it will not affect the normal operation of other streams. Simultaneously, it provides the architectural foundation for 16 concurrent 1080P video streams, far exceeding the 4-6 channel processing capacity of traditional solutions. The playback unit internally separates streaming, rendering, and recording into independent processes. Combined with an event listening thread and frame buffer queue, this further decouples functions, avoiding the need for additional streaming due to conflicts between playback and recording functions in traditional solutions. Furthermore, by reusing single-channel streaming data (decoding and storing it separately in playback and recording buffer queues), it saves over 50% of bandwidth, significantly improving resource utilization. From a functional module implementation perspective, the technology selection and operational logic of each process precisely address pain points related to latency, format compatibility, and functional expansion. The streaming process, based on the FFmpeg framework, not only supports almost all encoding formats such as H.264, H.265, and MJPEG, solving the problem of insufficient H.265 support in traditional solutions, but also reduces latency from the streaming source by configuring low-latency mode, TCP transmission protocol, and network connectivity detection. The rendering process, leveraging the OpenCV framework, achieves efficient processing of video frame format conversion and supports frame-by-frame overlay of custom graphics, overcoming the limitations of players like VLC and FFPlay in integrating visual recognition and status monitoring. Furthermore, OpenCV's open interface facilitates secondary development and enhances scalability. The recording process independently retrieves frames from a dedicated buffer queue for encoding, avoiding the impact of recording operations on playback smoothness. Filename rules and disk space monitoring ensure stable recording functionality, enhancing multi-functional processing capabilities. Moreover, dynamic buffer adjustment and adaptive network strategies provide crucial guarantees for low latency and high stability in self-organizing network environments.The dynamic cache adjustment mechanism monitors queue depth in real time and adjusts the rendering rate in intervals. When the cache is small, it slows down to accumulate frames to avoid stuttering, and when the cache is large, it speeds up to consume frames and reduce latency. Combined with network jitter monitoring, it dynamically adjusts the target queue depth to keep latency within 500ms. The adaptive network strategy accurately addresses the characteristics of self-organizing networks, such as limited bandwidth and dynamic topology changes. Ping detection monitors network connectivity in real time, and the automatic reconnection mechanism solves the problem of frequent packet loss and reconnection. The main and secondary stream switching ensures uninterrupted video streaming when bandwidth drops suddenly or the main stream fails. The three work together to ensure the stability of playback and recording when the self-organizing network status changes, avoiding stuttering or interruption caused by network fluctuations in traditional solutions. Ultimately, it achieves low latency, high stability, and multi-functional processing of RTSP video streams in self-organizing network environments. Attached Figure Description

[0007] Figure 1 This is a flowchart illustrating a method for playing and recording RTSP video streams based on an ad hoc network in one embodiment. Figure 2 This is a schematic diagram of a layered multi-instance system architecture in one embodiment; Figure 3 A flowchart of a single playback unit in one embodiment; Figure 4 The flowchart for the dynamic cache adjustment algorithm is shown in another embodiment. Detailed Implementation

[0008] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0009] In one embodiment, such as Figure 1 As shown, a method for playing and recording RTSP video streams based on ad hoc networks is provided, including the following steps: Step 102: Construct a layered multi-instance system architecture. From top to bottom, the architecture includes a self-organizing network environment layer, a main control layer, a playback unit layer, and a playback unit internal architecture layer. The self-organizing network environment layer contains multiple camera nodes, each providing real-time video streams via the RTSP protocol. The main control layer contains a playback unit manager, which manages and coordinates multiple playback unit instances within the playback unit layer. The playback unit layer adopts a multi-instance parallel architecture, with each playback unit processing one RTSP video stream independently. The playback unit internal architecture layer includes a streaming process, a frame buffer queue, a rendering process, a recording process, and an event listening thread.

[0010] like Figure 2As shown, a layered multi-instance system architecture is constructed, consisting of, from top to bottom, the self-organizing network environment layer, the main control layer, the playback unit layer, and the playback unit internal architecture layer: The ad hoc network environment layer, located at the top layer of the architecture, contains multiple camera nodes distributed throughout the ad hoc network. Each node provides a real-time video stream to the outside world via the RTSP protocol. These nodes can be network cameras from different manufacturers and of different models, deployed in various locations within the ad hoc network and interconnected via a wireless ad hoc network. Each camera node acts as a video source, continuously pushing RTSP video streams outward, forming a multi-parallel video data source.

[0011] The main control layer, acting as an intermediate scheduling layer, includes the playback unit manager, which is responsible for the unified management and coordination of multiple playback unit instances at the lower level. The playback unit manager receives user control commands, such as adding new video streams, deleting existing video streams, and switching between primary and secondary bitstreams, and distributes these commands to the corresponding playback units. Simultaneously, the manager is also responsible for the allocation and monitoring of system resources to ensure that the processing of multiple video streams does not exceed the system's capacity.

[0012] The playback unit layer employs a multi-instance parallel architecture, with each playback unit processing one RTSP video stream. Each playback unit is completely independent and does not affect others. The system can dynamically create or destroy playback unit instances as needed; the number of playback units is limited only by the computer's hardware performance. This design ensures that anomalies in a single video stream will not affect the normal playback of other video streams, improving system stability and reliability.

[0013] The internal architecture layer of the playback unit showcases the internal component structure of each playback unit. Each playback unit contains five core components: a streaming process, a frame buffer queue, a rendering process, a recording process, and an event listening thread. The streaming process uses the FFmpeg framework to pull RTSP streams from the corresponding camera node and decodes them; the decoded video frames are stored in a frame buffer queue implemented in shared memory; the rendering and recording processes run in parallel as independent processes, respectively retrieving frame data from the queue for display and storage; the event listening thread is responsible for handling user interaction events. This multi-process architecture completely decouples the streaming, rendering, and recording functions, improving the system's concurrent processing capabilities.

[0014] The entire architecture achieves efficient processing of multiple RTSP video streams through layered design and multi-instance parallelism, making it particularly suitable for multi-node video surveillance application scenarios in self-organizing network environments.

[0015] Step 104: For each playback unit, start the main control thread, initialize the streaming process, rendering process, recording process and event listening thread through the main control thread, and establish communication connections between the processes.

[0016] This method employs a multi-process separation architecture, separating different functional modules of video stream processing into independent processes and threads for execution, achieving functional decoupling and concurrent processing. Each playback unit, as an independent processing unit, contains the following core components: The main control thread acts as the central hub of the playback unit, responsible for initializing various sub-modules, coordinating communication between processes, and managing the lifecycle. The main control thread maintains the state machine of the playback unit and responds to external control commands, such as play, stop, record, and screenshot operations. The streaming process is a completely independent process, responsible for connecting to the RTSP stream source via the FFmpeg framework and continuously pulling video stream data. This process decodes the received raw video data and places it into the playback frame buffer queue. If recording is enabled, a copy is placed into the recording frame buffer queue for use by downstream processes. The rendering process is a completely independent process, performing video rendering and display using the OpenCV framework. This process retrieves video frames from the playback frame buffer queue, performs necessary image processing and overlay rendering, and then renders them to the specified window. The recording process is another independent process, dedicated to video recording. When recording is enabled, this process retrieves video frames from a dedicated recording frame buffer queue, encodes them, and writes them to the video file. The event listener thread is responsible for listening to and processing user interaction events, such as mouse clicks and keyboard input, and forwarding the events to the corresponding processing modules. For example... Figure 3 The diagram shown is a flowchart of a single playback unit.

[0017] Step 106: Start the streaming process. Use the FFmpeg framework to perform network connectivity checks on the target camera node. If the check passes, obtain the metadata parameters of the RTSP video stream, configure the streaming parameters, establish an RTSP connection, continuously pull video stream data and decode it, and store the decoded video frames in a frame buffer queue. If the recording function is enabled, copy the decoded video frames and store them in a dedicated recording frame buffer queue. Start the rendering process. Use the OpenCV framework to obtain video frames from the frame buffer queue, perform image format conversion and custom graphic overlay processing on the video frames, and then render them to the specified display window. At the same time, start an event listening thread to listen for user interaction events and forward them to the corresponding processing modules.

[0018] The streaming module uses the FFmpeg framework to acquire and decode RTSP streams. The specific implementation steps are as follows: Step 1: Before establishing an RTSP connection, first perform a network connectivity test on the target device. This involves sending ICMP packets (ping operations) to check network latency and packet loss rate. Set a reasonable timeout threshold; if the network latency exceeds the threshold or there is no response, the network is considered unreachable, thus avoiding invalid connection attempts.

[0019] The second step is to obtain video stream parameters. FFmpeg's probing tools are used to analyze the metadata of the RTSP stream, including key parameters such as video resolution (width and height), frame rate, encoding format, and pixel format. These parameters are used for subsequent decoder configuration and buffer allocation.

[0020] Step 3: Configure streaming parameters according to the characteristics of the self-organizing network: Select TCP mode as the transmission protocol to ensure data integrity in unstable network environments; enable the low latency flag to reduce internal buffering; dynamically calculate the buffer size based on the video resolution, generally set to a size that can hold 60 frames of data; set the log level to the warning level to reduce unnecessary log output. Step 4: Create an FFmpeg input stream object, specifying the RTSP URL and configuration parameters. Set the output format to the original video format, selecting YUV420P or RGB24 pixel format according to the actual situation. For ultra-high-definition video (resolution exceeding 2560 pixels), automatically perform resolution scaling to ensure smooth system processing.

[0021] Step 5: Start the asynchronous streaming process to continuously read video data from the RTSP source. Each time, one complete frame of video data is read; the data size is calculated based on the resolution and pixel format. For YUV420P format, the data size is width × height × 1.5 bytes; for RGB24 format, the data size is width × height × 3 bytes.

[0022] Step 6: Push data into the frame buffer queue. After decoding the received raw video data, put it into the playback frame buffer queue. If recording is enabled, make a copy and put it into the recording frame buffer queue for use by downstream processes.

[0023] The rendering process runs as an independent process, exchanging data with the FFmpeg streaming process through a shared playback frame buffer queue, and transmitting control signaling through pipes.

[0024] After the rendering process starts, an OpenCV display window is first created, and its properties are set to a resizable normal window. The OpenCV window is embedded into a specified parent window via the operating system API to achieve interface integration. A mouse callback function is set to capture user mouse events. The rendering process enters the main loop, waiting for the delay time calculated by the dynamic caching adjustment algorithm. Video frame data is retrieved from the playback frame buffer queue; if the queue is empty, the current rendering is skipped, and the previous frame is retained. The video stream's pixel format is converted accordingly. If it is YUV420P format, it needs to be converted to BGR format for display. The conversion process can utilize GPU acceleration to improve processing efficiency. Additional information is overlaid on the converted image, including but not limited to GPS location information display, timestamp overlay, alarm box drawing, target recognition box drawing, and custom text information. Overlay rendering is achieved by parsing a predefined drawing list, supporting adaptive scaling at different resolutions. The overlay rendering process can be accelerated using the GPU to improve processing efficiency.

[0025] The processed image is rendered to the window using OpenCV's display functions. A minimum wait time (1 millisecond) is set to ensure the interface responds to user actions.

[0026] Step 108: When a recording instruction is received, the recording process is started, video frames are obtained from the recording frame buffer queue, the video frames are format converted and image overlay processed, encoded into the specified format and written to the video file.

[0027] The recording process is independent of the rendering process, retrieving data from a dedicated recording frame buffer queue to avoid impacting playback smoothness. When recording is enabled, a video encoder is created, setting the encoding format to MP4V and configuring the same resolution and frame rate as the original video stream. Recording filenames are generated, including device information and timestamps to avoid filename conflicts. Raw frame data is retrieved from the recording queue and necessary format conversions are performed. If overlay information is needed in the recorded video, the same overlay drawing operations as in the rendering process are performed. Processed frames are written to the video file. Recording can be started, paused, and stopped. A maximum recording duration limit can be set, and files are automatically split upon timeout. Disk space is monitored during recording; recording automatically stops and an alarm is triggered when space is insufficient. Encoder resources are properly released upon completion of recording to ensure video file integrity. A recording file index is maintained for easy subsequent querying and management.

[0028] Step 110: During video stream processing, a dynamic cache adjustment mechanism is adopted to monitor the queue depth of the frame buffer queue in real time, adjust the rendering rate according to the queue depth range, and monitor network jitter to dynamically adjust the target queue depth according to the network jitter.

[0029] This application utilizes a dynamic cache adjustment mechanism, such as... Figure 4 As shown, using a linear feedback controller, the formula for adjusting the rendering rate is: μ(t)= F_base×(1 + k × r(t)) The adjustment coefficient r(t) is defined as the normalized deviation between the current queue depth and the target depth. r(t) = (Q(t) - Q_target) / Q_target The control gain k is set to 0.5, which is an empirical value verified through extensive experiments.

[0030] To ensure playback quality, the rendering rate is limited to a reasonable range: 0.7×F_base≤μ(t)≤1.5×F_base.

[0031] The queue depth space is divided into three control zones: an insufficient cache zone, a normal control zone, and an excessive cache zone. Each zone employs a different control strategy. Insufficient Cache Zone: When Q(t) < 0.5 × Q_target, the system determines that the cache is severely insufficient and reduces the rendering rate μ(t) to 0.7 times the base frame rate to allow the cache to accumulate quickly and avoid playback stuttering.

[0032] Normal control region: When 0.5×Q_target≤Q(t)≤2×Q_target, the system is in a normal state, adopts a linear feedback control model, and the rendering rate μ(t) is linearly adjusted according to the queue depth to achieve smooth rate changes.

[0033] Excessive caching: When Q(t) > 2 × Q_target, the system determines that there is too much caching and increases the rendering rate μ(t) to 1.5 times the base frame rate to quickly consume the backlog of cache and reduce playback latency.

[0034] To adapt to the network instability of ad hoc networks, a network jitter compensation mechanism is introduced. The arrival time interval of the most recent 10 frames is statistically analyzed, and the coefficient of variation CV = σ / μ_interval is calculated, where σ is the standard deviation of the time interval and μ_interval is the average value of the time interval. The target cache depth is dynamically adjusted based on the network jitter level: Q_target = F_base×(2 + CV); the greater the network jitter, the more target cache is used, thereby improving the system's jitter resistance.

[0035] During algorithm execution, the current state is measured, the current queue depth Q(t) is read, and the arrival time of the latest frame is recorded. Network jitter is calculated (updated every 500 milliseconds): the arrival time interval of the most recent 10 frames is calculated, the coefficient of variation CV is updated, and the target queue depth is adjusted. Based on the relationship between the current queue depth Q(t) and the target depth, the control interval in which the system is located is determined. A new rendering rate is calculated based on the corresponding control strategy applied to the current interval. To avoid sudden rate changes affecting playback quality, an exponential moving average is used for smoothing: μ_smooth(t) = 0.7×μ_smooth(t-1) + 0.3×μ(t). The waiting time for the next rendering cycle is calculated based on the smoothed rendering rate.

[0036] Step 112: When the self-organizing network status changes, start the adaptive network strategy, including detecting network connectivity by ping, performing automatic reconnection when the connection is abnormal, and performing primary and secondary bitstream switching operation according to network bandwidth or stream connection status, so as to ensure the stability of video stream playback and recording.

[0037] In a self-organizing network environment, network bandwidth may change dynamically, and this application supports seamless switching between primary and secondary bitstreams.

[0038] Switching trigger conditions include user-initiated switching: selecting the primary or secondary stream through the interface; automatic switching: automatically switching to the secondary stream when the primary stream fails consecutively for more than a threshold; and bandwidth adaptive switching: automatically selecting the appropriate stream based on network bandwidth measurement results. The switching process first saves the current playback state, including whether recording is in progress and the current playback position. Then, it stops the current streaming process but keeps the rendering and recording processes running. Based on the switching target (primary or secondary stream), it updates the RTSP connection address. If the primary stream address is empty, the secondary stream is used automatically; otherwise, it is. The streaming process is restarted, and a connection is established to the new RTSP stream. Once the connection is successful, the previously saved state is restored; if recording was in progress, recording is automatically resumed. During the switching process, the rendering process continues to display the last frame to avoid a black screen. A switching timeout timer is set; if the new stream connection fails, it reverts to the original stream.

[0039] Multi-process architectures require efficient inter-process communication mechanisms. This application employs a combination of various IPC technologies, setting up a shared memory queue for transmitting large amounts of video frame data. The streaming process places the decoded frame data into the shared memory queue, while the rendering and recording processes read data from their respective queues. The queue is implemented in a process-safe manner, supporting a multi-producer, multi-consumer model. Pipe communication is used to transmit control commands and events. A bidirectional pipe is established between the main process and child processes to transmit small amounts of information such as mouse events, keyboard events, and control commands. Shared variables are used to synchronize state information, such as recording status, screenshot trigger flags, and window sizes, which are synchronized between processes to avoid frequent message passing. A signal mechanism is used for process lifecycle management. Graceful process exit is achieved through system signals, ensuring proper resource release.

[0040] Furthermore, the ad hoc network environment is complex and ever-changing, making a robust anomaly handling mechanism crucial. In response to network anomalies, a reasonable connection timeout should be set, automatically retrying after a timeout. The number of empty frames should be checked, and reconnection should be triggered if a threshold is exceeded. A dynamic caching algorithm should be used to smooth out the impact of network fluctuations. An exponential backoff algorithm should be employed, with a 1-second wait time for the first reconnection, doubling the wait time for each subsequent failure, with a maximum wait time not exceeding 30 seconds. The number of reconnections should be recorded, and an alarm should be triggered and the connection stopped if the maximum number of reconnections is exceeded. Process and thread status should be checked regularly, zombie processes should be cleaned up, and a maximum queue depth should be set to prevent unlimited memory growth. When system resources are insufficient, video quality or frame rate should be automatically reduced; when the GPU is unavailable, processing should automatically switch to CPU; and when disk space is insufficient, recording should stop but playback should continue.

[0041] The aforementioned method for playing and recording RTSP video streams based on ad hoc networks addresses the fundamental issues of functional coupling and insufficient concurrency in traditional solutions by constructing a layered multi-instance architecture. The ad hoc network environment layer directly connects to multiple camera nodes, ensuring the supply of the original data source for the RTSP video streams. The main control layer uses a playback unit manager to uniformly schedule multiple playback units, avoiding resource overload and global coordination chaos. The core playback unit layer adopts a multi-instance parallel and independent design, with each unit corresponding to one video stream. This process-level isolation completely solves the problem of easy blocking in multi-channel concurrency in traditional single-process architectures. Even if a single video stream fails, it will not affect the normal operation of other streams. Simultaneously, it provides the architectural foundation for 16 concurrent 1080P video streams, far exceeding the 4-6 channel processing capacity of traditional solutions. The playback unit internally separates streaming, rendering, and recording into independent processes. Combined with an event listening thread and frame buffer queue, this further decouples functions, avoiding the need for additional streaming due to conflicts between playback and recording functions in traditional solutions. Furthermore, by reusing single-channel streaming data (decoding and storing it separately in playback and recording buffer queues), it saves over 50% of bandwidth, significantly improving resource utilization. From a functional module implementation perspective, the technology selection and operational logic of each process precisely address pain points related to latency, format compatibility, and functional expansion. The streaming process, based on the FFmpeg framework, not only supports almost all encoding formats such as H.264, H.265, and MJPEG, solving the problem of insufficient H.265 support in traditional solutions, but also reduces latency from the streaming source by configuring low-latency mode, TCP transmission protocol, and network connectivity detection. The rendering process, leveraging the OpenCV framework, achieves efficient processing of video frame format conversion and supports frame-by-frame overlay of custom graphics, overcoming the limitations of players like VLC and FFPlay in integrating visual recognition and status monitoring. Furthermore, OpenCV's open interface facilitates secondary development and enhances scalability. The recording process independently retrieves frames from a dedicated buffer queue for encoding, avoiding the impact of recording operations on playback smoothness. Filename rules and disk space monitoring ensure stable recording functionality, enhancing multi-functional processing capabilities. Moreover, dynamic buffer adjustment and adaptive network strategies provide crucial guarantees for low latency and high stability in self-organizing network environments.The dynamic cache adjustment mechanism monitors queue depth in real time and adjusts the rendering rate in intervals. When the cache is small, it slows down to accumulate frames to avoid stuttering, and when the cache is large, it speeds up to consume frames and reduce latency. Combined with network jitter monitoring, it dynamically adjusts the target queue depth to keep latency within 500ms. The adaptive network strategy accurately addresses the characteristics of self-organizing networks, such as limited bandwidth and dynamic topology changes. Ping detection monitors network connectivity in real time, and the automatic reconnection mechanism solves the problem of frequent packet loss and reconnection. The main and secondary stream switching ensures uninterrupted video streaming when bandwidth drops suddenly or the main stream fails. The three work together to ensure the stability of playback and recording when the self-organizing network status changes, avoiding stuttering or interruption caused by network fluctuations in traditional solutions. Ultimately, it achieves low latency, high stability, and multi-functional processing of RTSP video streams in self-organizing network environments.

[0042] In one embodiment, the custom graphics overlay processing includes overlaying at least one of GPS location information, timestamps, alarm boxes, target recognition boxes, and custom text information onto video frames; and the graphics overlay processing is implemented by parsing a predefined drawing list, supports adaptive scaling at different resolutions, and utilizes the GPU to accelerate the graphics overlay process to reduce CPU usage.

[0043] In one embodiment, the recording process further includes: generating a recording file name containing device information and a timestamp to avoid file name conflicts; setting a maximum recording duration limit and automatically splitting the video file when the recording duration reaches the limit; monitoring disk space in real time and automatically stopping recording and triggering an alarm when disk space is insufficient; releasing encoder resources after recording to ensure the integrity of the video file and maintaining a recording file index for subsequent queries.

[0044] In one embodiment, the dynamic cache adjustment mechanism calculates the rendering rate using a linear feedback control model. The linear feedback control model process is μ(t) = F_base × (1 + k × r(t)), where μ(t) is the current rendering rate, F_base is the original video frame rate, k is the control gain, r(t) is the normalized deviation between the current queue depth and the target queue depth, and r(t) = (Q(t) - Q_target) / Q_target, where Q(t) is the queue depth at the current moment, and Q_target is the target queue depth. At the same time, the range of the rendering rate is limited to 0.7 × F_base ≤ μ(t) ≤ 1.5 × F_base.

[0045] In one embodiment, adjusting the rendering rate based on the queue depth range includes: When Q(t) < 0.5 × Q_target, the system determines that the cache is severely insufficient and reduces the rendering rate μ(t) to 0.7 times the base frame rate; When 0.5×Q_target≤Q(t)≤2×Q_target, the system is in normal condition, adopts a linear feedback control model, and the rendering rate μ(t) is linearly adjusted according to the queue depth. When Q(t) > 2 × Q_target, the system determines that there is too much cache and increases the rendering rate μ(t) to 1.5 times the base frame rate.

[0046] In one embodiment, dynamically adjusting the target queue depth based on network jitter includes: Every 500 milliseconds, the arrival time interval of the most recent 10 video frames is counted, and the standard deviation σ and average value μ_interval of the time interval are calculated. The degree of network jitter is characterized by the coefficient of variation CV = σ / μ_interval. The target queue depth is adjusted according to the degree of network jitter. The adjustment formula is Q_target = F_base×(2 +CV), where Q_target is the adjusted target queue depth, F_base is the base frame rate, and CV is the coefficient of variation of network jitter.

[0047] In one embodiment, after adjusting the rendering rate, the process further includes smoothing the rendering rate, the smoothing process being as follows: μ_smooth (t) = 0.7μ_smooth(t-1)+0.3μ(t) Where μ_smooth(t) is the current smoothed rendering rate, μ_smooth(t-1) is the smoothed rendering rate at the previous time step, and μ(t) is the currently calculated rendering rate. Then, calculate the waiting time for the next rendering cycle based on the smoothed rendering rate. The waiting time is T_wait = 1000 / μ_smooth (t).

[0048] In one embodiment, the primary and secondary stream switching operation includes: saving the current playback state, stopping the current streaming process, updating the RTSP connection address according to the switching target, restarting the streaming process and establishing a new RTSP connection, and restoring the previously saved playback state; during the switching process, the rendering process continuously displays the last frame to avoid a black screen, and a switching timeout timer is set at the same time. If the new stream connection fails, it will fall back to the original stream.

[0049] In one embodiment, network connectivity detection is performed by sending ICMP packets to perform a ping operation, detecting network latency and packet loss rate, setting a timeout threshold, and determining that the network is unreachable and stopping the current RTSP connection attempt if the network latency exceeds the threshold or there is no response; the automatic reconnection operation adopts an exponential backoff algorithm, with the first reconnection waiting time being 1 second, and the waiting time doubling after each subsequent reconnection failure, with a maximum waiting time not exceeding 30 seconds, while recording the number of reconnections, and triggering an alarm and stopping reconnection when the number of reconnections exceeds the maximum number of reconnections threshold.

[0050] In one embodiment, the image format conversion includes: if the video frame is in YUV420P format, converting it to BGR format for display; the image format conversion process is accelerated by the GPU, and when the rendering process is initialized, an OpenCV display window is created and set to be resizable, the OpenCV window is embedded into a specified parent window through the operating system API to achieve interface integration, and a mouse callback function is set to capture user mouse events.

[0051] It should be understood that, although Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0052] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0053] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these modifications and improvements all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for playing and recording RTSP video streams based on ad hoc networks, characterized in that, The method includes: A layered, multi-instance system architecture is constructed, comprising, from top to bottom, a self-organizing network environment layer, a main control layer, a playback unit layer, and a playback unit internal architecture layer. The self-organizing network environment layer contains multiple camera nodes, each providing a real-time video stream via the RTSP protocol. The main control layer includes a playback unit manager for managing and coordinating multiple playback unit instances within the playback unit layer. The playback unit layer employs a multi-instance parallel architecture, with each playback unit processing one RTSP video stream independently. The playback unit internal architecture layer includes a streaming process, a frame buffer queue, a rendering process, a recording process, and an event listening thread. For each playback unit, the main control thread is started, and the streaming process, rendering process, recording process and event listening thread are initialized through the main control thread, and communication connections between the processes are established. The streaming process is initiated, and the network connectivity of the target camera node is checked using the FFmpeg framework. Upon successful detection, the metadata parameters of the RTSP video stream are obtained, streaming parameters are configured, and an RTSP connection is established. Video stream data is continuously pulled and decoded, and the decoded video frames are stored in the frame buffer queue. If recording is enabled, a copy of the decoded video frames is stored in a dedicated recording frame buffer queue. The rendering process is then initiated, and video frames are retrieved from the frame buffer queue using the OpenCV framework. After image format conversion and custom graphic overlay processing, the video frames are rendered to the specified display window. Simultaneously, the event listening thread is initiated to listen for user interaction events and forward them to the corresponding processing modules. When a recording instruction is received, the recording process is started, video frames are obtained from the recording frame buffer queue, the video frames are format converted and image overlay processed, encoded into a specified format and written to a video file; During video stream processing, a dynamic cache adjustment mechanism is adopted to monitor the queue depth of the frame buffer queue in real time, adjust the rendering rate according to the queue depth range, and monitor network jitter to dynamically adjust the target queue depth according to network jitter. When the self-organizing network status changes, an adaptive network strategy is activated, including detecting network connectivity through ping, performing automatic reconnection when a connection is abnormal, and performing primary and secondary stream switching operations based on network bandwidth or stream connection status, in order to ensure the stability of video stream playback and recording. Adjust the rendering rate based on the queue depth range, including: When Q(t) < 0.5 × Q_target, the system determines that the cache is severely insufficient and reduces the rendering rate μ(t) to 0.7 times the base frame rate; When 0.5×Q_target≤Q(t)≤2×Q_target, the system is in normal condition, adopts a linear feedback control model, and the rendering rate μ(t) is linearly adjusted according to the queue depth. When Q(t) > 2 × Q_target, the system determines that there is too much cache and increases the rendering rate μ(t) to 1.5 times the base frame rate; The dynamic cache adjustment mechanism calculates the rendering rate using a linear feedback control model. The linear feedback control model is defined as μ(t) = F_base × (1 + k × r(t)), where μ(t) is the current rendering rate, F_base is the video baseline frame rate, k is the control gain, and r(t) is the normalized deviation between the current queue depth and the target queue depth. Furthermore, r(t) = (Q(t) - Q_target) / Q_target, where Q(t) is the current queue depth and Q_target is the target queue depth. Simultaneously, the rendering rate is limited to a range of 0.7 × F_base. ≤ μ(t) ≤ 1.5×F_base.

2. The method according to claim 1, characterized in that, The custom graphics overlay processing includes overlaying at least one of GPS location information, timestamps, alarm boxes, target recognition boxes, and custom text information onto video frames; and the graphics overlay processing is implemented by parsing a predefined drawing list, supports adaptive scaling at different resolutions, and utilizes the GPU to accelerate the graphics overlay process to reduce CPU usage.

3. The method according to claim 1, characterized in that, The recording process also includes: generating recording file names containing device information and timestamps to avoid file name conflicts; setting a maximum recording duration limit, and automatically splitting the video file when the recording duration reaches the limit; monitoring disk space in real time, and automatically stopping recording and triggering an alarm when disk space is insufficient; releasing encoder resources after recording to ensure the integrity of the video file, and maintaining the recording file index for subsequent queries.

4. The method according to claim 1, characterized in that, After adjusting the rendering rate, the process also includes smoothing the rendering rate. The smoothing process is as follows: μ_smooth (t) = 0.7μ_smooth(t-1) + 0.3μ(t) Where μ_smooth(t) is the current smoothed rendering rate, μ_smooth(t-1) is the smoothed rendering rate at the previous time step, and μ(t) is the currently calculated rendering rate.

5. The method according to claim 1, characterized in that, The main / sub stream switching operation includes: saving the current playback state, stopping the current streaming process, updating the RTSP connection address according to the switching target, restarting the streaming process and establishing a new RTSP connection, and restoring the previously saved playback state. During the switching process, the rendering process continuously displays the last frame to avoid a black screen, and sets a switching timeout timer. If the new stream connection fails, it will fall back to the original stream.

6. The method according to claim 1, characterized in that, The network connectivity detection is performed by sending ICMP packets to perform a ping operation, detecting network latency and packet loss rate, and setting a timeout threshold. If the network latency exceeds the threshold or there is no response, the network is determined to be unreachable and the current RTSP connection attempt is stopped. The automatic reconnection operation adopts an exponential backoff algorithm. The initial reconnection waiting time is 1 second, and the waiting time doubles after each subsequent failed reconnection, with a maximum waiting time not exceeding 30 seconds. The number of reconnections is recorded, and when the number of reconnections exceeds the maximum reconnection threshold, an alarm is triggered and reconnection is stopped.

7. The method according to claim 1, characterized in that, The image format conversion includes: if the video frame is in YUV420P format, it is converted to BGR format for display; the image format conversion process is accelerated by GPU, and when the rendering process is initialized, an OpenCV display window is created and set to be resizable. The OpenCV window is embedded into a specified parent window through the operating system API to achieve interface integration, and a mouse callback function is set to capture user mouse events.

Images