A method and system for dynamically processing multiple video data streams by dual operating systems

Abstract

The present application belongs to the field of embedded system and video stream processing technology, in particular to a method and system for dynamically processing multi-channel video data stream by dual operating systems, the first operating system and the second operating system are simultaneously run in the heterogeneous multi-core processor subsystem, the second operating system collects hardware state in real time and reports, the first operating system executes global resource allocation optimization algorithm and then issues to the second operating system for accurate execution, and can trigger part of reconfiguration. The second operating system creates real-time processing tasks for each video stream and executes in parallel, and reports performance indicators to the first operating system for global optimization decision, and completes configuration switching in the video frame gap. The present application organically combines the flexible management capability of general operating system and the deterministic response of real-time operating system, and realizes high concurrency, low delay, dynamic adaptation and high reliable processing of multi-channel video stream.

Description

Technical Field

[0001] This invention relates to the field of embedded systems and video stream processing technology, and in particular to a method and system for dynamically processing multiple video data streams using dual operating systems. Background Technology

[0002] With the rapid development of applications such as intelligent monitoring, autonomous driving, industrial visual inspection, and telemedicine, the demand for real-time concurrent processing of multiple video data streams is increasing. Existing technical solutions mainly suffer from the following shortcomings: While multi-channel video processing solutions based on general-purpose processors offer flexible programming options, they are limited by serial execution architecture and memory bandwidth bottlenecks, making it difficult to achieve high-throughput concurrent processing of multiple video streams with low latency.

[0003] While FPGA video processing solutions implemented with pure hardware logic offer advantages such as extremely low latency and high parallelism, they lack operating system support, resulting in inflexible task scheduling, difficulties in resource management, and an inability to dynamically adapt to different numbers and resolutions of video stream inputs.

[0004] GPU solutions offer strong parallel processing capabilities, but they suffer from high power consumption, uncertain latency, and limited deployment in embedded scenarios.

[0005] Existing solutions for running a single operating system on an FPGA either use a general-purpose operating system (such as Linux) to obtain a rich software ecosystem but sacrifice real-time performance, or use a real-time operating system (such as FreeRTOS) to ensure deterministic response but lack complex task management and network communication capabilities. They cannot simultaneously meet the hard real-time constraints of video processing and the complex functional requirements of system management.

[0006] Existing FPGA + external processor solutions typically require additional chips, increasing system complexity and power consumption. Furthermore, inter-chip communication bandwidth becomes a performance bottleneck, making it impossible to achieve true on-chip integrated processing.

[0007] Therefore, there is an urgent need for a method and system that can run two operating systems with different characteristics on an FPGA platform simultaneously, each performing its own function and working together to dynamically process multiple video data streams. Summary of the Invention

[0008] This invention proposes a method and system for dynamically processing multiple video data streams using a dual-operating system, aiming to solve the technical problems of existing technologies where a single operating system cannot simultaneously achieve both hard real-time processing performance and flexible system management capabilities; and the use of FPGA plus external processors has technical problems such as inter-chip communication bottlenecks and high power consumption.

[0009] In a first aspect, the present invention provides a method for dynamically processing multiple video data streams using a dual-operating system, comprising: S1, Start the first operating system and the second operating system in the heterogeneous multi-core processor, establish a communication mechanism between the first operating system and the second operating system, and deploy a reconfigurable video processing pipeline hardware acceleration module; S2, the second operating system receives multiple video data streams concurrently through the multi-channel video input interface, detects the video stream access and obtains the underlying parameters, and notifies the first operating system to perform high-level registration and scheduling decisions through the communication mechanism. The first operating system then sends the scheduling decisions to the second operating system to execute hardware resource configuration. S3, the second operating system collects hardware status information in real time and reports it to the first operating system. The first operating system generates a resource allocation plan and sends it to the second operating system for execution. When the video stream changes, the first operating system triggers a reconfiguration decision, and the second operating system performs hardware adjustments through partial reconfigurable technology. S4, the second operating system creates a real-time processing task for each video stream, distributes video frames to the hardware acceleration module for processing according to the configuration scheme issued by the first operating system, and reports the processing completion status to the first operating system; S5, the second operating system continuously collects performance indicators and reports them to the first operating system, the first operating system executes global optimization decisions and sends them to the second operating system, and the second operating system completes configuration switching between video frames.

[0010] Furthermore, in S1, the first operating system is a Linux operating system, running on the application processor cluster of the heterogeneous multi-core processor subsystem; the second operating system is a FreeRTOS or Zephyr real-time operating system, running on the real-time processor cluster of the heterogeneous multi-core processor subsystem; the communication mechanism includes a shared memory region, an inter-core interrupt channel, and a hardware message queue.

[0011] Further, S3 specifically includes: the first operating system executing a global resource allocation optimization algorithm and distributing the generated resource allocation scheme to the second operating system for execution, wherein the global resource allocation optimization algorithm includes: the first operating system sorting active video streams by priority; calculating the required processing bandwidth and pipeline stages for each video stream; allocating the minimum set of resources that meet the requirements from the resource pool based on the available hardware resource information reported by the second operating system; and executing a degradation strategy when resources are insufficient, wherein the degradation strategy includes one or more of the following: reducing the frame rate of non-critical video streams, reducing the resolution of non-critical video streams, and merging the processing pipelines of low-priority video streams.

[0012] Furthermore, the partially reconfigurable technology in S3 includes: the first operating system making reconfiguration decisions and managing bitstream files; the second operating system performing reconfiguration operations within a precise time window between video frames, the reconfiguration operations including: pausing data transmission in the target area, saving the current processing state, loading a new partial bitstream through a configured access port, and restoring the processing state.

[0013] Furthermore, the partially reconfigurable technology also includes: the first operating system employs bitstream compression technology and a prefetching mechanism to reduce reconfiguration time; while the second operating system reconfigures one reconfigurable region, it ensures that other reconfigurable regions continue to work normally, thereby achieving pipelined reconfiguration.

[0014] Furthermore, the processing operations performed on the video frames by the hardware acceleration module in S4 include one or more combinations of noise reduction filtering, resolution scaling, color space conversion, video encoding and decoding, feature extraction, and object detection.

[0015] Furthermore, the global optimization decision-making strategy in S5 includes: adjusting task priorities, reallocating hardware acceleration modules, enabling or disabling the number of processing pipeline stages, and dynamically adjusting the frame rate and resolution; the second operating system completes configuration switching by utilizing the video frame intervals without interrupting the currently processed video frames.

[0016] Furthermore, the method also includes a dual-system fault isolation step: the second operating system monitors the running status of the first operating system through a heartbeat mechanism. When an abnormality is detected in the first operating system, the second operating system enters an autonomous running mode to maintain basic video stream processing and triggers the recovery process of the first operating system. The video processing is uninterrupted throughout the entire process.

[0017] Furthermore, the method also includes a hardware module failure handling step: when the second operating system detects a hardware acceleration module failure, it immediately isolates the faulty module and notifies the first operating system through the communication mechanism; the first operating system generates an emergency plan and sends it to the second operating system; the second operating system performs a configuration switch during the most recent video frame interval, transferring the video stream handled by the faulty module to other resources.

[0018] Secondly, the present invention provides a system for dynamically processing multiple video data streams using a dual-operating system, the system being used to execute the method, comprising: A heterogeneous multi-core processor subsystem includes an application processor cluster and a real-time processor cluster, wherein the application processor cluster is used to run a first operating system and the real-time processor cluster is used to run a second operating system. The dual-system communication subsystem includes a shared memory region, an inter-core interrupt controller, and a hardware message queue, which provides a communication channel for the first operating system and the second operating system to issue commands, report status, and share data. A reconfigurable video processing engine, comprising multiple dynamically reconfigurable video processing pipeline units, is directly controlled by the second operating system; The global resource scheduler, running on the first operating system, is used to perform system-level resource allocation decisions and global optimizations. A real-time execution scheduler, running on the second operating system, is used to convert the scheduling decisions generated by the global resource scheduler into hardware operation timing and ensure frame-level real-time performance.

[0019] The technical effects of the disclosed technical solution are as follows: 1. The dual systems complement each other: the general-purpose operating system provides a rich software ecosystem, network protocol stack, file system and support for complex algorithms, while the real-time operating system ensures microsecond-level deterministic response in video processing. The two work together to achieve a performance and functionality balance that a single system cannot achieve.

[0020] 2. Layered and decoupled architecture: The first operating system is responsible for global scheduling decisions, while the second operating system is responsible for hard real-time hardware control. The responsibilities are clear, the coupling is low, and it is easy to develop, upgrade and maintain independently.

[0021] 3. Dynamic adaptability: Supports hot-swapping of video channels and dynamic parameter changes. The first operating system quickly performs global rescheduling, and the second operating system seamlessly completes configuration switching between frames without downtime.

[0022] 4. High concurrency and low latency: The hard real-time scheduling of the second operating system ensures the deterministic timing of video processing, and the FPGA parallel architecture enables true concurrent processing, with single-frame processing latency controllable in milliseconds.

[0023] 5. On-chip integration: Both operating systems run on the FPGA platform processor, eliminating the need for additional chips, reducing inter-chip communication bottlenecks, and lowering system complexity and power consumption.

[0024] 6. High reliability: The dual-system operation is isolated, so even if the first operating system malfunctions, the second operating system can still maintain basic video stream processing, ensuring uninterrupted system operation. Furthermore, when the hardware acceleration module fails, the two systems can work together to isolate the fault and reallocate resources. Attached Figure Description

[0025] Figure 1 This is a flowchart illustrating a method for dynamically processing multiple video data streams using a dual-operating system, as proposed in an embodiment of the present invention. Figure 2This is a schematic diagram of the dual-system collaborative processing of multiple video streams provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the communication mechanism between two operating systems provided in an embodiment of the present invention. Detailed Implementation

[0026] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0027] This paper addresses the technical challenge mentioned in the background section: how to build a dual-operating system collaborative architecture on an FPGA platform, providing flexible system management and complex task coordination capabilities through a general-purpose operating system, while ensuring hard real-time determinism in multi-channel video data stream processing through a real-time operating system, thereby achieving organic collaboration between the two operating systems and enabling the system to adaptively reconfigure resources and optimize task scheduling based on the dynamic changes of the input video stream.

[0028] Based on this, this embodiment of the invention provides a method for dynamically processing multiple video data streams using dual operating systems. This embodiment uses the Xilinx Zynq UltraScale+ MPSoC series FPGA chip as an example for illustration. However, those skilled in the art should understand that this invention can also be implemented based on other FPGA or SoC chips with heterogeneous multi-core processor architectures.

[0029] refer to Figures 1 to 3 As shown, this embodiment constructs a dual-operating system collaborative processing architecture on an FPGA chip. The FPGA chip internally includes a heterogeneous multi-core processor subsystem and a programmable logic region. The heterogeneous multi-core processor subsystem is further divided into an application processor cluster and a real-time processor cluster.

[0030] The application processor cluster uses a quad-core ARM Cortex-A53 processor with a clock speed of 1.2GHz, running the primary operating system—Linux (kernel version 5.x). It is equipped with an independent DDR4 memory controller and connected to 2GB of DDR4 memory. The primary operating system is responsible for non-real-time or soft real-time tasks such as global resource scheduling, system management, user interaction, network communication, video output management, data storage, and remote control.

[0031] The real-time processor cluster uses a dual-core ARM Cortex-R5 processor with a clock speed of 533MHz. It runs a second operating system—FreeRTOS—which uses 256KB of tightly coupled memory (TCM) to store critical real-time code and data, and independently accesses 256KB of on-chip memory (OCM) for data sharing between the two systems. The second operating system is responsible for the hard real-time processing scheduling of the video data stream, hardware acceleration module control, frame-level precise timing management, interrupt response, and some reconfigurable execution.

[0032] The FPGA's programmable logic (PL) side is equipped with a reconfigurable video processing engine. This engine comprises eight reconfigurable video processing pipeline units (VPPs) and four partially reconfigurable regions (PRRs), supporting dynamic loading of different processing function modules at runtime. It is directly controlled by the second operating system (FreeRTOS) via the AXI bus. In addition, the PL side also integrates multiple video input interfaces (supporting protocols such as MIPICSI-2, HDMI, SDI, and GigEVision, with each interface equipped with an independent frame buffer controller and DMA engine), a high-speed interconnect subsystem (AXI interconnect matrix and on-chip network NoC), a frame buffer manager (supporting ping-pong buffering and multi-buffering mechanisms), and an output management module (running on the first operating system, supporting output modes such as independent output, picture-in-picture compositing, multi-window splicing, and network streaming media distribution).

[0033] The two operating systems interact through a dual-system communication subsystem, which employs a three-layer mechanism: the first layer is inter-core interrupt (IPI), with a latency of less than 1 microsecond, used for event notifications and emergency signals; the second layer is shared memory (using on-chip OCM), with a bandwidth of 128 bits at 533MHz, using a circular buffer and lock-free queues for exchanging commands, status, and configuration data; the third layer is a hardware message queue based on a programmable logic-based hardware FIFO, used for batch performance data reporting, with hardware guaranteeing no data loss and no blocking. The communication protocol defines a command channel (Linux to FreeRTOS, carrying scheduling instructions, configuration parameters, and reconfiguration commands) and a status channel (FreeRTOS to Linux, carrying hardware status, performance indicators, and alarm information). The synchronization mechanism uses hardware spinlocks and memory barrier instructions to ensure data consistency.

[0034] The following is combined Figure 2 The flowchart shown describes in detail the specific method steps for dynamically processing multiple video data streams using a dual operating system in this embodiment. Specifically, it includes: S1, the dual-operating system construction and initialization phase, specifically includes the following sub-steps.

[0035] S1.1 Building a heterogeneous multi-core processor platform: In this embodiment, within the heterogeneous multi-core processor subsystem of the FPGA chip, a quad-core ARM Cortex-A53 is configured as an application processor cluster, and a dual-core ARM Cortex-R5 is configured as a real-time processor cluster. The two processor clusters are connected via an on-chip interconnect bus (such as the AXI coherent extension interface) and independently access the DDR memory controller and on-chip memory, respectively.

[0036] S1.2, Start the first operating system: The first operating system is launched on the application processor cluster. In this embodiment, the first operating system is Linux (kernel version 5.x), and the kernel and root file system are loaded from flash memory or SD card. After Linux starts, necessary device drivers are loaded, including dual-system communication drivers, global resource scheduler kernel modules, some reconfigurable policy drivers, frame buffer management drivers, video output synthesis drivers, etc. Linux provides user space with application programming interfaces such as video stream management services, global resource scheduling daemons, performance analysis and visualization services, web configuration management interfaces, and network streaming media distribution services.

[0037] S1.3, Start the second operating system: A second operating system is launched on the real-time processor cluster. In this embodiment, the second operating system is FreeRTOS, which runs from tightly coupled memory (TCM). After FreeRTOS starts, the real-time task layer (including video frame synchronization task, DMA transfer scheduling task, hardware acceleration control task, performance acquisition task, and inter-core communication task), the reconfiguration execution layer (including ICAP / PCAP controller driver, bitstream loading manager, and state saving / restoration engine), and the hardware driver layer (including AXIVDMA driver, video processing IP core register driver, interrupt controller driver, and timer driver) are initialized.

[0038] S1.4 Establish a communication mechanism between the two operating systems: Configure shared memory region: Allocate a 256KB region in the on-chip OCM as shared memory for the dual systems, used to store command buffers, status buffers, and configuration parameters. Both Linux and FreeRTOS map this region to their respective address spaces. Configure inter-core interrupt (IPI) channel: Set up the IPI register so that either operating system can trigger an interrupt on the other system by writing to the IPI register. Configure hardware message queue: Instantiate a hardware FIFO queue on the programmable logic side for batch transmission of performance data from FreeRTOS to Linux. Communication protocol: The command channel (Linux→FreeRTOS) carries scheduling instructions, configuration parameters, and reconfiguration commands, using a producer-consumer model; the status channel (FreeRTOS→Linux) carries hardware status, performance indicators, and alarm information, transmitted through the hardware message queue; the synchronization mechanism uses hardware spinlocks and memory barrier instructions to ensure the consistency of shared data.

[0039] S1.5 Deploy a reconfigurable video processing pipeline hardware acceleration module array: Eight reconfigurable video processing pipeline units (VPPs) are deployed in the FPGA's programmable logic area. Each VPP includes a video decoding module, an image preprocessing module (such as denoising filtering, scaling, and color space conversion), an algorithm acceleration module (such as feature extraction and object detection), and a video encoding module. Four partial reconfigurable regions (PRRs) are also divided, each capable of independently loading different partial bitstreams, thus dynamically changing the functionality of the VPP unit at runtime. The VPP units are connected to the processor subsystem's DMA controller and interrupt controller via an AXI interconnect bus, ensuring that FreeRTOS can directly control the registers of these hardware modules.

[0040] (I) Hardware Structure of the VPP Unit: Each VPP unit adopts a multi-stage pipelined architecture, including an input FIFO, several processing stages (such as denoising, scaling, color conversion, etc.), a configuration register set, and an output FIFO. Each processing stage is executed in parallel via the pipeline, and one or more pixels can be processed in one clock cycle. The VPP unit is connected to the DMA controller and processor subsystem via the AXI bus, supporting FreeRTOS for control and status querying via register read / write. This pipelined video processing structure is a common technique in FPGA design.

[0041] (II) Implementation of Each Processing Module: Modules such as denoising filtering, resolution scaling, color space conversion, video encoding / decoding, feature extraction, and target detection all utilize mature FPGA IP cores or well-known hardware implementation methods. For example, denoising can employ median filtering or Gaussian filtering, scaling can use polyphase filters, and color conversion can use matrix multipliers. Those skilled in the art, after clarifying the functional requirements, can select or develop corresponding hardware modules. Specific circuit details are not innovative aspects of this invention and will not be elaborated further.

[0042] (iii) Coordination between the hardware acceleration module and the software scheduler. Hardware-software coordination is achieved through the following methods: FreeRTOS reads and writes the VPP's configuration registers via the AXI-lite bus to enable, set parameters, and reset modules.

[0043] The VPP module provides status registers (such as busy / idle, frame completion flag) and optional interrupt signals. FreeRTOS obtains the processing status through polling or interrupts.

[0044] DMA transfer is configured by FreeRTOS to move video frames from the input buffer to the VPP's input FIFO, and after processing, the DMA writes the result back to the output buffer.

[0045] The entire process is a loop of "configure DMA, start DMA transfer, wait for transfer completion interrupt, configure VPP, start VPP processing, wait for VPP completion interrupt, and notify Linux".

[0046] (iv) The flow of video frame data between VPP units: Typically, each video stream is assigned a fixed VPP unit, and each stream's VPP processes data in parallel without data flowing across units. When cascading processing is required (e.g., denoising before target detection), the output buffer address of the first-level VPP can be configured as the input source address of the second-level VPP, and the process can be completed via DMA relay. When multiple video streams need to be merged, the frame data from multiple streams can first be aggregated into the same splicing buffer via DMA, and then read and processed by a single VPP unit. Data flow is uniformly managed by FreeRTOS's DMA scheduling tasks, and automated transmission is achieved through DMA descriptor chains.

[0047] Through the initialization steps described above, this embodiment successfully built a dual-operating system runtime environment on a single FPGA chip. Linux, as a general-purpose operating system, provides a rich software ecosystem and flexible management capabilities, while FreeRTOS, as a real-time operating system, ensures the hard real-time responsiveness of subsequent video processing tasks. The dual systems achieve low-latency, highly reliable command and data exchange through an efficient three-layer communication mechanism. A reconfigurable video processing pipeline array provides the hardware foundation for subsequent dynamic resource adaptation.

[0048] Step S2, Multi-channel video stream access and dual-system collaborative registration stage.

[0049] S2.1 Concurrent reception of multiple video data streams: In this embodiment, up to eight video data streams are received concurrently through a multi-channel video input interface layer (e.g., four MIPICSI-2 interfaces and four HDMI interfaces). Each video stream is written to a frame buffer (located in DDR4 memory) via an independent DMA channel. The frame buffer manager allocates an independent input buffer for each video stream and supports a ping-pong buffering mechanism to eliminate frame transmission gaps.

[0050] S2.2, The second operating system detects new video stream access: When a FreeRTOS interrupt service routine detects a new video stream entering a video input interface (e.g., via the VSYNC sync signal or a hot-plug event), it reads the low-level parameters of the video stream, such as resolution, frame rate, and color space (e.g., YUV422, RGB888), and confirms the DMA channel status via hardware registers. Subsequently, FreeRTOS notifies Linux via inter-kernel communication (IPI interrupt) to encapsulate the low-level parameters of the video stream into a message and write it to the command channel in shared memory, thus triggering IPI.

[0051] S2.3, The first operating system performs high-level registration and scheduling decisions: Upon receiving an IPI interrupt, Linux wakes up the video stream management service in the lower half of the interrupt handling. This service reads the parameters of the new video stream from the shared memory command channel and completes high-level registration: assigning a globally unique stream identifier (StreamID), determining the processing strategy based on the application configuration (e.g., the video stream needs denoising, scaling, object detection, etc.), setting priorities (e.g., real-time monitoring video streams have higher priority than video recording / storage streams), and establishing a stream descriptor table (containing buffer addresses, processing pipeline templates, etc.). Then, Linux's global resource scheduler generates a scheduling decision (i.e., which hardware resources to allocate to the video stream) based on the current system load and the processing requirements of the new video stream. This scheduling decision is encapsulated into a command, written to the shared memory command channel, and notified to FreeRTOS via IPI.

[0052] S2.4, Second operating system configuration of underlying hardware resource mapping: Upon receiving the IPI, FreeRTOS reads the scheduling decision from shared memory and performs hardware resource configuration: allocating VPP units (e.g., VPP3), DMA channels (e.g., DMA channel 5), and output buffer regions (e.g., Buffer_Region_C) for the new video stream. Specific operations include: writing to the configuration register of VPP3 (setting the number of pipeline stages, filtering coefficients, etc.), configuring the descriptor of DMA channel 5 (source address, destination address, transfer length, loop mode), and setting the frame buffer address mapping table. After configuration, FreeRTOS marks the video stream as "ready" in the shared memory status channel and notifies Linux of the configuration completion via the IPI.

[0053] This system enables plug-and-play and dynamic registration of video streams. FreeRTOS handles microsecond-level interrupt responses and low-level parameter acquisition, while Linux handles millisecond-level policy decisions and resource allocation. This collaborative approach ensures both rapid response and complex stream management. This coordination mechanism allows the system to dynamically add or remove video streams during runtime without requiring a reboot.

[0054] Step S3, the dynamic resource assessment and allocation stage of dual-system collaboration: The second operating system collects hardware status information in real time and reports it to the first operating system. The first operating system generates a resource allocation plan and sends it to the second operating system for execution. When the video stream changes, the first operating system triggers a reconfiguration decision, and the second operating system performs hardware adjustments through partial reconfigurable technology. Specifically, this includes the following sub-steps.

[0055] S3.1, The second operating system collects hardware status information in real time: The resource monitoring module in FreeRTOS periodically (e.g., every 10 milliseconds) collects low-level hardware status information, including: FPGA logic resource utilization (the percentage of LUTs, FFs, and BRAMs used), memory bandwidth utilization (by monitoring the DDR controller's read / write counters), the load of each VPP processing pipeline (i.e., the ratio of processing time per frame to the frame period), and the status of each DMA channel (idle / busy / error). This status information is encapsulated into data packets and reported to Linux via a hardware message queue (a third-layer communication mechanism).

[0056] S3.2, The first operating system comprehensively analyzes and executes the resource allocation optimization algorithm: The dynamic resource scheduler in Linux receives hardware status information from FreeRTOS and simultaneously obtains the set of currently active video streams S={ , ,..., } and the processing requirement vector D for each video stream. = (resolution, frame rate, required pipeline stages, priority). The scheduler executes a global resource allocation optimization algorithm, the specific steps of which are as follows: The first step is to sort the active video streams in descending order according to their priority, resulting in an ordered set S'.

[0057] The second step is to process each video stream. ∈S': Calculate the required processing bandwidth BW( = Resolution width × Resolution height × Bytes per pixel × Frame rate; Determine the number and type of VPP units required based on the required pipeline stages; From the available hardware resource pool (based on the number of remaining VPPs, remaining memory bandwidth, idle DMA channels, etc. reported by FreeRTOS), for Allocate the minimum set of resources that meet its needs.

[0058] The third step is to implement a degradation strategy if the overall system resources are insufficient: reduce the frame rate of non-critical video streams (e.g., from 30fps to 15fps), or reduce the resolution of non-critical video streams (e.g., from 1080p to 720p), or merge multiple low-priority video streams into the same VPP processing pipeline for time-sharing processing.

[0059] Step 4: Output resource allocation mapping table M: →{ , , }

[0060] In Linux, the dynamic resource scheduler integrates hardware status information and the processing requirements of each video stream to generate a resource allocation scheme in the following manner: Priority ranking: The overall priority is determined based on the service type of the video stream (such as real-time monitoring, autonomous driving, video recording and storage, etc.) and frame rate requirements. Video streams with higher real-time requirements (such as obstacle detection in autonomous driving) have higher priority, while video recording and storage streams have the lowest priority. Under the same conditions, streams with higher frame rates have higher priority.

[0061] Bandwidth Calculation: For each video stream, calculate the required data processing bandwidth based on its resolution (width × height), color space format, and frame rate. Different color space formats have different numbers of bytes per pixel (e.g., RGB888 is 3 bytes, YUV422 is 2 bytes), and the processing bandwidth is proportional to the resolution, frame rate, and number of bytes per pixel.

[0062] Resource Allocation: Based on currently available VPP units, DMA channels, and memory bandwidth, allocate the minimum set of resources to each video stream to meet its bandwidth requirements. A greedy strategy is employed: resources are allocated sequentially from highest to lowest priority. Each video stream first selects the VPP unit with the lightest current load that can support its required processing functions (such as denoising, scaling, and object detection). If multiple VPP units can meet the requirements, the one with the remaining capacity after allocation that best matches the requirement is selected to improve resource utilization.

[0063] Degradation Strategy: When system resources are insufficient to meet the processing demands of all video streams, the following degradation measures are executed sequentially: first, reduce the frame rate of non-critical (low-priority) video streams (e.g., from 30fps to 15fps), then reduce their resolution (e.g., from 1080p to 720p), and finally merge multiple low-priority video streams into the same VPP unit for time-sharing processing. The degradation extent is dynamically determined based on the resource gap, aiming to just meet resource requirements and avoid excessive degradation. After degradation, resource allocation is re-executed; if it still cannot meet the requirements, newly connected video streams are rejected.

[0064] The degradation strategy is triggered when one of the following two situations occurs: Resource insufficiency determination: When executing the aforementioned resource allocation algorithm, if it is impossible to find a VPP unit that meets the processing capacity for the current video stream to be allocated (i.e., the remaining load space of all candidate VPPs is less than the load required by the video stream), or the total bandwidth requirement of all allocated video streams exceeds 80% of the remaining memory bandwidth, then it is determined that there is insufficient resources.

[0065] Performance overload determination: In subsequent S5 performance monitoring, if the actual frame rate of a running video stream is found to be lower than 85% of its target frame rate for multiple consecutive cycles (e.g., 3 frame cycles), or the processing latency exceeds 85% of the frame cycle for multiple consecutive cycles, then the video stream is determined to be overloaded and a degradation should be triggered.

[0066] When a degradation is triggered, the degradation measures are executed in the following order, and the degradation magnitude is dynamically determined based on the resource gap ratio or performance deviation ratio: reduce the frame rate of non-critical video streams.

[0067] Applicable to: Video streams with low priority (e.g., background recording, non-real-time preview).

[0068] Degradation Amount: Target Frame Rate = Original Frame Rate × (1 - Resource Gap Ratio), where the resource gap ratio is (Required Bandwidth - Available Bandwidth) / Required Bandwidth. The downgraded frame rate should not be lower than 50% of the original frame rate, and should not be lower than 15fps (ensuring acceptable smoothness for the human eye). If performance overload is triggered, the degradation amount is: Target Frame Rate = Original Frame Rate × (Target Frame Rate / Actual Frame Rate) × 0.9, that is, reduced to a frame rate slightly lower than the currently achievable frame rate.

[0069] Execution method: Linux sends frame rate control commands to the video source, or FreeRTOS selectively drops frames during DMA transfer.

[0070] S3.3, Issue and accurately execute the resource allocation plan: Linux encapsulates the resource allocation scheme as a command sequence, sends it to FreeRTOS via a shared memory command channel, and triggers an IPI interrupt. Upon receiving the scheme, FreeRTOS precisely executes the hardware resource configuration operations: for example, configuring the VPP3 registers for a video stream (setting filter coefficients, scaling, and output format), setting the descriptor for DMA channel 5 (including the source frame buffer address, target VPP input buffer address, and transfer length), and establishing the address mapping for the output frame buffer. All configuration operations are completed before the start of the next video frame.

[0071] S3.4 Dynamic reconfiguration processing: When the number or parameters of video streams change (e.g., a video stream switches from 1080p@30fps to 4K@60fps, or a new video stream is added), Linux's global scheduler detects the change and triggers a dynamic reconfiguration decision. Linux first determines whether the current resources meet the new requirements. If a change in processing functionality is needed (e.g., switching from simple scaling to object detection), it decides to partially reconfigure a specific PRR region. Linux loads the required bitstream file from flash memory or DDR and sends the reconfiguration command (specifying the PRR region, bitstream address, and loading method) to FreeRTOS via shared memory. FreeRTOS performs the reconfiguration operation within a precise time window between video frames: pausing data transmission in the target PRR region, loading the new partial bitstream via the ICAP (Internal Configuration Access Port) or PCAP (Processor Configuration Access Port) interface, resuming processing, and restarting the data stream. This embodiment employs bitstream compression technology to reduce typical reconfiguration time from 50ms to less than 10ms; simultaneously, Linux uses a prefetch mechanism (predicting the necessary bitstream based on historical patterns and loading it in advance from DDR to the cache), further compressing the effective reconfiguration time to 5ms. While reconfiguring one PRR region, FreeRTOS ensures that other PRR regions continue to function normally, achieving pipelined reconfiguration.

[0072] Through resource assessment and allocation via dual-system collaboration, dynamic optimization of hardware resources is achieved. Linux's global resource allocation algorithm comprehensively considers multiple factors such as priority, bandwidth requirements, and remaining hardware resources to generate the optimal allocation scheme. FreeRTOS precisely executes low-level configurations and supports partial reconfigurability, enabling the system to dynamically adjust hardware functions without interrupting overall services. This technical solution is the core of the invention's "dynamic processing" capability.

[0073] (i) Bitstream compression: To reduce data transfer time during reconfiguration, some bitstreams are stored using lossless compression algorithms (such as LZ77 series or dictionary compression). The Linux side performs offline compression on some bitstreams, and FreeRTOS integrates a corresponding decompression module that decompresses and writes to the configuration port simultaneously during loading. Compression can typically reduce the bitstream size by more than half, thus significantly shortening loading time.

[0074] (ii) Prefetching Mechanism: Linux maintains a bitstream usage record and predicts the next potentially needed bitstream based on historical call patterns (such as frequent consecutive calls to a function or reuse at fixed time intervals). When the system is idle, the predicted bitstream is preloaded from flash memory into a prefetch buffer in DDR. When the actual reconfiguration command is issued, if the target bitstream is already in the prefetch buffer, it can be used directly, further reducing reconfiguration latency; if it is not found, it is read from flash memory in real time. In practical applications, the prefetch hit rate can reach a high level, and the effective reconfiguration time can be compressed to the millisecond level.

[0075] (III) Pipeline-based reconfiguration coordination: When the system contains multiple PRR regions, FreeRTOS uses a pipelined approach to coordinate reconfiguration: while one PRR is being reconfigured, other PRR regions continue to process the video stream normally. FreeRTOS maintains a state machine (e.g., running, ready, configuring, and resuming) for each PRR and a reconfiguration request queue. When multiple PRRs need to be reconfigured, they are processed sequentially according to priority, with priority given to PRRs with longer frame periods (i.e., wider frame intervals) to ensure sufficient time for the operation to be completed. Mutual exclusion protection is implemented through hardware semaphores to ensure that other tasks do not access the target PRR during configuration.

[0076] (iv) Data stream pause and resume (no frame loss guarantee): FreeRTOS utilizes video frame intervals (vertical blanking period) to perform reconfiguration to ensure no video frames are lost. Frame gap detection: The start time of the frame gap is accurately identified through hardware VSYNC interrupt or line counter.

[0077] Pause data flow: Immediately after the start of the interframe gap, disable the input DMA channel of the target PRR, wait for the PRR internal pipeline to be drained, and save the current critical state (such as register configuration and processing counter) to memory.

[0078] Perform reconfiguration: Load a new partial bitstream via the ICAP or PCAP interface, during which the PRR output is in a high-impedance state.

[0079] Data Stream Resumption: After loading, restore the saved state (if the new configuration is compatible), reconfigure the DMA channel, enable input, and continue processing the next frame. No Frame Drop Guarantee: The entire pause, configuration, and resumption process is completed within the same frame interval. Therefore, no new video frames are submitted to the PRR being configured, and the frame data already written to the input buffer remains unchanged. After configuration, processing resumes from the current frame. If the configuration time exceeds the current frame interval due to a large bitstream, it is completed in steps using multiple frame intervals (through a state machine continuation). During this period, the input frame data is still normally buffered (through ping-pong buffering), and no frames are lost.

[0080] Through the above mechanism, the system ensures the continuity and integrity of the video stream while dynamically adjusting hardware functions.

[0081] Step S4, Dual-system driven concurrent video processing execution stage: The second operating system creates a real-time processing task for each video stream, distributes video frames to the hardware acceleration module for processing according to the configuration scheme issued by the first operating system, and reports the processing completion status to the first operating system. This specifically includes the following sub-steps.

[0082] S4.1, The second operating system creates real-time processing tasks: FreeRTOS creates an independent real-time processing task for each registered and resource-allocated video stream. Each task runs under FreeRTOS's hard real-time scheduler, employing a fixed-priority preemptive scheduling strategy. The video frame synchronization task is set to the highest priority to ensure timely response to frame start signals; the DMA transfer scheduling task is set to the second highest priority; the hardware acceleration control task is set to medium priority; and the performance acquisition task and inter-core communication task are set to lower priority. Each real-time processing task is bound to a corresponding VPP unit and DMA channel.

[0083] S4.2 Distribute video frames to the hardware acceleration module: Each real-time processing task, following the processing pipeline configuration scheme issued by Linux (e.g., processing order: denoising → scaling → color space conversion → object detection), controls the DMA controller to read video frame data from the input frame buffer and distribute it to the corresponding VPP unit. Specifically, the task waits for a frame synchronization signal (such as a VSYNC interrupt), then configures the DMA channel's source address to the current input frame buffer address and the destination address to the VPP unit's input FIFO address, and initiates DMA transfer. After the transfer is complete, the DMA controller generates an interrupt, and the task starts processing in the VPP unit within the interrupt service routine.

[0084] S4.3, The hardware acceleration module performs video processing: The VPP unit performs preset processing operations on video frames. In this embodiment, the VPP unit supports one or more combinations of the following operations: noise reduction filtering (such as median filtering, Gaussian filtering), resolution scaling (such as bilinear interpolation, bicubic interpolation), color space conversion (such as YUV422 to RGB888, RGB to YUV), video encoding and decoding (such as H.264 / H.265 hardware encoding), feature extraction (such as SIFT, HOG), and object detection (such as YOLO, hardware acceleration implementation of SSD). These operations process multiple pixels in parallel within one clock cycle through pipeline cascading, achieving extremely low latency.

[0085] S4.4 Write back the processing results and report the status: After the VPP unit finishes processing, FreeRTOS's task control DMA writes the processed video frame back from the VPP output FIFO to the output buffer (located in DDR). After the write-back is complete, the task reports the processing completion status to Linux through a shared memory status channel or a hardware message queue, including the frame number, processing time, and processing results (such as a list of bounding boxes for object detection).

[0086] S4.5, The first operating system manages output: Linux's output management module performs simultaneous output, merged output, or network distribution of multiple video streams based on application requirements. For example, in surveillance scenarios, multiple video streams can be combined into a single output display using picture-in-picture (PiP) or multi-window splicing; in autonomous driving scenarios, target detection results can be overlaid onto video frames and distributed via network streaming protocols (such as RTSP). The output management module calls the video output compositing driver, transmits the output frames to HDMI or DisplayPort interfaces via DMA, or sends them to network clients via the network card.

[0087] Technical achievements at this stage: FreeRTOS's hard real-time scheduling ensures that each video frame completes acquisition, processing, and output within a strict frame period (e.g., 33.3ms for 30fps), avoiding frame drops and jitter. The VPP pipeline on the FPGA side achieves true parallel processing. The total data bandwidth requirement for 8 channels of 1080p@30fps video is approximately 2.99GB / s, while DDR4-2400 provides a theoretical bandwidth of 19.2GB / s, leaving ample margin. Linux handles complex output synthesis and network distribution, fully leveraging its rich software ecosystem.

[0088] Step S5, the adaptive scheduling optimization phase with dual-system linkage: The second operating system continuously collects performance indicators and reports them to the first operating system. The first operating system executes global optimization decisions and sends them to the second operating system. The second operating system completes configuration switching between video frames. This specifically includes the following sub-steps.

[0089] S5.1, Second Operating System Continuously Collects Performance Metrics: FreeRTOS's performance monitoring module continuously collects real-time performance metrics for various video processing streams, including real-time frame rate (calculated by the number of tasks completed per second), processing latency (time from DMA read start to VPP completion), frame drop rate (number of frames dropped due to timeouts), and DMA transfer efficiency (the ratio of actual bandwidth to theoretical bandwidth). These metrics are updated every frame and reported in batches to Linux via a hardware message queue (Layer 3 communication).

[0090] S5.2, The first operating system comprehensively analyzes and executes global optimization decisions: Linux's intelligent scheduling service receives performance metrics and compares them to preset performance thresholds. When it detects that the performance of a video processing path is below the threshold (e.g., the actual frame rate is lower than 90% of the target frame rate, or the processing latency exceeds 80% of the frame period), a global optimization decision is triggered. The optimization strategies include: Adjust task priority: In FreeRTOS, increase the priority of the task corresponding to this video stream so that it can obtain CPU resources earlier.

[0091] Reassign the hardware acceleration module: migrate the video stream to a lighter-loaded VPP unit.

[0092] Enable / disable processing pipeline stages: If the current latency is too high, non-critical processing steps (such as noise reduction filtering) can be skipped to reduce the processing time per frame.

[0093] Dynamically adjust frame rate and resolution: Send control commands to the video source to reduce the frame rate or resolution of that video stream.

[0094] The trigger section is reconfigurable: if the current hardware acceleration algorithm is found to be inefficient, it can be reconfigured to a more efficient algorithm.

[0095] S5.3, Issue optimization decisions and perform atomic switching: Linux sends optimization decisions to FreeRTOS via a shared memory command channel, triggering an IPI (Initial Programming Interface). Upon receiving the decision, FreeRTOS atomically completes the configuration switch during the video frame interval (the vertical blanking period, typically hundreds of microseconds to milliseconds) without interrupting the currently processing video frame. For example, if a video stream needs to be migrated from VPP3 to VPP5, FreeRTOS first completes the configuration and preloads it on the new VPP5, then switches the DMA channel's target address to VPP5 at the start of the next frame, simultaneously releasing VPP3. The entire process does not lose any frames.

[0096] (a) Determining the precise time window for video frame intervals: The video frame gap, also known as the vertical blanking period (VBI), is the idle time between the end of one frame and the beginning of the next in a video signal. FreeRTOS determines this time window in the following way: The end time of each frame is detected by the hardware VSYNC interrupt signal.

[0097] The duration of the vertical blanking period is calculated based on the known format (resolution, frame rate) of the video stream or by measuring the time interval between two consecutive VSYNC interrupts.

[0098] To ensure operational safety, the actual time window used should be within 80% of the measurement duration, leaving a margin.

[0099] (ii) Implementation of atomic completion of configuration switching within the frame gap: "atomicity" means that a series of operations in configuration switching either all succeed or none are executed, and intermediate states are not allowed.

[0100] Upon detecting a VSYNC interrupt, FreeRTOS immediately enters a critical section (disabling interrupts or task scheduling) to ensure that subsequent operations are not interrupted.

[0101] A hardware double-buffering mechanism is adopted: two sets of configuration registers are set up for each reconfigurable hardware module. One set is currently in effect, while the other set is pre-written with new configurations in the background.

[0102] When the interframe gap arrives, FreeRTOS performs a single register write operation to switch the "Valid Configuration Group Selection" from the current group to the background group. This operation is guaranteed to be atomic by hardware and takes an extremely short time (microseconds).

[0103] After the switch is complete, exit the critical section, and the next frame of data will be automatically processed according to the new configuration. If the switch fails, retain the original configuration and report an error.

[0104] Technical benefits at this stage: This closed-loop adaptive optimization mechanism enables the system to dynamically adjust based on real-time operating conditions, always maintaining optimal performance. For example, when processing eight video streams simultaneously, if the scene of one video stream becomes more complex, leading to increased processing time, the system can automatically reduce its frame rate or migrate to a faster VPP unit, avoiding overall performance degradation. Compared to static configuration schemes, the dynamic optimization of this invention significantly improves the system's robustness and resource utilization.

[0105] In a specific scenario of dynamic channel addition and reduction, the dual-system collaborative workflow of the present invention is further illustrated through an example of dynamic channel addition and reduction.

[0106] Initial state: The system is stably running four channels of 4K@30fps video processing. Four VPPs in FreeRTOS each process one 4K video stream, with precise frame timing control. Linux monitors the overall resource utilization at 65%. FreeRTOS periodically reports performance metrics to Linux.

[0107] At time T1, FreeRTOS's interrupt service detects four new 1080p@60fps video streams entering the system. After parsing the video parameters, it notifies Linux via IPI. Linux's stream management service receives the notification and registers the new video streams.

[0108] At time T2, the Linux global scheduler calculates the additional processing bandwidth requirement: the bandwidth required for a single 1080p@60fps pipeline is approximately 1920×1080×3×60≈373MB / s, and the total additional bandwidth for 4 pipelines is approximately 1.49GB / s. The remaining resources are assessed (based on the hardware status reported by FreeRTOS, with 4 remaining VPP units and sufficient remaining memory bandwidth), and a decision is made to utilize the remaining resources to deploy a 4-way 1080p processing pipeline. A resource allocation plan is generated and distributed to FreeRTOS.

[0109] At time T3, FreeRTOS receives and parses the configuration scheme during the interframe interval, loads a 1080p processing configuration bitstream in the idle PRR region, configures the DMA channel and frame buffer, creates a new real-time processing task and starts it, and then notifies Linux that the configuration is complete.

[0110] At time T4, the system stably processed 4 channels of 4K and 4 channels of 1080p. FreeRTOS precisely controlled the frame synchronization and DMA transfer of the 8 video streams, while Linux monitored overall performance, achieving a resource utilization rate of 88%. All video streams met real-time requirements.

[0111] This embodiment also has dual-system fault isolation capability, which will be illustrated through two scenarios below.

[0112] Scenario 1: Handling Linux system exceptions.

[0113] FreeRTOS runs a heartbeat monitoring task to monitor the running status of the primary operating system (Linux). The specific implementation of the heartbeat mechanism is as follows: Heartbeat signal content: A fixed heartbeat region is defined in shared memory, containing a heartbeat counter. Linux periodically writes an incrementing value to this counter (e.g., every 0.5 to 1 second, the period can be configured according to system real-time requirements); simultaneously, Linux optionally triggers an IPI interrupt as an auxiliary heartbeat signal after each write. The FreeRTOS heartbeat monitoring task reads the value of this heartbeat counter and compares it with the previously read value.

[0114] Anomaly detection criteria: The heartbeat monitoring task reads the heartbeat counter at a preset check interval (e.g., 1.5 times the heartbeat cycle). If multiple consecutive reads (e.g., 2 or 3 times, the number can be configured according to system reliability requirements) show no change in the counter value and no expected IPI heartbeat interrupt is received, the Linux system is considered abnormal. To avoid false positives, the anomaly threshold can be dynamically configured: for high reliability scenarios, three consecutive no-change reads can be set to determine an anomaly; for fast recovery scenarios, two consecutive no-change reads can trigger recovery.

[0115] Timeout calculation: Actual timeout = Heartbeat cycle × Number of consecutive no-response times. For example, if the heartbeat cycle is 1 second and there are 3 consecutive no-response times, the timeout is 3 seconds. This timeout should be greater than the maximum response latency under normal Linux conditions (including interrupt response and task scheduling latency) to prevent false alarms.

[0116] Actions after anomaly confirmation: Upon anomaly detection, FreeRTOS immediately enters autonomous operation mode: maintaining basic processing of all current video streams, keeping the existing processing pipeline configuration unchanged, and continuing DMA transfer and frame synchronization scheduling; simultaneously suspending non-critical functions dependent on Linux (such as network streaming media distribution and the web configuration interface). FreeRTOS triggers a Linux system reboot via a hardware watchdog. After the Linux reboot is complete, it retrieves the current running state (including the list of active video streams, current configuration parameters, frame buffer addresses, etc.) saved by FreeRTOS from the shared memory state channel and restores global management functions. Video processing continues uninterrupted throughout the entire process.

[0117] Scenario 2: Dual-system collaborative processing when hardware acceleration module fails.

[0118] During operation, FreeRTOS detects a VPP3 processing anomaly (e.g., output frame verification failure or processing timeout). FreeRTOS immediately isolates the faulty module: suspends VPP3 DMA transfers, marks VPP3 as unavailable, and notifies Linux via an emergency IPI interrupt. Linux's global scheduler quickly evaluates and generates contingency plans, including: Plan A – transferring the video stream handled by VPP3 to an idle VPP (e.g., VPP7); Plan B – initiating PRR reconfiguration, reconfiguring the VPP3 region with a spare configuration bitstream to attempt repair; Plan C – temporary degradation processing (reducing the frame rate or resolution of the video stream). Linux distributes the contingency plan to FreeRTOS, which performs a configuration switch during the nearest video frame interval. Simultaneously, Linux logs the fault and notifies upper-layer applications via the user interface or remote interface. The entire process does not affect the normal processing of other video streams.

[0119] Although this embodiment uses the Xilinx Zynq UltraScale+ MPSoC platform as an example for detailed description, the present invention is also applicable to other FPGA or SoC platforms with heterogeneous multi-core processors. For example, based on Intel Agilex series FPGA chips, a quad-core ARM Cortex-A53 integrated in a hard-core processor system (HPS) can run Linux, while a soft-core Nios V processor running ZephyrRTOS can be instantiated in the FPGA logic. Dual-system communication is achieved through Avalon bus interconnect and shared on-chip SRAM. The core dual-system collaborative architecture, communication protocol, and processing flow remain unchanged. The second operating system can also be selected from different RTOS such as FreeRTOS, Zephyr, and RT-Thread, depending on the platform characteristics.

[0120] Based on the same inventive concept, embodiments of the present invention also provide a system for dynamically processing multiple video data streams using a dual-operating system, the system being used to execute the method, the system comprising: A heterogeneous multi-core processor subsystem includes an application processor cluster and a real-time processor cluster, wherein the application processor cluster is used to run a first operating system and the real-time processor cluster is used to run a second operating system. The dual-system communication subsystem includes a shared memory region, an inter-core interrupt controller, and a hardware message queue, which provides a communication channel for the first operating system and the second operating system to issue commands, report status, and share data. A reconfigurable video processing engine, comprising multiple dynamically reconfigurable video processing pipeline units, is directly controlled by the second operating system; The global resource scheduler, running on the first operating system, is used to perform system-level resource allocation decisions and global optimizations. A real-time execution scheduler, running on the second operating system, is used to convert the scheduling decisions generated by the global resource scheduler into hardware operation timing and ensure frame-level real-time performance.

[0121] Example embodiments have been disclosed herein, and while specific terminology has been used, it is for illustrative purposes only and should be construed as such, and is not intended to be limiting. In some instances, it will be apparent to those skilled in the art that features, characteristics, and / or elements described in conjunction with particular embodiments may be used alone, or in combination with features, characteristics, and / or elements described in conjunction with other embodiments, unless otherwise expressly indicated. Therefore, those skilled in the art will understand that various changes in form and detail may be made without departing from the scope of the invention as set forth in the appended claims.

Claims

1. A method for dynamically processing multiple video data streams using a dual operating system, characterized in that, include: S1, Start the first operating system and the second operating system in the heterogeneous multi-core processor, and establish a communication mechanism between the first operating system and the second operating system; And deploy reconfigurable video processing pipeline hardware acceleration modules; S2, the second operating system receives multiple video data streams concurrently through the multi-channel video input interface, detects the video stream access and obtains the underlying parameters, and notifies the first operating system to perform high-level registration and scheduling decisions through the communication mechanism. The first operating system then sends the scheduling decisions to the second operating system to execute hardware resource configuration. S3, the second operating system collects hardware status information in real time and reports it to the first operating system. The first operating system generates a resource allocation plan and sends it to the second operating system for execution. When the video stream changes, the first operating system triggers a reconfiguration decision, and the second operating system performs hardware adjustments through partial reconfigurable technology. S4, the second operating system creates a real-time processing task for each video stream, distributes video frames to the hardware acceleration module for processing according to the configuration scheme issued by the first operating system, and reports the processing completion status to the first operating system; S5, the second operating system continuously collects performance indicators and reports them to the first operating system, the first operating system executes global optimization decisions and sends them to the second operating system, and the second operating system completes configuration switching between video frames.

2. The method according to claim 1, characterized in that, In step S1, the first operating system is a Linux operating system, running on the application processor cluster of the heterogeneous multi-core processor subsystem; the second operating system is a FreeRTOS or Zephyr real-time operating system, running on the real-time processor cluster of the heterogeneous multi-core processor subsystem; the communication mechanism includes a shared memory region, an inter-core interrupt channel, and a hardware message queue.

3. The method according to claim 1, characterized in that, S3 specifically includes: the first operating system executing a global resource allocation optimization algorithm and distributing the generated resource allocation scheme to the second operating system for execution, wherein the global resource allocation optimization algorithm includes: the first operating system sorting active video streams by priority; calculating the required processing bandwidth and pipeline stages for each video stream; allocating the minimum set of resources that meet the requirements from the resource pool based on the available hardware resource information reported by the second operating system; and executing a degradation strategy when resources are insufficient, wherein the degradation strategy includes one or more of the following: reducing the frame rate of non-critical video streams, reducing the resolution of non-critical video streams, and merging the processing pipelines of low-priority video streams.

4. The method according to claim 1, characterized in that, The partially reconfigurable technology in S3 includes: the first operating system making reconfiguration decisions and managing bitstream files; the second operating system performing reconfiguration operations within a precise time window between video frames, the reconfiguration operations including: pausing data transmission in the target area, saving the current processing state, loading a new partial bitstream through a configured access port, and restoring the processing state.

5. The method according to claim 4, characterized in that, The reconfigurable technology further includes: the first operating system employs bitstream compression technology and a prefetching mechanism to reduce reconfiguration time; the second operating system ensures that other reconfigurable areas continue to work normally while reconfiguring one reconfigurable region, thus achieving pipelined reconfiguration.

6. The method according to claim 1, characterized in that, The hardware acceleration module in S4 performs processing operations on video frames including one or more combinations of noise reduction filtering, resolution scaling, color space conversion, video encoding and decoding, feature extraction, and object detection.

7. The method according to claim 1, characterized in that, The global optimization decision-making strategies in S5 include: adjusting task priorities, reallocating hardware acceleration modules, enabling or disabling the number of processing pipeline stages, and dynamically adjusting the frame rate and resolution; the second operating system completes configuration switching by utilizing the video frame intervals without interrupting the currently processed video frames.

8. The method according to claim 1, characterized in that, The method further includes a dual-system fault isolation step: the second operating system monitors the running status of the first operating system through a heartbeat mechanism. When an abnormality is detected in the first operating system, the second operating system enters an autonomous running mode to maintain basic video stream processing and triggers the recovery process of the first operating system. The video processing is uninterrupted throughout the entire process.

9. The method according to claim 1, characterized in that, The method further includes a hardware module fault handling step: when the second operating system detects a hardware acceleration module fault, it immediately isolates the faulty module and notifies the first operating system through the communication mechanism; The first operating system generates an emergency plan and distributes it to the second operating system; the second operating system performs a configuration switch during the most recent video frame interval, transferring the video stream handled by the faulty module to other resources.

10. A system for dynamically processing multiple video data streams using a dual operating system, characterized in that, The system is used to perform the method according to any one of claims 1-9, comprising: A heterogeneous multi-core processor subsystem includes an application processor cluster and a real-time processor cluster, wherein the application processor cluster is used to run a first operating system and the real-time processor cluster is used to run a second operating system. The dual-system communication subsystem includes a shared memory region, an inter-core interrupt controller, and a hardware message queue, which provides a communication channel for the first operating system and the second operating system to issue commands, report status, and share data. A reconfigurable video processing engine, comprising multiple dynamically reconfigurable video processing pipeline units, is directly controlled by the second operating system; The global resource scheduler, running on the first operating system, is used to perform system-level resource allocation decisions and global optimizations. A real-time execution scheduler, running on the second operating system, is used to convert the scheduling decisions generated by the global resource scheduler into hardware operation timing and ensure frame-level real-time performance.

Images