Intelligent multi-layer video storage bandwidth optimization system based on FPGA

Abstract

The invention discloses an intelligent multi-layer video storage bandwidth optimization system based on an FPGA (Field Programmable Gate Array). The intelligent multi-layer video storage bandwidth optimization system comprises an FPGA (Field Programmable Gate Array) main processing unit; an onboard DDR memory; a CXL controller; an external CXL memory device; and the intelligent multi-layer memory management unit is realized in the FPGA and is used for carrying out layer characteristic analysis on the video data, calculating a score for each layer according to a layer characteristic analysis result, determining a storage strategy and an access priority through a PDL and an arbiter, and carrying out compression or partial elimination on the corresponding layer according to the access priority, so that the video data is obtained. And processing the image layer data, writing the processed image layer data into the frame buffer selected by the storage strategy, reading the corresponding image layer data from each frame buffer, superposing, and outputting the superposed video data. Through the mode, the bandwidth utilization efficiency can be maximized while the video quality is ensured.

Description

Technical Field

[0001] The present application relates to the field of video processing technology, and in particular to an intelligent multi-layer video storage bandwidth optimization system based on FPGA. Background Art

[0002] Modern video processors, such as those implemented in FPGAs, often need to simultaneously process and composite multiple video layers. Each layer may undergo independent processing steps, such as cropping, scaling, and frame buffering, before being summed up for output. As video resolutions evolve from 1080p to 4Kp60 and beyond, and the number of layers increases, the bandwidth requirements for DDR (Double Data Rate SDRAM) memory have skyrocketed. For example, the read and write bandwidth requirements for a single channel of 4Kp60 (8-bit RGB) video are approximately 23.88 Gbps. If a system supports five such layers, the theoretical bandwidth requirement would exceed 100 Gbps. This presents a significant challenge for FPGA-based onboard DDR (e.g., 64-bit DDR4, which has a theoretical bandwidth of approximately 153.6 Gbps and an actual effective utilization rate of approximately 90%, or 138.24 Gbps). Insufficient bandwidth can easily lead to image flickering or processing delays.

[0003] The existing technology has the following problems:

[0004] 1. Limited hardware expansion: Simply increasing the number of DDR chips or controllers is strictly limited by the number of FPGA chip I / O pins, DDR controller resources, and PCB wiring complexity and cost.

[0005] 2. Simple layer compression: For example, when bandwidth is insufficient, lower-priority (usually obscured) layers are compressed. While this method reduces bandwidth, it sacrifices the image quality of these layers. If these layers are partially visible through methods such as alpha blending, the loss of image quality will affect the final output.

[0006] 3. Simple occlusion culling: For example, only the part of the layer that is not completely obscured by the upper layer is written to DDR. This method is completely ineffective when processing alpha blending or semi-transparent layers, because the "occluded" layer content still contributes to the final image.

[0007] Therefore, existing technologies have shortcomings in balancing multi-layer processing performance, image quality and hardware costs, especially the lack of an intelligent mechanism that can dynamically optimize bandwidth utilization based on layer characteristics and system load. Summary of the Invention

[0008] The FPGA-based intelligent multi-layer video storage bandwidth optimization system provided in this application can intelligently manage the allocation and access of multi-layer video data in DDR and CXL hybrid storage, and dynamically adjust the processing strategy according to the layer characteristics to maximize bandwidth utilization efficiency while ensuring video quality.

[0009] In a first aspect, the present application provides an FPGA-based intelligent multi-layer video storage bandwidth optimization system, which includes: an FPGA main processing unit, responsible for receiving video data, cropping and scaling each layer, frame buffer read and write control, layer overlay, and video output; onboard DDR memory, used for frame buffering; a CXL controller, integrated through the FPGA; an external CXL memory device, connected to the CXL controller, used for frame buffering; an intelligent multi-layer memory management unit, implemented within the FPGA, used to perform layer characteristic analysis on video data, and calculate a score for each layer based on the layer characteristic analysis results, and determine the storage strategy and access priority through the PDL and arbiter, and compress or partially eliminate the corresponding layer based on the access priority, and write the processed layer data to the frame buffer selected by the storage strategy, and read the corresponding layer data from each frame buffer, overlay it, and output the overlaid video data.

[0010] Among them, the intelligent multi-layer memory management unit includes: a layer characteristic analysis module, which is used to perform layer characteristic analysis on video data and obtain layer characteristic analysis results; a dynamic priority and resource allocation module, which is used to calculate a score for each layer based on the layer characteristic analysis results, and determine the storage strategy and access priority through a programmable delay line and an arbitrator; and an adaptive processing strategy module, which is used to compress or partially eliminate the corresponding layer based on the access priority.

[0011] The dynamic priority and resource allocation module includes: a scoring module for calculating a score for each layer based on the results of layer characteristic analysis; at least one programmable delay line, each of which is used to receive the score of the corresponding layer and output a delayed signal; an arbiter connected to the at least one programmable delay line for determining the layer with the highest current priority based on the order in which the delayed signals arrive; and a resource allocation unit for determining, based on the arbitration result output by the arbiter and the current total system bandwidth load, whether the frame buffer data for the layer should be stored in the onboard DDR memory or in an external CXL memory device via the CXL controller.

[0012] The resource allocation unit is further configured to place layers with high priority and access times higher than a threshold in the onboard DDR memory, and layers with low priority and capacity higher than a threshold in the external CXL memory device.

[0013] Among them, the programmable delay line is constructed by cascading lookup tables, or the programmable delay line is constructed by configuring a multiplexer through a lookup table and using the multiplexer to select a delay path, or the programmable delay line is constructed by a dedicated delay unit, or the programmable delay line is constructed by wiring resources inside the FPGA.

[0014] The arbiter is an asynchronous arbiter and / or is composed of a mutual exclusion latch array.

[0015] Among them, the adaptive processing strategy module is also used to perform lossy compression on the target layer when the target layer has a low priority and is blocked by a high-priority layer and the total system bandwidth is insufficient, or when the target layer is completely opaque and completely blocked by its upper layer, decide not to write the invalid part of the data of this layer into any frame buffer.

[0016] Among them, the adaptive processing strategy module is also used to dynamically adjust the parameters of the video processing IP according to the processing load and layer characteristics.

[0017] The adaptive processing strategy module is further used to calculate the decision factor of the i-th layer and select a processing strategy according to the decision factor.

[0018] The decision factor is calculated using the following formula:

[0019]

[0020] Among them, D i Indicates the decision factor for the processing strategy of the i-th video layer, α i represents the Alpha value of the i-th layer, bandwidth utilization represents the actual usage percentage of the system's current total storage bandwidth (DDR+CXL), target bandwidth utilization represents an expected bandwidth utilization threshold set by the system, layer score i represents the score calculated for the i-th layer by the dynamic priority and resource allocation module, average layer score represents the average of the scores of all currently active layers, occlusion ratio i represents the percentage of pixel area of the i-th layer that is occluded by its upper opaque or semi-transparent layer, K storage represents the adjustment factor given according to the storage medium currently allocated to the i-th layer and its characteristics, β1, β2, and β3 are positive weight coefficients used to adjust the influence of different factors in the decision-making.

[0021] The beneficial effects of the present application are as follows: Different from the prior art, the present application provides an FPGA-based intelligent multi-layer video storage bandwidth optimization system, which includes: an FPGA main processing unit, responsible for receiving video data, cropping and scaling each layer, frame buffer read and write control, layer overlay, and video output; onboard DDR memory for frame buffering; a CXL controller, integrated through the FPGA; an external CXL memory device, connected to the CXL controller, for frame buffering; an intelligent multi-layer memory management unit, implemented within the FPGA, for performing layer characteristic analysis on video data, and calculating a score for each layer based on the layer characteristic analysis results, and determining the storage strategy and access priority through the PDL and arbiter, and compressing or partially eliminating the corresponding layer based on the access priority, and writing the processed layer data to the frame buffer selected by the storage strategy, and reading the corresponding layer data from each frame buffer, overlaying it, and outputting the overlaid video data. It can intelligently manage the allocation and access of multi-layer video data in DDR and CXL hybrid storage, and dynamically adjust the processing strategy according to the layer characteristics to maximize bandwidth utilization efficiency while ensuring video quality. It effectively solves the bandwidth bottleneck problem in existing FPGA multi-layer video processing and significantly improves the overall performance and flexibility of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without inventive efforts. Among them:

[0023] Figure 1 This is a structural diagram of an embodiment of an FPGA-based intelligent multi-layer video storage bandwidth optimization system provided by the present application;

[0024] Figure 2 This is a schematic structural diagram of an embodiment of an intelligent multi-layer memory management unit provided by the present application;

[0025] Figure 3 This is a structural diagram of an embodiment of the dynamic priority and resource allocation module provided by this application

[0026] Figure 4 This is a schematic diagram of the workflow of the FPGA-based intelligent multi-layer video storage bandwidth optimization system of this application. DETAILED DESCRIPTION

[0027] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It will be understood that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for ease of description, only some, rather than all, structures related to the present application are shown in the drawings. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0028] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0029] See Figure 1 , Figure 1 This is a schematic diagram of an embodiment of an FPGA-based intelligent multi-layer video storage bandwidth optimization system provided by this application. The intelligent multi-layer video storage bandwidth optimization system 100 includes an FPGA main processing unit 10, onboard DDR memory 20, a CXL controller 11, an external CXL memory device 30, and an intelligent multi-layer memory management unit 12.

[0030] The FPGA main processing unit 10 is responsible for receiving video data, cropping and scaling of each layer, frame buffer read and write control, layer overlay and video output.

[0031] The onboard DDR memory 20 is used for frame buffering, that is, serves as one of the main frame buffer storage media.

[0032] The CXL controller 11 is integrated via FPGA.

[0033] An external CXL memory device 30 is connected to the CXL controller 11 for frame buffering. The FPGA-integrated CXL controller 11 (or a soft-core implementation) connects to an external CXL memory device 30 (such as a CXL Type 3 memory expander), forming a secondary or parallel frame buffer storage medium. The CXL interface utilizes the PCIe physical layer and supports the CXL.mem protocol, allowing the FPGA to directly access CXL memory via load / store instructions, providing high-bandwidth, low-latency memory expansion capabilities. This provides the system with storage resources far exceeding the capacity and potential bandwidth of onboard DDR.

[0034] In some embodiments, the CXL controller 11 (CXL Controller) is implemented in the FPGA as follows:

[0035] Function: The CXL controller 11 acts as a bridge and translator between the FPGA and external CXL memory devices 30. It encapsulates and transmits FPGA memory access requests according to the CXL protocol. It also receives data or status returned by the CXL memory device and converts it into a format understandable by the FPGA.

[0036] Implementation:

[0037] FPGA-Integrated Hard-Core CXL Controller 11: Some high-end FPGAs may already have integrated CXL controller 11 hard-core IP (Intellectual Property). This approach offers the best performance, lowest latency, and optimized power consumption because it is a specially designed hardware circuit.

[0038] Soft-core CXL controller 11: If the FPGA chip does not have an integrated hard-core CXL controller 11, or if more flexible customization is required, a soft-core CXL controller 11 can be implemented within the FPGA's programmable logic resources (such as LUTs, registers, and DSP units). This means building the functionality of the CXL controller 11 using the FPGA's logic units. This approach offers high flexibility, allowing functionality to be tailored to specific needs. However, compared to a hard-core implementation, its performance (such as maximum bandwidth and latency) may be slightly lower, and it consumes more FPGA logic resources.

[0039] Function: The CXL controller 11 must support key components of the CXL protocol stack, particularly the CXL.mem protocol related to memory access. It handles CXL link initialization and configuration, memory address mapping, command sending and receiving, and error handling.

[0040] The following describes the external CXL memory device 30 (CXL-connected Memory Device):

[0041] Type: CXL Type 3 Memory Expander. The CXL specification defines multiple device types, with Type 3 devices specifically designed for memory expansion. These devices lack computational capabilities and instead contain a large number of memory chips (which can be DDR4, DDR5, or even other memory technologies) connected to the host (in this case, the FPGA) via a CXL interface.

[0042] Connection method: The external CXL memory device 30 is connected to the CXL interface on the FPGA through a standard physical connector (similar to a PCIe slot or cable).

[0043] A second level or parallel frame buffer storage medium can be constructed:

[0044] Secondary Storage: External CXL memory devices 30 can serve as a lower level of storage than the onboard DDR. For example, the most frequently used and critical layer data can be placed in the onboard DDR for the lowest access latency, while layer data with larger capacity requirements or less frequent access can be placed in CXL memory.

[0045] Parallel Storage: In some designs, external CXL memory device 30 can operate in parallel with onboard DDR to share frame buffering tasks. The intelligent multi-layer memory management unit 12 dynamically decides whether to store a specific layer's data in onboard DDR or external CXL memory device 30, based on the layer's characteristics and real-time requirements.

[0046] The technical characteristics of the CXL interface are as follows:

[0047] Leveraging the PCIe Physical Layer: The CXL protocol reuses the proven PCI Express (PCIe) physical layer. This means CXL devices can use the same electrical characteristics, connectors, and signal transmission methods as PCIe devices. This significantly reduces the implementation threshold and cost of CXL technology and enables it to take advantage of the increasing bandwidth of the PCIe physical layer (e.g., PCIe 4.0, 5.0, 6.0, etc.).

[0048] Support for CXL.mem protocol: CXL.mem is a key component of the CXL protocol suite. It allows the host (FPGA) to access memory connected to the CXL device as if it were local memory.

[0049] Direct access to CXL memory via load / store instructions: This is one of the core advantages of the CXL.mem protocol. Processing units within the FPGA (for example, if the FPGA is running an embedded processor or has a dedicated DMA engine) can directly read and write data to CXL memory using "load" and "store" instructions similar to those used to access regular memory. Compared to traditional I / O device access (such as transferring data blocks via DMA), this approach provides finer access granularity (up to byte level) and lower latency because instructions can be directly mapped to memory operations on the CXL bus.

[0050] Provides high-bandwidth, low-latency memory expansion capabilities:

[0051] High bandwidth lies in: CXL utilizes the multi-lane PCIe physical layer to provide very high theoretical bandwidth. For example, a x16 PCIe Gen5 CXL link can provide a theoretical bidirectional bandwidth of 128GB / s.

[0052] Low latency: While CXL memory typically has higher latency than onboard DDR memory directly connected to the FPGA, its optimized protocol stack and load / store access model keep it significantly lower than network-connected remote memory or other traditional I / O storage devices. CXL memory latency is typically in the tens to hundreds of nanoseconds.

[0053] The advantages of the system are:

[0054] Far Exceeding Onboard DDR Capacity: CXL Type 3 memory expanders can provide terabytes of memory capacity, far exceeding the DDR capacity directly integrated into FPGA chips. This enables the system to process more and higher-resolution video layers or cache video data for longer periods of time.

[0055] Potential bandwidth increase: While the bandwidth of a single CXL link may be comparable to or higher than the total bandwidth of the FPGA's onboard DDR controller, by using multiple CXL devices or wider CXL links, a huge aggregate bandwidth can be achieved. More importantly, CXL memory can work in conjunction with onboard DDR, effectively increasing the total bandwidth available to the system for frame buffering.

[0056] In summary, by integrating the CXL controller 11 (either hard-core or soft-core) into the FPGA and connecting it to an external CXL memory device (such as a CXL Type 3 memory expander), the system gains powerful memory expansion capabilities. This extended memory can be accessed by the FPGA in a manner similar to local memory (load / store instructions) via the efficient CXL.mem protocol, providing high bandwidth and relatively low latency. This not only solves the problem of insufficient onboard DDR capacity, but also provides a wider resource allocation space for the intelligent multi-layer memory management unit 12, thereby more effectively optimizing the bandwidth bottleneck of multi-layer video processing.

[0057] The intelligent multi-layer memory management unit 12 is implemented inside the FPGA and is used to perform layer characteristic analysis on the video data. Based on the layer characteristic analysis results, a score is calculated for each layer. The storage strategy and access priority are determined through the PDL and the arbitrator. Based on the access priority, the corresponding layer is compressed or partially eliminated. The processed layer data is written to the frame buffer selected by the storage strategy, and the corresponding layer data is read from each frame buffer for superposition, and the superimposed video data is output.

[0058] See Figure 2 The intelligent multi-layer memory management unit 12 includes: a layer characteristic analysis module 121, a dynamic priority and resource allocation module 122 and an adaptive processing strategy module 123.

[0059] The layer characteristics analysis module 121 is used to perform layer characteristics analysis on the video data to obtain layer characteristics analysis results. The layer characteristics analysis module 121 receives metadata for each video input layer, including resolution, position in the final image, size, transparency (alpha value), user-set priority, and real-time occlusion information from the layer overlay module (to determine whether a layer is covered by an upper completely opaque layer). The analysis then generates layer characteristics analysis results.

[0060] The dynamic priority and resource allocation module 122 is used to calculate a score for each layer based on the layer characteristic analysis results, and determine the storage strategy and access priority through the programmable delay line and arbiter.

[0061] See Figure 3 The dynamic priority and resource allocation module 122 includes: a scoring module 1221 , at least one programmable delay line 1222 , an arbiter 1223 and a resource allocation unit 1224 .

[0062] Scoring module 1221 is used to calculate a score for each layer based on the layer characteristic analysis results. For example, a dynamic "bandwidth requirement / importance combined score" is calculated for each layer based on the layer characteristic analysis results. For example, layers with high resolution, foreground (unobstructed or with high alpha values), and high user priority will receive higher scores.

[0063] Each programmable delay line 1222 is configured to receive a score of a corresponding layer and output a delayed signal.

[0064] The programmable delay line 1222 is constructed by cascading lookup tables, or the programmable delay line 1222 configures a multiplexer through a lookup table and uses the multiplexer to select a delay path, or the programmable delay line 1222 is constructed by a dedicated delay unit, or the programmable delay line 1222 is constructed by wiring resources inside the FPGA.

[0065] The programmable delay line 1222 (PDL) is further described as follows:

[0066] Basic Concepts: A programmable delay line (PDL) is a digital circuit or module whose core function is to apply a controllable, programmable time delay to an input signal before outputting it. Simply put, after an input signal enters a PDL, it does not appear immediately at the output. Instead, it "waits" for a period of time determined by external control signals or internal configuration.

[0067] Implementation principle (in FPGA): In FPGA, PDL is usually not an independent hard-core IP (although some specialized FPGAs may have it), but is built using the programmable logic resources within the FPGA. Common implementation methods include:

[0068] Look-up table (LUT) cascade:

[0069] Using LUTs as delay units: A single LUT has inherent propagation delay. By connecting multiple LUTs in series (connecting the output of one LUT to the input of the next), the signal accumulates a certain amount of delay each time it passes through a LUT.

[0070] Using LUTs as Multiplexers (MUX): You can design a structure consisting of multiple parallel delay paths, each with a different fixed delay (for example, achieved by a different number of buffers or LUTs in series). Then, use one or more LUTs configured as a multiplexer to select one of the paths to direct the input signal to the output based on an external "programming" signal. This way, by selecting different paths, you can achieve different total delays.

[0071] Dedicated Delay Cells: Some FPGA architectures may include dedicated, configurable delay cells, which typically have more precise and predictable delay characteristics than PDLs built with general-purpose logic.

[0072] Programmable routing resources: The routing resources within an FPGA also have inherent delays. By cleverly utilizing programmable routing switches, signals can be directed through routing paths of varying lengths or types, indirectly adjusting the delay. However, precise control of this approach is complex.

[0073] The meaning of "programmable": "Programmable" means that the delay amount of PDL is not fixed, but can be changed through external control signals or configuration registers when the system is working or designed.

[0074] For example, static programming: setting the delay value during FPGA configuration (when burning the bitstream) or system initialization.

[0075] For example, dynamic programming allows for dynamic adjustment of delay values based on real-time requirements during system operation. For example, in the IMMU of the present invention, each layer's "combined bandwidth requirement / importance score" is dynamically converted into programming input for the corresponding PDL, thereby changing the delay of that layer's signal in real time.

[0076] The application of the programmable delay line 1222 in this application is as follows:

[0077] Converting "score" to "time": Scoring module 1221 calculates a comprehensive score for each video layer. This score is used as a control input for the corresponding PDL. The PDL is designed so that higher-scoring layers have shorter PDL latency, while lower-scoring layers have longer latency.

[0078] Providing a basis for arbitration: After PDL processing, signals representing different layer priorities (these signals may simply be a pulse indicating that the layer is ready) will arrive at the arbitrator 1223 at different times. This time difference directly reflects the layer priority.

[0079] The key features and challenges of PDL are:

[0080] Delay range and accuracy: The maximum delay and minimum delay step (resolution) that the PDL can provide are important parameters.

[0081] Consistency and predictability: In FPGAs, due to process variations (PVT—Process, Voltage, Temperature variations) and routing differences, accurately controlling and predicting PDL delay can be challenging. Therefore, calibration mechanisms or robust designs may be required.

[0082] Resource consumption: Implementing a PDL with a large delay range and high accuracy may consume more FPGA logic resources.

[0083] Arbiter 1223 is connected to at least one programmable delay line 1222 and is used to determine the layer with the highest current priority based on the order in which delayed signals arrive. Specifically, priority determination is implemented using programmable delay line 1222 (PDL) and arbiter 1223: each layer's comprehensive score is converted into a control input for the PDL, so that layers with higher scores experience shorter delays in the PDL. The PDL outputs of multiple layers are connected to a single hardware arbiter 1223. Arbiter 1223 quickly determines the layer with the highest current priority based on the order in which signals arrive (i.e., the magnitude of the delay).

[0084] The arbiter 1223 is an asynchronous arbiter 1223 and / or is composed of a mutex latch array.

[0085] The arbiter 1223 is further described as follows:

[0086] Basic Concepts: Arbitrator 1223 is a logic circuit used to resolve access conflicts between multiple requesters for a single shared resource. When multiple requesters simultaneously or nearly simultaneously request access to a shared resource, arbitrator 1223 selects one requester and grants access based on a pre-defined arbitration policy, forcing the remaining requesters to wait.

[0087] Working principle:

[0088] Request Inputs: Arbiter 1223 has multiple inputs, each corresponding to a request source. When a request source needs to access a shared resource, it sends a request signal to arbiter 1223.

[0089] Grant Outputs: Arbitrator 1223 also has multiple outputs, each corresponding to a request source. When arbitrator 1223 decides to grant resource access rights to a request source, it sends a grant signal to the output corresponding to the request source.

[0090] Arbitration Logic: This is the core of the arbitrator 1223. It determines which request should be prioritized based on an internal arbitration algorithm or strategy.

[0091] Resource Busy / Acknowledge: Usually, a shared resource has a "busy" signal when it is in use, or a "acknowledge" signal after a request is processed. These signals can be fed back to the arbitrator 1223 so that the arbitrator 1223 knows when to process the next request.

[0092] Common arbitration strategies are as follows:

[0093] Fixed Priority: Each request source is assigned a fixed priority. When a conflict occurs, the request source with the highest priority is granted. This is simple but can lead to starvation of lower-priority requests.

[0094] Round-Robin: Request sources are granted authorization in turn. This approach offers better fairness but may not be suitable for systems with real-time requirements.

[0095] Time-based arbitration (FCFS): The earliest request source is granted access. In the present invention, the PDL converts "priority" into "arrival time." Therefore, arbitrator 1223 is actually performing time-based arbitration.

[0096] Least Recently Used (LRU): Prioritizes requests that have not been served for the longest time.

[0097] Weighted Arbitration: Allocates access opportunities based on the weight of the request source.

[0098] The application of the arbitrator 1223 in this application is as follows:

[0099] Receiving signals from PDL: The “priority signals” of multiple video layers (after being delayed by their respective PDLs) arrive at the respective request input terminals of the arbitrator 1223 simultaneously or successively.

[0100] Arbitration based on arrival time: The arbitrator 1223 is designed to respond to the first valid signal to arrive. Since the PDL has converted the high-priority layer signal into the early-arriving signal, the arbitrator 1223 naturally selects the layer with the highest current priority.

[0101] Decision Output: The output of arbiter 1223 (e.g., a one-hot code or an index value) indicates which layer currently has the highest access rights. This output is then used by the dynamic resource allocation module (e.g., to determine whether data should be stored in DDR or CXL memory) and the adaptive processing strategy module 123.

[0102] Hardware implementation of the arbiter 1223: In an FPGA, this arrival time-based arbiter 1223 can be implemented very efficiently. For example:

[0103] Array of Mutual Exclusion Latches (SRLatches): Each request input is connected to the Set terminal of an SR latch. When the first request arrives, the corresponding latch is set, and cross-coupled logic prevents other latches from being set. This way, only one latch output is high, indicating that the request has won.

[0104] Asynchronous arbiter 1223: For applications that are very sensitive to timing, asynchronous arbitration circuits can also be designed. They do not rely on the global clock and have faster response speed.

[0105] The key features and challenges of Arbitrator 1223 are:

[0106] Fairness and starvation problem: According to the arbitration strategy, it is necessary to consider whether all requests can be responded to in a timely manner.

[0107] Response time: The decision delay of the arbitrator 1223 itself needs to be small enough to meet the real-time requirements of the system.

[0108] Metastability: When multiple request signals arrive almost simultaneously, arbiter 1223 (especially latch-based ones) may enter a metastable state, resulting in uncertain or delayed output. Designing a robust arbiter 1223 requires addressing or mitigating metastability. In this solution, one of the PDL design goals is to minimize the arrival time differences between signals of different priority levels to reduce the probability of metastability.

[0109] Scalability: The arbitrator 1223 needs to be able to handle the expected number of request sources.

[0110] In summary, in this application, the programmable delay line 1222 (PDL) and the arbiter 1223 (Arbiter) form a key hardware decision-making subsystem. The PDL is responsible for converting the abstract layer "priority score" into a physical "time delay"; the arbiter 1223 quickly and efficiently selects the layer with the highest current priority based on these time delays (i.e., the order in which the signals arrive). This combination utilizes the characteristics of the time domain for calculations and decision-making, is very suitable for implementation in FPGAs, and provides hardware-level support for intelligent management of the IMMU.

[0111] This application uses PDL for dynamic priority decision making for video layer processing. The PDL unit can be composed of a cascade of lookup tables (LUTs) within the FPGA, which is implemented by configuring the LUTs as multiplexers or delay units and precisely controlling the routing delay. The arbiter 1223 can be based on a mutex latch (such as an SR latch) structure, responding to the first valid signal to arrive.

[0112] Based on the arbitration result and the current total system bandwidth load, the frame buffer data for that layer is stored in onboard DDR or in CXL memory via the CXL interface. A general strategy is to place high-priority, frequently accessed layers in DDR, where latency is potentially lower, and high-capacity, lower-priority layers in CXL memory.

[0113] In some embodiments, the resource allocation unit 1224 is configured to determine whether the frame buffer data of the layer is stored in the onboard DDR memory 20 or in the external CXL memory device 30 via the CXL controller 11 based on the arbitration result output by the arbiter 1223 and the current total bandwidth load of the system.

[0114] The resource allocation unit 1224 is further configured to place layers with high priority and access times exceeding a threshold in the onboard DDR memory 20 , and layers with low priority and capacity exceeding a threshold in the external CXL memory device 30 .

[0115] The adaptive processing strategy module 123 is used to compress or partially remove corresponding layers according to access priorities.

[0116] The above-mentioned adaptive processing strategy module 123 is also used to perform lossy compression on the target layer when the target layer has a low priority and is blocked by a high-priority layer and the total system bandwidth is insufficient, or when the target layer is completely opaque and completely blocked by its upper layer, decide not to write the invalid part of the data of this layer into any frame buffer.

[0117] The aforementioned adaptive processing strategy module 123 is also used to dynamically adjust the parameters of the video processing IP according to the processing load and layer characteristics.

[0118] For conditional compression / culling: combine the output of the priority arbitration module and the occlusion information of the layer characteristic analysis module 121.

[0119] If a layer has a low priority and is significantly obscured by a high-priority layer (for example, if more than a threshold percentage of pixels are covered, as determined by pixel-level occlusion), and the system's total bandwidth is limited, the adaptive processing strategy module 123 can instruct the video processing pipeline to perform moderate lossy compression (such as reducing bit depth, selectively dropping frames, or reducing resolution before scaling) on the layer before writing it to the frame buffer (whether DDR or CXL memory). This is more sophisticated than "Solution 1" in the background art.

[0120] If a layer is completely opaque and completely occluded by its upper layer, the adaptive processing strategy module 123 can decide not to write the invalid portion of data of this layer to any frame buffer, saving bandwidth and being more thoughtful because it is based on real-time, dynamic priority and occlusion judgment.

[0121] Dynamic IP Configuration and Measurement: The adaptive processing strategy module 123 can dynamically adjust the parameters of certain video processing IPs (such as layer scalers and compression units) based on processing load and layer characteristics, and can even selectively bypass certain processing units in more complex systems. For example, if a layer is determined to be significantly compressible, the compression IP parameters can be dynamically enabled or adjusted.

[0122] In this solution, DDR and CXL memory can be considered two distinct "storage layers." The IMMU manages the flow and processing of data between these layers through the FPGA's internal logic and the CXL interface. The IMMU records information ("measurements") such as the actual bandwidth used by each layer, the selected storage medium, and the compression strategy employed, for subsequent performance analysis or strategy optimization.

[0123] The above-mentioned adaptive processing strategy module 123 is also used for:

[0124] For the i-th layer, calculate its decision factor and select a processing strategy based on the decision factor.

[0125] For example, the decision factor is calculated using the following formula:

[0126]

[0127] Chinese expressions and explanations of various parameters in the formula:

[0128] D i : The "processing strategy application decision factor" for the i-th video layer. A larger value means a higher tendency to apply a "bandwidth-saving" processing strategy (such as compression, frame rate reduction, or resolution reduction) to this layer.

[0129] α i (Alpha value / transparency): Alpha value of the i-th layer, ranging from [0,1]. 0 means completely transparent, 1 means completely opaque. (1-α i ) represents the "visible contribution" of the layer. The smaller the Alpha value (the more transparent), the smaller its direct visual contribution to the final image may be, and its processing priority may also be reduced.

[0130] Bandwidth utilization: The actual usage percentage of the system's total storage bandwidth (DDR+CXL).

[0131] Target bandwidth utilization: A desired bandwidth utilization threshold set by the system (e.g. 80%). When the actual utilization exceeds this target, the system needs to take measures to reduce the bandwidth.

[0132] Layer score i: The "combined bandwidth demand / importance score" calculated by the "dynamic priority and resource allocation module 122" for the i-th layer. The higher the score, the more important the layer or the higher the bandwidth demand.

[0133] Average Layer Rating: The average of the Layer Ratings of all currently active layers. This is used for normalization, providing a relative comparison benchmark for individual layer ratings.

[0134] Occlusion ratio i: The percentage of pixel area of the i-th layer that is obscured by the opaque or semi-transparent layer above it, in the range [0,1]. 0 means completely unobstructed, and 1 means completely obscured.

[0135] K storage (storage medium adjustment factor): an adjustment factor given based on the storage medium (DDR or CXL memory) currently allocated to the i-th layer and its characteristics.

[0136] For example, if layer i is stored in DDR with lower latency but limited capacity, and DDR bandwidth is tight, K_Storage may be large, tending to apply policies to free up DDR resources.

[0137] If layer i is stored in CXL memory, which has larger capacity but slightly higher latency, and there is enough CXL bandwidth, K_storage can be smaller.

[0138] This factor can also take into account the energy consumption characteristics of different storage media.

[0139] β1, β2, β3 (weight coefficients): These are positive weight coefficients used to adjust the influence of different factors in the decision. These coefficients can be adjusted and optimized based on the specific application scenario and performance goals.

[0140] β1 adjusts the impact of "system overall bandwidth pressure". The larger β1 is, the tighter the system bandwidth is, and the larger the decision factor D is.

[0141] β2 adjusts the inverse effect of the layer's own importance. The higher the layer score (the more important it is), the smaller its -β2 power term is, the smaller the decision factor D is, and the less likely it is to apply the bandwidth-saving strategy to it.

[0142] β3 adjusts the influence of the “layer occlusion degree”. The higher the occlusion ratio, the larger the decision factor D, and the more likely the strategy is to be applied to it.

[0143] The creativity of the formula is reflected in:

[0144] Nonlinear fusion of multidimensional factors: This formula doesn't simply linearly add the factors together. Instead, it uses power functions and products to reflect the complex, nonlinear interactions between different factors. For example, when bandwidth utilization exceeds the target value, its impact increases dramatically as the β1 exponent increases.

[0145] Introduction of relative importance: By comparing the layer score with the average layer score, the decision is not only based on the absolute importance of the layer, but also takes into account its relative position among all current layers.

[0146] Storage media-aware decision making: The introduction of the K storage factor enables decisions to take into account the characteristics and load conditions of the actual data storage location, which is a direct response to hybrid storage architectures.

[0147] Collaborative consideration of transparency and occlusion: The transparency of the layer itself is also considered. i The occlusion ratio_i and the ratio of layers blocked by other layers more comprehensively evaluate the actual visual contribution of the layer. Even if a semi-transparent layer is partially blocked, its unblocked part still contributes.

[0148] Adjustable flexibility: By adjusting the weight coefficients β1, β2, and β3, the focus of the decision-making strategy can be flexibly changed to adapt to different application scenarios (for example, whether to prioritize the highest quality of all layers or prioritize the smoothness of key layers when bandwidth is extremely limited).

[0149] How to use the formula:

[0150] For each layer i, calculate its decision factor D i . One or more thresholds can then be set:

[0151] Threshold 1 (e.g., mild compression threshold): If D i >Threshold 1, a mild bandwidth saving strategy is applied to layer i (such as slightly reducing the bit depth or selectively discarding some non-key frames).

[0152] Threshold 2 (e.g., significant compression threshold, Threshold 2 > Threshold 1): If D i >Threshold 2, a more significant bandwidth saving strategy is applied to layer i (such as significantly reducing the resolution and then scaling, or using a lossy compression algorithm with a higher compression ratio).

[0153] Threshold_reject (e.g., completely reject / do not process thresholds much greater than threshold 2): If D i Beyond a very high threshold (for example, the layer is almost completely transparent and completely obscured, and the system bandwidth is extremely tight), consider not processing or transmitting the data for that layer at all (use with caution).

[0154] These thresholds themselves can also be dynamically adjusted, for example by learning and adjusting based on historical system performance data or user preferences.

[0155] This formula provides a quantitative and configurable decision framework, enabling the "adaptive processing strategy module 123" to make intelligent and informed bandwidth optimization decisions based on complex and dynamically changing system states and layer characteristics.

[0156] See Figure 4 , the system workflow is as follows:

[0157] Step 41: Video data streams #1, #2...#N are input into the FPGA.

[0158] Step 42: Each data stream undergoes pre-processing such as initial cropping and scaling.

[0159] Step 43: The layer characteristic analysis module of the IMMU obtains information of each layer.

[0160] Step 44: The dynamic priority and resource allocation module calculates a score for each layer and determines the storage policy (DDR / CXL) and access priority through the PDL and arbitrator.

[0161] Step 45: The adaptive processing strategy module instructs compression or partial elimination of certain layers as needed.

[0162] Step 46: The processed layer data is written to the selected frame buffer (DDR or CXL memory).

[0163] Step 47: The layer overlay module reads layer data from each frame buffer and performs overlay.

[0164] Step 48: Output the final synthesized video data.

[0165] Step 49: The DDR controller and CXL controller manage physical access to the DDR and CXL memories, respectively.

[0166] The beneficial effects are as follows:

[0167] a. Significantly improve effective bandwidth and storage capacity: By introducing CXL memory as an extension, the system's total available bandwidth and storage capacity far exceed solutions that rely solely on onboard DDR, enabling support for more and higher-definition video layers.

[0168] b. Intelligent bandwidth allocation and priority management: The IMMU utilizes hardware mechanisms such as the PDL and arbiter 1223 to dynamically allocate storage resources and access priorities in real time based on layer importance and system load, ensuring the processing performance and quality of critical layers.

[0169] c. Improved layer processing strategy: Compared with the simple compression or culling in the background technology, the adaptive processing strategy of the present invention is more sophisticated and flexible. It can target and minimize the impact on the quality of non-critical or obscured layers when bandwidth is tight, resulting in a better overall visual effect.

[0170] d. Enhanced system adaptability and configurability: The IMMU has the ability to dynamically adjust processing strategies and record operating status, enabling the system to better adapt to different application scenarios and video content, and providing data support for system optimization.

[0171] e. Maintaining a relatively simple hardware design: Compared to significantly increasing the number of onboard DDR chips, using CXL for memory expansion places less pressure on FPGA pins and PCB design, and is more cost-effective.

[0172] Furthermore, this application is expected to achieve the following technical effects through the combination and innovation of the above technical means:

[0173] a. Introduction and collaborative management of a hybrid storage architecture: For the first time, onboard DDR and external memory connected via the CXL interface (such as CXL Type 3 devices) are combined within FPGA video processing to form a unified, hierarchical hybrid storage system. This fundamentally differs from existing technologies that rely solely on DDR or simply consider DDR expansion.

[0174] b. Dynamic layer priority arbitration based on time-domain logic: The programmable delay line 1222 (PDL) and arbiter 1223 mechanism are innovatively applied to dynamic priority determination and resource allocation in multi-layer video processing. By mapping layer "importance scores" to time delays and utilizing the hardware arbiter 1223 for high-speed decision-making, this is a novel technical approach that differs from traditional priority management based on software or complex state machines.

[0175] c. Intelligent Multi-layer Memory Management Unit (IMMU) Design: The IMMU, a comprehensive management unit integrating layer characteristic analysis, dynamic priority arbitration, adaptive processing strategy decision-making, and DDR / CXL memory path selection, is the core of this invention. It integrates technologies from various fields (CXL memory technology, time-domain computing logic, and dynamic configuration) to form a unique solution for optimizing video processing bandwidth.

[0176] In summary, this application overcomes the limitations of existing technologies and achieves a breakthrough through the combination and innovation of the above-mentioned technical means: when facing the DDR bandwidth bottleneck, the existing technology has obvious defects in its solutions (increasing DDR, simple compression, simple elimination). This application is not a simple improvement on the existing solution, but proposes a new system architecture and management mechanism. When faced with the same problem, those skilled in the art usually consider optimizing DDR access scheduling, improving compression algorithms, or exploring more efficient DDR types, and will not directly think of combining CXL memory technology with time domain computing hardware (PDL / arbitrator 1223) from the field of machine learning for dynamic management of video layers and bandwidth optimization.

[0177] And a unique combination of technical means: Using CXL for FPGA video frame buffer expansion is novel in itself, but more importantly, how the intelligent multi-layer memory management unit 12 "intelligently" manages this extended memory. The use of PDL and arbiter 1223 for priority decision-making is not a prior art in the field of video processing. The application scenario of the relevant technology is low-complexity machine learning. Migrating it and applying it to real-time, data-intensive video layer management requires overcoming many technical obstacles (such as scoring mechanism design, close coupling with the video processing flow, performance requirements for hardware implementation, etc.), which are not easily conceived by those skilled in the art.

[0178] The intelligent multi-layer memory management unit 12 organically combines the "capacity / bandwidth advantages" of CXL memory with the "fast dynamic decision-making capabilities" and "flexibility of adaptive processing strategies" of the PDL / arbitrator 1223. The synergistic effect of this multi-technology integration enables the system to achieve results in bandwidth optimization, image quality preservation and dynamic adaptability that are difficult to achieve with existing single technologies or simple combinations.

[0179] Significantly improved technical effects: This application not only solves the problem of insufficient bandwidth, but also, through intelligent management, enables refined control and efficient utilization of system resources in complex video scenarios (such as multi-layer high-resolution and alpha blending). Its comprehensive improvement in performance, effects, and system flexibility exceeds the expectations of those skilled in the art based on existing technologies. For example, traditional priority schemes are typically based on fixed or software-updated weights, while the hardware PDL / arbitrator 1223 of this solution can achieve faster and more dynamic hardware-level priority response.

[0180] In summary, this application effectively solves the bandwidth bottleneck problem in existing FPGA multi-layer video processing through an innovative hybrid storage architecture, a unique dynamic priority arbitration mechanism based on time domain logic, and an intelligent adaptive processing strategy, and significantly improves the overall performance and flexibility of the system, with outstanding creativity and non-obviousness.

[0181] Further, the terms of this application are explained as follows:

[0182] FPGA (Field Programmable Gate Array): Field Programmable Gate Array.

[0183] DDR (Double Data Rate SDRAM): Double Data Rate Synchronous Dynamic Random Access Memory.

[0184] CXL (Compute Express Link): An open, high-speed interconnect standard for processors, memory expansion, and accelerators.

[0185] CXL.mem: A sub-protocol of the CXL protocol that allows the host processor to access memory connected to the CXL device through load / store instructions.

[0186] CXLType 3Device: A device type defined in the CXL specification, typically used as a memory expander or memory buffer pool.

[0187] PDL (Programmable Delay Line): Programmable delay line, a circuit that can change the signal transmission delay through programming.

[0188] Arbiter 1223 (Arbiter): A logic circuit used to resolve access conflicts between multiple requesters on shared resources.

[0189] IMMU (Intelligent Multi-layer Memory Unit): Intelligent multi-layer memory management unit, the core module proposed in this application.

[0190] LUT (Look-Up Table): Look-up table, the basic unit for implementing combinational logic in FPGA.

[0191] Alpha Blending: An image synthesis technology that controls the transparency of the image through the alpha channel value to achieve smooth transitions and overlays between layers.

[0192] In the several embodiments provided in this application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the modules or units is merely a logical functional division. In actual implementation, other division methods may be used, such as combining or integrating multiple units or components into another system, or ignoring or not implementing certain features.

[0193] If the integrated units in the above other embodiments are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0194] The above description is only an implementation method of the present application and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the description and drawings of this application, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. An intelligent multi-layer video storage bandwidth optimization system based on FPGA, characterized in that: The intelligent multi-layer video storage bandwidth optimization system includes: The FPGA main processing unit is responsible for receiving video data, cropping and scaling each layer, frame buffer read and write control, layer overlay, and video output; Onboard DDR memory for frame buffer; CXL controller, integrated via FPGA; an external CXL memory device connected to the CXL controller for frame buffering; The intelligent multi-layer memory management unit is implemented inside the FPGA and is used to analyze the layer characteristics of the video data. Based on the layer characteristic analysis results, a score is calculated for each layer. The storage strategy and access priority are determined through the PDL and arbitrator. Based on the access priority, the corresponding layer is compressed or partially eliminated. The processed layer data is written to the frame buffer selected by the storage strategy, and the corresponding layer data is read from each frame buffer for superposition, and the superimposed video data is output.

2. The intelligent multi-layer video storage bandwidth optimization system according to claim 1, characterized in that: The intelligent multi-layer memory management unit includes: A layer characteristic analysis module is used to perform layer characteristic analysis on video data to obtain the layer characteristic analysis result; a dynamic priority and resource allocation module, configured to calculate a score for each layer based on the layer characteristic analysis results, and determine a storage strategy and access priority through a programmable delay line and the arbiter; The adaptive processing strategy module is used to compress or partially eliminate the corresponding layer according to the access priority.

3. The intelligent multi-layer video storage bandwidth optimization system according to claim 2, characterized in that: The dynamic priority and resource allocation module includes: A scoring module, configured to calculate a score for each layer based on the layer characteristic analysis results; At least one programmable delay line, each of which is used to receive the score of the corresponding layer and output a delayed signal The arbiter is connected to the at least one programmable delay line and is used to determine the layer with the highest current priority according to the order in which the delayed signals arrive; The resource allocation unit is used to determine whether the frame buffer data of the layer is stored in the onboard DDR memory or in the external CXL memory device through the CXL controller based on the arbitration result output by the arbitrator and the current total bandwidth load of the system.

4. The intelligent multi-layer video storage bandwidth optimization system according to claim 3, characterized in that: The resource allocation unit is further configured to place layers with high priority and access times exceeding a threshold in the onboard DDR memory, and layers with low priority and capacity exceeding a threshold in the external CXL memory device.

5. The intelligent multi-layer video storage bandwidth optimization system according to claim 3, characterized in that: The programmable delay line is constructed by cascading lookup tables, or the programmable delay line is constructed by configuring a multiplexer through a lookup table and selecting a delay path using the multiplexer, or the programmable delay line is constructed by a dedicated delay unit, or the programmable delay line is constructed by wiring resources inside an FPGA.

6. The intelligent multi-layer video storage bandwidth optimization system according to claim 3, characterized in that: The arbiter is an asynchronous arbiter and / or is composed of a mutual exclusion latch array.

7. The intelligent multi-layer video storage bandwidth optimization system according to claim 2, characterized in that: The adaptive processing strategy module is further configured to perform lossy compression on the target layer when the target layer has a low priority and is obscured by a high priority layer and the total system bandwidth is insufficient, or to decide not to write invalid data of the layer into any frame buffer when the target layer is completely opaque and completely obscured by its upper layer.

8. The intelligent multi-layer video storage bandwidth optimization system according to claim 2, characterized in that: The adaptive processing strategy module is also used to dynamically adjust the parameters of the video processing IP according to the processing load and layer characteristics.

9. The intelligent multi-layer video storage bandwidth optimization system according to claim 2, characterized in that: The adaptive processing strategy module is further configured to: For the i-th layer, its decision factor is calculated, and a processing strategy is selected according to the decision factor.

10. The intelligent multi-layer video storage bandwidth optimization system according to claim 9, characterized in that: The decision factor is calculated using the following formula: Among them, D i Indicates the decision factor for the processing strategy of the i-th video layer, α i represents the Alpha value of the i-th layer, bandwidth utilization represents the actual usage percentage of the system's current total storage bandwidth (DDR+CXL), target bandwidth utilization represents an expected bandwidth utilization threshold set by the system, layer score i represents the score calculated for the i-th layer by the dynamic priority and resource allocation module, average layer score represents the average of the scores of all currently active layers, occlusion ratio i represents the percentage of pixel area of the i-th layer that is occluded by its upper opaque or semi-transparent layer, K storage represents the adjustment factor given according to the storage medium currently allocated to the i-th layer and its characteristics, β1, β2, and β3 are positive weight coefficients used to adjust the influence of different factors in the decision-making.

Images