Adaptive Reconstruction Method, System, and Media for AI Inference Data Preprocessing Pipeline

Abstract

This invention discloses an adaptive reconstruction method, system, and medium for AI inference data preprocessing pipelines, belonging to the field of artificial intelligence inference optimization technology. The method includes: a control layer receiving an inference task description and decoding it into a list of kernel functional requirements; generating an optimal kernel combination sequence and configuration parameter set based on a policy rule base; evaluating the occupancy of dynamically reconfigurable logical regions; and controlling the loading of target kernels into corresponding slots. The loaded kernels are then connected in series through a unified streaming data interface to form a pipeline, performing non-disk-based streaming step-by-step data processing. Adjacent kernels use elastic buffer units for data caching and rate matching. The data processed through the complete processing pipeline is output to the inference engine in a format conforming to the inference engine's input specifications for inference computation. This invention achieves dynamic, on-demand reconstruction and flexible combination of data preprocessing functions, eliminating intermediate data disk-based overhead and significantly reducing processing latency.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence inference optimization technology, and in particular to an adaptive reconstruction method, system and storage medium for AI inference data preprocessing pipeline. Background Technology

[0002] With the rapid development of deep learning inference technology, AI visual inference systems are widely used in key areas such as autonomous driving perception, medical image-assisted diagnosis, industrial quality inspection, security monitoring, and remote sensing data analysis. One of the core challenges faced by these applications is the deep contradiction between the diversity of input data and the strict constraints on the input specifications of the inference model.

[0003] On the inference side, different neural network models have precise and stringent requirements for the format of input data. Taking image classification models as an example, the ResNet series typically requires a 224×224 pixel RGB three-channel image as input, normalized with ImageNet mean and variance; the YOLOv8 object detection model requires a 640×640 square image as input; medical image segmentation models (such as U-Net and SegNet) often require a 512×512 grayscale image with specific histogram equalization processing; and remote sensing scene understanding models may process 4K or even higher resolution multispectral images. At the same time, in the trade-off between inference accuracy and hardware resources, the requirements for quantization accuracy vary in different scenarios. For example, high-precision medical applications tend to use FP32 or FP16 quantization, while edge device inference with extremely high real-time requirements relies on INT8 or even INT4 quantization to save computational resources.

[0004] On the acquisition side, the sources of raw data are extremely diverse: for example, RAW format Bayer array images output by industrial cameras, JPEG / H.264 compressed images transmitted over the network, DICOM format images from medical equipment, and 4K high frame rate video streams from cameras mounted on drones. Their resolution, color space, encoding format, and bit depth all differ significantly from the input format expected by the inference model. Therefore, before the raw data is fed into the inference engine, it must undergo a complex and multi-stage preprocessing process, which typically includes: data decoding and format conversion, ROI cropping, resolution scaling, color space conversion (RGB to BGR, RGB to YUV, grayscale, etc.), and pixel precision quantization and normalization.

[0005] Currently, AI inference data preprocessing mainly includes three solutions: pure software, GPU acceleration, and dedicated hardware, but all have significant shortcomings. For example, pure software solutions rely on the CPU to execute image processing algorithms such as OpenCV. Although development is flexible, preprocessing and inference share CPU resources, which can easily create performance bottlenecks in 4K / 8K high-resolution and high-frame-rate scenarios, resulting in high end-to-end latency. GPU acceleration solutions leverage CUDA / OpenCL to improve parallel processing capabilities, but GPU resources must simultaneously handle inference and preprocessing tasks, leading to resource contention. Additionally, data transfer between the host and device introduces additional PCIe transmission latency and consumes more power. While ASIC dedicated hardware solutions offer advantages in high throughput and low latency, their processing capabilities, quantization accuracy, and input specifications cannot be modified after tape-out. If multiple application scenarios are to be compatible from the initial design stage, it will inevitably lead to issues in power consumption, area, and timing, making it difficult to adapt to the rapid iteration of AI models and the deployment requirements of multiple scenarios.

[0006] Furthermore, existing solutions generally employ a "store-and-forward" model, where the results of each preprocessing stage must be written back to memory before being read by the next stage. This results in multiple memory accesses and data copies, significantly increasing system I / O latency and bandwidth pressure, and causing cache pollution and pipeline stalls. In multi-stage processing chains, I / O wait times can even exceed actual computation time, severely limiting the real-time performance of AI inference systems. Summary of the Invention

[0007] In view of this, the purpose of this invention is to address the shortcomings of the prior art by providing an adaptive reconstruction method, system, and medium for AI inference data preprocessing pipelines, which can adaptively reconstruct the data preprocessing pipeline according to the needs of AI inference tasks, eliminate intermediate data disk write overhead, have a sound flow control mechanism, and are easy to expand.

[0008] To achieve the above objectives, the present invention adopts the following technical solution:

[0009] Firstly, this application provides an adaptive reconstruction method for an AI inference data preprocessing pipeline, including:

[0010] The control layer receives the inference task description and decodes the inference task description into a list of kernel function requirements.

[0011] The control layer generates the optimal Kernel combination sequence and the configuration parameter set of each Kernel based on the Kernel functional requirement list and the predefined policy rule base.

[0012] The control layer assesses the current occupancy status of the dynamically reconfigurable logical region and determines the target kernel to be loaded and the loading timing from the kernel combination sequence;

[0013] The control layer controls the loading of the dynamic reconfigurable logic of the target Kernel into the corresponding Kernel slot and writes configuration parameters to the target Kernel.

[0014] The loaded kernels are connected in series through a unified data flow interface to form a pipeline, which processes the input data in a non-disk streaming manner step by step. Adjacent kernels use elastic buffer units to cache data and match rates.

[0015] The data, after being processed through the complete processing chain, is output to the inference engine in a format that conforms to the input specifications of the inference engine for inference calculation;

[0016] The processing throughput of each loaded kernel, the level of the elastic buffer units between the slots of each loaded kernel, and the frame rate performance metrics of the inference engine are continuously collected.

[0017] The performance metrics are fed back to the policy rule base for closed-loop optimization of subsequent kernel combination decisions;

[0018] The switching of each loaded kernel is triggered by at least one of the following events: forced reconfiguration request, automatic demand change detection, performance optimization trigger, and fault recovery trigger. Different events have different priorities, and higher priority events can preempt the switching process of lower priority events.

[0019] In one implementation, the forced reconfiguration request is triggered by a model switching command explicitly sent by the upper layer of the inference framework, the model switching command carrying a target model identifier;

[0020] The automatic demand change detection is triggered by the inference demand parser detecting that the kernel sequence required by a newly arrived inference task does not match the configuration of any currently loaded kernel in any kernel slot.

[0021] The performance optimization is triggered when the performance monitoring module detects that the processing throughput of the Kernel configuration loaded in any Kernel slot is continuously lower than a set threshold.

[0022] The fault recovery is triggered by an unrecoverable internal error occurring during the operation of a kernel loaded in any kernel slot.

[0023] Secondly, this application provides an adaptive reconstruction system for an AI inference data preprocessing pipeline, comprising:

[0024] The control layer is located in the static logic area of ​​the chip. The control layer is configured to generate a corresponding data processing kernel combination sequence and configuration parameter set according to the received inference task description, and control the dynamic loading and switching of the kernel.

[0025] The data processing layer is located in the dynamically reconfigurable logic area of ​​the chip. The data processing layer includes multiple kernel slots. Each kernel slot is used to load a data processing kernel. After loading, each kernel is connected in series through a unified data flow interface to form a pipeline. An elastic buffer unit is provided between adjacent kernels. The elastic buffer unit is used to cache intermediate data between kernels and realize data flow rate matching.

[0026] An inference engine layer is used to receive preprocessed data output by the data processing layer and perform inference calculations.

[0027] The Kernel library stores multiple dynamic reconfigurable logical files of available Kernels and metadata records corresponding to each Kernel.

[0028] The control layer communicates with the data processing layer through a standard interface. The control layer sends configuration parameters to each kernel and obtains running status information from each kernel and the inference engine layer.

[0029] The Kernel library supports online updates, and new Kernels can be queried and called by the system after registration.

[0030] The metadata record shall include at least one of the following fields: Kernel unique identifier, Kernel function type label, dynamic reconfigurable logic file path and checksum, dynamic reconfigurable logic file size, minimum area of ​​required logic region, input data bit width requirement, output data bit width requirement, maximum operating clock frequency, configuration parameter register definition table, version number and release timestamp.

[0031] The unified data flow interface adopts an interface protocol based on pipelined data transmission. Data is transmitted between adjacent kernels through point-to-point data paths. Data is processed sequentially by each kernel within the data processing layer. In the complete processing path within the data processing layer, the data is transmitted only through the data paths between kernels and the elastic buffer unit.

[0032] In one implementation, the control layer includes:

[0033] An inference requirement parser is used to receive an inference task description and decode the inference task description into a list of kernel function requirements.

[0034] The core combination strategy engine is used to generate the optimal Kernel combination sequence and the configuration parameter set of each Kernel based on the Kernel functional requirement list and the predefined strategy rule base.

[0035] The dynamic loading arbitrator is used to assess the current occupancy of dynamically reconfigurable logical regions, decide whether to trigger a reconfiguration operation, and select the target logical region and loading timing.

[0036] The configuration distribution bus is used to write the configuration parameters of each kernel into the corresponding register group;

[0037] The performance monitoring and feedback module is used to collect the processing throughput, elastic buffer level and frame rate performance metrics of each kernel and the inference engine, and feed the performance metrics back to the core combination strategy engine.

[0038] In one implementation, the inference task description includes at least one of model identifier, input resolution requirement, quantization accuracy requirement, and color format requirement;

[0039] The Kernel functional requirements list includes a description of the requirements for data processing function types;

[0040] The signal type, signal width, and timing requirements of the standard interface are fixed during the system design phase. The standard interface has consistent signal definitions and timing specifications for all loaded content in the dynamically reconfigurable logic region.

[0041] In one implementation, the elastic buffer unit is configured as follows:

[0042] Real-time monitoring of data backlog in the buffer;

[0043] When the data backlog exceeds a preset high water level threshold, a deceleration signal is sent to the upstream kernel, instructing the upstream kernel to reduce the output rate.

[0044] When the data backlog level is lower than a preset low water level threshold, the upstream kernel is allowed to output at the current rate;

[0045] When the data backlog in the buffer exceeds a preset time threshold and is not fully consumed, a timeout protection process is triggered. The timeout protection process includes: triggering reconfiguration optimization when the backlog level is at the first level, and performing controlled frame dropping and reporting the abnormal event to the control layer when the backlog level is at the second level.

[0046] Thirdly, this application provides a dynamic loading control device for AI inference data preprocessing kernel, comprising:

[0047] The Kernel library stores dynamic reconfigurable logical files for multiple available Kernels and metadata records corresponding to each Kernel.

[0048] A switching triggering unit is used to generate a switching instruction in response to a switching triggering event. The switching of the Kernel is triggered by at least one of the following triggering events: forced reconfiguration request, automatic demand change detection, performance optimization trigger, and fault recovery trigger. Different triggering events have different priorities, and high-priority events can preempt the switching process of low-priority events.

[0049] The pipeline evacuation control unit is used to send a pipeline evacuation command to the kernel to be switched before switching.

[0050] The logic loading control unit is used to read the dynamic reconfigurable logic file of the target Kernel from the Kernel library and write it to the target dynamic reconfigurable logic region through the configuration controller.

[0051] The initialization control unit is used to write the configuration parameters into the configuration register of the new kernel after the logic loading is completed, and to confirm that the new kernel is ready.

[0052] The recovery control unit is used to notify the upstream kernel to resume output, so that the streaming data interface of the new kernel can automatically connect with the upstream and downstream elastic buffers.

[0053] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method described above.

[0054] Compared with the prior art, the present invention has at least one of the following beneficial effects:

[0055] The adaptive reconfiguration method for AI inference data preprocessing pipeline in this application achieves end-to-end automated mapping and dynamic execution from inference tasks to hardware resource configuration through the technical processes of steps S1 to S6. Each step is linked by a kernel combination sequence and configuration parameter set, and uses a unified data flow interface as the data transmission channel, forming a closed-loop technical logic from requirement perception, strategy decision-making, resource arbitration, precise deployment, streaming execution, to inference output. Specifically, in step S1, the control layer decodes the inference task description, including model identifier, input resolution, quantization accuracy, and color format, into a structured list of kernel functional requirements through an inference requirement parser, enabling the system to accurately understand the upper-layer inference requirements. The functional requirements of the task for data preprocessing provide clear semantic input for subsequent hardware resource configuration. This differs from existing technologies that require manual configuration of preprocessing module parameters based on model specifications or reliance on preset templates, achieving automated transformation from the natural semantic layer to the functional requirement layer. In step S2, the core combination strategy engine generates the optimal kernel combination sequence and the configuration parameter set for each kernel based on the functional requirement list and a predefined strategy rule base (including kernel compatibility rules, resource constraints, and performance optimization objectives). This achieves multi-objective optimization mapping from the "functional requirement list" to the "hardware kernel combination sequence." The strategy engine uses the functional classes recorded in the metadata of the kernel library. Candidate kernels are filtered by type tags, and the matching of upstream and downstream data widths is verified by input / output bit width requirements. Resource feasibility is assessed by the minimum area of ​​the required logical region, ensuring that the generated combination sequence achieves the optimization goal of minimizing end-to-end latency or maximizing throughput under multiple constraints of functional completeness, interface compatibility, and resource feasibility. In step S3, the control layer evaluates the current occupancy of the dynamically reconfigurable logical region and decides the target kernel to be loaded and the loading timing from the kernel combination sequence. The dynamic loading arbitrator maintains a dynamic region resource occupancy table to record the current occupancy status of each dynamic sub-region, the type of loaded kernel, and the occupied area in real time. It combines the current combination sequence with the already loaded kernels. The configuration degree is determined by a triple judgment, which solves the problems of avoiding resource conflicts and optimizing loading timing in multi-kernel concurrent scenarios. In step S4, the control layer controls the loading of the dynamic reconfigurable logic of the target kernel into the corresponding kernel slot and writes the configuration parameters to the target kernel. After the logic loading control unit verifies the integrity of the bitstream file through the check code, it writes the dynamic reconfigurable logic into the target dynamic area through the internal configuration access port of the configuration controller. The configuration distribution bus accurately writes the configuration parameters such as the output frame size, processing parameters, and elastic buffer depth into the configuration register of the target kernel according to the configuration parameter register definition table in the metadata, realizing the safe deployment and parameter initialization of hardware logic in the dynamic area.This application replaces only a single kernel in the target slot without affecting the normal operation of other loaded kernels, refining the reconfiguration granularity from the overall pipeline to individual functional stages. The reconfiguration time is proportional to the logical area occupied by a single kernel, rather than the total area of ​​the entire pipeline, saving 60% to 80% of reconfiguration time compared to the overall replacement scheme. In step S5, the loaded kernels are connected in series through a unified data flow interface to form a pipeline, processing the input data in a non-disk-based streaming manner. Adjacent kernels use elastic buffer units for data caching and rate matching. The unified data flow interface adopts an interface protocol based on pipelined data transmission, allowing data to be directly transferred between adjacent kernels without writing to external storage media, completely eliminating the IO latency and bandwidth consumption introduced by the write wait and reread of each processing result in the traditional store-and-forward mode. The elastic buffer unit monitors the data backlog in real time through a water level sensing mechanism. When the water level exceeds a preset high water level threshold, it automatically sends a deceleration signal to the upstream kernel to control data injection from the source. To prevent overflow, the back pressure is automatically released when the water level falls below the low water level threshold, allowing the upstream to output at full speed and avoiding downstream idle waiting. A timeout protection mechanism triggers tiered degradation processing during persistent speed mismatches, thus simultaneously ensuring high throughput and reliability of the pipeline in two dimensions: "eliminating IO wait" and "maintaining link stability." In step S6, the data processed by the complete preprocessing link is output to the inference engine in a format conforming to the inference engine's input specifications for inference computation. Since this data has already undergone complete preprocessing, its resolution, quantization accuracy, and color format fully meet the inference engine's input requirements. The inference engine can directly perform inference computation without performing any additional data adaptation operations, eliminating the additional latency introduced by the inference engine's need to perform additional data format conversion in existing technologies. Furthermore, the inference engine's frame rate and other performance indicators can be fed back to the control layer through a standard interface, allowing the processing experience of each inference task to be transformed into optimization basis for subsequent kernel combination decisions. This enables the system to continuously converge to the optimal preprocessing configuration during long-term operation under different inference tasks, different data characteristics, and different load conditions. Information is transferred between each step through kernel combination sequences and configuration parameter sets, zero-copy data transfer is achieved through a unified data flow interface, and closed-loop optimization is achieved through performance feedback. Together, they construct a complete adaptive system and method that can independently complete the entire process of "understanding the task, formulating strategies, executing safely, evaluating the effect, and optimizing and iterating". Compared with the open-loop fixed configuration mode of existing technologies, it has significant technical progress in terms of reconfiguration granularity, processing latency, resource utilization efficiency, and long-term adaptability. Attached Figure Description

[0056] The preferred embodiments will now be described in a clear and easy-to-understand manner, in conjunction with the accompanying drawings, to further explain the above-mentioned characteristics, technical features, advantages, and implementation methods of the present invention.

[0057] Figure 1 A flowchart of a pipeline adaptive reconstruction method for AI inference data preprocessing provided in an embodiment of the present invention;

[0058] Figure 2 This is a structural block diagram of the pipeline adaptive reconstruction system for AI inference data preprocessing in an embodiment of the present invention;

[0059] Figure 3 This is a structural block diagram of the AI ​​inference data preprocessing kernel dynamic loading control device in an embodiment of the present invention;

[0060] Figure 4 This is a structural layout diagram of the pipeline adaptive reconstruction system for AI inference image preprocessing in an embodiment of the present invention;

[0061] Figure 5 This is a schematic diagram of the structure of some kernel libraries commonly used in image preprocessing in this embodiment of the invention;

[0062] Figure 6 This is a schematic diagram of the Kernel loading and switching process in an embodiment of the present invention;

[0063] Figure 7 This is a schematic diagram illustrating the process of image data being transferred between kernels in a streaming manner within the streaming processing architecture provided in this embodiment of the invention.

[0064] Figure 8 This is a schematic diagram illustrating the corresponding kernel switching and adaptation for changes in some requirements within the same scenario, provided by an embodiment of the present invention.

[0065] Figure 9 This is a schematic diagram illustrating flexible combination and dynamic adaptation of kernels for different scenarios, provided in an embodiment of the present invention.

[0066] Explanation of reference numerals in the attached figures:

[0067] 10-Control Layer; 110-Inference Requirement Parser; 120-Core Combinatorial Strategy Engine; 130-Dynamically Loading Arbitrator; 140-Configuration Distribution Bus; 150-Performance Monitoring and Feedback Module;

[0068] 20 - Data processing layer; 210 - Kernel slot; 220 - Unified streaming data interface; 230 - Flexible buffer unit;

[0069] 30 - Inference Engine Layer;

[0070] 40 - Kernel library; 41 - Metadata records.

[0071] 100 - Kernel Library; 200 - Switching Trigger Unit; 300 - Pipeline Drain Control Unit; 400 - Logic Load Control Unit; 500 - Initialization Control Unit; 600 - Recovery Control Unit. Detailed Implementation

[0072] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the specific implementation methods of the present invention will be described below with reference to the accompanying drawings. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings and other implementation methods can be obtained based on these drawings without any creative effort.

[0073] To keep the drawings concise, each figure only schematically shows the parts relevant to the invention, and these do not represent the actual structure of the product. Furthermore, to facilitate understanding, in some figures, only one of components with the same structure or function is schematically depicted, or only one is labeled. In this document, "one" not only means "only one," but can also mean "more than one."

[0074] It should also be further understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0075] In this document, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to fixed connections, detachable connections, or integral connections; they can refer to mechanical connections or electrical connections; they can refer to direct connections or indirect connections through an intermediate medium; and they can refer to the internal communication between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0076] Furthermore, in the description of this application, the terms "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0077] It should be noted that the above embodiments can be freely combined as needed. The above are merely preferred embodiments of the present invention. It should be pointed out that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

[0078] Existing technologies, such as the printing visual recognition and positioning system based on dynamic partial reconfiguration disclosed in CN121639441A, employ FPGA dynamic partial reconfiguration technology to achieve online switching of visual processing algorithms. In the hardware architecture of this solution, the internal logic resources of the FPGA chip are physically divided into a static system area and a dynamically reconfigurable functional area. The static system area has its functions permanently fixed during system operation and integrates a high-speed image acquisition interface module, a data flow scheduling and cache management module, a dynamic reconfiguration controller, and a system service and communication core module. The dynamically reconfigurable functional area acts as an algorithm container, dynamically loading corresponding dedicated visual processing algorithm hardware modules online according to different printing tasks (different materials, different markings).

[0079] The workflow of the existing technical solution is as follows: When a production task is switched, the task parsing and scheduling unit receives the new task instruction and parses out the corresponding target algorithm module identifier. The dynamic reconfiguration controller responds to the algorithm switching control signal, reads the target local configuration bitstream file from the configuration memory, and logically reconfigures the dynamically reconfigurable functional area, thereby achieving a complete switch from one visual processing algorithm to another. During the reconfiguration process, the data stream scheduling and cache management module pauses the input image data stream to the dynamic area and caches the input data; after the reconfiguration is completed, the data stream is routed to the reconfigured hardware logic module, and processing resumes.

[0080] While the aforementioned existing technical solutions achieve online switching of algorithms based on FPGA dynamic partial reconfiguration, they have the following technical limitations:

[0081] First, the dynamically reconfigurable functional area adopts a "whole replacement" working mode, requiring the entire image processing algorithm hardware module to be loaded into the dynamic area for each algorithm switch. This means that when only a certain functional link in the processing pipeline needs to be replaced (for example, only adjusting the quantization accuracy while keeping the scaling method unchanged), the entire algorithm module still needs to be replaced, resulting in long reconfiguration time and high resource consumption, making it difficult to adapt to real-time inference scenarios that require fast response.

[0082] Second, the dynamically reconfigurable functional area forms a complete processing pipeline optimized for a specific scenario after each reconfiguration. The functional combination of this pipeline is fixed in the bitstream file during the design phase. The system cannot independently adjust and flexibly combine the processing links in the pipeline according to the real-time requirements of the inference task (such as changes in model accuracy, input resolution, and preprocessing requirements), and lacks fine-grained adaptive capability for different AI inference tasks.

[0083] Third, the data stream processing adopts the traditional store-and-forward mechanism. The processing modules rely on external memory for data interaction. The processing results of each stage need to be written back to memory before being read by the next stage, resulting in multiple memory accesses and data copies, which generates significant IO latency and bandwidth pressure.

[0084] Fourth, the lack of effective cache management and backpressure control mechanisms makes it prone to data congestion and frame loss in multi-task switching or data burst scenarios, making it difficult to guarantee the real-time performance and data integrity of the system.

[0085] Therefore, please refer to the appendix. Figures 1-9 This application provides an adaptive reconstruction method, system, apparatus, and medium for AI inference data preprocessing pipelines, which can be used to solve the above-mentioned technical problems.

[0086] The following explanation is based on the accompanying diagram:

[0087] Reference Appendix Figure 1 This application illustrates a pipeline adaptive reconstruction method for AI inference data preprocessing provided by an embodiment of this application. The pipeline adaptive reconstruction method includes:

[0088] Step S1: The control layer receives the inference task description and decodes the inference task description into a kernel functional requirement list.

[0089] In this step, the inference requirement parser decodes the diverse inference task descriptions (such as model ID, resolution requirements, and quantization accuracy) into a structured list of kernel functional requirements. This enables the system to accurately understand the specific requirements of the upper-level inference task for data preprocessing, providing a clear input basis for subsequent hardware resource configuration.

[0090] Step S2: The control layer generates the optimal Kernel combination sequence and the configuration parameter set of each Kernel based on the Kernel functional requirement list and the predefined policy rule base.

[0091] In this step, the core combination strategy engine generates the optimal kernel combination sequence and the configuration parameter set of each kernel based on the functional requirement list and the predefined strategy rule library. Through the pre-configured kernel compatibility rules and resource constraints in the strategy rule library, it ensures that the generated combination sequence meets the feasibility constraints of hardware resources while maintaining full functionality, thus realizing intelligent mapping from "functional requirements" to "hardware kernel sequence".

[0092] Step S3: The control layer assesses the current occupancy status of the dynamically reconfigurable logical region and decides on the target kernel to be loaded and the loading timing.

[0093] In this step, the control layer assesses the current occupancy status of the dynamically reconfigurable logical region and decides on the target kernel and loading timing. By querying the dynamic region resource occupancy table in real time through the arbitrator, conflicting loading of already occupied regions is avoided, ensuring the safety of the reconfiguration operation.

[0094] Step S4: The control layer loads the dynamic reconfigurable logic of the target Kernel into the corresponding Kernel slot and writes configuration parameters to the target Kernel.

[0095] In this step, the control layer loads the dynamic reconfigurable logic of the target kernel into the corresponding kernel slot and writes the configuration parameters, realizing the precise deployment and parameter initialization of hardware logic in the dynamic area. Unlike the existing overall replacement mode of a single kernel without affecting related technologies (such as CN121639441A), this application only replaces other loaded kernels in the target slot, making the reconfiguration granularity refined from "overall pipeline" to "single functional link", significantly reducing reconfiguration time and resource overhead.

[0096] Step S5: After loading, each kernel is connected in series through a unified data flow interface to form a pipeline. The input data is processed step by step in a non-disk streaming manner. Adjacent kernels use elastic buffer units to cache data and match rates.

[0097] In this step, each loaded kernel is connected in series through a unified data flow interface to form a pipeline and performs non-disk streaming processing of data at each stage. Data is directly transferred between adjacent kernels without being written to external storage media, completely eliminating the IO latency introduced by the write wait and re-read of each processing result in the traditional store-and-forward mode. At the same time, the elastic buffer unit between adjacent kernels caches data in real time and implements bidirectional backpressure control based on water level awareness. When the processing speed of a certain kernel is lower than the upstream output rate, the elastic buffer unit automatically feeds back a deceleration signal to the upstream to prevent data overflow and frame loss. Thus, the high throughput and reliability of the pipeline are guaranteed in both dimensions of "eliminating IO wait" and "maintaining link stability".

[0098] Step S6: The data processed by the complete processing link is output to the inference engine in a format that conforms to the input specifications of the inference engine for inference calculation.

[0099] In this step, the data processed by the complete processing chain is output to the inference engine in a format that conforms to the input specifications of the inference engine. This allows the inference engine to directly obtain preprocessed data that meets its data format requirements, avoiding additional data adaptation operations on the inference engine side. At the same time, the performance indicators such as the frame rate of the inference engine can be fed back to the control layer in a closed loop to optimize subsequent kernel combination decisions.

[0100] On the one hand, this embodiment divides the data processing layer into multiple independent kernel slots, each of which can independently load the corresponding data processing kernel, thereby achieving "independent replacement of a single link" rather than "overall replacement", significantly shortening reconfiguration time and improving flexibility. On the other hand, an inference requirement parser and a core combination strategy engine are introduced to automatically decode AI inference tasks into a list of kernel functional requirements and generate the optimal combination sequence, realizing automated mapping from inference tasks to hardware resource configuration. Furthermore, each kernel is connected in series through a unified streaming data interface to form a zero-copy pipeline, and elastic buffer units are set between adjacent kernels to achieve bidirectional backpressure control, fundamentally eliminating the IO overhead of the store-and-forward mode and ensuring stable and reliable transmission of data streams.

[0101] More specifically, in a preferred embodiment of the present invention, the adaptive reconstruction method further includes:

[0102] Step S7: Continuously collect the processing throughput of each loaded kernel, the water level of the elastic buffer units between slots of each loaded kernel, and the frame rate performance metrics of the inference engine.

[0103] Among them, the processing throughput reflects the actual data processing speed of each kernel under the current configuration, and is used to identify the performance bottleneck in the pipeline (for example, when the throughput of the scaling kernel is lower than that of its upstream ROI splitting kernel, it indicates that the scaling kernel has become the bottleneck restricting the overall pipeline throughput). The level of the elastic buffer unit reflects the degree of real-time data backlog between adjacent kernels, and is used to quantify the severity of the mismatch in processing speed among kernels (for example, if the level is consistently high, it indicates that the upstream processing capacity far exceeds the downstream consumption capacity, and there is a risk of overflow; if the level is consistently low, it indicates that the upstream supply is insufficient and the downstream kernels are idle and waiting). The frame rate of the inference engine reflects the overall processing capacity of end-to-end preprocessing and inference, and is a direct measure of the final output performance of the system.

[0104] Step S8: Feed the frame rate performance metrics back to the policy rule base for closed-loop optimization of subsequent kernel combination decisions.

[0105] In this step, the feedback mechanism enables the core combination strategy engine to dynamically adjust subsequent kernel combination strategies based on the actual performance of the current pipeline. For example, when the performance monitoring module detects that the processing throughput of a certain kernel is consistently lower than the threshold corresponding to the target frame rate, the strategy engine can replace it with an optimized high-throughput version in subsequent decisions (such as replacing a bicubic interpolation scaling kernel with a bilinear interpolation scaling kernel to improve processing speed); when the elastic buffer unit level is consistently high, the strategy engine can adjust the rate matching parameters of adjacent kernels or reallocate kernel combinations in subsequent decisions to balance the processing capabilities of each stage; when the frame rate of the inference engine is detected to be decreasing, the strategy engine can comprehensively determine whether it is a bottleneck in the preprocessing stage or a change in the computing power requirements of the inference task itself, and trigger the corresponding optimization strategy accordingly.

[0106] In one embodiment of the present invention, the kernel switching is triggered by at least one of the following events: forced reconfiguration request, automatic requirement change detection, performance optimization trigger, and fault recovery trigger. The forced reconfiguration request is triggered by a model switching command explicitly sent by the upper layer of the inference framework. This model switching command carries a target model identifier. This mechanism enables upper-layer applications to directly drive hardware pipeline reconfiguration at the business level, providing the system with a highest-priority on-demand forced switching channel and ensuring that critical task switching is not interfered with by resource contention or delays in performance optimization logic.

[0107] Automatic requirement change detection is triggered by the inference requirement parser detecting a mismatch between the kernel sequence required by a newly arrived inference task and the configuration of the kernel already loaded in any kernel slot. This mechanism enables the system to automatically identify the differences between the current hardware configuration and the requirements of the new task based on the real-time parsing of the inference task description without explicit instructions from the upper layer, and to autonomously initiate reconfiguration accordingly. This avoids manual intervention or additional scheduling overhead from upper-layer applications in dynamic multi-task hybrid deployment scenarios (such as edge devices simultaneously receiving inference requests from different clients), and achieves system-level autonomous adaptation capabilities.

[0108] The performance optimization trigger is triggered when the performance monitoring module detects that the processing throughput of the currently loaded kernel configuration in any kernel slot is continuously lower than a set threshold. This mechanism enables the system to proactively initiate optimized reconfiguration when the inference task has not changed but the operating environment (such as fluctuations in input image resolution or data complexity causing changes in the actual kernel processing cycle) or hardware status (such as chip frequency reduction caused by temperature) changes. This works in conjunction with the closed-loop feedback mechanism in steps S7 and S8.

[0109] The performance monitoring module identifies bottlenecks and triggers reconfiguration based on elastic buffer level and throughput data. The policy engine then makes new decisions and loads a better kernel, thereby extending the system's adaptive capability from "passive response during task switching" to "proactive optimization during long-term operation," effectively solving the performance degradation problem caused by dynamic changes in workload.

[0110] The fault recovery mechanism is triggered by an unrecoverable internal error occurring during the operation of a kernel loaded in any kernel slot. This mechanism enables the system to quickly restart the slot and load a backup kernel or a different version with the same function when a kernel fails, thus avoiding the collapse of the entire inference pipeline due to a single point of failure. Compared with the existing technology where a fault causes the entire pipeline to stop or requires the entire FPGA to be reprogrammed, the fault recovery mechanism of this application greatly improves the reliability and serviceability of the system.

[0111] Different triggering events have different priorities. High-priority events can preempt the switching process of low-priority events. By assigning differentiated priorities to the four types of triggering events, when the system is executing a low-priority switching process, the arrival of a high-priority event can interrupt the current process and immediately respond to the high-priority request.

[0112] For example, if the system is performing a kernel replacement due to performance optimization, and the upper-layer application issues a forced reconfiguration request (the new model needs to be deployed immediately), the system can safely interrupt the current optimization reconfiguration (since the performance optimization trigger does not involve urgent data integrity requirements) and immediately respond to the forced reconfiguration, ensuring the real-time nature of critical business switching. This priority preemption mechanism works in conjunction with the aforementioned steps three (resource assessment and loading timing decision) and four (precise loading) to enable the switching arbitrator to make optimal decisions in multi-event concurrent scenarios: the arbitrator selects the target kernel and loading timing based on the priority of the triggering event, and high-priority events can obtain priority allocation rights for dynamic area resources, thereby ensuring the priority execution of critical switching tasks in a resource-constrained multi-task competition environment.

[0113] Reference Appendix Figure 2 Another embodiment of the present invention also provides a pipeline adaptive reconstruction system for AI inference data preprocessing, the pipeline adaptive reconstruction system comprising a control layer 10, a data processing layer 20, and an inference engine layer 30, wherein:

[0114] The control layer 10 is located in the static logic area of ​​the chip. The control layer 10 is configured to generate a corresponding data processing kernel combination sequence and configuration parameter set according to the received inference task description, and control the dynamic loading and switching of the kernel.

[0115] The data processing layer 20 is located in the dynamically reconfigurable logic area of ​​the chip. The data processing layer 20 includes multiple kernel slots 210. Each kernel slot 210 is used to load a data processing kernel. After loading, each kernel is connected in series through a unified data flow interface 220 to form a pipeline. An elastic buffer unit 230 is provided between adjacent kernels. The elastic buffer unit 230 is used to cache intermediate data between kernels and realize data flow rate matching.

[0116] The inference engine layer 30 is used to receive preprocessed data output from the data processing layer 20 and perform inference calculations;

[0117] The control layer 10 communicates with the data processing layer 20 through a standard interface. The control layer 10 sends configuration parameters to each kernel and obtains running status information from each kernel and the inference engine layer 30.

[0118] Specifically, the static zone deployment scheme ensures that the logical functions of the control layer 10 remain permanently fixed during system operation and are not affected by the reconfiguration operation of the dynamic zone (the static zone and the dynamic zone communicate through a predefined standard interface, the interface specification of which is fixed during the system design phase). This ensures that the system's "decision center" always operates stably in scenarios where the data processing layer 20 is frequently reconfigured, effectively avoiding system-level failures such as control signal jitter or configuration bus interruption caused by dynamic zone reconfiguration.

[0119] The design of multiple kernel slots in the data processing layer 20 allows it to flexibly load different numbers and types of kernels (such as ROI splitting, scaling, quantization, histogram processing, etc.) according to the needs of inference tasks. The serial relationship between the slots is standardized through a unified data flow interface (the input and output interfaces of each kernel follow the same signal definition, data width, and timing specifications). Any kernel that conforms to the interface specifications can be inserted into any slot without modifying adjacent modules, thus completely solving the inherent problems of high integration complexity and poor scalability of preprocessing modules in traditional solutions at the architectural level. The elastic buffer unit set between adjacent kernels is used to cache intermediate data between kernels and achieve data flow rate matching. This design allows the elastic buffer unit to temporarily cache the data output by the upstream kernel when there is a difference in processing speed between upstream and downstream kernels (for example, the processing speed of the upstream ROI splitting kernel is higher than that of the downstream scaling kernel). The elastic buffer unit can also feed back a deceleration signal to the upstream kernel through a water level sensing mechanism to prevent data overflow, thus ensuring the stable operation of the pipeline under speed mismatch conditions without the intervention of external storage media.

[0120] This is fundamentally different from the existing technology where the dynamic area is used as a "single algorithm container" and needs to be replaced as a whole each time it is switched, and the data interaction between processing modules depends on external memory.

[0121] The inference engine layer 30 receives the preprocessed data output by the data processing layer 20 and performs inference calculations. The preprocessed data it receives has undergone streaming processing of the complete processing chain, and the data format fully meets the input specification requirements of the inference engine. The inference engine layer 30 can directly perform inference calculations without performing any additional data adaptation operations, thereby eliminating the additional latency introduced by the inference engine having to perform additional data format conversion in traditional solutions.

[0122] The control layer 10 and the data processing layer 20 communicate via a standard interface. The control layer 10 sends configuration parameters to each kernel and obtains operational status information from each kernel and the inference engine layer 30. This bidirectional communication mechanism enables the control layer 10 to not only issue configuration commands downwards (such as switching kernels and adjusting processing parameters), but also to obtain real-time operational status information such as the processing throughput of each kernel in the data processing layer, the level of the elastic buffer unit, and the frame rate of the inference engine layer. This provides a communication foundation for the closed-loop feedback optimization in the aforementioned steps S7 and S8, enabling the system to possess complete logic of perception, decision-making, execution, and re-perception.

[0123] Therefore, after generating the kernel combination sequence based on the inference task description, the control layer 10 configures each kernel slot of the data processing layer 20 through a standard interface. After completing the data preprocessing according to the configuration, the data processing layer 20 outputs standard format data to the inference engine layer 30. After performing inference calculations, the inference engine layer 30 feeds back its performance indicators such as frame rate to the control layer 10 through the same standard interface. The control layer 10 then optimizes subsequent kernel combination decisions accordingly. This heterogeneous architecture design allows the system to balance stability (static area is not affected by reconfiguration), flexibility (dynamic area loads different kernels on demand), and scalability (unified interface supports hot-swapping of new kernels) on the same chip, representing a significant technological advancement in flexibility, reliability, and scalability compared to existing technologies.

[0124] More specifically, in a preferred embodiment of the present invention, the control layer 10 includes an inference requirement parser 110, a core combination strategy engine 120, a dynamic loading arbitrator 130, a configuration distribution bus 140, and a performance monitoring and feedback module 150, wherein:

[0125] Inference requirement parser 110 is used to receive inference task description and decode the inference task description into a Kernel functional requirement list;

[0126] The core combination strategy engine 120 is used to generate the optimal Kernel combination sequence and the set of configuration parameters for each Kernel based on the Kernel functional requirement list and the predefined strategy rule library.

[0127] The dynamic loading arbitrator 130 is used to evaluate the current occupancy of the dynamically reconfigurable logical region, decide whether to trigger a reconfiguration operation, and select the target logical region and loading timing.

[0128] The configuration bus 140 is used to write the configuration parameters of each kernel into the corresponding register group;

[0129] The performance monitoring and feedback module 150 is used to collect the processing throughput, elastic buffer level and frame rate performance indicators of each kernel and the inference engine, and feed the frame rate performance indicators back to the core combination strategy engine 120.

[0130] In one specific embodiment, the inference requirement parser 110 identifies parameters carried in the inference task description, such as model ID, input resolution requirements (e.g., 224×224, 640×640, or 512×512), quantization accuracy requirements (e.g., INT8, FP16, or FP32), and color format requirements (e.g., RGB, BGR, or grayscale), and transforms them into a list of kernel function requirements with defined semantics. This transforms the abstract description of the upper-level inference task into a hardware-executable kernel function sequence, solving the inherent problem in the prior art of lacking an automated mapping mechanism between inference task requirements and hardware preprocessing capabilities.

[0131] The core combination strategy engine 120 uses pre-configured kernel compatibility rules (such as the ROI splitting kernel output must match the input bit width of the scaling kernel), resource constraints (such as whether the remaining area of ​​the dynamic region is sufficient to accommodate the target kernel), and performance optimization objectives (such as minimizing end-to-end latency or maximizing throughput) in the strategy rule base to select the optimal kernel combination from the kernel library under the premise of satisfying functional integrity and resource feasibility. This achieves multi-objective optimization mapping from the functional requirement list to the hardware kernel combination sequence.

[0132] The dynamic loading arbitrator 130 makes a triple decision by maintaining a dynamic region resource occupancy table (recording the current occupancy status, loaded kernel type, and occupied area of ​​each dynamic sub-region) and combining the matching degree between the current kernel combination sequence and the loaded kernels:

[0133] Whether to trigger reconfiguration: No reconfiguration is needed when the target combination is completely consistent with the currently loaded kernel, thus saving time overhead; Which logical region to select: Find a suitable free region in the dynamic area based on the "minimum area of ​​the required logical region" of the target kernel; When to load: Select the switching time based on the current pipeline frame boundary state to ensure data integrity, solving the problems of avoiding resource conflicts and optimizing loading time in multi-kernel concurrent loading scenarios.

[0134] The configuration distribution bus 140 serves as the configuration execution channel between the control layer 10 and the data processing layer 20. It accurately writes the configuration parameters generated by the core combination strategy engine 120 into the configuration register of the target kernel in each kernel slot through a standard bus protocol (such as AXI4-Lite), ensuring that each kernel is in the correct parameter initialization state before executing the processing task.

[0135] The performance monitoring and feedback module 150 continuously collects the actual processing speed of each kernel, the real-time level of the elastic buffer unit, and the end-to-end frame rate of the inference engine. This enables the core combination strategy engine 120 to verify the effectiveness of its previous decisions based on the actual performance of the current pipeline and make targeted adjustments in subsequent decisions (for example, when it is detected that the throughput of the quantized kernel is much lower than expected, the strategy engine replaces it with an optimized high-throughput version or adjusts its configuration parameters in subsequent decisions). This achieves a complete closed-loop logic within the control layer 10, from decision-making, execution, evaluation to re-decision-making.

[0136] Therefore, the output of the inference requirement parser 110 (Kernel functional requirement list) serves as the input of the core combination strategy engine 120, and the output of the strategy engine (Kernel combination sequence and configuration parameters) serves as the input of the dynamic loading arbitrator 130 and the configuration distribution bus 140. After the arbitrator 130 ensures the safety of the reconfiguration operation, the configuration distribution bus 140 performs precise configuration. The performance data collected by the performance monitoring and feedback module 150 during the execution phase is then sent back to the strategy engine 120 through the feedback channel to form a closed loop.

[0137] More specifically, in a preferred embodiment of the present invention, the inference task description includes at least one of model identifier, input resolution requirements, quantization accuracy requirements, and color format requirements; the kernel functional requirements list contains a description of the requirements for data processing function types.

[0138] Specifically, in this embodiment, by establishing a precise mapping bridge from the semantic layer of the AI ​​inference task to the kernel hardware functional requirements layer, the control layer 10 can automatically derive the complete underlying kernel configuration requirements based on limited high-level parameter inputs: the model identifier is used to uniquely determine the type and version of the target inference model (such as YOLOv8, ResNet-50, U-Net). Since different models have fixed their input layer data specifications during training, for example, YOLOv8 requires a 640×640 RGB three-channel image, ResNet... ET-50 requires a 224×224 RGB three-channel image normalized with ImageNet mean and variance, while U-Net requires a 512×512 single-channel grayscale image with histogram equalization. Therefore, the model identifier is the primary index for the inference requirement parser 110 to derive the kernel functional requirements. By querying the preprocessing requirement mapping table associated with the model identifier, a set of baseline kernel functional requirements can be obtained (e.g., for U-Net, it is "grayscale conversion → histogram equalization → scaling to 512×512 → FP32 quantization").

[0139] The input resolution requirement specifies the width and height of the input image expected by the inference engine (e.g., 224×224, 640×640, or 512×512). This parameter directly determines the output frame size configuration of the scaling kernel (i.e., the target width and height register values ​​of the ResizeKernel) and also affects the granularity configuration of the ROI segmentation kernel. When the original image resolution is much higher than the target input resolution (e.g., a 4K×4K remote sensing image input to a semantic segmentation model that only requires 512×512), the ROI segmentation kernel needs to segment the original large image into multiple smaller images and set an appropriate overlap rate to ensure the spatial continuity of the segmentation results. However, when the original image resolution is close to the target input resolution (e.g., a 600×400 industrial camera output input to a 640×640 YOLOv8 model), ROI segmentation may not be necessary or only a small scaling is required, thus reducing unnecessary kernel loading and processing overhead.

[0140] The quantization precision requirement specifies the numerical precision format used for inference computation (such as INT8, FP16, or FP32). This parameter directly determines the type of quantization kernel to choose. The INT8 quantization kernel requires configuration of scaling factors and zero-point offsets to achieve fixed-point quantization, the FP16 quantization kernel requires configuration of floating-point format conversion logic, and the FP32 quantization kernel may perform pass-through processing. Different quantization precision choices also affect the bit width configuration of the elastic buffer unit (8-bit data path in INT8 mode, 16-bit path in FP16 mode, and 32-bit path in FP32 mode) and the subsequent computational resource allocation strategy of the inference engine.

[0141] The color format requirement specifies the input color space (such as RGB, BGR, or grayscale) expected by the inference engine. This parameter determines the type of color space conversion kernel and whether a grayscale kernel is needed. For example, when the original image is in RGB format but the inference model requires BGR format (such as the BGR order commonly used in OpenCV), an RGB-to-BGR conversion kernel needs to be configured in the preprocessing chain. When a color image is input to a grayscale model (such as some medical image segmentation models), an RGB-to-GRAY conversion kernel needs to be configured and appropriate color space transformation parameters need to be set.

[0142] The four types of parameters mentioned above are not isolated but rather have mutual constraints and linkages. For example, the model identifier determines the baseline input resolution and color format requirements, but the input resolution requirements can cover the model's default configuration to adapt to different inference accuracy-speed trade-off strategies (such as YOLOv8 accepting 416×416 input to improve inference speed, at the cost of a slight decrease in detection accuracy). The quantization accuracy requirements and color format requirements together determine the total bit width of the data path and the matching of data width between kernels (such as RGB three-channel × FP16 accuracy = 48-bit data width, while single-channel grayscale × INT8 = 8-bit data width, and the elastic buffer unit and unified data stream interface need to dynamically adapt their bit width configuration according to the actual combination).

[0143] The inference requirement parser 110, by comprehensively parsing these four types of parameters and identifying the linkage constraints between them, can completely transform the inference task description into a complete list of kernel function requirements, including "which functional types of kernels are needed (decoding, ROI segmentation, scaling, color space conversion, quantization, histogram processing, etc.), the specific values ​​of the configuration parameters of each kernel (output size, scaling ratio, quantization format, color matrix coefficients, etc.), and the data width matching relationship between kernels (bit width consistency verification and necessary bit width adaptation of upstream and downstream kernels)". This allows the core combination strategy engine to accurately retrieve and combine the required kernels in the kernel library based solely on this requirement list, thereby achieving fully automatic mapping from the high-level semantics of the inference task to the specific configuration of the hardware preprocessing pipeline. Compared with the existing technology, which requires manual configuration of preprocessing module parameters according to model specifications or reliance on preset templates for limited scene matching, the four-parameter linkage parsing mechanism of this application significantly improves the system's ability to quickly adapt to new models and new scenes.

[0144] Therefore, when deploying a new inference model, only the above four types of parameters need to be provided in the inference task description (or a pre-configured template associated with the model identifier) ​​and the system can automatically complete the entire configuration from requirement analysis to kernel combination, achieving accurate support for the preprocessing requirements of the new model without manual intervention.

[0145] Furthermore, in a preferred embodiment of the present invention, the static logic region and the dynamic reconfigurable logic region of the chip communicate through a predefined standard interface. The signal type, signal width, and timing requirements of the standard interface are fixed during the system design phase and do not change due to changes in the content of the dynamic reconfigurable logic region. This means that the electrical characteristics (level standard, drive capability, pull-up / pull-down configuration) and logical functions (signal effective level, signal timing relationship) of all control signals (such as configuration write enable, register selection, interrupt request), data signals (such as configuration data write, status data read) and handshake signals (such as VALID / READY handshake, frame boundary marker SOF / EOF) between the static and dynamic regions are determined during the design phase. Regardless of the kernel bitstream loaded in the dynamic region, the signal type on the interface remains unchanged, so that the control layer 10 of the static region does not need to perceive the specific functional changes of the internal logic of the dynamic region to complete the configuration distribution and status reading through the same set of signals.

[0146] The fact that the signal width is fixed during the system design phase means that the bit width of the data path between the static area and the dynamic area (such as 32-bit for the configuration bus, 32-bit for the status readback bus, and 512-bit for the streaming data interface) is fixed during the chip design phase and will not change due to the type of kernel loaded in the dynamic area (8-bit INT8 quantization kernel or 16-bit FP16 quantization kernel). This allows the configuration distribution bus 140 in the static area and the performance monitoring and feedback module 150 to interact with all dynamic area kernels in a unified bit width format, avoiding the data alignment logic complexity and additional bit width conversion overhead introduced by dynamic bit width changes.

[0147] The timing requirements being fixed during the system design phase means that the signal setup time, hold time, clock frequency, and clock domain cross synchronization strategy between the static and dynamic regions have been determined and verified by static timing analysis during the system design phase. Regardless of the configuration logic of the kernel loaded in the dynamic region, the interface timing meets the design constraints. This effectively avoids hardware reliability issues such as metastability and sampling errors caused by changes in interface timing margin due to changes in dynamic region logic.

[0148] Those skilled in the art will understand that in a dynamic partial reconfiguration system, if the interface specifications between the static and dynamic areas are not fixed during the design phase, the interface timing may change due to different layout and routing results after each dynamic area logic switch, requiring re-timing verification or even redesign of the static area logic, which is unacceptable in high-reliability application scenarios. While existing technologies use standard bus protocols such as AXI-Stream for communication interfaces between the static and dynamic reconfigurable functional areas, they do not explicitly decouple and fix the interface signal type, signal width, and timing requirements from the internal logic of the dynamic area during the design phase. Since the dynamic area loads a "complete algorithm hardware module" (including a complete pipeline for preprocessing and identification / positioning) each time, the interface specifications between different algorithm modules are inconsistent. Consistency relies on each module adhering to the same specification during design, rather than being enforced by system-level fixed interface constraints. This means that when modules developed by different teams or different versions of modules are connected, even minor inconsistencies in the interface can lead to system-level failures. In contrast, this application solidifies the interface specifications during the system design phase and forces all communication between the static and dynamic areas to be conducted through this predefined standard interface. This makes the logic of the static area completely transparent to the content of the dynamic area. Whenever it is necessary to update or expand system functions (such as registering a new image processing kernel in the kernel library), developers only need to ensure that the new kernel follows the standard interface specification (interface signal type matching, signal width matching, timing constraints are met) to connect to the system. The control layer logic of the static area does not need to be modified.

[0149] Compared to traditional solutions that require redesigning the static area or modifying control logic to adapt to differences in new module interfaces, the interface-fixed design of this application significantly reduces the complexity of system upgrades and expansions, and improves the maintainability and long-term evolution capability of the system in different deployment scenarios. At the same time, the existence of this standard interface allows each kernel to seamlessly access the pipeline through a unified data flow interface and configuration interface. Any kernel that conforms to the interface specification can be inserted into any kernel slot without affecting the normal operation of upstream and downstream modules. This provides physical and link layer deterministic guarantees for the aforementioned step S5, "each kernel is connected in series through a unified data flow interface to form a pipeline". As a result, the entire system can maintain the stable operation of the static area logic and the interconnection between each kernel in the dynamic area in scenarios where the dynamic area content changes frequently (multiple inference task switching), achieving a balance between stability and flexibility.

[0150] Furthermore, in a preferred embodiment of the present invention, the pipeline adaptive reconfiguration system further includes a Kernel library 40, which stores dynamic reconfigurable logic files of multiple available Kernels and metadata records corresponding to each Kernel; the Kernel library supports online updates, and new Kernels can be queried and called by the system after registration.

[0151] Therefore, by using the Kernel library 40 as a centralized storage and management platform, and by adopting a metadata-driven intelligent retrieval mechanism and online update capability, this application constructs a scalable and extensible hardware algorithm resource management system, enabling the system to have a new type of "plug-and-play" Kernel hot-swappable access capability, fundamentally solving the inherent problems of fixed hardware functions, difficulty in integrating algorithm modules, and poor system scalability in the existing technology.

[0152] Those skilled in the art will understand that if metadata is missing or incomplete, the policy engine will have to obtain kernel attributes through deep parsing of the bitstream file, which not only significantly increases decision latency but is also difficult to implement in practical engineering. The kernel library supports online updates, and new kernels can be queried and called by the system after registration. This online update mechanism allows the reconfigurable logic file and corresponding metadata records to be directly registered into the library after a new data processing algorithm is developed and verified by functional simulation and logic synthesis without system downtime or redeployment. Moreover, the registration process does not affect the currently executing inference task (the data processing layer continues to process data with the existing kernel configuration). After registration, the new kernel can be queried and selected by the policy engine in subsequent kernel combination decisions.

[0153] like Figure 5As shown, Kernel Library 40 is a centralized storage and management module for all available Kernel dynamic reconfigurable logical files.

[0154] The kernel library 40 is stored in non-volatile memory (such as Flash, SD card, or solid-state drive). Each kernel entry is described in the kernel library as a metadata record 41, containing the following fields:

[0155] Kernel_ID: A unique identifier for the kernel, such as "KRNL_RESIZE_001";

[0156] Kernel feature type tags: such as "scaling", "ROI splitting", "quantization", "histogram equalization";

[0157] Dynamically reconfigurable logical file path and checksum: such as " / kernels / resize_001.bit", the checksum uses CRC32 or SHA-256 algorithm;

[0158] Dynamically reconfigurable logical file size: in bytes;

[0159] Minimum area of ​​the required logical region: expressed in terms of the number of lookup tables (LUTs) or the number of equivalent logical units;

[0160] Input data bit width requirements: such as "24-bit RGB", "8-bit grayscale";

[0161] Output data bit width requirements: such as "24-bit RGB", "32-bit floating point";

[0162] Maximum operating clock frequency: e.g., "300MHz";

[0163] Configuration parameter register definition table: describes the type, address offset, and value range of all configurable parameters of this kernel;

[0164] Version number and release timestamp.

[0165] Kernel library 40 supports online updates. Newly developed kernels, after functional simulation verification and logic synthesis, are registered into the library as dynamic reconfigurable logic files. The registration process includes: writing the reconfigurable logic file to the storage medium, adding a metadata record 41 to Kernel library 40, and performing integrity verification. The entire registration process requires no system downtime, and the system can select the new kernel during the next reconfiguration scheduling.

[0166] Figure 3The document lists some common kernels used in image preprocessing: image decoding kernel (supports JPEG, PNG, etc.), ROI segmentation kernel, scaling kernel (supports nearest neighbor, bilinear, bicubic interpolation), color space conversion kernel (RGB to BGR, RGB to YUV, RGB to grayscale, etc.), quantization kernel (supports INT8, FP16, FP32 precision, etc.), histogram processing kernel (histogram equalization, contrast enhancement), and normalization kernel. User-defined kernels can be encapsulated according to unified encapsulation requirements and registered in the kernel library.

[0167] Furthermore, in a preferred embodiment of the present invention, the metadata record 41 includes at least one of the following fields: Kernel unique identifier, Kernel function type label, dynamic reconfigurable logic file path and checksum, dynamic reconfigurable logic file size, minimum area of ​​required logic region, input data bit width requirement, output data bit width requirement, maximum operating clock frequency, configuration parameter register definition table, version number and release timestamp.

[0168] Among them, the unique identifier of the kernel and the function type label enable the strategy engine to accurately locate all candidate kernels in the kernel library based on the function requirement list output by the inference requirement parser 110, and combine the version number and timestamp to realize multi-version selection and gray-scale upgrade; the file path and checksum ensure that the logic loading control unit accurately locates the bit stream and verifies the file integrity before loading to prevent corrupted data from being written to the dynamic area; the minimum area of ​​the required logical area supports the arbitrator to quickly complete the region adaptation decision by comparing the current dynamic area's free resources (avoiding loading failure or resource fragmentation problems due to insufficient area); the input and output bit width must be... The strategy engine should be able to verify the data width matching between upstream and downstream kernels and automatically insert a conversion kernel or select an alternative when a mismatch occurs. Simultaneously, when a single-slot kernel is replaced with a new kernel, the unified streaming data interface should automatically adapt to the new bit width configuration, achieving automatic data path adaptation after a single pipeline stage replacement rather than manual adjustment. The maximum operating clock frequency should assist the arbiter in evaluating the timing feasibility of the target kernel under the current chip operating conditions and, if necessary, perform frequency reduction processing through the clock management unit to avoid timing violations. The configuration parameter register definition table should provide a complete configuration distribution bus. The configuration instruction generation basis (including parameter type, address offset, and value range) enables the bus to complete precise register initialization without needing to understand the internal implementation details of the kernel. Simultaneously, in performance optimization-triggered reconfiguration, it supports lightweight reconfiguration by modifying only configuration parameters rather than replacing the entire kernel. The complementary information among the above ten types of fields constitutes the complete data foundation for the decision-making link between the policy engine 120, the arbitrator 130, and the configuration distribution bus 140. The policy engine relies on identifiers, function tags, version numbers, input / output bit widths, and register definition tables to complete kernel retrieval and combination optimization. The arbitrator relies on… The system completes the loading feasibility assessment and region selection based on file path, checksum, file size, required area, and maximum clock frequency. The configuration bus relies on the register definition table to complete the configuration sequence generation. Compared with the existing technology where the configuration memory stores bit streams at preset addresses without structured metadata description, and the kernel selection relies on a simple mapping of preset identifiers and file names without a structured description of functional attributes, resource requirements, and interface specifications, this application realizes full-link automated decision-making from kernel retrieval to deployment through the complete definition of ten types of fields, enabling the system to have standardized, scalable, and traceable hardware resource management capabilities.

[0169] Furthermore, in a preferred embodiment of the present invention, the unified data flow interface 220 adopts an interface protocol based on pipelined data transmission, and data is directly transmitted between adjacent kernels; the data is processed sequentially through each kernel in a streaming manner within the data processing layer 20, without needing to pass through an external storage medium throughout the process.

[0170] Therefore, this application establishes an end-to-end zero-copy streaming data path within the data processing layer, fundamentally eliminating the multiple memory accesses, data copies, and IO wait delays introduced by the traditional store-and-forward mode, where the processing results of each level are written back to the external storage medium and then read by the next level. Direct data transfer between adjacent kernels means that after the first kernel (such as the ROI segmentation kernel) completes the processing of a row or a pixel block, it can immediately push it to the second kernel (such as the scaling kernel) through a unified data flow interface to start processing, without waiting for the entire frame to be processed before writing it to the off-chip DDR. This streaming design, which uses pixel words as the basic transmission unit, data lines as the intermediate granularity, and frame boundary markers (SOF / EOF) as the complete semantic unit, allows multiple kernels to completely overlap in the time dimension.

[0171] Those skilled in the art will understand that in 4K / 8K high-resolution real-time inference scenarios, the amount of intermediate data in each frame can reach tens of megabytes. If the preprocessing results at each stage need to be written back to off-chip DDR and then read back, each write to disk and read back requires competition and scheduling of PCIe or memory controller bandwidth. The additional latency and bandwidth consumption introduced as a result can be several times the actual computation time. However, this application constrains the data path of the complete processing link to a closed path of "Kernel, unified data stream interface, elastic buffer unit, unified data stream interface, next Kernel", enabling the system to complete the entire preprocessing process without the intervention of external storage media.

[0172] Furthermore, in a preferred embodiment of the present invention, the elastic buffer unit 230 is configured as follows:

[0173] Real-time monitoring of data backlog in the buffer;

[0174] When the data backlog exceeds a preset high water level threshold, a deceleration signal is sent to the upstream kernel, instructing the upstream kernel to reduce the output rate.

[0175] When the data backlog level is below a preset low water level threshold, the upstream kernel is allowed to output at the current rate.

[0176] Specifically, by using the elastic buffer unit 230 to monitor the data backlog in real time and trigger deceleration signals and allow normal output based on high / low water level thresholds respectively, this application establishes an adaptive bidirectional flow regulation mechanism based on water level perception between adjacent kernels, enabling the system to automatically match the speed difference between upstream and downstream kernels without external intervention.

[0177] When the processing rate of an upstream kernel (such as an ROI splitting kernel) is higher than the consumption rate of a downstream kernel (such as a bicubic interpolation scaling kernel), data gradually accumulates in the elastic buffer unit. Once the water level exceeds a preset high water level threshold, the elastic buffer unit immediately sends a deceleration signal to the upstream kernel (by invalidating the READY signal in the VALID / READY handshake protocol or reducing its effective frequency). The upstream kernel then reduces its output rate accordingly, thereby controlling the amount of data injected from the source to prevent buffer overflow and data loss.

[0178] When the water level drops below the low water level threshold due to upstream deceleration or downstream acceleration, the elastic buffer unit releases the back pressure (restoring the effective frequency of the READY signal), allowing the upstream kernel to output at full speed to avoid the downstream kernel being idle due to insufficient data supply. This mechanism, combined with the zero-copy streaming transmission of the aforementioned unified data stream interface, enables the pipeline to maintain continuous and stable operation under speed mismatch conditions without the need for large-capacity external frame buffers between kernels (the buffer depth can be configured to buffer only a few image lines instead of the entire frame, significantly reducing storage resource overhead). Furthermore, it completely eliminates the need for manual configuration of rate matching parameters or reliance on external controller intervention, offering significant advantages in pipeline stability and resource utilization efficiency compared to existing technologies.

[0179] Furthermore, in a preferred embodiment of the present invention, the elastic buffer unit 230 is further configured to trigger timeout protection processing when the data backlog in the buffer exceeds a preset time threshold and is still not fully consumed; the timeout protection processing includes: triggering reconfiguration optimization when the backlog level is at the first level, and performing controlled frame dropping and reporting abnormal events to the control layer when the backlog level is at the second level.

[0180] Compared to data corruption or pipeline deadlock caused by buffer overflow without a timeout protection mechanism (the downstream kernel waits for input while the upstream stops output due to back pressure, and the system enters a dead state that cannot recover on its own), the hierarchical timeout protection in this application solves structural rate mismatch through "reconfiguration optimization" and ensures the continuity of the system's frame rate and overall serviceability in extreme cases by sacrificing the integrity of single frame data through "controlled frame dropping," so that the system still has self-healing ability when facing unpredictable load fluctuations or local failures.

[0181] like Figure 6 As shown, in the streaming processing architecture of this embodiment of the invention, image data is transmitted between kernels in a streaming manner.

[0182] The basic transmission unit of a data stream is a pixel word. Each pixel word contains data of several consecutive pixels and position indication information. The position indication information includes: the column index of the current pixel in the row, the row index of the current row in the frame, the row end marker, and the frame end marker.

[0183] A data line consists of several consecutive pixel words. At the line boundary, the system notifies the downstream kernel that the current line has been transmitted by an end-of-line (EOL) marker.

[0184] An image frame consists of several lines of data. The start of a frame is marked by a Start of Frame (SOF) marker, and the end of a frame is marked by an End of Frame (EOF) marker.

[0185] Each kernel is connected to a flexible buffer unit 230. The flexible buffer unit 230 is implemented by a dual-port RAM and has a configurable depth (by default, it supports caching 2-4 lines of image data).

[0186] The elastic buffer unit 230 monitors its data backlog (water level) in real time. Water level monitoring is achieved through a read / write pointer counter, and the difference between the write pointer and the read pointer is the amount of data in the current buffer.

[0187] When the water level exceeds a preset high water level threshold (e.g., 80% of the buffer depth), the back pressure control logic sends a deceleration signal to the upstream kernel (invalidating the READY signal or reducing its effective frequency), and the upstream kernel reduces its output rate accordingly. When the water level is below a low water level threshold (e.g., 20% of the buffer depth), the back pressure control logic allows the upstream kernel to output at full speed (the READY signal remains valid).

[0188] This water level-aware adaptive flow regulation mechanism ensures the stable operation of the entire streaming link under conditions of varying kernel processing speeds.

[0189] In addition, the elastic buffer unit 230 is equipped with a timeout protection mechanism. Each buffer has a timeout counter, which triggers a timeout interrupt when the buffer remains non-empty for more than a preset threshold (e.g., 10 milliseconds). The control layer performs degradation processing based on the backlog level: for minor backlog, it triggers reconfiguration optimization (switching to a higher throughput alternative kernel); for severe backlog, it performs controlled frame dropping (discarding part or all of the data in the current frame) and reports the abnormal event to the control plane.

[0190] More specifically, in a preferred embodiment of the present invention, the kernel switching is triggered by at least one of the following triggering events, which are arranged in order of priority:

[0191] Forced reconfiguration requests are triggered by model switching commands explicitly sent by the upper layer of the inference framework; automatic requirement change detection is triggered by the inference requirement parser 110 detecting that the kernel sequence required by a newly arrived inference task does not match the current loaded configuration; performance optimization is triggered by the performance monitoring module detecting that the processing throughput under the current kernel configuration is continuously lower than a set threshold; fault recovery is triggered by an unrecoverable internal error occurring during kernel operation.

[0192] More specifically, in a preferred embodiment of the present invention, the kernel switching process includes:

[0193] Data boundary alignment before switching: Send a pipeline emptying command to the kernel to be switched so that the kernel to be switched will no longer receive new input data after completing the processing of the current frame;

[0194] Upstream data flow pause: Send a pause output command to the upstream Kernel of the Kernel to be switched, so that the upstream Kernel stops generating new output data;

[0195] Dynamic reconfigurable logic loading: Read the target Kernel's dynamic reconfigurable logic file from the Kernel library and write it to the target dynamic reconfigurable logic region;

[0196] New Kernel Initialization: Write the configuration parameters to the new Kernel's configuration register and confirm that the new Kernel is ready;

[0197] Interface re-establishment and data flow recovery: Notify the upstream kernel to resume output, and the new kernel's streaming data interface will automatically connect with the upstream and downstream elastic buffers;

[0198] Post-switch verification: Monitor whether the processing output of the new Kernel is normal. If the verification is successful, the switch is considered successful.

[0199] More specifically, in a preferred embodiment of the present invention, the number of each kernel slot in the data processing layer and the type of kernel loaded in the kernel slot can be configured and adjusted according to the actual application scenario and hardware capabilities.

[0200] like Figure 6 As shown, the kernel loading and switching process is divided into six stages, each of which is executed in strict sequence.

[0201] Phase 1: Data boundary alignment before switching.

[0202] The control layer sends a "pipeline drain" command to the kernel to be reconfigured. This command is written to the kernel's control register via the configuration interface. Upon receiving the drain command, the kernel to be reconfigured completes processing of the current frame, stops accepting new input data, and pushes all processed data into the downstream elastic buffer. The control layer confirms the drain is complete by reading the kernel's status register.

[0203] For example, suppose the current Slot 1 is loading a bicubic interpolation scaling kernel, and it needs to be switched to a bilinear interpolation scaling kernel. The control layer sends a drain command to the current kernel in Slot 1. After the kernel completes the scaling processing of the current frame, it stops working and pushes the processing result into the elastic buffer between Slot 1 and Slot 2.

[0204] Phase Two: Upstream data flow is suspended.

[0205] The control layer sends a "pause output" command to the upstream kernel of the kernel to be reconfigured. After completing the transmission of the current data packet, the upstream kernel stops generating new VALID pulses and enters a waiting state. At the same time, the elastic buffer between the kernel to be reconfigured and the upstream kernel has been cleared by the drain operation in Phase 1, ensuring that the reconfiguration area enters a data-free state.

[0206] Continuing the example above, the control layer sends a pause output command to the ROI segmentation kernel of Slot 0. After the kernel of Slot 0 completes the transmission of the current data packet, it pauses output. The elastic buffer between Slot 0 and Slot 1 is now empty.

[0207] Phase 3: Dynamic reconfigurable logic loading and logical region reconfiguration.

[0208] The arbiter reads the target kernel's dynamic reconfigurable logic file from the kernel library and writes the dynamic reconfigurable logic into the target dynamic reconfigurable logic area by configuring the controller's internal configuration access port (such as ICAP or PCAP interface).

[0209] The dynamic reconfigurable logic write process is guaranteed by the configuration controller hardware. During the write, the logical function of the target area is in an undefined state, but through static and dynamic area isolation mechanisms (such as DFX Decoupler), the process is completely transparent to other logical areas and data paths, without causing any interference.

[0210] Typical reconfiguration loading time is proportional to the area of ​​the target region. For a medium-sized kernel (approximately 10K LUTs), the loading time is approximately 10-50 milliseconds with standard configuration interface bandwidth.

[0211] Phase 4: New Kernel initialization and configuration parameter writing.

[0212] After the dynamic reconfigurable logic is loaded, the new kernel's logic functions are ready, but all internal registers are in a default reset state. The control layer then writes all configuration registers of the new kernel sequentially through the configuration interface, according to the pre-generated set of configuration parameters.

[0213] Configuration parameters include: output frame size (such as width and height), processing parameters (such as scaling ratio, quantization coefficient, color matrix coefficient, etc.), elastic buffer depth, timeout threshold, backpressure strategy, etc.

[0214] After the write operation is complete, the control layer reads the kernel's STATUS register to confirm its readiness status. A "ready" flag in the STATUS register being 1 indicates that the kernel is ready.

[0215] Phase 5: Interface re-establishment and data flow recovery.

[0216] Once the new kernel is confirmed to be ready, the control layer notifies the upstream kernel to resume output, and the data stream is re-established. The new kernel's streaming data interface automatically connects with the upstream and downstream elastic buffers, and the handshake mechanism (VALID / READY) takes effect immediately. The data stream begins normal transmission from the first pixel of the next frame.

[0217] Because of the unified interface protocol, upstream and downstream modules can seamlessly interoperate with the new kernel without any modifications. This demonstrates the core advantage of a unified streaming data interface—any kernel conforming to the interface specification can be inserted into any slot in the pipeline.

[0218] Phase Six: Post-Switch Verification and Alarm Clearing.

[0219] After the data stream is restored, the monitoring module of the control layer continuously observes whether the processing output of the new kernel is normal. Verification indicators include: whether the frame rate reaches the expected value, whether the output data width is correct, and whether pixel statistics (such as mean and variance) are within a reasonable range.

[0220] If the verification passes, the switchover is considered successful, and the relevant alarm status is cleared. If the verification fails (e.g., abnormal output frame format or sudden drop in frame rate), a switchover failure interruption is triggered, and the rollback process is started—reloading the original kernel and restoring it to the state before the switchover.

[0221] Figure 8 The diagram illustrates the verification of the present invention's kernel switching adaptation to changes in some preprocessing requirements within the same inference scenario. Figure 5Taking the quantization precision switching from INT8 to FP16 as an example, this application demonstrates how, when only the requirements of a single preprocessing stage change during model inference (such as adjusting the quantization precision due to increased precision requirements or changes in hardware resource status), this application can quickly and accurately replace a single kernel without changing other logic. This verifies the switching efficiency, data security, and end-to-end functionality of the system and method described in this invention in the scenario of "independent replacement of a single stage".

[0222] In practical AI inference deployments, quantization accuracy is not static. The same inference task may have different trade-offs between accuracy and speed under different operating environments. For example, at the edge, INT8 quantization is used to improve throughput due to power consumption and bandwidth limitations. However, in some complex scenarios (such as low-light conditions or densely packed small targets), the accuracy loss caused by INT8 quantization may be unacceptable, requiring a temporary switch to FP16 quantization to ensure detection accuracy. Similarly, when chip temperature rises or power supply voltage fluctuations reduce timing margins, it may be necessary to switch from FP16 to INT8 to reduce the power consumption and heat generation of the computing unit. In traditional solutions, if ASIC is used, the quantization accuracy cannot be changed after tape-out. If FPGA is used, the entire preprocessing pipeline needs to be resynthesized and reprogrammed, with switching times ranging from several seconds to tens of minutes. Figure 5 The core technical problem to be verified is: when only the quantization accuracy needs to be changed while keeping other preprocessing steps such as ROI segmentation and scaling unchanged, can the system complete the accurate replacement of a single kernel within a few milliseconds to tens of milliseconds without interfering with other running kernels or losing image data?

[0223] like Figure 8 As shown, in the initial state, the system's data processing layer 20 contains three kernel slots: Slot 0 loads the ROI segmentation kernel, Slot 1 loads the scaling kernel, and Slot 2 loads the INT8 quantization kernel. These three kernels are connected in series through a unified data flow interface 220 to form a complete preprocessing pipeline. The raw image is sequentially processed through ROI segmentation, scaling, and INT8 quantization before being output to the inference engine. At this point, the quantization precision requirement specified in the inference task description is INT8, the system is in a steady-state operation, and the video stream is continuously input and processed normally.

[0224] When the requirements of the inference task change, for example, if the upper-layer application detects that the current scene complexity is too high and the accuracy loss caused by INT8 quantization exceeds the acceptable range, and decides to increase the quantization accuracy from INT8 to FP16, the control layer completes the single-slot precise replacement of the quantization kernel through the following steps:

[0225] Step S501: The control layer receives the quantization precision switching instruction and parses the requirements.

[0226] The inference requirement parser 110 receives a new inference task description (in which the quantization precision requirement changes from INT8 to FP16, while the model identifier, input resolution requirement, and color format requirement remain unchanged), and decodes it into a list of kernel feature requirements. The core combinatorial strategy engine compares the new requirement list with the currently loaded kernel configuration, identifying that only the quantization kernel type of Slot 2 needs to be changed (replaced from INT8 to FP16), while the configurations of the ROI splitting kernel of Slot 0 and the scaling kernel of Slot 1 remain unchanged. Based on this, a new kernel combinatorial sequence is generated: Slot 0 retains the ROI splitting kernel, Slot 1 retains the scaling kernel, and Slot 2 needs to be replaced with an FP16 quantization kernel.

[0227] Step S502: Data boundary alignment before switching (pipeline emptying).

[0228] The control layer sends a "pipeline drain" command to the INT8 quantization kernel in Slot 2. This command is written to the control register of the INT8 quantization kernel through the configuration interface, causing the kernel to stop accepting new input data after completing the quantization processing of the current frame, and to push all processed data into the elastic buffer unit downstream of Slot 2. The control layer confirms the drain is complete by reading the kernel's STATUS register. At this point, the current data boundaries are aligned, and there are no intra-frame interrupts.

[0229] Step S503: Upstream data flow is paused.

[0230] The control layer sends a "pause output" command to the scaling kernel of Slot 1 (i.e., the upstream kernel of the kernel to be replaced). After completing the transmission of the current data packet, the scaling kernel of Slot 1 no longer generates new VALID pulses and enters a waiting state. At the same time, the elastic buffer unit between Slot 1 and Slot 2 has been cleared by the drain operation in step S502, ensuring that the reconfiguration area of ​​Slot 2 enters a data-free state.

[0231] It is worth noting that the ROI splitting kernel of Slot 0 is still running normally, and its output data is temporarily stored in the elastic buffer unit between Slot 0 and Slot 1. Since Slot 1 has stopped outputting, the water level of this buffer unit will gradually rise. However, due to the back pressure mechanism of the elastic buffer, the rise in water level will be fed back to Slot 0 to reduce its output rate, thereby completing the smooth pause of the upstream data flow without losing data.

[0232] Step S504: FP16 quantizes the dynamic reconfigurable logic loading of the Kernel.

[0233] The dynamic loading arbiter queries the metadata record of the FP16 quantization kernel from the kernel library to obtain information such as its dynamically reconfigurable logic file path, file size, minimum required logical region area, and checksum. Based on the "minimum required logical region area" field, the arbiter assesses whether the available resources in the dynamic sub-region where Slot 2 is located meet the loading requirements of the FP16 quantization kernel (since FP16 quantization logic typically consumes more logic resources than INT8 quantization logic, the arbiter needs to confirm that the area of ​​the Slot 2 region is sufficient to accommodate the new kernel). After confirming resource availability, the logic loading control unit verifies the integrity of the FP16 quantization kernel's bitstream file using the checksum. Upon successful verification, the dynamically reconfigurable logic is written to the dynamically reconfigurable logic region where Slot 2 is located via the configuration controller's internal configuration access port (such as ICAP or PCAP interface). During the write process, the logical function of the Slot 2 region is in an undefined state, but through a static / dynamic region isolation mechanism (such as DFXDecoupler), this process is completely transparent to Slot 0, Slot 1, and the data path, causing no interference.

[0234] Step S505: FP16 quantization kernel initialization and configuration parameter writing.

[0235] After the dynamic reconfigurable logic is loaded, the FP16 quantization kernel's logic functions are ready, but all internal registers are in a default reset state. The control layer then uses the configuration distribution bus to sequentially write all configuration registers of the FP16 quantization kernel—including quantization format (FP16), scaling factor, quantization range, elastic buffer depth, timeout threshold, etc.—according to the configuration parameter set pre-generated by the core combinational strategy engine. After writing, the control layer reads the FP16 quantization kernel's STATUS register to confirm the ready state (the "ready" flag in the STATUS register is 1).

[0236] Step S506: Re-establish the interface and restore the data flow.

[0237] Once the FP16 quantization kernel is confirmed to be ready, the control layer notifies the scaling kernel in Slot 1 to resume output, and the data stream is re-established. The streaming data interface of the FP16 quantization kernel automatically interfaces with the upstream and downstream elastic buffer units, and the handshake mechanism (VALID / READY) takes effect immediately. The data stream begins normal transmission from the first pixel of the next frame. Due to the consistent unified data stream interface protocol, the scaling kernel in Slot 1 and the inference engine in Slot 3 can seamlessly interoperate with the new FP16 quantization kernel without any modifications.

[0238] Step S507: Verify and clear alarms after switching.

[0239] After the data stream is restored, the performance monitoring and feedback module of the control layer continuously observes whether the processing output of the FP16 quantization kernel is normal. Verification indicators include whether the output frame rate reaches the expected value, whether the output data width is correct (FP16 format is 16-bit, different from INT8's 8-bit; the bit width configuration of the unified data stream interface has been automatically adapted), and whether pixel statistics (such as mean and variance) are within a reasonable range. After successful verification, the switch is deemed successful, relevant alarm states are cleared, and the system continues to process subsequent image frames with FP16 precision.

[0240] Throughout the switchover process, the ROI splitting kernel of Slot 0 and the scaling kernel of Slot 1 remained untouched, and their logical functions, configuration parameters, and operating status remained unchanged before and after the switchover. The control layer initiated a reconfiguration operation only for Slot 2, while the kernels of other slots continued to operate normally during the switchover (only through the backpressure mechanism of the elastic buffer for flow regulation).

[0241] Therefore, the above process fully verifies the "single-stage independent replacement" capability achieved by the present invention through the multi-Kernel slot architecture: when only the quantization accuracy needs to be adjusted, the system does not need to replace the entire preprocessing pipeline as in the prior art, but only needs to load and replace the bit stream corresponding to the FP16 quantization Kernel. The reconfiguration time is proportional to the logic area occupied by the quantization Kernel alone (rather than proportional to the total area of ​​the entire pipeline), which can save 60%-80% of the reconfiguration time compared to the overall replacement scheme.

[0242] Figure 9 The diagram illustrates the kernel combination matching and end-to-end processing verification for three different AI inference application scenarios according to the present invention. Figure 7By applying the same system architecture (control layer decision-making + kernel library resources + multi-slot pipeline) to three typical scenarios with different input sources, different preprocessing requirements and different inference output formats—target detection, defect segmentation and remote sensing image analysis—the invention fully verifies the system and method's universal adaptability, combinatorial flexibility and end-to-end processing effectiveness in multiple scenarios.

[0243] Scenario 1: Kernel group matching verification for object detection.

[0244] This scenario simulates object detection inference tasks in applications such as security monitoring or autonomous driving. The raw image input (such as 1080P or 4K resolution video frames captured by a surveillance camera) first enters Slot 0 of the data processing layer, where the ROI segmentation kernel is loaded. The ROI segmentation kernel performs block processing on the original large image.

[0245] For high-resolution input images, directly feeding them into the inference engine not only exceeds the 640×640 input specification limit of the YOLO model, but also wastes valuable inference computing power due to a large number of background areas without targets. The ROI segmentation kernel uses a sliding window strategy to crop the original image into several M×N sub-images (the sub-image size can be set according to the configuration, such as 640×640 or 416×416). Overlapping areas (such as 10% overlap rate) can be configured between the sub-images to ensure that targets located at the segmentation boundaries are not truncated. After Slot 0 is processed, the sub-image data flows into Slot 1 through a unified data flow interface. Slot 1 loads the scaling kernel. The scaling kernel scales the sub-images output from Slot 0 to the precise input size required by the YOLO model (such as 640×640). This scaling operation uses the interpolation algorithm specified in the configuration parameters (such as bilinear interpolation or bicubic interpolation) to ensure that the scaled image meets the tensor dimension requirements of the model input layer while preserving the target features. After Slot 1 processing, the scaled 640×640 sub-image data flows into Slot 2 through a unified data flow interface. Slot 2 loads the quantization kernel. The quantization kernel quantizes the scaled sub-image according to the inference accuracy requirements of the YOLO model (e.g., INT8 quantization), mapping floating-point pixel values ​​to INT8 fixed-point representations and configuring appropriate scaling factors and zero-point offset parameters to reduce quantization accuracy loss. After Slot 2 processing, the image data processed through the complete preprocessing chain (ROI segmentation → scaling → INT8 quantization) is output to the inference engine layer in a format conforming to the inference engine's input specifications. The inference engine loads the YOLO model to perform object detection inference calculations, ultimately outputting the target bounding box (including the target's category label, confidence score, and bounding box coordinates), completing the end-to-end object detection inference task.

[0246] This scenario demonstrates that for target detection tasks requiring a preprocessing combination of "large image to small image, scaling to a fixed size, and fixed-point quantization," the embodiments of this application can complete the reconstruction of the entire preprocessing pipeline without modifying the system hardware architecture by loading the ROI segmentation kernel in Slot 0, the scaling kernel in Slot 1, and the INT8 quantization kernel in Slot 2. Furthermore, zero-copy streaming is achieved between the kernels through a unified data stream interface, enabling high-resolution video streams to be continuously input and processed at a real-time frame rate.

[0247] Scenario 2: Kernel assembly fit verification for defect segmentation in automated optical inspection (AOI).

[0248] This scenario simulates PCB defect detection or surface flaw detection tasks in industrial quality inspection. In this scenario, the input image source may come from various interface types (Ethernet, optical, HDMI, etc.) and use specific transmission protocols (such as GigEVision, CameraLink, or HDMI embedded information). Therefore, the first stage of the preprocessing pipeline is different from that in Scenario 1.

[0249] The raw input data first enters Slot 0, where the decoding kernel is loaded. The decoding kernel performs the corresponding decoding operations based on the input source's transmission protocol and encoding format (such as MJPEG compressed stream, RAW Bayer array, or H.264 encoded stream). For MJPEG streams input via Ethernet, the decoding kernel performs JPEG decompression; for RAW data input via optical port, the decoding kernel performs Bayer interpolation to reconstruct the RGB image; for HDMI input, the decoding kernel performs color space conversion and timing recovery, thus uniformly converting input data from different interfaces and encoding formats into standard format image data for subsequent processing. After Slot 0 processing is complete, the decoded image data flows into Slot 1, where the ROI segmentation kernel is loaded. Similar to Scenario 1, the ROI segmentation kernel performs block cropping on the high-resolution raw image, but for defect segmentation tasks, a higher overlap rate (e.g., 20%) can be set to ensure that small defects located at the segmentation boundaries are not missed. Simultaneously, the segmentation size must match the input specifications of the SegNet segmentation model. After Slot 1 is processed, the data from each sub-image flows into Slot 2, where the quantization kernel is loaded. Unlike Scenario 1, defect segmentation typically requires higher accuracy than object detection. Therefore, the quantization kernel can be configured with FP16 precision to maintain high segmentation accuracy while moderately reducing computational overhead, or configured with INT8 quantization to improve processing speed based on actual accuracy requirements. After Slot 2 is processed, the image data, processed through the complete preprocessing chain (decoding, ROI segmentation, quantization), is output to the inference engine layer in a format conforming to the inference engine's input specifications. The inference engine loads the SegNet segmentation model to perform inference calculations, ultimately outputting a segmentation mask for the defect region (identifying the pixel-level contour, defect type, and severity), completing the end-to-end AOI defect segmentation task.

[0250] Scenario 2 demonstrates that when changes in the input source interface type lead to changes in preprocessing requirements (Scenario 1 requires no decoding, Scenario 2 requires kernel decoding), this application only needs to switch the Slot 0 kernel from "ROI segmentation" to "decoding kernel" and adjust the Slot 1 and Slot 2 kernel configurations accordingly. This allows for rapid adaptation to new input source types without altering the overall system architecture. Compared to existing technologies that require redesigning the acquisition interface or replacing a dedicated acquisition card, this application's unified streaming interface and multi-slot independent replacement mechanism significantly reduce the system's adaptation cost and switching time for different input sources.

[0251] Scenario 3: Kernel combination matching verification for remote sensing imagery and scene understanding.

[0252] Scenario 3 simulates semantic understanding tasks for high-resolution images in remote sensing image interpretation, panoramic monitoring, or geographic information systems. High-resolution image input (such as 4K×4K or even 8K×8K remote sensing satellite imagery or drone aerial images) first enters Slot 0, where the ROI segmentation kernel is loaded. Unlike Scenario 1 and Scenario 2, the resolution of remote sensing images is much higher than that of conventional images, requiring the ROI segmentation kernel to adaptively adjust its segmentation strategy based on the semantic regions. Non-overlapping segmentation can be used for uniform regions to improve efficiency, while overlapping segmentation can be used for complex regions to ensure semantic continuity. The segmentation size must match the input specifications of the scene understanding model. After Slot 0 is processed, the sub-image data flows into Slot 1, where the quantization kernel is loaded. Remote sensing image analysis typically uses INT4 or INT8 quantization to maintain high inference speed while adapting to the computational constraints of edge devices. The quantization kernel is configured with appropriate quantization parameters based on the model's accuracy requirements. Unlike Slot 1 and Slot 2, Slot 2 loads a contrast enhancement kernel (or histogram processing kernel) instead of an additional scaling or quantization kernel. This is because the spectral differences between different land cover types (forest, water, urban areas, farmland) in remote sensing imagery can be subtle, and directly feeding them into the inference engine may lead to a decrease in semantic segmentation accuracy. Therefore, after quantization and before feeding them into the inference engine, the image needs to undergo contrast enhancement or histogram equalization to stretch the spectral differences between land cover types. After processing in Slot 2, the image data processed through the complete preprocessing chain (ROI segmentation, quantization, contrast enhancement) is output to the inference engine layer in a format that conforms to the input specifications of the inference engine. The inference engine loads a scene understanding model (such as DeepLab or PSPNet) to perform inference calculations, and finally outputs the region labeling results (pixel-level classification and labeling of each semantic region, such as forest labeled as green, water labeled as blue, urban areas labeled as gray, and farmland labeled as yellow), completing the end-to-end remote sensing image scene understanding task.

[0253] Scenario 3 demonstrates that when the inference task shifts from object detection / defect segmentation to scene understanding, the preprocessing requirements undergo a substantial change: Scenario 1 and Scenario 2 both require scaling the kernel to match the model's fixed input size, while in Scenario 3, if the semantic segmentation model supports arbitrary resolution input and the primary requirement is ground feature spectral enhancement, then scaling the kernel is unnecessary, but contrast enhancement kernels are required. This application flexibly switches from a combination of "quantization kernel + scaling kernel" to "contrast enhancement kernel" through Slot 2, fully verifying the flexibility of the multi-kernel slot independent replacement mechanism under different preprocessing logics. The three scenarios respectively demonstrate three different kernel combinations: Scenario 1 is "ROI segmentation, scaling, quantization," Scenario 2 is "decoding, ROI segmentation, quantization," and Scenario 3 is "ROI segmentation, quantization, contrast enhancement." Each combination is automatically generated through the inference requirement parsing of the control layer and the policy engine, completing the mapping from requirements to hardware configuration without manual intervention. Furthermore, each kernel is loaded and replaced independently; a kernel change in one slot does not affect the normal operation of other slots.

[0254] Therefore, when switching scenes (such as switching from object detection in scene 1 to AOI defect segmentation in scene 2), the inference requirement parser 110 of the control layer regenerates the kernel function requirement list according to the new inference task description (scene 2 has more decoding requirements and fewer scaling requirements than scene 1). The core combination strategy engine generates a new optimal kernel combination sequence accordingly. The dynamic loading arbitrator evaluates the current slot occupancy and decides which kernels need to be replaced. For example, Slot 0 in scene 1 loads the ROI segmentation kernel, while Slot 0 in scene 2 needs to load the decoding kernel. Therefore, the arbitrator only triggers the reconfiguration operation for Slot 0, and the kernels of Slot 1 (ROI segmentation) and Slot 2 (quantization) are retained and continue to work. Similarly, when switching from scene 2 to scene 3, the arbitrator only reconfigures Slot 1 from "ROI segmentation" to "quantization" and Slot 2 from "quantization" to "contrast enhancement", while other slots remain unchanged. This incremental reconfiguration mechanism, which "replaces only the changed parts," makes the reconfiguration time during scene switching proportional to the number of kernels that need to be replaced (the replacement time for a single kernel is 10-50 milliseconds), rather than proportional to the total size of the entire pipeline.

[0255] Reference Appendix Figure 3Another embodiment of the present invention also provides an AI inference data preprocessing kernel dynamic loading control device, including a kernel library 100, a switching trigger unit 200, a pipeline emptying control unit 300, a logic loading control unit 400, an initialization control unit 500, and a recovery control unit 600, wherein:

[0256] Kernel library 100 stores multiple available kernels in a dynamically reconfigurable logical file and the metadata records corresponding to each kernel.

[0257] The switching trigger unit 200 is used to generate a switching command in response to a switching trigger event;

[0258] The pipeline evacuation control unit 300 is used to send a pipeline evacuation command to the kernel to be switched before switching.

[0259] The logic loading control unit 400 is used to read the dynamic reconfigurable logic file of the target Kernel from the Kernel library 100 and write it into the target dynamic reconfigurable logic region through the configuration controller;

[0260] The initialization control unit 500 is used to write the configuration register of the new kernel according to the set of configuration parameters after the logic loading is completed, and to confirm that the new kernel is ready.

[0261] The recovery control unit 600 is used to notify the upstream kernel to resume output, so that the streaming data interface of the new kernel can automatically connect with the upstream and downstream elastic buffers.

[0262] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the AI ​​inference data preprocessing pipeline adaptive reconstruction method of any of the above embodiments.

[0263] It should be noted that the above embodiments can be freely combined as needed. The above are merely preferred embodiments of the present invention. It should be pointed out that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. An AI inference data preprocessing pipeline adaptive reconstruction method, characterized in that, include: The control layer receives the inference task description and decodes the inference task description into a list of kernel function requirements; The control layer generates the optimal Kernel combination sequence and the configuration parameter set of each Kernel based on the Kernel functional requirement list and the predefined policy rule base. The control layer assesses the current occupancy status of the dynamically reconfigurable logical region and determines the target kernel to be loaded and the loading timing from the kernel combination sequence; The control layer controls the loading of the dynamic reconfigurable logic of the target Kernel into the corresponding Kernel slot and writes configuration parameters to the target Kernel. The loaded kernels are connected in series through a unified data flow interface to form a pipeline, which processes the input data in a non-disk streaming manner step by step. Adjacent kernels use elastic buffer units to cache data and match rates. The data, after being processed through the complete processing chain, is output to the inference engine in a format that conforms to the input specifications of the inference engine for inference calculation; The processing throughput of each loaded kernel, the level of the elastic buffer units between the slots of each loaded kernel, and the frame rate performance metrics of the inference engine are continuously collected. The performance metrics are fed back to the policy rule base for closed-loop optimization of subsequent kernel combination decisions; The switching of each loaded kernel is triggered by at least one of the following events: forced reconfiguration request, automatic demand change detection, performance optimization trigger, and fault recovery trigger. Different events have different priorities, and higher priority events can preempt the switching process of lower priority events.

2. The adaptive reconstruction method for AI inference data preprocessing pipeline according to claim 1, characterized in that, The forced reconfiguration request is triggered by a model switching command explicitly sent by the upper layer of the inference framework, and the model switching command carries the target model identifier. The automatic demand change detection is triggered by the inference demand parser detecting that the kernel sequence required by a newly arrived inference task does not match the configuration of any currently loaded kernel in any kernel slot. The performance optimization is triggered when the performance monitoring module detects that the processing throughput of the Kernel configuration loaded in any Kernel slot is continuously lower than a set threshold. The fault recovery is triggered by an unrecoverable internal error occurring during the operation of a kernel loaded in any kernel slot.

3. An AI inference data preprocessing pipeline adaptive reconstruction system, characterized in that, include: The control layer is located in the static logic area of ​​the chip. The control layer is configured to generate a corresponding data processing kernel combination sequence and configuration parameter set according to the received inference task description, and control the dynamic loading and switching of the kernel. The data processing layer is located in the dynamically reconfigurable logic area of ​​the chip. The data processing layer includes multiple kernel slots. Each kernel slot is used to load a data processing kernel. After loading, each kernel is connected in series through a unified data flow interface to form a pipeline. An elastic buffer unit is provided between adjacent kernels. The elastic buffer unit is used to cache intermediate data between kernels and realize data flow rate matching. An inference engine layer is used to receive preprocessed data output by the data processing layer and perform inference calculations. The Kernel library stores multiple dynamic reconfigurable logical files of available Kernels and metadata records corresponding to each Kernel. The control layer communicates with the data processing layer through a standard interface. The control layer sends configuration parameters to each kernel and obtains running status information from each kernel and the inference engine layer. The Kernel library supports online updates, and new Kernels can be queried and called by the system after registration. The metadata record shall include at least one of the following fields: Kernel unique identifier, Kernel function type label, dynamic reconfigurable logic file path and checksum, dynamic reconfigurable logic file size, minimum area of ​​required logic region, input data bit width requirement, output data bit width requirement, maximum operating clock frequency, configuration parameter register definition table, version number and release timestamp. The unified data flow interface adopts an interface protocol based on pipelined data transmission, and data is transmitted between adjacent kernels through point-to-point data paths. The data is processed sequentially through each kernel within the data processing layer. The data is transmitted only through the data path between each kernel and the elastic buffer unit in the complete processing path within the data processing layer.

4. The AI inference data preprocessing pipeline adaptive reconstruction system of claim 3, wherein, The control layer includes: An inference requirement parser is used to receive an inference task description and decode the inference task description into a list of kernel function requirements. The core combination strategy engine is used to generate the optimal Kernel combination sequence and the configuration parameter set of each Kernel based on the Kernel functional requirement list and the predefined strategy rule base. The dynamic loading arbitrator is used to evaluate the current occupancy of dynamically reconfigurable logical regions, decide whether to trigger a reconfiguration operation, and select the target logical region and loading timing. The configuration distribution bus is used to write the configuration parameters of each kernel into the corresponding register group; The performance monitoring and feedback module is used to collect the processing throughput, elastic buffer level and frame rate performance metrics of each kernel and inference engine, and feed the performance metrics back to the core combination strategy engine.

5. The AI inference data preprocessing pipeline adaptive reconstruction system of claim 4, wherein, The inference task description includes at least one of the following: model identifier, input resolution requirement, quantization accuracy requirement, and color format requirement; The Kernel functional requirements list includes a description of the requirements for data processing function types; The signal type, signal width, and timing requirements of the standard interface are fixed during the system design phase. The standard interface has consistent signal definitions and timing specifications for all loaded content in the dynamically reconfigurable logic region.

6. The AI inference data preprocessing pipeline adaptive reconstruction system of claim 3, wherein, The elastic buffer unit is configured as follows: Real-time monitoring of data backlog in the buffer; When the data backlog exceeds a preset high water level threshold, a deceleration signal is sent to the upstream kernel, instructing the upstream kernel to reduce the output rate; When the data backlog level is lower than a preset low water level threshold, the upstream kernel is allowed to output at the current rate; If the data in the buffer has not been fully consumed after exceeding a preset time threshold, timeout protection is triggered. The timeout protection process includes: triggering reconfiguration optimization when the backlog level is at the first level, and performing controlled frame dropping and reporting abnormal events to the control layer when the backlog level is at the second level.

7. A dynamic loading control device for AI inference data preprocessing kernel, characterized in that, include: The Kernel library stores dynamic reconfigurable logical files for multiple available Kernels and metadata records corresponding to each Kernel. A switching triggering unit is used to generate a switching instruction in response to a switching triggering event. The switching of the Kernel is triggered by at least one of the following triggering events: forced reconfiguration request, automatic demand change detection, performance optimization trigger, and fault recovery trigger. Different triggering events have different priorities, and high-priority events can preempt the switching process of low-priority events. The pipeline evacuation control unit is used to send a pipeline evacuation command to the kernel to be switched before switching. The logic loading control unit is used to read the dynamic reconfigurable logic file of the target Kernel from the Kernel library and write it to the target dynamic reconfigurable logic region through the configuration controller. The initialization control unit is used to write the configuration register of the new kernel according to the set of configuration parameters after the logic loading is completed, and to confirm that the new kernel is ready. The recovery control unit is used to notify the upstream kernel to resume output, so that the streaming data interface of the new kernel can automatically connect with the upstream and downstream elastic buffers.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1-2.

Images