An intelligent image processing system and method based on FPGA

Abstract

This invention discloses an intelligent image processing system and method based on FPGA, belonging to the field of image processing technology. The system includes: an intelligent interface adaptation module for converting image data from different interfaces into a unified format video stream; a scene adaptive binding engine for dynamically determining the binding relationship between the video stream and the processing module, and the data bit width of the video bus, based on the processing task and the status of each module; and an AHB bus control module for configuring the video bus bit width according to decisions and sending configuration information to the processing module. This invention achieves plug-and-play access to multi-source heterogeneous images, dynamic intelligent reconstruction of the processing pipeline, and online optimization of bus parameters, significantly improving the system's flexibility, resource utilization, and overall performance.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically to an intelligent image processing system and method based on FPGA. Background Technology

[0002] Existing FPGA-based image processing systems typically employ an architecture where dedicated acquisition modules with fixed interfaces are directly connected to specific processing pipelines. This architecture has significant drawbacks: First, the system cannot adaptively access image sources employing different physical layer or data link layer protocols, requiring custom hardware design for each interface, resulting in complex system integration, high costs, and poor scalability. Second, the binding relationship between image data streams and processing modules is fixed during the system design phase, making it impossible to dynamically adjust according to real-time task requirements or system load, hindering optimized resource scheduling and reuse. Furthermore, the system bus width is usually fixed; when dealing with high-resolution, high-frame-rate image data, increasing the clock frequency is often necessary to ensure bandwidth, posing a significant challenge to FPGA timing convergence and limiting system performance improvements.

[0003] The aforementioned problems result in traditional systems lacking flexibility, having low resource utilization, and high operation and maintenance costs in scenarios with multiple inputs and dynamic demands, such as intelligent security, industrial vision, and medical imaging, making it difficult to meet the actual needs of intelligent processing. Summary of the Invention

[0004] This invention provides an intelligent image processing system and method based on FPGA to solve the technical problems of rigid interface adaptation, fixed processing flow and module binding, non-dynamic adjustment of bus width, and reliance on manual intervention for system reconfiguration in traditional FPGA image processing systems.

[0005] This invention is achieved through the following technical solution:

[0006] A first aspect of the present invention provides an intelligent image processing system based on FPGA, comprising:

[0007] The intelligent interface adapter module is located at the entrance of the video bus and is configured to identify the interface type of at least one image data source, and convert image data from different interface types into a unified format video data stream based on the identification result.

[0008] The scene-adaptive binding engine is configured to determine the binding relationship between the video data stream and at least one image processing module based on at least one processing task to be executed and the working status of each image processing module inside the FPGA, and to determine the data bit width of the video bus.

[0009] The AHB bus control module is communicatively connected to the scene adaptive binding engine and is configured to configure the data width of the video bus via the AHB bus based on the data width, so that the video bus transmits the video data stream with the configured data width; and to send configuration information to the image processing module via the AHB bus based on the binding relationship, so that the image processing module processes the video data stream based on the configuration information.

[0010] Furthermore, the intelligent interface adaptation module includes:

[0011] The protocol feature acquisition unit is configured to acquire the data format features and timing features of the image data in real time.

[0012] The interface type recognition unit is configured to match the data format features and temporal features of the image data with a pre-stored feature template; if the match is successful, the standard conversion logic corresponding to the feature template is invoked; if the match fails, a temporary conversion logic for the image data is generated based on the data format features and temporal features.

[0013] The protocol conversion unit is configured to convert the image data into a video data stream of a unified format through the standard conversion logic or the temporary conversion logic.

[0014] Furthermore, the interface type identification unit includes:

[0015] The timing analysis subunit is configured to deduce the generation rules of line synchronization, field synchronization and pixel clock based on the timing characteristics of the image data when a match fails.

[0016] The data analysis subunit is configured to determine the arrangement order and bit width of pixel data based on the data format characteristics when a match fails.

[0017] The logic generation subunit is configured to generate the temporary conversion logic based on the generation rules and the arrangement order and bit width of the pixel data.

[0018] Furthermore, the scene adaptive binding engine includes:

[0019] The status monitoring unit is configured to continuously monitor the data flow of the video bus and the working status of each image processing module, including resource utilization and processing queue length.

[0020] The attribute archive stores the priority, resource consumption coefficient, and processing latency tolerance of each processing task, as well as the processing functions and performance indicators of each image processing module.

[0021] The binding algorithm unit is configured to establish a constrained algorithm model with the optimization objectives of maximizing system throughput, minimizing total processing latency, and balancing resource load. The algorithm model is input with the currently monitored data flow, the working status of each image processing module, and the data retrieved from the attribute archive. The built-in optimization algorithm is used to solve the optimal binding relationship between the video data stream and at least one image processing module within a preset time window.

[0022] Furthermore, the AHB bus control module is also configured to:

[0023] The running status data is periodically read from the status register of each image processing module via the AHB bus. The running status data includes at least one of the following: processing frame rate, resource utilization rate, and error flag.

[0024] The running status data is fed back to the scene adaptive binding engine so that the scene adaptive binding engine can update the working status of each image processing module.

[0025] Furthermore, the configuration information includes the identifier of the video data stream, the processing function enable signal, and the processing parameters;

[0026] The identifier of the video data stream is used to indicate the video data stream that the image processing module needs to process; the processing function enable signal is used to enable or disable the processing function of the image processing module; and the processing parameters are used to adjust the operational behavior of the image processing module.

[0027] Furthermore, it also includes a dynamic triggering and reconfiguration module, which is configured to send a rebinding request to the scene adaptive binding engine in response to scene change events, system status events and / or external command events, and trigger the AHB bus control module to update the configuration information based on the new binding relationship.

[0028] Furthermore, the dynamic triggering and reconfiguration module includes:

[0029] The trigger condition monitor is configured to continuously monitor scene change events, system status events, and / or external command events;

[0030] The hierarchical reconfiguration unit is configured to select a reconfiguration level based on the event type, wherein the reconfiguration level includes a first reconfiguration level, a second reconfiguration level, and a third reconfiguration level.

[0031] Specifically, the first configuration level triggers the AHB bus control module to update the processing parameters in the binding configuration information, the second configuration level triggers the adjustment of the bit width of the video bus, and the third configuration level triggers the scene adaptive binding engine to update the binding relationship.

[0032] Furthermore, the video bus includes a bit width configuration register; the AHB bus control module is also configured to write a bit width configuration value to the bit width configuration register via the AHB bus to adjust the data bit width of the video bus.

[0033] A second aspect of the present invention provides an intelligent image processing method based on FPGA, comprising:

[0034] Identify the interface type of at least one image data source, and convert image data from different interface types into a video data stream in a unified format based on the identification results;

[0035] Based on at least one processing task to be executed and the working status of each image processing module inside the FPGA, the binding relationship between the video data stream and at least one image processing module is determined, and the data bit width of the video bus is determined.

[0036] Based on the data bit width, the data bit width of the video bus is configured via the AHB bus so that the video bus transmits the video data stream with the configured data bit width;

[0037] Based on the binding relationship, configuration information is sent to the image processing module via the AHB bus, so that the image processing module processes the video data stream based on the configuration information.

[0038] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0039] This invention constructs an intelligent image processing system with self-awareness, dynamic optimization, and real-time reconfiguration capabilities by introducing a collaborative design of an intelligent interface adaptation module, a scene-adaptive binding engine, and an AHB bus control module. It achieves automatic identification and unified format conversion of multiple interface protocols, breaking down barriers to heterogeneous device access. The scene-adaptive binding engine dynamically determines the optimal data flow-processing module binding relationship and bus width by sensing task requirements and module status in real time, enabling flexible pipeline reconfiguration and on-demand resource allocation. The AHB bus control module, as a high-efficiency execution unit, translates the binding engine's decisions into precise configurations of the video bus and various processing modules within milliseconds. This significantly improves the system's flexibility, resource utilization, and overall performance when processing multi-source heterogeneous images. Attached Figure Description

[0040] To more clearly illustrate the technical solutions of the exemplary embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly described below. It should be understood that the following drawings only show some embodiments of the present invention and should not be considered as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort. In the drawings:

[0041] Figure 1 This is a diagram of an FPGA-based image processing architecture according to an embodiment of the present invention;

[0042] Figure 2 This is a block diagram of an intelligent image processing system based on FPGA according to an embodiment of the present invention;

[0043] Figure 3 This is a block diagram of an intelligent interface adaptation module according to an embodiment of the present invention;

[0044] Figure 4 This is a schematic diagram of an intelligent image processing method based on FPGA according to an embodiment of the present invention. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the embodiments and accompanying drawings. The illustrative embodiments and descriptions of the present invention are only used to explain the present invention and are not intended to limit the present invention.

[0046] It should be noted that the terms "comprising" and "having" and any variations thereof in the specification, claims, and accompanying drawings of this invention are intended to cover non-exclusive inclusion, for example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to other steps or units inherent in the device.

[0047] The terminology used in the various embodiments of the invention is for the purpose of describing particular embodiments only and is not intended to limit the various embodiments of the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which the various embodiments of the invention pertain. The terms (such as those defined in commonly used dictionaries) are to be interpreted as having the same meaning as in the context of the relevant technical field and are not to be interpreted as having an idealized or overly formal meaning, unless clearly defined in the various embodiments of the invention.

[0048] This invention aims to address the technical problems of rigid interface adaptation, fixed processing flow, static resource configuration, and reliance on manual system reconfiguration in traditional FPGA image processing systems. It proposes an intelligent image processing system and method with capabilities for intelligent interface recognition, scene-adaptive binding, dynamic bit-width configuration, and real-time status monitoring. This system is suitable for complex processing scenarios with multi-source heterogeneous image inputs, such as intelligent security monitoring, industrial visual inspection, and real-time medical image processing, and helps improve system flexibility, resource utilization, and overall intelligence.

[0049] The system of this invention can be deployed in application scenarios that include heterogeneous image inputs (such as multiple cameras or video capture cards with different interface protocols) and diverse processing requirements (such as noise reduction, edge detection, and target recognition). The image processing system is physically implemented using one or more FPGA chips, within which the functional modules described in this invention are constructed through programmable logic.

[0050] Figure 1 This invention presents a multi-source heterogeneous image processing architecture based on FPGA, comprising N image input modules, M image processing modules, a video bus, and an AHB bus. In this invention, a modular design concept is adopted within the FPGA to uniformly connect all input modules to the video bus (video_bus). Regardless of the image source type and interface, all modules can be transmitted via a unified format video bus through a specific conversion mechanism. All image processing modules use the video bus (video_bus) as their data source, and a dedicated conversion module can convert the output data to the corresponding output interface format to adapt to different display devices or external systems. Furthermore, an AHB bus is introduced into the FPGA to precisely allocate address space for each module (AHB slave modules). The AHB bus manages the unique identifier of each address space, where the Src id identifies the initiating module of the data / transaction, and the dev id identifies the destination module of the data / transaction, ensuring a one-to-one correspondence between data / modules and address space IDs. The AHB bus serves as the control core, managing bus access and coordinating module communication through the AHB arbitration module. Simultaneously, the AHB bus communicates with external device applications, which provide a software interface for users to interact with the system by issuing commands, configuring parameters, and receiving data. The AHB arbitration module interfaces with the software layer for configuration and status monitoring, controlling and monitoring all modules within the FPGA.

[0051] In this invention, the video bus and AHB bus are data transmission and control channels with specific intelligent management functions, constructed using the programmable interconnect resources and bus protocol IP cores within the FPGA. Specifically, the video bus achieves dynamic optimization of data transmission bandwidth and intelligent orientation of data flow by introducing configurable bit-width registers and programmable crossbar switching routing logic.

[0052] Each module is allocated a dedicated configuration and status address space via the AHB bus, and a dedicated binding execution instruction format is designed to achieve centralized and efficient management of system configuration and status information. The AHB bus supports bidirectional data interaction.

[0053] (1) Downlink configuration channel: After parsing the configuration instructions (such as image binding relationship, processing parameters, etc.) from the software layer, they are written into the address space of the target module through the AHB bus to achieve microsecond-level configuration response.

[0054] (2) Uplink status channel: The module's working status (such as processing frame rate, RAM utilization rate), fault flag bits and other information are stored in the address space in real time, and the software periodically reads and converts them into visual data.

[0055] like Figure 2 As shown, the FPGA-based intelligent image processing system provided by this invention includes an intelligent interface adaptation module 201, a scene adaptive binding engine 202, and an AHB bus control module 203. The specific implementation and functions of each module are described below.

[0056] (1) The intelligent interface adapter module 201 is set at the entrance of the video bus and is configured to identify the interface type of each image data source and convert the image data of different interface types into a unified format video data stream according to the identification result.

[0057] Specifically, the intelligent interface adapter module 201 can physically be located in the input I / O Bank area of ​​the FPGA chip or near the high-speed transceiver. Its main function is to serve as a unified access layer for the system. When image data sources from different interfaces (such as SDI, CameraLink, and Gigabit Ethernet video streams) are connected to the FPGA, the intelligent interface adapter module 201 first identifies the connected image data. For example, for a connected CameraLink signal, the module detects the LVDS signal characteristics on its Channel Link clock and data lines; for an SDI signal, it detects its serial bit rate and embedded synchronization header. After identification, the module calls or generates the corresponding conversion logic to strip all input data from the physical layer and link layer protocols, extracts the valid pixel data, horizontal and vertical sync signals, and pixel clocks, and packages them into an internally defined video data stream format.

[0058] A unified format video data stream contains at least a stream identifier (Stream ID), pixel data validity bits, line synchronization (Hsync), field synchronization (Vsync), and additional timestamps and data quality flags. The intelligent interface adapter module 201 sends the converted unified format video data stream to the video bus entry point, and then transmits it to the internal processing module via the video bus.

[0059] (2) The scene adaptive binding engine 202 is configured to determine the binding relationship between the video data stream and at least one image processing module based on at least one processing task to be executed and the working status of each image processing module inside the FPGA, and to determine the data bit width of the video bus.

[0060] The scene-adaptive binding engine 202 can be implemented by a soft-core processor or a dedicated state machine within the FPGA. Its core responsibility is to act as the intelligent scheduling center of the system. The scene-adaptive binding engine 202 receives a list of processing tasks from the upper-layer application software. At the same time, it acquires the working status of each image processing module in real time through an internal monitoring channel, and performs two key decisions based on task requirements and module status: First, it determines the binding relationship, that is, decides which video data stream (identified by its Stream ID) to which image processing module(s) for processing. When multiple image processing modules are bound simultaneously, it specifies the processing order to form a processing pipeline. Second, it determines the data bit width of the video bus.

[0061] Taking an intelligent traffic monitoring scenario as an example, the system receives two processing tasks: Task 1 (high priority) requires "vehicle detection and tracking" of the stream from "Intersection A camera"; Task 2 requires "license plate recognition" of the stream from "Intersection B camera". The scene adaptive binding engine 202 receives task attributes (priority, required algorithm), obtains the status of each image processing module (module X supports "object detection" and is currently idle; module Y supports "optical character recognition" and has a low load; module Z supports "feature extraction" and is running other tasks), and uses a built-in optimization algorithm to solve for the allocation of input data streams to processing modules that meet the task processing requirements and internal resource supply. The decision result may be: binding the video stream from "Intersection A" to module X to perform vehicle detection, and then further binding the output stream of module X to a dedicated "tracking algorithm module"; at the same time, directly binding the video stream from "Intersection B" to module Y for license plate recognition. This forms a dynamically constructed processing pipeline that includes serial and parallel branches.

[0062] The operational status of an image processing module can be a comprehensive indicator, including: whether the module is idle, the current resource utilization percentage, the backlog of frames in the processing queue, and whether the self-test status is normal. For example, when the system needs to process multiple 4K high-definition streams, the scene adaptive binding engine 202 decides to configure the video bus to a 256-bit width to maintain a lower clock frequency under high data throughput, thereby reducing the difficulty of timing convergence.

[0063] (3) The AHB bus control module 203 is configured to configure the data bit width of the video bus through the AHB bus based on the data bit width; and to send configuration information to the image processing module through the AHB bus based on the binding relationship.

[0064] The AHB bus control module 203 can function as a dedicated configuration controller connected to the AHB bus, or as part of the AHB bus interconnect structure embedded within the FPGA. Its core functions are configuration execution and status collection. It communicates with the scene adaptive binding engine 202, receiving binding relationship decisions and converting them into specific configuration information understandable by each image processing module. Subsequently, the AHB bus control module 203, acting as the master device of the AHB bus, initiates write transactions to write the configuration information into the configuration registers within the target image processing module. Simultaneously, it periodically initiates read transactions to read operational status data from the status registers of each module.

[0065] In this embodiment, the image processing module is a hardware-accelerated IP core within the FPGA that executes specific image processing algorithms (such as filtering, transformation, and feature extraction). Each module is designed with a standardized video stream input / output interface. Each module contains a configuration register space to receive configuration information from the AHB bus. Based on the configuration information, the module is informed which Stream ID's data stream to process and which processing functions to enable. Enable signals control the disabling of redundant module interfaces; for example, enabling "Gaussian denoising" and disabling "median filtering" for noise reduction tasks. Processing parameters, such as the Gaussian kernel size and sigma value, are configured. After configuration, the module begins listening to and capturing data packets of the specified Stream ID from the video bus for corresponding processing, and outputs the results back to the video bus for use by subsequent modules or output to an external display device.

[0066] The video bus is configured to transmit video data streams to the bound image processing modules with a configured data bit width, based on the binding relationship determined by the scene adaptive binding engine 202.

[0067] In this embodiment, the video bus, consisting of a series of configurable wiring resources, crossbars, and FIFO buffers, serves as a data highway within the FPGA. It routes video streams to their bound modules based on the binding relationships between N inputs and M image processing modules. For example, a binding relationship might indicate that "the data stream with stream ID=1 must pass through module A (denoising) and module B (edge ​​detection) sequentially." The crossbars within the video bus execute the corresponding binding configuration, ensuring that data packets with ID=1, after being output from the smart interface adapter module, are first sent to the input port of module A. The output data processed by module A is then routed to the input port of module B. Simultaneously, the bit width of the video bus is configurable, organizing data transmission according to the bit width determined during binding.

[0068] Specifically, during the hardware design phase, the video bus defines a sufficiently large maximum physical bit width. Through the bit width configuration register, it controls the data packer / unpacker and byte enable logic in the data path, thereby changing the data organization method and enabling the bit width of the part, and realizing the line-of-sight transmission bit width configuration.

[0069] The bit width configuration register stores the currently active bit width mode value (e.g., 00 corresponds to 64 bits, 01 to 128 bits, 10 to 256 bits, with a maximum physical bit width of 256 bits). When the system operates at the lower configuration bit width of 64 bits, the packing logic at the video bus source end packs the 64 bits of valid data transmitted each clock cycle into the designated location of the 256-bit physical channel, indicating the valid portion via an accompanying byte enable signal. The unpacking logic at the destination end then extracts the valid data based on this enable signal. When configured with a higher bit width, more physical bits are used.

[0070] Furthermore, bit width configuration can also affect the transmission clock strategy. For example, in 64-bit mode, to transmit the same amount of data as in 256-bit mode, the system can keep the bus clock frequency unchanged, but it will require four clock cycles to transmit the equivalent of one clock cycle of data in 256-bit mode. Alternatively, the bus clock frequency can be reduced by clock division or gating while maintaining the same data throughput. The specific strategy is determined by the scenario-adaptive binding engine based on bandwidth requirements and timing margins, and is reflected in the bit width configuration value.

[0071] Furthermore, such as Figure 3 As shown, the intelligent interface adaptation module 201 includes a protocol feature acquisition unit 2011, an interface type identification unit 2012, and a protocol conversion unit 2013.

[0072] The protocol feature acquisition unit 2011 is configured to acquire the data format and timing features of the incoming image data in real time. This unit consists of a high-speed sampling circuit and feature extraction logic. For parallel interfaces, timing features are extracted by statistically analyzing the signal transition patterns within a specific time period using latched data lines. Examples of features include the approximate frequency of the pixel clock, the effective pulse width and period of the horizontal sync signal, and the effective pulse width and period of the vertical sync signal. For serial interfaces, the clock and data are first recovered using a clock data recovery circuit. Then, the data stream is decoded, and specific synchronization words (SAV / EAV) are searched to extract timing and data format features, such as the number of effective pixels per line, the number of effective lines per frame, pixel bit depth (10-bit, 12-bit), and color space (RGB).

[0073] The interface type recognition unit 2012 is configured to match the data format features and temporal features of the image data with the pre-stored feature template. If the match is successful, the standard conversion logic corresponding to the feature template is called. If the match fails, temporary conversion logic is generated based on the data format features and temporal features of the image data.

[0074] This unit contains a small lookup table (LUT) or read-only memory (ROM) that stores feature templates for various standard interfaces. Each template defines the typical timing parameter range (such as line frequency range, field frequency range) and data format of the interface. The interface type identification unit 2012 compares the features extracted by the protocol feature acquisition unit 2011 with all feature templates. If all key features (line synchronization period, pixel bit depth, etc.) fall within the tolerance range of a certain feature template, it is considered a successful match, and the corresponding interface type identifier is output. If it does not match any of the pre-stored templates, it is determined to be an unknown or custom interface, triggering a temporary logic generation process.

[0075] When an interface is identified as unknown, approximate rules are derived based on the collected timing characteristics. For example, the horizontal sync signal is active low, with a pulse width of approximately T_p*10 cycles (T_p represents the stable period on the data line); the vertical sync signal is also active low, with a pulse width of approximately T_h*2 cycles (T_h represents the time interval T_h between two horizontal sync pulses); valid pixel data begins to appear after a delay of 5 T_p cycles following the invalidation of the horizontal sync, and lasts for T_h-15*T_p cycles. These rules are formally described as the state transition conditions of a finite state machine. The arrangement order of the pixel data is inferred based on the data format characteristics (analyzing the value change patterns on continuous data lines), and hardware description code or configuration bitstream for temporary conversion logic is generated. For example, on a 16-bit data line, is the high 8 bits R and the low 8 bits G, or are they interleaved, and what is the corresponding effective bit width?

[0076] The protocol conversion unit 2013 consists of a series of reconfigurable data path circuits, configured to convert image data into a unified format video data stream via standard conversion logic or temporary conversion logic. For a successfully matched standard interface, the unit loads the corresponding standard conversion logic from a pre-compiled IP core library based on the interface type identifier. For example, for the "CL_FULL" type, a Camera Link deserializer, decoder, and channel alignment circuitry are loaded. For an unknown interface, the configuration bitstream generated by the temporary conversion logic generation process is loaded. Regardless of the method, the protocol conversion unit 2013 ultimately regenerates standardized line synchronization (Hsync), field synchronization (Vsync), and pixel clock (Pclk) signals based on the recovered timing; rearranges and packages the input raw pixel data according to a unified format, adds assigned identifiers, and forms a unified video data stream output to the video bus.

[0077] Furthermore, the interface type identification unit 2012 also includes a timing analysis subunit, a data analysis subunit, and a logic generation subunit. The timing analysis subunit is configured to deduce the generation rules for line synchronization, field synchronization, and pixel clocks based on the timing characteristics of the image data when a match fails. The data analysis subunit is configured to determine the arrangement order and bit width of the pixel data based on the data format characteristics. The logic generation subunit is configured to generate temporary conversion logic based on the generation rules and the arrangement order and bit width of the pixel data. The protocol conversion unit 2013 generates hardware description code or configuration bitstream for the temporary conversion logic and downloads it to the FPGA to dynamically configure the finite state machine and data alignment circuit to perform a unified format conversion on the image data.

[0078] The timing analysis subunit is implemented using a set of configurable counters and comparators. For example, it uses a high-frequency clock to oversample the input signal, and the counters count the intervals of the horizontal sync pulses, the intervals of the vertical sync pulses, and the stable clock cycles during the data validity period. By comparing these counts with a preset threshold range, an approximate timing rule is derived. The data analysis subunit is implemented using a small FIFO (First-In, First-Out) buffer in conjunction with pattern matching logic. For example, it continuously buffers several rows of pixel data and analyzes the stability or variation patterns of data bits at specific locations to infer the pixel bit width and arrangement order. The logic generation subunit can be a small program running on the soft core of the FPGA, or a pre-built, parameterizable hardware template.

[0079] The logic generation subunit receives the results of timing and data analysis as parameters. It compiles the number of states, transition conditions, and outputs obtained from the timing analysis into a netlist description of a finite state machine. It then substitutes the bit width and arrangement order parameters obtained from the data analysis into a general data alignment and reassembly hardware template to generate a short piece of hardware description code or a partial configuration bitstream describing the temporary transition logic. Subsequently, the system loads this temporary transition logic into the reserved reconfigurable area within the protocol conversion unit 2013 via the FPGA's partial reconfiguration interface, instantiating a finite state machine and data alignment circuit for this unknown interface. This process can be completed during system initialization or when a new device is connected. After completion, the interface can be used as a new feature template, correctly recognized and used by the system.

[0080] Furthermore, the scene adaptive binding engine 202 includes a state monitoring unit, an attribute archive, and a binding algorithm unit, and its specific operation is as follows:

[0081] The status monitoring unit is connected to the video bus and the counters inside each image processing module via a lightweight bus or dedicated signal line, and is configured to continuously read: (1) the video bus data flow, specifically by using a bandwidth counter to estimate the data throughput of each video stream; (2) the working status of the image processing module, including the module's internal reports of its LUT utilization, DSP utilization, BRAM utilization, etc., as well as the processing queue length.

[0082] The AHB bus control module 205 periodically initiates a series of AHB read transactions to access the status register group of each image processing module, read the operating status data of each module, package the read operating status data, and feed it back to the status monitoring unit of the scene adaptive binding engine through the internal interface or shared memory. This allows the scene adaptive binding engine to update its understanding of the working status of each image processing module and provide a more accurate real-time basis for the next round of binding decisions.

[0083] The operational status data includes at least one of the following: processing frame rate, resource utilization rate, and error flags (data verification error, algorithm overflow, timeout, and other abnormal flags). The processing frame rate is calculated by counting the number of processed frames using an internal counter in the image processing module; the resource utilization rate includes the utilization rates of the real-time LUT, BRAM, and DSP reported by the module's internal monitoring circuit.

[0084] The attribute archive can be a data structure stored in on-chip memory or shared memory with the soft core. It contains two types of information:

[0085] (1) Processing task attributes: Predefined static attributes for each task type, including the priority of processing tasks, resource consumption coefficient and processing delay tolerance.

[0086] Task priorities can be weighted to make decisions in case of resource conflicts; for example, the weight of the "object recognition" task is set to 10, and the weight of the "image enhancement" task is set to 3. The resource consumption coefficient estimates the typical proportion of various hardware resources used by the task when executed on a standard processing module (e.g., DSP: 30%; LUT: 20%; BRAM: 10%). The processing latency tolerance indicates the maximum end-to-end processing latency allowed for the task (e.g., 33ms, corresponding to 30fps).

[0087] (2) Image processing module attributes: Describe the capabilities and static performance of each specific hardware module, including processing functions (a list of task types supported by the module) and performance indicators (such as the nominal processing speed and maximum throughput (MP / s) of the module when processing a specific task).

[0088] The binding algorithm unit is periodically triggered to perform an optimization solution. Its core is a hardware implementation of a built-in multi-objective optimization algorithm. The optimization objectives of the algorithm model include maximizing system throughput, minimizing total processing latency, and balancing resource load. A multi-objective optimization function is established using a constrained weighted sum method based on these optimization objectives. Constraints include capacity constraints (a video data stream can only be bound to a module that supports its required processing task type), capacity constraints (the sum of the resource consumption coefficients of all tasks allocated to a module cannot exceed the current idle resource capacity of that module), and latency constraints (for tasks with latency tolerance requirements, their estimated processing latency must be less than the tolerance).

[0089] During the solution process, the algorithm takes the currently monitored data flow, the working status of each module (resource utilization, queue length), and the task priority / resource consumption coefficient / latency tolerance and module function / performance indicators called from the attribute archive as input. Within a preset time window (e.g., 1ms), it searches for the binding relationship matrix (N rows and M columns, representing the mapping relationship and task allocation between N streams and M modules) of N video streams to M processing modules, so that the weighted optimization objective function value is optimal under the above constraints.

[0090] The solution algorithm can be a heuristic algorithm, such as a greedy algorithm or a simplified genetic algorithm, which searches the assignment matrix Assign{N,M} within a preset time to maximize the objective function value.

[0091] The solution matrix of the algorithm model represents the optimal binding relationship and includes suggested values ​​for the video bus data bit width to meet throughput and latency targets. For example, after calculating the total bandwidth requirement, the lowest clock frequency corresponding to the required bit width is selected from the static bit width configuration of the video bus. The video bus bit width supports unified static configurations such as 64-bit, 128-bit, and 256-bit. Increasing the bit width can reduce the clock frequency while maintaining transmission bandwidth. When increasing from 64-bit to 256-bit, the clock frequency can be reduced to one-quarter of the original for the same data volume requirement, reducing timing convergence problems caused by high clock frequencies. Higher data bit width leads to higher transmission bandwidth, enabling the system to achieve high-speed data transmission at lower clock frequencies, improving data processing efficiency, reducing the difficulty of FPGA internal timing design, and improving system stability and reliability.

[0092] The AHB bus control module 203 obtains the optimal binding relationship and video bus width determined by the scene adaptive binding engine 202 through an internal interface or shared memory, and encodes the above information into specific configuration information and writes it into the configuration register of the target module.

[0093] Furthermore, the configuration information includes the identifier of the video data stream, the processing function enable signal, and the processing parameters.

[0094] The AHB bus control module 203 writes the identifier (Stream ID) of the video data stream into the target stream ID register of the image processing module, which can then filter data from the video bus.

[0095] The AHB bus control module 203 writes the processing function enable signal into the corresponding bit of the function control register of the image processing module. The image processing module controls the activation of the processing function based on the enable signal, such as setting bit 0 to enable noise reduction and bit 1 to enable edge detection.

[0096] The AHB bus control module 203 writes processing parameters into the parameter register group of the image processing module. For example, for denoising tasks, it writes the Gaussian kernel size and Sigma value; for edge detection, it writes the high and low thresholds. The image processing module adjusts the operational behavior of the algorithm kernel based on these parameters.

[0097] The AHB bus control module 203 writes the data bit width into the bit width configuration register of the video bus, and the video bus transmits data with the configured data bit width value.

[0098] Furthermore, the system also includes a dynamic triggering and reconfiguration module, comprising a trigger condition monitor and a reconfiguration unit. The trigger condition monitor can be implemented by a monitoring thread running on a soft-core processor, used to monitor the following three types of events:

[0099] A. Scene change events: Receive signals from external sensors, such as a light sensor indicating a switch from day to night, or a mode switching command sent by an upper-layer application, such as switching from regular monitoring to intrusion detection mode.

[0100] B. System status events: monitoring the resource utilization rate of a certain image processing module for a certain period of time, or the temperature sensor reporting that the FPGA junction temperature is too high;

[0101] C. External command events: Users input reconfiguration commands through the software interface, such as system initialization, or adding / changing input interfaces.

[0102] In response to scene change events, system status events, and / or external command events, the reconfiguration unit sends a rebinding request to the scene adaptive binding engine 202 and triggers the AHB bus control module 203 to update the configuration information based on the new binding relationship.

[0103] Furthermore, the dynamic triggering and reconfiguration module is configured to perform different levels of reconfiguration based on the detected event type and severity, in order to balance response speed and impact on the system.

[0104] The dynamic triggering and reconfiguration module also includes a hierarchical reconfiguration unit, which is configured to select a reconfiguration level based on the event type, including a first-level, second-level, and third-level reconfiguration. Specifically, the first-level reconfiguration triggers the AHB bus control module to update the processing parameters in the binding configuration information. The second-level reconfiguration triggers an adjustment to the bit width of the video bus, and the third-level reconfiguration triggers the scene adaptive binding engine to update the binding relationships.

[0105] For example, in response to a scene change event switching from daytime to nighttime mode, the binding relationship may not need to be changed, but the processing parameters need to be adjusted (such as increasing image gain in low light). The hierarchical reconfiguration unit triggers the first reconfiguration level and sends a request to the scene adaptive binding engine 202. After making a decision, the scene adaptive binding engine notifies the AHB bus control module 203 to update only the processing parameter registers of the relevant modules (such as adjusting the denoising intensity and edge detection threshold).

[0106] For example, in response to a system status event indicating excessive resource utilization, the hierarchical reconfiguration unit triggers a second and / or third configuration level. At the second configuration level, the scene adaptive binding engine 202 disables the binding algorithm solution and only reassesses bandwidth requirements based on monitoring data. At the third configuration level, the scene adaptive binding engine 202 not only resolves the binding relationships and migrates some tasks to idle modules, but also reassesses bandwidth requirements and decides to adjust the bit width of the video bus to reduce clock frequency and alleviate timing pressure. The hierarchical reconfiguration unit triggers the AHB bus control module 203 to write a new bit width value to the video bus bit width configuration register. Simultaneously, the AHB bus control module updates the configuration information of modules involved in the binding relationship change.

[0107] For example, severe system status events responding to module failure alarms or external command events involving core algorithm replacement may involve activating backup modules or switching processing pipeline structures. The hierarchical reconfiguration unit triggers the third configuration level, requiring the scene adaptive binding engine 202 to perform a comprehensive rebinding calculation and thorough re-optimization. After the binding engine calculates the new binding relationship, the AHB control module will perform a wide-ranging configuration information update, potentially even triggering a reconfiguration of the intelligent interface adapter module.

[0108] Through a hierarchical reconfiguration triggering mechanism, the system can adapt to dynamic changes both internally and externally at the most appropriate cost and speed, maintaining efficient and stable operation.

[0109] Based on the same inventive concept, this invention provides an FPGA-based intelligent image processing method, applicable to any embodiment of the FPGA-based intelligent image processing system of this invention, the method as follows: Figure 4 As shown.

[0110] S1 identifies the interface type of at least one image data source and converts image data from different interface types into a unified format video data stream based on the identification result.

[0111] This step is performed by the intelligent interface adaptation module 201. When the system powers on or a new device is connected, the intelligent interface adaptation logic within the FPGA begins operation. It samples the physical characteristics of the input signal and matches these characteristics with built-in templates to identify the interface type. If it is a standard interface, it loads preset conversion logic; if it is an unknown interface, it derives protocol rules based on the characteristics and dynamically generates conversion logic. Finally, it converts all input data into a unified format video data stream with a unique Stream ID.

[0112] S2, based on at least one processing task to be executed and the working status of each image processing module inside the FPGA, determine the binding relationship between the video data stream and at least one image processing module, and determine the data bit width of the video bus.

[0113] After the system starts running, the scene adaptive binding engine 202 periodically executes a scheduling loop. It first collects information, including: receiving a list of processing tasks from the upper-layer application, reading real-time data traffic from the video bus from the status monitoring unit, obtaining the latest operating status of each image processing module from the AHB bus control module, and querying the static attributes of tasks and modules from the attribute archive. Then, using a built-in optimization algorithm, with the goals of maximizing throughput, minimizing latency, and balancing load, and constrained by capacity, capacity, and latency, it solves for the optimal binding relationship between the current video data stream and each image processing module, and determines the optimal video bus data bit width that meets performance requirements.

[0114] S3, Based on the data bit width, configure the data bit width of the video bus through the AHB bus so that the video bus transmits the video data stream with the configured data bit width.

[0115] S4. Based on the binding relationship, configuration information is sent to the image processing module via the AHB bus so that the image processing module processes the video data stream based on the configuration information.

[0116] Steps S3 and S4 are executed by the AHB bus control module 203, which receives the binding decision from the scene adaptive binding engine 202. Based on the bit width value of the decision, it configures the bit width register of the video bus, and the crossbar switches inside the video bus complete the routing table configuration according to the binding relationship. Simultaneously, the binding relationship is encoded into configuration information for each module and written to the configuration register of the corresponding module via the AHB bus. After configuration, each image processing module, based on the configuration information in its configuration register, captures the specified data stream from the video bus for processing. The processed result data stream is returned to the video bus and flows to the next processing stage or output interface.

[0117] During this process, the AHB bus control module 203 periodically reads the data from the status registers of each module and feeds back these operating status data to the scene adaptive binding engine 202 to update its understanding of the system status, thereby initiating the next binding decision cycle.

[0118] Furthermore, during system operation, the dynamic triggering and reconfiguration module continuously monitors for triggering events such as scene changes, abnormal system status, or external commands. Once detected, it assesses the required reconfiguration level based on the event type and triggers the execution of the corresponding level of reconfiguration operation. The process then jumps to step S2 or S3 to dynamically adjust the binding relationship, bus width, or module configuration parameters.

[0119] The above embodiments are merely preferred embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the appended claims.

Claims

1. An intelligent image processing system based on FPGA, characterized in that, include: The intelligent interface adapter module is located at the entrance of the video bus and is configured to identify the interface type of at least one image data source, and convert image data from different interface types into a unified format video data stream based on the identification result. The scene-adaptive binding engine is configured to determine the binding relationship between the video data stream and at least one image processing module based on at least one processing task to be executed and the working status of each image processing module inside the FPGA, and to determine the data bit width of the video bus. The AHB bus control module is communicatively connected to the scene adaptive binding engine and is configured to configure the data width of the video bus via the AHB bus based on the data width, so that the video bus transmits the video data stream with the configured data width. Based on the binding relationship, configuration information is sent to the image processing module via the AHB bus, so that the image processing module processes the video data stream based on the configuration information; The intelligent interface adaptation module includes: The protocol feature acquisition unit is configured to acquire the data format features and timing features of the image data in real time. The interface type recognition unit is configured to match the data format features and temporal features of the image data with a pre-stored feature template; if the match is successful, the standard conversion logic corresponding to the feature template is invoked; if the match fails, a temporary conversion logic for the image data is generated based on the data format features and temporal features. The protocol conversion unit is configured to convert the image data into a video data stream of a unified format through the standard conversion logic or the temporary conversion logic; The scene adaptive binding engine includes: The status monitoring unit is configured to continuously monitor the data flow of the video bus and the working status of each image processing module, including resource utilization and processing queue length. The attribute archive stores the priority, resource consumption coefficient, and processing latency tolerance of each processing task, as well as the processing functions and performance indicators of each image processing module. The binding algorithm unit is configured to establish a constrained algorithm model with the optimization objectives of maximizing system throughput, minimizing total processing latency, and balancing resource load. The algorithm model is input with the currently monitored data flow, the working status of each image processing module, and the data retrieved from the attribute archive. The built-in optimization algorithm is used to solve the optimal binding relationship between the video data stream and at least one image processing module within a preset time window.

2. The FPGA-based intelligent image processing system according to claim 1, characterized in that, The interface type identification unit includes: The timing analysis subunit is configured to deduce the generation rules of line synchronization, field synchronization and pixel clock based on the timing characteristics of the image data when a match fails. The data analysis subunit is configured to determine the arrangement order and bit width of pixel data based on the data format characteristics when a match fails. The logic generation subunit is configured to generate the temporary conversion logic based on the generation rules and the arrangement order and bit width of the pixel data.

3. The FPGA-based intelligent image processing system according to claim 1, characterized in that, The AHB bus control module is also configured to: The running status data is periodically read from the status register of each image processing module via the AHB bus. The running status data includes at least one of the following: processing frame rate, resource utilization rate, and error flag. The running status data is fed back to the scene adaptive binding engine so that the scene adaptive binding engine can update the working status of each image processing module.

4. The FPGA-based intelligent image processing system according to claim 1, characterized in that, The configuration information includes the identifier of the video data stream, the processing function enable signal, and the processing parameters; The identifier of the video data stream is used to indicate the video data stream that the image processing module needs to process; the processing function enable signal is used to enable or disable the processing function of the image processing module; and the processing parameters are used to adjust the operational behavior of the image processing module.

5. The FPGA-based intelligent image processing system according to claim 1, characterized in that, It also includes a dynamic triggering and reconfiguration module, which is configured to send a rebinding request to the scene adaptive binding engine in response to scene change events, system status events and / or external command events, and trigger the AHB bus control module to update the configuration information based on the new binding relationship.

6. The FPGA-based intelligent image processing system according to claim 5, characterized in that, The dynamic triggering and reconfiguration module includes: The trigger condition monitor is configured to continuously monitor scene change events, system status events, and / or external command events; The hierarchical reconfiguration unit is configured to select a reconfiguration level based on the event type, wherein the reconfiguration level includes a first reconfiguration level, a second reconfiguration level, and a third reconfiguration level. Specifically, the first configuration level triggers the AHB bus control module to update the processing parameters in the configuration information, the second configuration level triggers the adjustment of the bit width of the video bus, and the third configuration level triggers the scene adaptive binding engine to update the binding relationship.

7. The FPGA-based intelligent image processing system according to claim 1, characterized in that, The video bus includes a bit width configuration register; the AHB bus control module is also configured to write bit width configuration values ​​to the bit width configuration register via the AHB bus to adjust the data bit width of the video bus.

8. An intelligent image processing method based on FPGA, characterized in that, The method is applied to the FPGA-based intelligent image processing system according to any one of claims 1-7, comprising: Identify the interface type of at least one image data source, and convert image data from different interface types into a video data stream in a unified format based on the identification results; Based on at least one processing task to be executed and the working status of each image processing module inside the FPGA, the binding relationship between the video data stream and at least one image processing module is determined, and the data bit width of the video bus is determined. Based on the data bit width, the data bit width of the video bus is configured via the AHB bus so that the video bus transmits the video data stream with the configured data bit width; Based on the binding relationship, configuration information is sent to the image processing module via the AHB bus, so that the image processing module processes the video data stream based on the configuration information.

Images