FPGA heterogeneous data processing method based on digital-intelligent integrated platform

Abstract

The invention relates to the technical field of data processing, and discloses an FPGA (Field Programmable Gate Array) heterogeneous data processing method based on a digital-intelligent integrated platform, which comprises the following steps that: a platform management layer responds to a data processing task request, schedules an acceleration node from an FPGA resource pool, and issues a pre-compilation hardware acceleration bit stream and configuration parameters; loading the hardware acceleration bit stream by an FPGA acceleration node, and dynamically configuring to form a hardware processing pipeline matched with the data processing task request; the method comprises the following steps: receiving an original data stream input by an external data source through a high-speed network interface, and writing the original data stream into an onboard memory of the high-speed network interface; the hardware processing pipeline carries out real-time processing on the original data stream in the onboard memory; and writing the processed result data into a storage device for persistence, and sending the generated metadata and event data to the platform management layer. High-speed processing of data streams is realized in a mode of reconstructing an FPGA hardware processing assembly line.

Description

Technical Field

[0001] This invention relates to the field of data processing technology and discloses an FPGA heterogeneous data processing method based on a digital intelligence integrated platform. Background Technology

[0002] With the rapid development of big data, artificial intelligence, and the Internet of Things, the total amount of global data is experiencing explosive growth. Against this backdrop, data processing tasks face severe challenges, including massive data volumes, high real-time processing requirements, and complex and diverse data formats. Traditional CPU-centric data processing architectures often encounter performance bottlenecks when handling such high-throughput, low-latency streaming data due to the limited energy efficiency and parallel processing capabilities of general-purpose computing models.

[0003] In existing technologies, FPGA-based data processing solutions typically suffer from rigid resource management, redundant data processing links, and low platform integration. FPGA accelerators are often statically bound to specific servers, with fixed functions that cannot be flexibly reconfigured according to dynamic tasks, resulting in low resource utilization. Simultaneously, the process of data transmission from the network to the processing unit often requires CPU intervention, introducing additional latency and overhead, and lacks efficient data flow management and verification mechanisms. Furthermore, the entire system lacks a unified intelligent management layer, leading to inefficient task deployment, resource configuration, and monitoring and maintenance, making it difficult to meet the stringent requirements of integrated digital platforms for real-time performance, throughput, and flexibility. Summary of the Invention

[0004] This application provides an FPGA heterogeneous data processing method based on a digital intelligence integrated platform to solve the technical problems mentioned in the background.

[0005] To achieve the above objectives, this invention provides a heterogeneous data processing method for FPGAs based on a digital intelligence integration platform, the method specifically including: The platform management layer responds to data processing task requests by scheduling FPGA acceleration nodes from the FPGA resource pool and distributing pre-compiled hardware acceleration bitstreams and configuration parameters. The FPGA acceleration node loads the pre-compiled hardware acceleration bitstream and dynamically configures it to form a hardware processing pipeline that matches the data processing task request. The system receives raw data streams from external data sources via a high-speed network interface and writes the raw data streams into the onboard memory of the high-speed network interface. The hardware processing pipeline processes the raw data stream in the onboard memory in real time; The processed result data is written to a storage device for persistence, and the generated metadata and event data are sent to the platform management layer.

[0006] Furthermore, the platform management layer responds to data processing task requests by scheduling acceleration nodes from the FPGA resource pool and issuing pre-compiled hardware acceleration bitstreams and configuration parameters, specifically including: Receive the data processing task request and parse the processing type, performance indicators and resource requirements contained in the data processing task request; Based on the analysis results, an idle acceleration node that meets the requirements is matched from the FPGA resource pool, and resource reservation and scheduling are performed. Retrieve the pre-compiled bitstream file corresponding to the processing type from the pre-stored hardware-accelerated bitstream library; The pre-compiled bitstream file, along with the associated driver and configuration parameters, is distributed to the scheduled FPGA acceleration node via the management network.

[0007] Furthermore, the FPGA acceleration node loads the pre-compiled hardware-accelerated bitstream and dynamically configures it to form a hardware processing pipeline that matches the data processing task request, specifically including: Receive and parse the pre-compiled hardware-accelerated bitstream and configuration parameters issued by the platform management layer; Based on the pre-compiled bitstream file, the programmable logic resources of the FPGA are reconstructed through an internal configuration interface; Based on the configuration parameters, the functional modules in the hardware processing pipeline are initialized. After configuration, a dedicated hardware pipeline architecture containing at least one data processing stage is formed inside the FPGA.

[0008] Furthermore, receiving the raw data stream from an external data source via a high-speed network interface and writing the raw data stream into the onboard memory of the high-speed network interface specifically includes: A data transmission channel is established with the external data source through the high-speed network interface; On the FPGA acceleration node, the data receiving controller directly writes the raw data stream arriving at the high-speed network interface into a designated buffer in the onboard memory via DMA. The data receiving controller performs integrity verification and sorting on the original data stream based on the data packet sequence number or timestamp. Independent buffer regions are allocated in the onboard memory for different types of data streams, and the buffer status descriptors are updated for the hardware processing pipeline to read.

[0009] Furthermore, on the FPGA acceleration node, the data receiving controller directly writes the raw data stream arriving at the network interface into a designated buffer in the onboard memory via DMA, specifically including: The data receiving controller monitors the network interface receiving status and triggers a DMA transfer request when the data packet arrives. Based on the pre-configured buffer descriptor information, establish a DMA transfer channel from the network interface to the specified buffer in onboard memory; During DMA transfer, the data receiving controller directly manages the data movement without the CPU participating in data transmission; After a single DMA transfer is completed, the corresponding buffer status descriptor is updated to mark the data as ready.

[0010] Furthermore, the data receiving controller performs integrity verification and sorting of the original data stream based on the data packet sequence number or timestamp, specifically including: Parse the header information of the input data packet and extract the data packet sequence number or timestamp field; The sequence number or timestamp of the current data packet is compared with the expected value to detect whether there is packet loss, out-of-order delivery, or duplication. When out-of-order data packets are detected, they are rearranged into the correct order based on their sequence number or timestamp. For data packets that fail verification or cannot be recovered, an error flag is generated and recorded in the error status register; Only data streams that pass integrity checks and are in the correct order are written to the designated buffer in the onboard memory.

[0011] Furthermore, allocating independent buffer regions for different types of data streams in the onboard memory and updating the buffer state descriptors for the hardware processing pipeline to read specifically includes: Based on the data processing task request, logically independent buffer areas are divided in the onboard memory for data streams from different sources or in different formats. Create a corresponding buffer status descriptor for each independent buffer region. The descriptor shall at least include the buffer base address, capacity, current write position and data validity status. After data is successfully written to a specific buffer area, the current write position and data validity status in the corresponding buffer status descriptor are updated in real time. The hardware processing pipeline monitors changes in the buffer state descriptor through polling or interruption, and reads valid data from the corresponding buffer area for processing based on the updated descriptor information.

[0012] Furthermore, the hardware processing pipeline performs real-time processing on the raw data stream in the onboard memory, specifically including: Raw data blocks are read sequentially from the onboard memory using pipelined control logic; At least one preset operation is performed in at least one processing stage of the hardware processing pipeline; Each processing stage interacts with data via an on-chip high-speed bus or registers, forming a parallel processing architecture; The processing status of the hardware processing pipeline is monitored in real time, and abnormal events are recorded in the status register.

[0013] Furthermore, in at least one processing stage of the hardware processing pipeline, at least one preset operation is executed, specifically including: Data format conversion: Convert the raw data from the first format to the second format specified by the platform; Rule-based filtering: Matching and filtering data content based on a pre-loaded set of filtering rules; Data compression: Real-time compression of data streams using lossless or lossy compression algorithms; AI reasoning: Performs reasoning analysis on data through a pre-built neural network computing engine.

[0014] Furthermore, the processed result data is written to a storage device for persistence, and the generated metadata and event data are sent to the platform management layer, specifically including: The processed result data is written to the designated persistent storage device in the form of data blocks through a dedicated storage controller; Generate descriptive metadata corresponding to the result data and event messages related to the processing procedure; The metadata and event messages are uploaded to the platform management level in real time through an independent management channel; The platform management layer updates the data catalog based on the received metadata and performs task status monitoring and resource scheduling optimization based on event messages.

[0015] Compared with the prior art, the beneficial effects of the present invention are as follows: Using a digital intelligence integrated platform as a carrier, end-to-end optimization of heterogeneous data processing of FPGAs is achieved. Dispersed FPGA resources are integrated into a unified schedulable resource pool. The platform management layer can intelligently schedule nodes and remotely distribute bit streams based on dynamic task requests, enabling dynamic reconfiguration of FPGA hardware functions and greatly improving resource utilization flexibility and efficiency. Through DMA direct writing, data stream integrity verification and sorting, and descriptor-based buffer management, an efficient and reliable data path from the network interface to onboard memory is constructed, significantly reducing CPU overhead and data transmission latency. The hardware processing pipeline realizes parallel pipelined processing of operations such as data format conversion, filtering, compression, and AI inference, greatly improving the real-time performance and throughput of data processing. Finally, the processing results are persisted and interact with metadata and event data with the platform management layer in real time, forming a closed-loop management system. This enables transparent monitoring of task status and continuous resource optimization, thereby comprehensively solving the core problems of resource rigidity, link redundancy, and inefficient management in existing technologies. Attached Figure Description

[0016] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings: Figure 1 A flowchart illustrating the FPGA heterogeneous data processing method based on a digital intelligence integration platform in an embodiment of the present invention is shown. Detailed Implementation

[0017] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and are not intended to limit the scope of the invention.

[0018] In the description of this application, it should be understood that the terms "center", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this application.

[0019] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0020] In the description of this application, it should be noted that, unless otherwise expressly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection between two components. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0021] The following is a description of preferred embodiments of the present invention in conjunction with the accompanying drawings.

[0022] like Figure 1 As shown, embodiments of the present invention disclose an FPGA heterogeneous data processing method based on a digital intelligence integrated platform. To further illustrate the technical concept of the present invention, the technical solution of the present invention will now be described in conjunction with specific application scenarios.

[0023] This invention provides a method for heterogeneous data processing on an FPGA based on a digital intelligence integrated platform, specifically including: The platform management layer responds to data processing task requests by scheduling FPGA acceleration nodes from the FPGA resource pool and distributing pre-compiled hardware acceleration bitstreams and configuration parameters. The FPGA acceleration node loads the pre-compiled hardware acceleration bitstream and dynamically configures it to form a hardware processing pipeline that matches the data processing task request. The system receives raw data streams from external data sources via a high-speed network interface and writes the raw data streams into the onboard memory of the high-speed network interface. The hardware processing pipeline processes the raw data stream in the onboard memory in real time; The processed result data is written to a storage device for persistence, and the generated metadata and event data are sent to the platform management layer.

[0024] In this embodiment, the data processing task request specifically includes: the type of data to be processed (such as video stream, sensor data, etc.), the required processing operations (such as format conversion, AI inference, data filtering, etc.), and specific requirements for processing performance indicators (such as throughput, latency) and resource requirements.

[0025] In this embodiment, the hardware processing pipeline is a dedicated hardware architecture dynamically reconfigured by the FPGA acceleration node according to task requirements. It forms a high-performance data path within the FPGA, containing multiple processing stages, by loading a pre-compiled bitstream. This pipeline directly reads raw data from onboard memory and utilizes its parallel processing capabilities to execute specific operations such as data format conversion, rule filtering, compression, or AI inference sequentially or in parallel, ultimately outputting the processing results efficiently. This achieves high throughput and low latency processing through software-defined hardware execution.

[0026] The beneficial effects of the above technical solution are as follows: By constructing a software-hardware collaborative digital intelligence integrated platform, end-to-end optimization of FPGA heterogeneous data processing is achieved. Its core beneficial effect is that the method integrates scattered FPGA resources into a unified schedulable resource pool, enabling the platform management layer to intelligently schedule nodes and remotely distribute bit streams based on dynamic task requests, realizing dynamic reconfiguration of FPGA hardware functions and greatly improving the flexibility and utilization rate of resource utilization. At the same time, by constructing an efficient processing path from the network interface through onboard memory to the dedicated hardware pipeline, and combining it with the automated management of event data and metadata, a complete closed loop from task distribution, real-time processing to status monitoring is formed, thereby significantly improving the throughput, real-time performance, and overall energy efficiency of the system.

[0027] S110: In some embodiments of the present invention, the platform management layer responds to data processing task requests, schedules acceleration nodes from the FPGA resource pool, and issues pre-compiled hardware acceleration bitstreams and configuration parameters, specifically including: Receive the data processing task request and parse the processing type, performance indicators and resource requirements contained in the data processing task request; Based on the analysis results, an idle acceleration node that meets the requirements is matched from the FPGA resource pool, and resource reservation and scheduling are performed. Retrieve the pre-compiled bitstream file corresponding to the processing type from the pre-stored hardware-accelerated bitstream library; The pre-compiled bitstream file, along with the associated driver and configuration parameters, is distributed to the scheduled FPGA acceleration node via the management network.

[0028] In this embodiment, matching idle acceleration nodes that meet the requirements from the FPGA resource pool and performing resource reservation and scheduling specifically includes: the platform management layer uses the processing type, performance indicators, and resource requirements obtained from parsing the task request as filtering conditions to intelligently match acceleration nodes that are currently idle and whose hardware specifications meet the requirements from the FPGA resource pool. Then, resource reservation and scheduling are immediately performed on the node, and its status is marked as "allocated" to ensure the exclusivity of resources and the determinism of task execution, thereby laying the foundation for subsequent bit stream distribution and hardware configuration.

[0029] The beneficial effects of the above technical solution are as follows: Through an intelligent resource matching and scheduling mechanism, efficient and accurate allocation of FPGA heterogeneous computing resources is achieved. This enables the platform management layer to dynamically select acceleration nodes with matching hardware specifications and in an idle state from the resource pool based on the specific processing type, performance indicators, and resource requirements of the task request, and immediately execute resource reservation and state locking. This not only ensures the immediate availability and exclusivity of the computing resources required by the task and avoids resource conflicts, but also provides a stable and reliable hardware foundation for subsequent bitstream distribution and hardware pipeline construction. Furthermore, it significantly improves the overall utilization rate of the FPGA resource pool and the determinism of task execution at the system level, realizing a fundamental shift from "static binding" to "dynamic on-demand allocation".

[0030] S120: In some embodiments of this application, the FPGA acceleration node loads the pre-compiled hardware-accelerated bitstream and dynamically configures it to form a hardware processing pipeline that matches the data processing task request, specifically including: Receive and parse the pre-compiled hardware-accelerated bitstream and configuration parameters issued by the platform management layer; Based on the pre-compiled bitstream file, the programmable logic resources of the FPGA are reconstructed through an internal configuration interface; Based on the configuration parameters, the functional modules in the hardware processing pipeline are initialized. After configuration, a dedicated hardware pipeline architecture containing at least one data processing stage is formed inside the FPGA.

[0031] In this embodiment, the configuration parameters typically include key initialization information such as the operating mode, threshold, address mapping, and algorithm coefficients of each functional module in the hardware pipeline (such as data interface, processing engine, memory controller, etc.).

[0032] In this embodiment, the functional module includes a data interface unit, a processing engine unit, and a memory controller unit.

[0033] In this embodiment, the dedicated hardware pipeline architecture is a high-performance data path formed by reconfiguring logic resources within an FPGA, containing multiple levels of data processing stages, which can perform customized hardware acceleration processing of the data stream in a parallel pipeline manner.

[0034] The beneficial effects of the above technical solution are as follows: Through the dynamic reconfiguration and parameterized configuration of FPGA logic resources, on-demand customization and efficient deployment of hardware acceleration capabilities are achieved. By receiving and parsing the pre-compiled bitstream and configuration parameters containing key information such as module operating mode and address mapping issued by the platform, the FPGA acceleration node can quickly reconfigure its programmable logic resources and accurately initialize core units such as data interfaces, processing engines, and memory controllers, thereby building a dedicated hardware pipeline architecture that is highly matched to the current data processing task inside the chip. This architecture decomposes complex data processing tasks into multi-stage pipelines, realizing full-process hardware parallel acceleration of data from access, processing to output. This not only significantly improves data processing throughput and real-time performance but also greatly enhances the flexibility and adaptability of the entire system to cope with diverse computing tasks.

[0035] S130: In some embodiments of this application, receiving the raw data stream input from an external data source through a high-speed network interface and writing the raw data stream into the onboard memory of the high-speed network interface specifically includes: A data transmission channel is established with the external data source through the high-speed network interface; On the FPGA acceleration node, the data receiving controller directly writes the raw data stream arriving at the high-speed network interface into a designated buffer in the onboard memory via DMA. The data receiving controller performs integrity verification and sorting on the original data stream based on the data packet sequence number or timestamp. Independent buffer regions are allocated in the onboard memory for different types of data streams, and the buffer status descriptors are updated for the hardware processing pipeline to read.

[0036] In this embodiment, the data transmission channel refers to a stable, high-bandwidth logical communication link negotiated and established with an external data source through a high-speed network interface, providing a foundation for the reliable transmission of subsequent raw data streams.

[0037] In this embodiment, the data receiving controller running on the FPGA acceleration node directly writes the raw data stream arriving at the network interface into a designated buffer in the onboard memory via DMA, specifically including: The data receiving controller monitors the network interface receiving status and triggers a DMA transfer request when the data packet arrives. Based on the pre-configured buffer descriptor information, establish a DMA transfer channel from the network interface to the specified buffer in onboard memory; During DMA transfer, the data receiving controller directly manages the data movement without the CPU participating in data transmission; After a single DMA transfer is completed, the corresponding buffer status descriptor is updated to mark the data as ready.

[0038] In this embodiment, the data receiving controller performs integrity verification and sorting of the original data stream based on the data packet sequence number or timestamp, specifically including: Parse the header information of the input data packet and extract the data packet sequence number or timestamp field; The sequence number or timestamp of the current data packet is compared with the expected value to detect whether there is packet loss, out-of-order delivery, or duplication. When out-of-order data packets are detected, they are rearranged into the correct order based on their sequence number or timestamp. For data packets that fail verification or cannot be recovered, an error flag is generated and recorded in the error status register; Only data streams that pass integrity checks and are in the correct order are written to the designated buffer in the onboard memory.

[0039] In this embodiment, allocating independent buffer areas for different types of data streams in the onboard memory and updating the buffer state descriptor for the hardware processing pipeline to read specifically includes: Based on the data processing task request, logically independent buffer areas are divided in the onboard memory for data streams from different sources or in different formats. Create a corresponding buffer status descriptor for each independent buffer region. The descriptor shall at least include the buffer base address, capacity, current write position and data validity status. After data is successfully written to a specific buffer area, the current write position and data validity status in the corresponding buffer status descriptor are updated in real time. The hardware processing pipeline monitors changes in the buffer state descriptor through polling or interruption, and reads valid data from the corresponding buffer area for processing based on the updated descriptor information.

[0040] The beneficial effects of the above technical solution are as follows: By constructing a highly automated data receiving and preprocessing channel, performance optimization is achieved from data access to processing readiness. The data receiving controller uses DMA to directly move data arriving at the network interface to onboard memory, effectively avoiding the latency and resource overhead caused by CPU intervention. Through a sequence number or timestamp-based integrity verification and sorting mechanism, the integrity and order of the data stream are guaranteed. Simultaneously, by dividing different data streams into independent buffers and using dynamically updated state descriptors, both data classification management and an efficient and reliable data supply mechanism are achieved for the backend hardware processing pipeline. This series of technical measures collectively constructs a high-throughput, low-latency, and reliable data transmission path, laying a solid foundation for subsequent real-time processing.

[0041] S140: In some embodiments of this application, the hardware processing pipeline performs real-time processing on the raw data stream in the onboard memory, specifically including: Raw data blocks are read sequentially from the onboard memory using pipelined control logic; At least one preset operation is performed in at least one processing stage of the hardware processing pipeline; Each processing stage interacts with data via an on-chip high-speed bus or registers, forming a parallel processing architecture; The processing status of the hardware processing pipeline is monitored in real time, and abnormal events are recorded in the status register.

[0042] In this embodiment, at least one preset operation is executed in at least one processing stage of the hardware processing pipeline, specifically including: Data format conversion: Convert the raw data from the first format to the second format specified by the platform; Rule-based filtering: Matching and filtering data content based on a pre-loaded set of filtering rules; Data compression: Real-time compression of data streams using lossless or lossy compression algorithms; AI reasoning: Performs reasoning analysis on data through a pre-built neural network computing engine.

[0043] The beneficial effects of the above technical solution are as follows: by constructing a hardware processing pipeline, a qualitative leap in data processing performance is achieved, forming a true parallel processing architecture; this architecture can flexibly perform various operations such as data format conversion, rule filtering, data compression, and AI inference, transforming complex software algorithms into hardware parallel computing, thereby greatly improving data processing throughput and real-time performance; at the same time, by monitoring the processing status in real time and recording anomalies, the reliability and maintainability of the processing process are ensured, ultimately providing a low-latency, high-energy-efficiency hardware acceleration solution for high-load data applications.

[0044] S150: In some embodiments of this application, the processed result data is written to a storage device for persistence, and the generated metadata and event data are sent to the platform management layer, specifically including: The processed result data is written to the designated persistent storage device in the form of data blocks through a dedicated storage controller; Generate descriptive metadata corresponding to the result data and event messages related to the processing procedure; The metadata and event messages are uploaded to the platform management level in real time through an independent management channel; The platform management layer updates the data catalog based on the received metadata and performs task status monitoring and resource scheduling optimization based on event messages.

[0045] The beneficial effects of the above technical solution are as follows: By constructing a closed-loop system for data output and management, collaborative optimization with intelligent system management is achieved. A dedicated storage controller directly writes the result data into persistent storage devices in the form of data blocks, ensuring high throughput and reliability of data storage. Simultaneously, descriptive metadata and processing event messages are automatically generated and transmitted back to the platform management layer in real time through an independent management channel. This enables the platform to dynamically update the data catalog to maintain a global data view and optimize task status monitoring and resource scheduling strategies based on accurate event feedback. Thus, while ensuring the integrity of data processing, the observability, automation level, and resource utilization efficiency of the entire system are significantly improved.

[0046] In the description of the above embodiments, specific features, structures, materials, or characteristics may be combined in any suitable manner in one or more embodiments or examples.

[0047] Although the invention has been described above with reference to embodiments, various modifications can be made and components can be replaced with equivalents without departing from the scope of the invention. In particular, as long as there is no structural conflict, the features in the embodiments disclosed in this invention can be combined with each other in any way. The fact that not all of these combinations are described in this specification is merely for the sake of brevity and resource conservation.

[0048] It will be understood by those skilled in the art that the above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for heterogeneous data processing using FPGA based on a digital intelligence integrated platform, characterized in that: include: The platform management layer responds to data processing task requests by scheduling FPGA acceleration nodes from the FPGA resource pool and distributing pre-compiled hardware acceleration bitstreams and configuration parameters. The FPGA acceleration node loads the pre-compiled hardware acceleration bitstream and dynamically configures it to form a hardware processing pipeline that matches the data processing task request. The system receives raw data streams from external data sources via a high-speed network interface and writes the raw data streams into the onboard memory of the high-speed network interface. The hardware processing pipeline processes the raw data stream in the onboard memory in real time; The processed result data is written to a storage device for persistence, and the generated metadata and event data are sent to the platform management layer.

2. The FPGA heterogeneous data processing method based on a digital intelligence integrated platform according to claim 1, characterized in that, The platform management layer responds to data processing task requests by scheduling acceleration nodes from the FPGA resource pool and distributing pre-compiled hardware acceleration bitstreams and configuration parameters, specifically including: Receive the data processing task request and parse the processing type, performance indicators and resource requirements contained in the data processing task request; Based on the analysis results, an idle acceleration node that meets the requirements is matched from the FPGA resource pool, and resource reservation and scheduling are performed. Retrieve the pre-compiled bitstream file corresponding to the processing type from the pre-stored hardware-accelerated bitstream library; The pre-compiled bitstream file, along with the associated driver and configuration parameters, is distributed to the scheduled FPGA acceleration node via the management network.

3. The FPGA heterogeneous data processing method based on a digital intelligence integrated platform according to claim 1, characterized in that, The FPGA acceleration node loads the pre-compiled hardware-accelerated bitstream and dynamically configures it to form a hardware processing pipeline that matches the data processing task request, specifically including: Receive and parse the pre-compiled hardware-accelerated bitstream and configuration parameters issued by the platform management layer; Based on the pre-compiled bitstream file, the programmable logic resources of the FPGA are reconstructed through an internal configuration interface; Based on the configuration parameters, the functional modules in the hardware processing pipeline are initialized. After configuration, a dedicated hardware pipeline architecture containing at least one data processing stage is formed inside the FPGA.

4. The FPGA heterogeneous data processing method based on a digital intelligence integrated platform according to claim 1, characterized in that, The process includes receiving raw data streams from external data sources via a high-speed network interface and writing the raw data streams into the onboard memory of the high-speed network interface. A data transmission channel is established with the external data source through the high-speed network interface; On the FPGA acceleration node, the data receiving controller directly writes the raw data stream arriving at the high-speed network interface into a designated buffer in the onboard memory via DMA. The data receiving controller performs integrity verification and sorting on the original data stream based on the data packet sequence number or timestamp. Independent buffer regions are allocated in the onboard memory for different types of data streams, and the buffer status descriptors are updated for the hardware processing pipeline to read.

5. The FPGA heterogeneous data processing method based on a digital intelligence integrated platform according to claim 4, characterized in that, The data receiving controller running on the FPGA acceleration node directly writes the raw data stream arriving at the network interface into a designated buffer in the onboard memory via DMA, specifically including: The data receiving controller monitors the network interface receiving status and triggers a DMA transfer request when the data packet arrives. Based on the pre-configured buffer descriptor information, establish a DMA transfer channel from the network interface to the specified buffer in onboard memory; During DMA transfer, the data receiving controller directly manages the data movement without the CPU participating in data transmission; After a single DMA transfer is completed, the corresponding buffer status descriptor is updated to mark the data as ready.

6. The FPGA heterogeneous data processing method based on a digital intelligence integrated platform according to claim 4, characterized in that, The data receiving controller performs integrity verification and sorting of the original data stream based on the data packet sequence number or timestamp, specifically including: Parse the header information of the input data packet and extract the data packet sequence number or timestamp field; The sequence number or timestamp of the current data packet is compared with the expected value to detect whether there is packet loss, out-of-order delivery, or duplication. When out-of-order data packets are detected, they are rearranged into the correct order based on their sequence number or timestamp. For data packets that fail verification or cannot be recovered, an error flag is generated and recorded in the error status register; The data stream that has passed integrity verification and is in the correct order is written to the designated buffer in the onboard memory.

7. The FPGA heterogeneous data processing method based on a digital intelligence integrated platform according to claim 4, characterized in that, In the onboard memory, independent buffer areas are allocated for different types of data streams, and the buffer status descriptors are updated for the hardware processing pipeline to read. Specifically, this includes: Based on the data processing task request, logically independent buffer areas are divided in the onboard memory for data streams from different sources or in different formats. Create a corresponding buffer status descriptor for each independent buffer region. The descriptor shall at least include the buffer base address, capacity, current write position and data validity status. After data is successfully written to a specific buffer area, the current write position and data validity status in the corresponding buffer status descriptor are updated in real time. The hardware processing pipeline monitors changes in the buffer state descriptor through polling or interruption, and reads valid data from the corresponding buffer area for processing based on the updated descriptor information.

8. The FPGA heterogeneous data processing method based on a digital intelligence integrated platform according to claim 1, characterized in that, The hardware processing pipeline performs real-time processing on the raw data stream in the onboard memory, specifically including: Raw data blocks are read sequentially from the onboard memory using pipelined control logic; At least one preset operation is performed in at least one processing stage of the hardware processing pipeline; Each processing stage interacts with data via an on-chip high-speed bus or registers, forming a parallel processing architecture; The processing status of the hardware processing pipeline is monitored in real time, and abnormal events are recorded in the status register.

9. The FPGA heterogeneous data processing method based on a digital intelligence integrated platform according to claim 8, characterized in that, In at least one processing stage of the hardware processing pipeline, at least one preset operation is executed, specifically including: Data format conversion: Convert the raw data from the first format to the second format specified by the platform; Rule-based filtering: Matching and filtering data content based on a pre-loaded set of filtering rules; Data compression: Real-time compression of data streams using lossless or lossy compression algorithms; AI reasoning: Performs reasoning analysis on data through a pre-built neural network computing engine.

10. The FPGA heterogeneous data processing method based on a digital intelligence integrated platform according to claim 1, characterized in that, The processed result data is written to a storage device for persistence. When the generated metadata and event data are sent to the platform management layer, the specific steps include: The processed result data is written to the designated persistent storage device in the form of data blocks through a dedicated storage controller; Generate descriptive metadata corresponding to the result data and event messages related to the processing procedure; The metadata and event messages are uploaded to the platform management level in real time through an independent management channel; The platform management layer updates the data catalog based on the received metadata and performs task status monitoring and resource scheduling optimization based on event messages.

Images