A T+0 real-time data collection and service processing device for a data middle platform

Abstract

The utility model discloses to solve the problem of the existing technology, and the assembly is complicated, and data flow link is long, not only greatly increases operation and maintenance management complexity, but also leads to end-to-end data delay generally at minute level, cannot break through time efficiency bottleneck, and difficult to achieve the required second level even millisecond level T+0 real-time processing requirement of data middle platform, provide a kind of T+0 real-time data acquisition and service processing device for data middle platform.The device includes by backplane circuit integration in same physical cabinet: unified access module, stream processing engine module, embedded service module, control and monitoring panel;Cabinet built-in 12V DC power module and cooling fan group.Relying on FPGA core processing circuit and PCIe4.0 direct connection architecture, compress data flow link, realize data cleaning, correlation and other real-time processing, break through minute level delay bottleneck, achieve second level / millisecond level T+0 processing.

Description

Technical Field

[0001] This utility model relates to the field of hardware acceleration and real-time processing technology for data middleware, and in particular to a T+0 real-time data acquisition and service-oriented processing device for data middleware. Background Technology

[0002] A data platform is a core infrastructure for enterprise digital transformation. Its core value lies in integrating multi-source, heterogeneous data scattered across various business systems, breaking down data silos, and transforming raw data into reusable, standardized data assets through unified data governance and service-oriented encapsulation. This provides real-time data support for business decision-making, intelligent analysis, and precise operations. Essentially, it constructs a closed-loop process for data collection, processing, storage, and service, requiring data flow to possess high compatibility, low latency, high security, and controllability. This makes it a key support for enterprises to improve the efficiency of data value transformation and quickly respond to market demands.

[0003] Traditional data platforms often employ a T+1 batch processing model or achieve near real-time processing through a combination of multiple independent tools such as Canal / Kafka for data collection, Flink / Spark for processing, and Redis / API gateway services. This approach involves complex components and lengthy data flow chains, significantly increasing operational complexity and resulting in end-to-end data latency typically in the minutes range. It fails to overcome timeliness bottlenecks and cannot meet the second- or even millisecond-level T+0 real-time processing requirements of a data platform, severely hindering the immediate conversion of data value and rapid business response. Utility Model Content

[0004] This invention addresses the problems in existing technologies where complex components and lengthy data flow links significantly increase the complexity of operation and maintenance management, and result in end-to-end data latency generally in the minute range, making it impossible to break through the timeliness bottleneck and achieve the second-level or even millisecond-level T+0 real-time processing requirements of data middleware. It provides a T+0 real-time data acquisition and service-oriented processing device for data middleware.

[0005] The technical solution adopted in this utility model is:

[0006] A T+0 real-time data acquisition and service-oriented processing device for a data middleware platform, characterized in that it includes components integrated into the same physical chassis via backplane circuitry:

[0007] The unified access module includes a multi-protocol adaptation circuit, a multi-threaded memory buffer circuit, and a data verification circuit. The multi-protocol adaptation circuit integrates a MySQLBinlog listening interface, an Oracle log receiving interface, a MongoDBoplog adaptation interface, and an HTTP stream access port. The multi-threaded memory buffer circuit uses a DDR4 memory chip and an FPGA control circuit to form a circular queue structure. The data verification circuit includes a CRC32 verification module.

[0008] The streaming processing engine module includes an FPGA core processing circuit, a PCIe 4.0 bus interface circuit, and a processing logic configuration interface. The FPGA core processing circuit is divided into a data cleaning unit, a formatting unit, a dimension association unit, and a lightweight aggregation unit. It is directly connected to the unified access module through the PCIe 4.0 bus interface circuit.

[0009] The embedded service module includes a memory database chip, an API gateway interface circuit, an ESAM security control module, and a flow control circuit. The memory database chip uses a Redis-compatible DDR4 high-speed memory array. The API gateway interface circuit integrates a RESTful interface chip and a gRPC protocol processing chip. The ESAM security control module includes a hardware encryption engine and a key storage unit. The embedded service module is connected to the streaming processing engine module through a shared memory interface.

[0010] The control and monitoring panel includes an ARM main control chip, an LCD display circuit, an audible and visual alarm circuit, and an Ethernet configuration interface. The ARM main control chip communicates with each module via an I2C bus, and the audible and visual alarm circuit includes a red LED indicator and a buzzer.

[0011] The chassis has a built-in 12V DC power supply module and a cooling fan assembly.

[0012] Furthermore, the multi-protocol adapter circuit uses an STM32F407 microcontroller as the core for protocol parsing, and each data source interface is configured with an independent signal conditioning circuit, including an impedance matching resistor network and a surge protection diode.

[0013] Furthermore, the FPGA core processing circuit uses the Xilinx Alveo U50 chip, and its internal logic units are divided into: a data receiving submodule, a data processing submodule, and a shared memory interface submodule. Each submodule achieves data synchronization through an internal FIFO buffer.

[0014] Furthermore, the memory database chip uses dual-port DDR4 memory, and the parallel access of the streaming processing engine module and the API gateway interface circuit is realized through the memory controller circuit. The memory address space is divided into data storage area and index area, and a circular overlay storage mechanism is adopted.

[0015] Furthermore, the ESAM security control module uses the HT2804 chip, which integrates an AES-256 encryption engine, a true random number generator, and an anti-tampering detection circuit. It is connected to the memory database chip via the SPI bus to achieve hardware-level permission verification for data access.

[0016] Furthermore, the flow control circuit uses the MAX32660 chip, which has a built-in flow statistics register and threshold comparator. When the API query QPS exceeds the preset value, the flow limiting mechanism is triggered by a hardware interrupt signal.

[0017] Furthermore, the circular queue structure of the multi-threaded memory buffer circuit implements read and write pointer management through FPGA control circuit, and configures a non-volatile SRAM chip to store breakpoint location information, supporting data continuation after power failure.

[0018] Furthermore, the ARM main control chip of the control and monitoring panel adopts the STM32H750 chip, and collects the operating parameters of each module through the Modbus protocol. The operating parameters include bus bandwidth utilization, FPGA operation cycle, memory usage and interface error count.

[0019] Furthermore, the backplane circuit adopts a 6-layer PCB design, integrating PCIe 4.0 signal traces, power distribution network and ground shielding layer. Each module is connected to the backplane through gold finger slots, supporting hot-swapping.

[0020] Furthermore, the 12V DC power supply module uses the LM2596 step-down chip, with an output voltage accuracy of ±2%, and is equipped with a capacitor filter array and overcurrent protection circuit to provide an independent power supply circuit for each module.

[0021] The beneficial effects of this utility model are:

[0022] This utility model provides a T+0 real-time data acquisition and service-oriented processing device for a data middleware platform. It adopts an integrated design of "unified access + streaming processing + embedded service-oriented" modules, replacing the multi-tool splicing mode. The entire link is integrated within the same chassis and backplane circuit, significantly reducing component connection links and simplifying operation and maintenance complexity. Relying on the FPGA core processing circuit and PCIe 4.0 direct connection architecture, it compresses the data flow link, enabling real-time processing such as data cleaning and correlation, breaking through minute-level latency bottlenecks and achieving second / millisecond-level T+0 processing. Multi-protocol adaptation circuits integrate various data source interfaces, allowing compatibility with heterogeneous data without additional adapters, improving access adaptability. DDR4 + FPGA circular queues and CRC32 verification ensure efficient and complete data transmission. ESAM security control and flow control circuits strengthen security defenses and prevent overload. Combined with centralized monitoring by the ARM main controller, it ensures stable operation, comprehensively meeting the core requirements of the data middleware platform for real-time performance, simplicity, security, and reliability. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a T+0 real-time data acquisition and service-oriented processing device used in a data middle platform. Detailed Implementation

[0025] In the description of this utility model, it should be understood that the terms "center", "longitudinal", "transverse", "length", "width", "thickness", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", "clockwise", "counterclockwise", "axial", "radial", "circumferential", etc., indicating the orientation or positional relationship are based on the orientation or positional relationship shown in the accompanying drawings, and are only for the convenience of describing this utility model and simplifying the description, and are not intended to indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this utility model.

[0026] The following disclosure provides many different embodiments or examples for implementing various structures of this invention. To simplify the disclosure, specific examples of components and arrangements are described below. Of course, these are merely examples and are not intended to limit the scope of this invention.

[0027] The embodiments of the utility model will now be described in detail with reference to the accompanying drawings.

[0028] As attached Figure 1 As shown, the T+0 real-time data acquisition and service-oriented processing device for data middleware provided in this embodiment includes: a unified access module 1, a streaming processing engine module 2, an embedded service-oriented module 3, and a control and monitoring panel 4.

[0029] Unified access module 1 includes a multi-protocol adaptation circuit, a multi-threaded memory buffer circuit, and a data verification circuit. The multi-protocol adaptation circuit integrates a MySQLBinlog listening interface, an Oracle log receiving interface, a MongoDBoplog adaptation interface, and an HTTP stream access port. The multi-threaded memory buffer circuit uses a DDR4 memory chip and an FPGA control circuit to form a circular queue structure. The data verification circuit includes a CRC32 verification module.

[0030] The streaming processing engine module 2 includes an FPGA core processing circuit, a PCIe 4.0 bus interface circuit, and a processing logic configuration interface. The FPGA core processing circuit is divided into a data cleaning unit, a formatting unit, a dimension association unit, and a lightweight aggregation unit. It is directly connected to the unified access module 1 (through the backplane circuit) via the PCIe 4.0 bus interface circuit.

[0031] Embedded service module 3 includes a memory database chip, an API gateway interface circuit, an ESAM security control module, and a flow control circuit. The memory database chip uses a Redis protocol-compatible DDR4 high-speed memory array. The API gateway interface circuit integrates a RESTful interface chip and a gRPC protocol processing chip. The ESAM security control module includes a hardware encryption engine and a key storage unit. Embedded service module 3 is connected to streaming processing engine module 2 via a shared memory interface (through a backplane circuit).

[0032] The control and monitoring panel 4 includes an ARM main control chip, an LCD display circuit, an audible and visual alarm circuit, and an Ethernet configuration interface. The ARM main control chip communicates with each module through the I2C bus. The audible and visual alarm circuit includes a red LED indicator and a buzzer.

[0033] The unified access module 1, streaming engine module 2, embedded service module 3, and control and monitoring panel 4 are integrated into the same physical chassis via backplane circuitry. The chassis has a built-in 12V DC power supply module and a cooling fan assembly.

[0034] The beneficial effects of the above technical solution are as follows: The T+0 real-time data acquisition and service-oriented processing device for the data platform provided in this embodiment adopts an integrated design of "unified access + streaming processing + embedded service" modules, replacing the multi-tool splicing mode. The entire link is integrated in the same chassis and backplane circuit, which greatly reduces the component connection links and simplifies the operation and maintenance complexity. Relying on the FPGA core processing circuit and PCIe4.0 direct connection architecture, the data flow link is compressed to realize real-time processing such as data cleaning and correlation, breaking through the minute-level latency bottleneck and achieving second-level / millisecond-level T+0 processing. The multi-protocol adaptation circuit integrates various data source interfaces, which can be compatible with heterogeneous data without additional conversion equipment, improving access adaptability. DDR4 + FPGA circular queue and CRC32 check ensure efficient and complete data transmission. ESAM security control and flow control circuit build a solid security defense and avoid overload. With the ARM main control centralized monitoring, stable operation is ensured, and the core requirements of the data platform for real-time performance, simplicity, security and reliability are fully met.

[0035] Furthermore, the multi-protocol adapter circuit uses an STM32F407 microcontroller as the core for protocol parsing, and each data source interface is configured with an independent signal conditioning circuit, including an impedance matching resistor network and a surge protection diode.

[0036] Furthermore, the FPGA core processing circuit uses the Xilinx Alveo U50 chip, and its internal logic units are divided into: a data receiving submodule, a data processing submodule, and a shared memory interface submodule. Each submodule achieves data synchronization through an internal FIFO buffer.

[0037] Furthermore, the memory database chip uses dual-port DDR4 memory, and the parallel access between the streaming processing engine module 2 and the API gateway interface circuit is realized through the memory controller circuit. The memory address space is divided into a data storage area and an index area, and a circular overlay storage mechanism is adopted.

[0038] Furthermore, the ESAM security control module uses the HT2804 chip, which integrates an AES-256 encryption engine, a true random number generator, and an anti-tampering detection circuit. It is connected to the memory database chip via the SPI bus to achieve hardware-level permission verification for data access.

[0039] Furthermore, the flow control circuit uses the MAX32660 chip, which has a built-in flow statistics register and threshold comparator. When the API query QPS exceeds the preset value, the flow limiting mechanism is triggered by a hardware interrupt signal.

[0040] Furthermore, the circular queue structure of the multi-threaded memory buffer circuit implements read and write pointer management through FPGA control circuit, and configures a non-volatile SRAM chip to store breakpoint location information, supporting data continuation after power failure.

[0041] Furthermore, the ARM main control chip of the control and monitoring panel 4 adopts the STM32H750 chip, and collects the operating parameters of each module through the Modbus protocol. The operating parameters include bus bandwidth utilization, FPGA operation cycle, memory usage and interface error count.

[0042] Furthermore, the backplane circuit adopts a 6-layer PCB design, integrating PCIe 4.0 signal traces, power distribution network and ground shielding layer. Each module is connected to the backplane through gold finger slots, supporting hot-swapping.

[0043] Furthermore, the 12V DC power supply module uses the LM2596 step-down chip, with an output voltage accuracy of ±2%, and is equipped with a capacitor filter array and overcurrent protection circuit to provide an independent power supply circuit for each module.

[0044] The working principle of the T+0 real-time data acquisition and service-oriented processing device for the data platform provided in this embodiment is as follows:

[0045] This T+0 real-time data acquisition and service-oriented processing device for the data middle platform operates on the core logic of "real-time access - high-speed processing - secure service - full-process monitoring." Through the coordinated operation of various functional modules, it achieves end-to-end real-time data processing from acquisition to service-oriented output. During operation, the unified access module 1 first assumes the responsibility of data entry. Its multi-protocol adaptation circuit, based on the STM32F407 microcontroller, adapts to different data sources through various dedicated interfaces such as the MySQLBinlog listening interface and the Oracle log receiving interface. Impedance matching resistor networks and surge protection diodes configured for each interface ensure stable signal transmission. The acquired data first enters a circular queue-style multi-threaded memory buffer circuit composed of DDR4 memory chips and FPGA control circuitry. The FPGA control circuit achieves efficient caching through read / write pointer management. The breakpoint location information stored in the non-volatile SRAM chip supports data continuation after power failure. Subsequently, the CRC32 verification module performs data integrity verification, filtering out invalid data.

[0046] Data that passes verification is transmitted at high speed to the streaming engine module 2 via the PCIe 4.0 bus interface circuit. The core processing circuit of this module uses the Xilinx Alveo U50 chip. The internally divided data receiving submodule, data processing submodule, and shared memory interface submodule achieve data synchronization through an internal FIFO buffer. The data processing submodule further removes redundant information through the data cleaning unit, unifies the data format through the formatting unit, establishes data association relationships through the dimension association unit, and completes real-time statistical calculations through the lightweight aggregation unit, thereby realizing real-time data processing and meeting the T+0 processing time requirement.

[0047] The processed structured data is transmitted to the embedded service module 3 via a shared memory interface. The memory database chip (compatible with the Redis protocol) composed of dual-port DDR4 memory supports parallel access between the streaming processing engine module 2 and the API gateway interface circuit through the memory controller circuit. Its divided data storage area and index area adopt a circular overwrite storage mechanism to ensure efficient data reading and writing. The RESTful interface chip and gRPC protocol processing chip integrated in the API gateway interface circuit encapsulate the data into standardized service interfaces for downstream applications of the data platform to call. At the same time, the ESAM security control module (HT2804 chip) performs hardware-level permission verification for access to the memory database through the SPI bus using an AES-256 encryption engine, a true random number generator, and an anti-tampering detection circuit. The flow control circuit (MAX32660 chip) monitors the API query QPS through a flow statistics register and a threshold comparator. When the QPS exceeds the preset value, a hardware interrupt is triggered to limit the flow and ensure service stability.

[0048] The entire device's operation is controlled by the control and monitoring panel 4. The ARM main control chip (STM32H750) collects operating parameters such as bus bandwidth utilization, FPGA operation cycle, and memory usage of each module via the I2C bus and Modbus protocol, which are displayed in real time on the LCD display circuit. When abnormal conditions such as excessive interface error count are detected, the red LED indicator and buzzer of the audible and visual alarm circuit trigger an alarm. Each module is physically integrated through a 6-layer PCB backplane circuit (integrating PCIe 4.0 signal traces, power distribution network, and grounding shielding layer). The gold finger slot supports hot-swapping of modules. The 12V DC power supply module (LM2596 step-down chip) provides independent power supply with an accuracy of ±2% to each module through a capacitor filter array and overcurrent protection circuit. The cooling fan group ensures long-term stable operation of the device, ultimately achieving real-time, safe, and reliable operation of the entire process of data acquisition, processing, and service output.

Claims

1. A T+0 real-time data acquisition and service-oriented processing device for a data middleware platform, characterized in that, This includes circuitry integrated into the same physical chassis via backplane circuitry. The unified access module includes a multi-protocol adaptation circuit, a multi-threaded memory buffer circuit, and a data verification circuit. The multi-protocol adaptation circuit integrates a MySQLBinlog listening interface, an Oracle log receiving interface, a MongoDBoplog adaptation interface, and an HTTP stream access port. The multi-threaded memory buffer circuit uses a DDR4 memory chip and an FPGA control circuit to form a circular queue structure. The data verification circuit includes a CRC32 verification module. The streaming processing engine module includes an FPGA core processing circuit, a PCIe 4.0 bus interface circuit, and a processing logic configuration interface. The FPGA core processing circuit is divided into a data cleaning unit, a formatting unit, a dimension association unit, and a lightweight aggregation unit. It is directly connected to the unified access module through the PCIe 4.0 bus interface circuit. The embedded service module includes a memory database chip, an API gateway interface circuit, an ESAM security control module, and a flow control circuit. The memory database chip uses a Redis protocol-compatible DDR4 high-speed memory array. The API gateway interface circuit integrates a RESTful interface chip and a gRPC protocol processing chip. The ESAM security control module includes a hardware encryption engine and a key storage unit. The embedded service module connects to the streaming engine module via a shared memory interface; The control and monitoring panel includes an ARM main control chip, an LCD display circuit, an audible and visual alarm circuit, and an Ethernet configuration interface. The ARM main control chip communicates with each module via an I2C bus, and the audible and visual alarm circuit includes a red LED indicator and a buzzer. The chassis has a built-in 12V DC power supply module and a cooling fan assembly.

2. The T+0 real-time data acquisition and service-oriented processing device for a data middleware platform according to claim 1, characterized in that, The multi-protocol adapter circuit uses an STM32F407 microcontroller as the core for protocol parsing, and each data source interface is configured with an independent signal conditioning circuit, including an impedance matching resistor network and a surge protection diode.

3. The T+0 real-time data acquisition and service-oriented processing device for a data middleware platform according to claim 1, characterized in that, The FPGA core processing circuit uses the Xilinx Alveo U50 chip, and its internal logic units are divided into: a data receiving submodule, a data processing submodule, and a shared memory interface submodule. Each submodule achieves data synchronization through an internal FIFO buffer.

4. The T+0 real-time data acquisition and service-oriented processing device for a data middleware platform according to claim 1, characterized in that, The memory database chip uses dual-port DDR4 memory and enables parallel access between the streaming engine module and the API gateway interface circuit through the memory controller circuit. The memory address space is divided into a data storage area and an index area, and a circular overlay storage mechanism is adopted.

5. The T+0 real-time data acquisition and service-oriented processing device for a data middleware platform according to claim 1, characterized in that, The ESAM security control module uses the HT2804 chip, which integrates an AES-256 encryption engine, a true random number generator, and an anti-tampering detection circuit. It connects to the memory database chip via the SPI bus to achieve hardware-level permission verification for data access.

6. The T+0 real-time data acquisition and service-oriented processing device for a data middleware platform according to claim 1, characterized in that, The flow control circuit uses the MAX32660 chip, which has a built-in flow statistics register and threshold comparator. When the API query QPS exceeds the preset value, the flow limiting mechanism is triggered by a hardware interrupt signal.

7. The T+0 real-time data acquisition and service-oriented processing device for a data middleware platform according to claim 1, characterized in that, The circular queue structure of the multi-threaded memory buffer circuit implements read and write pointer management through FPGA control circuit, and configures a non-volatile SRAM chip to store breakpoint location information, supporting data continuation after power failure.

8. The T+0 real-time data acquisition and service-oriented processing device for a data middleware platform according to claim 1, characterized in that, The ARM main control chip of the control and monitoring panel uses the STM32H750 chip. It collects the operating parameters of each module through the Modbus protocol. The operating parameters include bus bandwidth utilization, FPGA operation cycle, memory usage and interface error count.

9. The T+0 real-time data acquisition and service-oriented processing device for a data middleware platform according to claim 1, characterized in that, The backplane circuit adopts a 6-layer PCB design, integrating PCIe 4.0 signal traces, power distribution network and ground shielding layer. Each module is connected to the backplane through gold finger slots, supporting hot-swapping.

10. The T+0 real-time data acquisition and service-oriented processing device for a data middleware platform according to any one of claims 1-9, characterized in that, The 12V DC power supply module uses the LM2596 step-down chip, with an output voltage accuracy of ±2%. It is equipped with a capacitor filter array and overcurrent protection circuit to provide an independent power supply circuit for each module.

Images