FPGA-based streaming data cleaning accelerator

Abstract

The present application relates to the technical field of data cleaning, and more particularly to a stream data cleaning accelerator based on FPGA, which comprises a coarse-grained reconfigurable array cleaning pipeline arranged in a programmable logic region of an FPGA chip and comprising a plurality of processing units; each processing unit is internally provided with an arithmetic logic operation core, a local register stack, a crossbar router and an instruction buffer; a configuration controller is used to issue function configuration instructions to each processing unit in the coarse-grained reconfigurable array cleaning pipeline; a quantitative perception stream neighbor interpolation engine; an interpolation value generation module is used to generate interpolation values according to a precise neighbor set; and a data interface is used to receive original data streams to be cleaned and output cleaned data records. The problems of insufficient throughput of traditional data cleaning schemes and excessively high computational complexity of K-neighbor interpolation methods are effectively solved.

Description

Technical Field

[0001] This invention belongs to the field of data cleaning technology, specifically relating to an FPGA-based streaming data cleaning accelerator. Background Technology

[0002] Field-Programmable Gate Arrays (FPGAs), as a reconfigurable hardware computing platform, offer advantages such as low power consumption, low latency, and high throughput. In recent years, they have seen increasing applications in data center acceleration, network packet processing, and high-frequency trading in finance. By directly mapping computational logic to hardware circuits, FPGAs avoid the overhead of instruction decoding and scheduling in software solutions, achieving deterministic processing latency and extremely high energy efficiency. However, the fine-grained programmable architecture of traditional FPGAs requires reloading the entire configuration bitstream during function switching, with reconfiguration times typically on the order of milliseconds. This makes them unsuitable for the dynamic adjustment of processing strategies based on data characteristics required in data cleaning scenarios.

[0003] In terms of missing value imputation techniques, simple imputation methods include mean imputation, mode imputation, and fixed constant imputation. These methods are computationally simple but do not consider the correlation between data records, resulting in low accuracy. K-nearest neighbor (KNN) based imputation methods find the few records most similar to the record to be imputed in the complete data record set, using the corresponding field values ​​of the nearest neighbor records to infer the missing value, thus better preserving the statistical distribution characteristics of the data. However, accurate KNN retrieval requires calculating the distance between the record to be imputed and all reference records. The computational complexity is linearly related to the number of reference records. When the reference data scale reaches hundreds of thousands of records, the KNN retrieval time for a single record may reach tens of milliseconds or even longer, failing to meet the real-time requirements of streaming processing.

[0004] To address the efficiency issue of exact K-nearest neighbor (KNN) search, academia and industry have proposed various approximate nearest neighbor search techniques. Locality-sensitive hashing (LSH) designs a special family of hash functions that map similar data points to the same hash bucket with a high probability, thus transforming nearest neighbor search into a local search within the hash bucket and significantly reducing search complexity. Product quantization decomposes a high-dimensional vector into multiple low-dimensional sub-vectors and performs vector quantization on each, achieving fast approximate distance calculation through a pre-computed codeword distance lookup table. While these techniques have achieved good results in software implementation, efficient implementation on hardware platforms such as field-programmable gate arrays (FPGAs) still faces many challenges, including parallel computation of hash functions, efficient storage and access of codebooks, and on-chip caching of distance lookup tables, which have not yet been systematically resolved. Summary of the Invention

[0005] The main objective of this invention is to provide an FPGA-based streaming data cleaning accelerator that achieves data-level parallelism and pipeline parallelism through a coarse-grained reconfigurable array. By employing a two-stage nearest neighbor retrieval strategy, high-dimensional distance calculations are transformed into integer index lookup operations, effectively solving the problems of insufficient throughput in traditional data cleaning schemes and excessive computational complexity of the K-nearest neighbor interpolation method. This results in high-throughput, low-latency, and high-precision streaming data cleaning processing.

[0006] To solve the above problems, the technical solution of the present invention is implemented as follows:

[0007] An FPGA-based streaming data cleaning accelerator includes:

[0008] The coarse-grained reconfigurable array cleaning pipeline is located within the programmable logic region of the FPGA chip and contains multiple processing units arranged in a two-dimensional grid topology. Adjacent processing units are interconnected through bidirectional data channels. Each processing unit contains an arithmetic logic operation core, a local register file, a crossbar switch router, and an instruction buffer.

[0009] The configuration controller is used to issue function configuration instructions to each processing unit in the coarse-grained reconfigurable array cleaning pipeline, so as to configure different processing units as processing unit groups that perform different functions.

[0010] The quantization-aware streaming nearest neighbor interpolation engine, deployed inside the FPGA chip, includes a locality-sensitive hash coarse screening module, a product quantization fine ranking module, and an interpolation value generation module. The locality-sensitive hash coarse screening module generates a composite hash signature based on the query feature vector of the data record to be interpolated and retrieves a candidate nearest neighbor set. The product quantization fine ranking module performs approximate distance calculations based on a subspace distance lookup table on the candidate data records in the candidate nearest neighbor set and generates a finely ranked nearest neighbor set. The interpolation value generation module generates interpolation values ​​based on the finely ranked nearest neighbor set.

[0011] The data interface is used to receive the raw data stream to be cleaned and to output the cleaned data records.

[0012] Furthermore, the coarse-grained reconfigurable array cleaning pipeline comprises 16 processing units arranged in a 4x4 two-dimensional grid topology. Adjacent processing units are interconnected via a 32-bit bidirectional data channel. Each processing unit contains an arithmetic logic unit, a local register file containing 32 32-bit registers, a crossbar router with 4 input ports and 4 output ports, and an instruction buffer with a capacity of 512 bytes. The four input ports of the crossbar router are connected to bidirectional data channels from the adjacent processing units above, below, left, and right, respectively. The four output ports of the crossbar router are also connected to bidirectional data channels leading to the adjacent processing units above, below, left, and right, respectively.

[0013] Furthermore, the configuration controller configures the four processing units in the first row as a format verification processing unit group, the four processing units in the second row as a missing detection processing unit group, the four processing units in the third row as a nearest neighbor interpolation processing unit group, and the four processing units in the fourth row as a deduplication output processing unit group.

[0014] Furthermore, the format verification processing unit group performs field boundary verification on the raw data stream input through the data interface, and passes the valid data records that pass the verification down to the missing data detection processing unit group through the cross switch router; the missing data detection processing unit group scans the valid data records field by field to identify missing fields, marks the data records containing missing fields as data records to be imputed and passes them to the nearest imputed processing unit group through the cross switch router, and marks the data records that do not contain missing fields as complete data records and passes them to the deduplication output processing unit group through the cross switch router.

[0015] Furthermore, the Locality Sensitive Hash (LSH) coarse screening module includes a first BRAM storage area, which stores eight sets of random projection vectors, each set containing 16 random projection vectors. The LSH coarse screening module extracts the values ​​of all non-missing fields from the data record to be imputed to form a query feature vector. The query feature vector is then multiplied element-wise with each of the 16 random projection vectors in each set, and the results are accumulated to obtain 16 inner product results. Each inner product result is compared with zero. If the inner product result is greater than or equal to zero, a binary bit 1 is recorded; if the inner product result is less than zero, a binary bit 0 is recorded. The 16 binary bits are concatenated to form a 16-bit local hash code. The eight sets of random projection vectors generate a total of eight 16-bit local hash codes. The eight local hash codes are concatenated to form a 128-bit composite hash signature.

[0016] Furthermore, the locality-sensitive hash coarse screening module also includes a second BRAM storage area, which is used to store a hash bucket index table. The hash bucket index table uses a 128-bit composite hash signature as the index key and the list of storage addresses of data records in the reference data buffer as the index value. The locality-sensitive hash coarse screening module retrieves the corresponding list of storage addresses in the hash bucket index table based on the 128-bit composite hash signature of the query feature vector, and reads all the data records pointed to by the list of storage addresses from the reference data buffer to form a candidate nearest neighbor set.

[0017] Furthermore, the product quantization and fine-ranking module includes a third BRAM storage area, which stores an orthogonal rotation matrix. The number of rows and columns of the orthogonal rotation matrix are equal to the dimension of the query feature vector. The product quantization and fine-ranking module performs matrix-vector multiplication on the query feature vector and the orthogonal rotation matrix to obtain the rotated query vector. The matrix-vector multiplication is completed collaboratively by the arithmetic logic operation kernels of the four processing units in the nearest neighbor interpolation processing unit group in a pulsating array manner. Each processing unit is responsible for calculating the element values ​​of a consecutive quarter dimension in the rotated query vector. The product quantization and fine-ranking module evenly divides the rotated query vector into eight subspace segments along the dimensional direction.

[0018] Furthermore, the product quantization fine-ranking module also includes a fourth BRAM storage area, which stores eight subspace codebooks. Each subspace codebook corresponds to one subspace segment, and each subspace codebook contains 256 codeword vectors. The dimension of each codeword vector is the same as the dimension of the corresponding subspace segment. For each subspace segment of the rotated query vector, the product quantization fine-ranking module subtracts the subspace segment from the 256 codeword vectors in the corresponding subspace codebook element by element, squares the results, and sums them to calculate the squared Euclidean distance. The index of the codeword vector with the smallest squared Euclidean distance is selected as the quantization codeword index. A total of eight quantization codeword indices are generated from the eight subspace segments to form the quantization encoding of the rotated query vector.

[0019] Furthermore, the product quantization fine ranking module also includes a fifth BRAM storage area, which stores eight subspace distance lookup tables. Each subspace distance lookup table is a two-dimensional array of 256 rows and 256 columns, with row and column indices ranging from 0 to 255. Each element in the two-dimensional array stores the squared Euclidean distance between the corresponding row index codeword vector and column index codeword vector. For each candidate data record in the candidate nearest neighbor set, the product quantization fine ranking module uses the i-th quantized codeword index in the quantized encoding of the rotated query vector as the row index and the i-th quantized codeword index in the quantized encoding of the candidate data record as the column index. It reads the corresponding element value from the i-th subspace distance lookup table as the approximate distance contribution value of the i-th subspace. The approximate distance contribution values ​​of the eight subspaces are accumulated to obtain the approximate squared Euclidean distance between the rotated query vector and the candidate data record. All candidate data records in the candidate nearest neighbor set are sorted in ascending order according to the approximate squared Euclidean distance value, and the top eight sorted candidate data records are selected to form the finely ranked nearest neighbor set.

[0020] The FPGA-based streaming data cleaning accelerator described in this invention has the following beneficial effects:

[0021] This invention employs a coarse-grained reconfigurable array cleaning pipeline as its core computing architecture. Multiple processing units are arranged in a two-dimensional grid topology and interconnected via bidirectional data channels. Each processing unit integrates an arithmetic logic core, a local register file, a cross-connect router, and an instruction buffer, forming a fully functional computing node. This architecture allows different stages of data cleaning to be mapped to processing unit groups in different rows. Processing units within the same row process multiple data records in parallel, achieving data-level parallelism. Processing units in different rows perform different functions such as format verification, missing data detection, nearest neighbor interpolation, and deduplication output, achieving pipeline parallelism. This fully leverages the parallelism of the data cleaning process and significantly improves system throughput. The configuration controller issues function configuration instructions to each processing unit to achieve runtime function reconfiguration. Compared to the traditional FPGA reconfiguration method that requires reloading the entire bitstream, this invention only needs to update the instruction buffer content of the processing unit to complete the function switch, greatly reducing reconfiguration overhead and enabling the system to dynamically adjust the cleaning strategy based on data characteristics.

[0022] The quantization-aware streaming nearest neighbor interpolation engine designed in this invention employs a two-stage retrieval strategy, effectively solving the problem of excessive computational complexity in traditional K-nearest neighbor interpolation methods. The locality-sensitive hash coarse screening module generates a composite hash signature through inner product operations and symbolic quantization of random projection vectors. It then uses a hash bucket index table to quickly locate the candidate nearest neighbor set, narrowing the search scope from all reference data records to at least a few candidate records. The product quantization fine sorting module eliminates correlations between feature dimensions through orthogonal rotation matrices, divides the rotated query vector into multiple subspace segments, and performs vector quantization. It utilizes a pre-computed subspace distance lookup table to transform high-dimensional distance calculations into integer index lookup and accumulation operations, significantly reducing computational complexity while maintaining high approximation accuracy. The interpolation value generation module uses a bitonetic sorting network and median calculation to generate interpolated values, exhibiting stronger outlier robustness compared to mean-filling methods. The entire nearest neighbor interpolation process fully utilizes the BRAM storage resources within the FPGA chip and the parallel computing capabilities of the processing unit, achieving high-precision, low-latency real-time missing value filling. Attached Figure Description

[0023] Figure 1 A schematic diagram of the core computing region of a streaming data cleaning accelerator based on a field-programmable gate array provided in an embodiment of the present invention;

[0024] Figure 2 A schematic diagram illustrating the internal microarchitecture design and data path connection relationship of a single processing unit in a coarse-grained reconfigurable array provided in this embodiment of the invention;

[0025] Figure 3 This is a schematic diagram of the hardware logic and processing flow of the local sensitive hash coarse screening module in the quantization-aware streaming nearest neighbor interpolation engine provided in an embodiment of the present invention. Detailed Implementation

[0026] An FPGA-based streaming data cleaning accelerator includes:

[0027] The coarse-grained reconfigurable array cleaning pipeline is located within the programmable logic region of the FPGA chip and contains multiple processing units arranged in a two-dimensional grid topology. Adjacent processing units are interconnected through bidirectional data channels. Each processing unit contains an arithmetic logic operation core, a local register file, a crossbar switch router, and an instruction buffer.

[0028] The configuration controller is used to issue function configuration instructions to each processing unit in the coarse-grained reconfigurable array cleaning pipeline, so as to configure different processing units as processing unit groups that perform different functions.

[0029] The quantization-aware streaming nearest neighbor interpolation engine, deployed inside the FPGA chip, includes a locality-sensitive hash coarse screening module, a product quantization fine ranking module, and an interpolation value generation module. The locality-sensitive hash coarse screening module generates a composite hash signature based on the query feature vector of the data record to be interpolated and retrieves a candidate nearest neighbor set. The product quantization fine ranking module performs approximate distance calculations based on a subspace distance lookup table on the candidate data records in the candidate nearest neighbor set and generates a finely ranked nearest neighbor set. The interpolation value generation module generates interpolation values ​​based on the finely ranked nearest neighbor set.

[0030] The data interface is used to receive the raw data stream to be cleaned and to output the cleaned data records.

[0031] The coarse-grained reconfigurable array cleaning pipeline is located within the programmable logic region of the FPGA chip. The core consideration for adopting a coarse-grained reconfigurable architecture is that while the fine-grained programmable logic units of traditional FPGAs offer extremely high flexibility, their configuration overhead is significant. Each function switch requires reloading a large number of configuration bitstreams, typically taking milliseconds, which is insufficient to meet the stringent real-time requirements of streaming data cleaning scenarios. The coarse-grained reconfigurable array pre-combines multiple basic logic units into relatively complete processing units, allowing function switches to require only modifications to the instruction sequence within the processing unit. This reduces configuration time to the microsecond level, thereby significantly reducing reconfiguration latency while maintaining architectural flexibility.

[0032] The coarse-grained reconfigurable array cleaning pipeline comprises multiple processing units; in one specific embodiment, the number of processing units is set to 16. The selection of 16 processing units is based on the fact that the data cleaning process typically includes four main stages: format verification, missing data detection, numerical imputation, and deduplication filtering. Allocating 4 processing units to each stage enables parallel processing within that stage. Furthermore, the 16 processing units occupy approximately 35% to 45% of the available FPGA logic resources of a medium-sized FPGA chip, fully utilizing chip resources while reserving sufficient space for other auxiliary circuits. In other alternative embodiments, the number of processing units can be adjusted according to the resource size of the target FPGA chip and the complexity of the data cleaning task. For example, on a resource-constrained low-cost FPGA chip, a 3x3 array of 9 processing units can be used, while on a high-end FPGA chip, this can be expanded to 25 or 36 processing units to support more complex cleaning processes.

[0033] Multiple processing units are arranged in a two-dimensional grid topology. In an implementation with 16 processing units, these units are organized in a 4x4 matrix. The two-dimensional grid topology offers unique advantages over one-dimensional linear or fully interconnected topologies: in a one-dimensional linear topology, data transmission from one end to the other requires passing through all intermediate nodes, and communication latency increases linearly with array size; while a fully interconnected topology has the lowest communication latency, the number of interconnects is proportional to the square of the number of processing units, resulting in unacceptable wiring complexity and resource consumption. The two-dimensional grid topology strikes a balance between these two extremes. The maximum number of communication hops between any two processing units is the sum of the number of rows and columns minus 2. For a 4x4 array, the maximum number of communication hops is 6. Furthermore, each processing unit only needs to maintain connections with its four adjacent processing units (up, down, left, and right), resulting in a total number of interconnects that is only approximately twice the number of processing units. More importantly, the two-dimensional grid topology naturally aligns with the phased nature of the data cleaning process. Data can flow sequentially from row 1 to row 4, with processing units in each row processing the same batch of data records in parallel, forming a pipelined cascade relationship between rows.

[0034] Adjacent processing units are interconnected via bidirectional data channels. The physical implementation of these channels uses a pair of opposing unidirectional buses, each containing a data signal line, a valid signal line, and a ready signal line. In one specific implementation, the bidirectional data channel width is set to 32 bits, meaning there are 32 data signal lines. The choice of 32 bits is based on the following considerations: in data cleaning tasks, the value of a single field is typically represented in single-precision floating-point or 32-bit fixed-point format, and a 32-bit width allows for the transmission of one field value within a single clock cycle; simultaneously, a 32-bit width allows for efficient utilization of the FPGA chip's native 32-bit or 36-bit data path resources during internal routing, avoiding resource waste caused by width mismatch. In other optional implementations, the bidirectional data channel width can be adjusted to 64 bits or 128 bits according to data format requirements to support double-precision floating-point operations or vectorized data transmission.

[0035] The bidirectional data channel employs a handshake protocol for flow control. The transmitting side sets the valid signal line high when data is valid, and the receiving side sets the ready signal line high when it has receiving capability. Data transmission only occurs when both the valid and ready signal lines are high simultaneously. This handshake mechanism ensures that adjacent processing units with different processing speeds will not experience data loss or overwriting due to speed differences, achieving automatic flow balancing.

[0036] Each processing unit contains an arithmetic logic unit (ALU), which is the core component for performing data cleaning calculations. The ALU integrates integer and floating-point arithmetic units. The integer arithmetic unit supports addition, subtraction, multiplication, division, logical AND, logical OR, logical XOR, logical NOT, left shift, and right shift operations for 32-bit integers. The floating-point arithmetic unit supports addition, subtraction, multiplication, and division operations for single-precision floating-point numbers conforming to the IEEE 754 standard. The addition and logical operations of the integer arithmetic unit are implemented using pure combinational logic and can be completed within a single clock cycle. The multiplication operation is implemented using the DSP hard core embedded in the FPGA chip, also within a single clock cycle. The division operation uses an iterative algorithm and requires multiple clock cycles. All operations in the floating-point arithmetic unit are implemented using a pipelined architecture. The pipeline depth varies depending on the operation type: addition and subtraction have a pipeline depth of 4 stages, multiplication has a pipeline depth of 3 stages, and division has a pipeline depth of 12 stages.

[0037] refer to Figure 1 This diagram illustrates the core computational region of a streaming data cleaning accelerator based on a field-programmable gate array (FPGA), namely the coarse-grained reconfigurable array cleaning pipeline. Located in the chip's programmable logic resource center, this region abandons the traditional control-flow-driven execution mode of a central processing unit (CPU) and instead adopts a dataflow-driven spatial computing architecture. From a macroscopic layout perspective, this core region includes... Homogeneous processing units, these processing units according to OK The columns are physically arranged in a two-dimensional grid topology. Within this two-dimensional grid, row indices... The range of values ​​is to Column index The range of values ​​is also... to The physical location of any processing unit can be determined by coordinates. Unique and definitive. This regularized array structure is highly compatible with the underlying logic array blocks and wiring resources of the chip, minimizing signal transmission delay within the chip.

[0038] The internal interconnect network of this array employs a statically configured mesh network. Each processing unit is not directly interconnected with all its neighbors; instead, it is connected to physically adjacent processing units in the four directions (up, down, left, and right) via point-to-point bidirectional data channels. Specifically, located at coordinates... The processing unit, through a dedicated The bit-width parallel bus is connected to the coordinates respectively. , , and The neighboring cells. For processing cells located at the array edge, their ports facing outwards are either left floating or connected to the boundary scan chain, depending on the configuration. The bit width of the data channel is set to... The bit width is designed to match the data width of single-precision floating-point numbers or standard integers, ensuring that a complete field value can be transmitted each clock cycle. This neighbor-to-neighbor interconnection method maintains data locality transmission efficiency while keeping the complexity of interconnect resources within a linear growth range, avoiding the exponential area expansion problem caused by a fully interconnected architecture.

[0039] A deep dive into the microstructure of a single processing unit reveals that each unit integrates four key functional modules: an arithmetic logic core, a local register file, a crossbar router, and an instruction buffer. The arithmetic logic core is the power source for performing data cleaning operations, and it integrates... Bit-integer arithmetic logic and conformance to the Institute of Electrical and Electronics Engineers (IEEE) It uses standard single-precision floating-point arithmetic logic. This core can complete addition, subtraction, and bitwise operations in a single cycle, and supports multi-cycle multiplication and division operations through pipelining. The local register file acts as a high-speed cache for near-memory computations, containing... indivual A general-purpose register with a bit width is used to temporarily store intermediate calculation results or unprocessed data passed from neighbors, greatly reducing the frequency of access to the chip's external memory. The crossbar switch router is a data exchange hub with five input ports and five output ports, responsible for establishing non-blocking data paths between the local processing core and its neighbors in the east, south, west, and north directions. The instruction buffer is a [missing information - likely a capacity value]. The on-chip static random access memory of a byte is used to store the sequence of microinstructions that the current processing unit needs to execute during a specific cleaning phase.

[0040] From the perspective of functional mapping, this The line processing unit is configured as a line with a depth of Coarse-grained production line. OK Each processing unit is configured as a format verification level, responsible for receiving the raw data stream and using its internal arithmetic logic core to check the type matching and numerical range boundaries of the data fields in parallel. OK Each processing unit is configured as a missing value detection level. It receives valid data from the previous row and uses bitmasking operations to quickly scan for null values ​​or non-numeric markers in the fields, marking records containing missing values ​​as pending imputation. OK The processing units constitute the nearest neighbor interpolation stage, which is the most computationally dense region of the entire array. This row of units is configured to perform vector distance calculations and weighted average operations to fill in missing field values. OK The first processing unit, acting as the end of the pipeline, is configured as a deduplication output stage. It is responsible for calculating the fingerprint hash value of data records and filtering duplicate data based on fingerprint collisions. The data flow in the array exhibits an overall top-down trend, meaning data records flow from the first... The input line is processed in succession by each processing unit, and finally from the first line... Output rows. Within each row, data can also be swapped between neighbors in the horizontal direction according to algorithm requirements. For example, in the nearest neighbor interpolation stage, some data needs to be swapped in the first row. The processing units within a row perform pulsed transfers to accumulate the global distance. This hybrid computing model of inter-row pipelined and intra-row parallelism fully leverages the data-level parallelism and task-level parallelism in streaming data cleaning tasks, enabling the accelerator to achieve extremely high data throughput at very low clock frequencies.

[0041] The operation selection of the arithmetic logic core is controlled by opcodes. Each opcode is 6 bits wide and can encode 64 different operation types. The current implementation uses 23 of these opcodes, with the remaining opcodes reserved for future feature expansion. The arithmetic logic core has two source operand input ports and one destination operand output port. Source operands can come from the local register file or immediate values, and the destination operand is written to the local register file.

[0042] Each processing unit also has a local register file, which provides high-speed operand storage and access capabilities for the arithmetic logic core. In one specific implementation, the local register file contains 32 registers, each with a bit width of 32 bits, for a total capacity of 128 bytes. This configuration of 32 registers references the design experience of classic reduced instruction set processors, satisfying the storage requirements of most data cleaning algorithms for temporary variables without increasing access latency due to an excessive number of registers. The local register file is implemented using the distributed random access memory resources within the FPGA chip, possessing dual-port read and single-port write capabilities. It can simultaneously read two source operands and write one destination operand within a single clock cycle, supporting fully pipelined operation of the arithmetic logic core.

[0043] Of the 32 registers in the local register file, register 0 is hardwired to a constant zero. Any write operations to it are ignored, and reading from this register always returns zero. This design simplifies instruction coding for operations requiring zero values. Registers 1 through 8 are designated as temporary data registers, used to store data received from or about to be sent to adjacent processing units. Registers 9 through 28 are general-purpose registers, freely usable by instructions. Registers 29 through 31 are designated as control registers, used to store the current processing unit's functional configuration status, data validity flags, and error status flags, respectively.

[0044] Each processing unit also contains a crossbar router, which manages data exchange between the processing unit and its neighboring units. The crossbar router has four input ports and four output ports. In an implementation with 16 processing units arranged in a 4x4 grid, the four input ports are connected to the receiving ends of bidirectional data channels from the adjacent processing unit above, below, left, and right, respectively. The four output ports are connected to the transmitting ends of the bidirectional data channels to the adjacent processing units above, below, left, and right, respectively. For processing units located at the array edge, no input or output ports in the direction of adjacent processing units are left floating or connected to a data interface.

[0045] refer to Figure 2 The processing unit, as the basic computing node of the streaming data cleaning accelerator, has its core logic encapsulated within a clearly defined physical boundary, communicating with other units in the array through standardized interfaces. From an overall layout perspective, the processing unit integrates five key functional modules: a five-port crossbar router at the center, an arithmetic logic unit on the left, a local register file on the right, an instruction buffer at the top, and data input / output ports distributed around the perimeter. This layout aims to minimize the length of interconnects between modules, thereby reducing signal transmission latency.

[0046] At the heart of this processing unit is a five-port crossover router, which serves as the data flow scheduling hub. This router is configured with five data receiving ports: North Input, South Input, West Input, East Input, and Local Input; and correspondingly, five data transmitting ports: North Output, South Output, West Output, East Output, and Local Output. Internally, the router contains a multiplexer network controlled by configuration bits, capable of establishing non-blocking connections between any input port and any output port based on routing configuration commands issued from the command buffer. Bit-width parallel data paths. For example, a data stream from the north input port can be directly routed to the south output port to be passed to the next row processing unit, or routed to the local output port to enter the arithmetic logic core for computation.

[0047] The arithmetic logic unit located on the left side of this processing unit is the power engine for executing specific cleaning algorithms. This core operates through two... The bit-wide operand bus is connected to the local output port and the read port of the local register file of the crossbar switch router. The arithmetic core integrates an integer arithmetic unit and a single-precision floating-point arithmetic unit, supporting addition, subtraction, multiplication, division, and bitwise logic operations. When performing a calculation task, the arithmetic core obtains the source operand from the input bus, completes the calculation within a single clock cycle or pipeline cycle, and writes the calculation result back to the local input port of the crossbar switch router via the result bus, which then distributes it to other units or stores it in the local register file. To improve computational efficiency, this architecture designs a dedicated fast operand channel between the arithmetic logic core and the local register file, allowing the arithmetic core to directly read register data without router forwarding, thereby reducing the latency of register read operations.

[0048] The local register file located to the right of this processing unit provides high-speed temporary data storage capabilities. This register file contains... There are 1 physical register, each with a bit width of 1. The total capacity is sufficient to accommodate intermediate variables and sliding window data in complex cleaning algorithms. This register file has dual-port read / write capabilities, enabling simultaneous operand reading and writing of the previous calculation result within one clock cycle for the arithmetic logic core. The instruction buffer located above this processing unit stores the microinstruction sequence for the current stage. This buffer is connected to the router, arithmetic core, and register file via control decoding logic, responsible for parsing instructions and generating corresponding control signals in each clock cycle, such as routing selection signals, opcodes, and register read / write enable signals, thereby driving the entire processing unit to work collaboratively. For external data interaction, this processing unit interconnects with adjacent units through interfaces in four directions: north, south, west, and east. The north and south ports are primarily used for vertical data flow transmission, while the west and east ports are primarily used for horizontal data exchange or broadcasting. All data channels utilize... The bit width is designed to match the storage format of standard data fields.

[0049] The crossbar switch router employs a full crossbar switch matrix internal structure. Data from any input port can be routed to any one or more output ports, or to the local register file within the processing unit. Data in the local register file can also be sent to any one or more output ports via the crossbar switch router. The routing configuration of the crossbar switch matrix is ​​controlled by a set of configuration registers, the contents of which determine the connection relationships between each input and output port. The crossbar switch router supports three basic routing modes: in cut-through mode, data is forwarded directly from one input port to its corresponding output port without processing by the local processing unit; in aggregation mode, data from multiple input ports is aggregated into the local register file for processing; and in distribution mode, data generated by the local processing unit is distributed to multiple output ports. These three modes can be combined to meet complex data flow scheduling requirements.

[0050] Each processing unit also has an instruction buffer, which stores the sequence of instructions required for the processing unit to perform data cleaning tasks. In one specific implementation, the capacity of the instruction buffer is set to 512 bytes. Each instruction is 32 bits long, or 4 bytes, so the instruction buffer can store 128 instructions. The capacity of 128 instructions is sufficient to accommodate the implementation of typical algorithms for each stage of data cleaning. For example, the deduplication algorithm based on the cuckoo filter requires approximately 45 instructions in hardware implementation, and the nearest neighbor search algorithm based on locality-sensitive hashing requires approximately 78 instructions, both of which are within the capacity range of the instruction buffer. The instruction buffer is implemented using the block random access memory resources inside the FPGA chip, and has single-port read capability, allowing one instruction to be read per clock cycle for execution by the arithmetic logic core.

[0051] The instruction buffer is equipped with an instruction pointer register, the value of which indicates the address of the currently executing instruction within the instruction buffer. During normal execution, the instruction pointer register automatically increments by 1 after each instruction is executed. When a jump instruction is executed, the instruction pointer register is updated to the jump target address. When a conditional branch instruction is executed, the value of the instruction pointer register is determined based on the condition judgment result, either by incrementing by 1 or updating to the branch target address. The contents of the instruction buffer are written by the configuration controller during the processing unit function configuration phase and remain unchanged during the data cleaning and execution phase.

[0052] The configuration controller issues function configuration commands to each processing unit in the coarse-grained reconfigurable array cleaning pipeline. It plays a centralized management role in the entire system, coordinating the function allocation and task scheduling of each processing unit. The physical implementation of the configuration controller can utilize an embedded processor core within an FPGA chip, such as an ARM Cortex-M series soft-core processor, or a dedicated finite state machine. The advantage of using an embedded processor core is that the configuration logic can be implemented through software programming, facilitating function iteration and debugging; the advantage of using a finite state machine is that the timing of the configuration process can be precisely controlled, resulting in lower configuration latency and greater determinism.

[0053] The configuration controller communicates with each processing unit via a dedicated configuration bus. The configuration bus uses a star topology, with the configuration controller located at the center. Each processing unit is connected to the configuration controller via an independent branch of the configuration bus. The configuration bus is 32 bits wide, comprising 8 address signal lines, 8 command signal lines, and 16 data signal lines. The address signal lines specify the target processing unit number; in an implementation with 16 processing units, the effective address range is 0 to 15. The command signal lines specify the type of configuration operation, supporting operations including writing to the instruction buffer, writing to the configuration register, reading the status register, and resetting the processing unit. The data signal lines are used to transmit configuration data or status data.

[0054] The controller is configured to group different processing units into groups that perform different functions. In one specific implementation, the controller configures the four processing units in the first row as a format verification processing unit group, the four processing units in the second row as a missing data detection processing unit group, the four processing units in the third row as a nearest neighbor interpolation processing unit group, and the four processing units in the fourth row as a deduplication output processing unit group. This row-based functional configuration strategy allows data to flow sequentially through each cleaning stage in the coarse-grained reconfigurable array cleaning pipeline, forming a natural pipeline structure. The four processing units in the same row perform the same function but process different data records, achieving data-level parallelism within a stage; the processing units in different rows perform different functions and process data records in different cleaning stages, achieving pipeline parallelism between stages.

[0055] The functional configuration process includes the following steps: First, the configuration controller broadcasts a reset command to all processing units, causing each unit to enter its initial state and clearing any residual data in the instruction buffer and local register file. Next, the configuration controller sequentially writes the instruction sequence corresponding to its respective processing unit group to each processing unit. For example, it writes the instruction sequence for the format verification algorithm to the four processing units in the first row, and the instruction sequence for the missing data detection algorithm to the four processing units in the second row. The configuration controller also needs to configure the routing table of the cross-switch routers within each processing unit, setting the data flow path between processing units. For example, it configures the cross-switch routers of the processing units in the first row to send processing results to the output port below, and configures the cross-switch routers of the processing units in the second row to receive data from the input port above. After completing all configuration operations, the configuration controller sends a start command to each processing unit. Each processing unit then begins executing instructions from the starting address of its respective instruction buffer, entering the data cleaning working state.

[0056] In another optional implementation, the configuration controller supports dynamic reconfiguration, which can dynamically adjust the functional allocation of processing unit groups based on data characteristics during the data cleaning process. For example, when a low proportion of missing values ​​is detected in the input data stream, the configuration controller can reconfigure some processing units originally configured as nearest neighbor interpolation processing units as deduplication output processing units to improve the throughput of the deduplication stage. During dynamic reconfiguration, the configuration controller first pauses instruction execution of the processing unit to be reconfigured, waits for its internal pipeline to be drained, then writes a new instruction sequence and configuration parameters, and finally resumes instruction execution. Due to the limited capacity of the instruction buffer and the use of block random access memory, the reconfiguration time of a single processing unit is approximately tens of microseconds, far lower than the millisecond-level time required for traditional full-chip reconfiguration of FPGAs.

[0057] The quantization-aware streaming nearest neighbor imputation engine is deployed within an FPGA chip. Its design goal is to perform real-time numerical imputation of data records with missing fields in streaming data cleaning scenarios. Traditional missing value imputation methods typically use mean imputation or fixed constant imputation. While these methods are computationally simple, they ignore the inherent correlation between data records, and the imputation results often deviate significantly from the true values. The K-nearest neighbor-based imputation method finds the few records most similar to the data record to be imputed in the existing complete data records, and uses the values ​​of these nearest neighbor records in the missing field to infer the imputation value, which can better preserve the statistical characteristics and distribution patterns of the data. However, accurate K-nearest neighbor retrieval requires calculating the distance between the data record to be imputed and all reference data records. The computational complexity is proportional to the number of reference data records, making it difficult to meet real-time requirements in large-scale data scenarios. The quantization-aware streaming nearest neighbor interpolation engine uses a two-stage retrieval strategy to resolve this contradiction: the first stage uses locality-sensitive hashing to quickly filter out a small number of candidate nearest neighbors, narrowing the search scope from all reference data records to the candidate nearest neighbor set; the second stage uses product quantization technology to finely sort the candidate nearest neighbor set, obtaining high-quality K-nearest neighbor results with lower computational cost.

[0058] The Locality Sensitive Hashing (LSH) coarse-screening module generates a composite hash signature based on the query feature vector of the data record to be imputed and retrieves a candidate nearest neighbor set. The core idea of ​​LSH is to design a special family of hash functions such that data points that are close in distance in the original high-dimensional space are likely to fall into the same hash bucket after hash mapping, while data points that are far apart are likely to fall into different hash buckets. This characteristic allows for a quick determination of whether two data points are likely neighbors by comparing their hash signatures, thus avoiding the need for precise distance calculations for all data points.

[0059] The Locality Sensitive Hashing (LSH) coarse screening module first extracts the values ​​of all non-missing fields from the data records to be imputed, forming a query feature vector. The reason for extracting non-missing fields instead of all fields is that the values ​​of missing fields are unknown and cannot participate in distance calculation and hash signature generation. Assume the data records to be imputed contain... There are 10 fields, among which If any field is missing, then the dimension of the query feature vector is... ,in This indicates the total number of fields in the data record. This indicates the number of missing fields. In one specific implementation, the data record contains 32 fields, each a 32-bit single-precision floating-point number. When there are 4 missing fields, the dimension of the query feature vector is 28.

[0060] The Locality Sensitive Hash (LSH) coarse screening module includes a first BRAM storage area, which stores multiple sets of random projection vectors. In one specific implementation, the first BRAM storage area stores eight sets of random projection vectors, each set containing 16 random projection vectors. Each element of the random projection vectors follows a standard normal distribution and is generated by the host system during system initialization and written to the first BRAM storage area via the configuration controller. The purpose of using eight sets of random projection vectors instead of a single set is to improve the distinguishability of the hash signature. A single set of random projection vectors generates a limited number of hash codes, leading to a higher probability of different data points generating the same hash code, which can easily cause a large number of non-nearest neighbor data points to be misclassified as candidate nearest neighbors. Multiple sets of random projection vectors generate hash codes separately and are then concatenated to form a composite hash signature, effectively reducing the probability of misclassification. Each set containing 16 random projection vectors is a trade-off between hash code length and computational complexity; 16 random projection vectors generate 16-bit hash codes, which has a moderate computational load and provides sufficient distinguishability.

[0061] The Locality Sensitive Hashing (LSH) coarse screening module performs an element-wise multiplication of the query feature vector with each of the 16 random projection vectors in each set of random projection vectors, followed by summing the results. Let the query feature vector be... A certain random projection vector is The result of the inner product operation is ,in This represents the query feature vector. Represents a random projection vector. Represents the vector dimension. Represents the first feature vector of the query One element, The first element of the random projection vector is... One element, This represents the result of the inner product operation. The geometric meaning of the inner product operation is to calculate the projection length of the query feature vector onto the direction of the random projection vector. This projection length reflects the degree of correlation between the query feature vector and that random direction.

[0062] refer to Figure 3 This diagram details the hardware logic and processing flow of the Locality Sensitive Hash (LSH) coarse screening module in the Quantization-Aware Streaming Nearest Neighbor Imputation Engine. The module's design goal is to leverage hardware parallel computing capabilities to quickly map high-dimensional input feature vectors into low-dimensional binary hash codes, enabling rapid screening of the nearest neighbor candidate set. From a data flow perspective, the entire processing flow proceeds from left to right through input buffering, projection calculation, symbolic quantization, and hash code storage. Each stage employs a pipelined design to meet the real-time processing requirements of streaming data.

[0063] At the beginning of the process, an input feature vector buffer is used to temporarily store query vectors to be processed. Because data records typically have high feature dimensions, this cache is designed to accommodate... A 3D single-precision floating-point vector. This input buffer is accessed via a high-bandwidth... The dimensional data broadcast bus connects to the subsequent computing array, ensuring that each dimension component in the vector can be accessed by all computing units simultaneously, thus avoiding the bandwidth bottleneck caused by serial reading. Parallel to this is the projection matrix storage area, implemented using on-chip block random access memory, used to store pre-generated random projection matrices. The size of the matrix is ,Include indivual A random projection vector of dimension. This storage area transmits the components of the projection vector to the computing array through a dedicated read port and data path.

[0064] The core computing area consists of It consists of several parallel digital signal processing cores, each corresponding to a random projection vector. These digital signal processing cores are configured to perform vector inner product operations, that is, to compute the input query vector. With the random projection vectors dot product between In terms of hardware implementation, each digital signal processing core contains a multiply-accumulate tree or pipelined multiply-accumulate unit, capable of efficiently performing multiplication and accumulation operations on corresponding components. Because... Each nucleus is physically independent, therefore this The intra-group product operation is performed in parallel within the same time window, which greatly improves the throughput of hash mapping.

[0065] Following the digital signal processing core is the symbol extraction module, which consists of... It consists of several parallel comparator circuits. Each comparator receives the inner product scalar value output by the corresponding digital signal processing core and compares it with a zero threshold. According to the principle of locality-sensitive hashing, if the inner product result is greater than or equal to... The comparator outputs logic levels. If the inner product result is less than Then output logic level This process achieves a quantization mapping from continuous real number space to discrete binary space, and the generated binary bits are the local hash bits in that projection direction. .

[0066] At the end of the process, from The output bits of the comparator to It is transmitted in parallel to the hash code register. This register is a A high-speed register with a bit width, responsible for latching this on the rising edge of one clock cycle. Each binary bit forms a complete binary digit. A local hash code. This hash code is then used as an index key to access the external hash bucket index table to obtain a list of candidate nearest neighbor addresses. The entire... Figure 3 The hardware logic demonstrated effectively solves the problems of high computational latency and large instruction overhead in traditional software implementations by decomposing high-dimensional vector operations into parallel multiply-add and compare operations and using a dedicated bus for data distribution, thus providing efficient coarse screening support for subsequent precise interpolation.

[0067] For each set of 16 random projection vectors, inner product operations are performed to obtain 16 inner product results. The locality-sensitive hashing coarse-screening module compares each inner product result with zero; a result greater than or equal to zero is recorded as a 1, and a result less than zero is recorded as a 0. This sign quantization operation discretizes consecutive inner product results into binary bits. The basis of sign quantization is that when two data points are close in high-dimensional space, their projection symbols in the same random direction have a high probability of being the same; conversely, data points that are far apart tend to have independent projection symbols in random directions. The 16 binary bits are concatenated in a fixed order to form a 16-bit local hash code. Eight sets of random projection vectors generate eight 16-bit local hash codes in this way. These eight local hash codes are concatenated in ascending order of group number to form a 128-bit composite hash signature. The 128-bit composite hash signature can distinguish... The different hash states are sufficient to guarantee a high probability that different data points will have different signatures in practical applications.

[0068] Internally, the inner product operation is implemented through the arithmetic logic core of the processing units in the nearest neighbor interpolation processing unit group. Since the eight sets of random projection vectors are independent, the eight inner product operations can be distributed among different processing units for parallel execution. In a nearest neighbor interpolation processing unit group containing four processing units, two rounds of iteration can be used to complete the eight inner product operations. In each round, each of the four processing units is responsible for one set, and the 16 inner product operations within each set are executed serially within the processing unit. Signed quantization and bit concatenation operations are performed through the logical operation instructions and shift instructions of the arithmetic logic core, resulting in extremely low computational overhead.

[0069] The locality-sensitive hash coarse screening module also includes a second BRAM storage area, which stores the hash bucket index table. The hash bucket index table uses a 128-bit composite hash signature as the index key and a list of storage addresses of data records in the reference data buffer as the index values. The reference data buffer is a pre-stored set of complete data records that do not contain missing fields; these records serve as candidate sources for nearest neighbor retrieval. During system initialization, the host system calculates the composite hash signature for each data record in the reference data buffer and organizes the storage addresses of data records with the same composite hash signature into a linked list or array, storing them in the corresponding entries of the hash bucket index table.

[0070] The hash bucket index table is organized using a direct mapping method. Since the value space of a 128-bit composite hash signature far exceeds the number of actual reference data records, directly using the 128-bit signature as the index key would result in an extremely sparse hash bucket index table. To solve this problem, signature truncation or signature hashing can be used to compress the 128-bit signature into a shorter index key. In one specific implementation, the lower 20 bits of the 128-bit composite hash signature are taken as the index key of the hash bucket index table, which contains... That is, approximately 1 million entries, each storing a pointer to a list of storage addresses and a list length counter. When multiple different 128-bit composite hash signatures are truncated to the same 20-bit index key, the storage addresses of the data records corresponding to these signatures are merged into the same storage address list. During retrieval, it is necessary to further verify the complete 128-bit signature to filter out false matches.

[0071] The Locality Sensitive Hashing (LSH) coarse-screening module retrieves the corresponding list of storage addresses in the hash bucket index table based on the 128-bit composite hash signature of the query feature vector. It then reads all the data records pointed to by these storage addresses from the reference data buffer to form a candidate nearest neighbor set. Due to the probabilistic nature of LSH, the candidate nearest neighbor set includes both data records that are truly close to the query feature vector and a small number of data records that are farther away but happen to have the same or similar hash signatures. The size of the candidate nearest neighbor set depends on the fill level of the hash buckets. With a configuration of 100,000 data records in the reference data buffer and 1 million entries in the hash bucket index table, each hash bucket contains an average of approximately 0.1 data records. Considering the uneven distribution of data, the actual size of the retrieved candidate nearest neighbor set is typically between 1 and 50 records.

[0072] The product quantization fine-ranking module performs approximate distance calculations based on a subspace distance lookup table on candidate data records in the candidate nearest neighbor set and generates a finely ranked nearest neighbor set. Product quantization is a vector compression technique whose core idea is to decompose a high-dimensional vector into multiple low-dimensional sub-vectors, perform vector quantization on each sub-vector independently, and replace the original sub-vector with the quantized codeword index. This decomposition and compression strategy can significantly reduce storage overhead and distance calculation complexity while maintaining high approximation accuracy.

[0073] The product quantization and ranking module includes a third BRAM storage area, which stores the orthogonal rotation matrix. The number of rows and columns of the orthogonal rotation matrix is ​​equal to the dimension of the query feature vector. The purpose of introducing the orthogonal rotation matrix is ​​to optimize the approximate accuracy of product quantization. Original product quantization directly divides the vector into sub-vectors according to their dimensions. This division method does not consider the correlation between dimensions. When there is a strong correlation between adjacent dimensions, the variance distribution of the sub-vectors is uneven, and the quantization error of some sub-vectors will be significantly higher than that of others. The orthogonal rotation matrix, by performing an orthogonal transformation on the vectors, maps the original feature space to a new feature space. In the new space, the correlation between dimensions is eliminated or weakened, resulting in a more uniform variance distribution for the divided sub-vectors, thereby reducing the overall quantization error. The orthogonal rotation matrix is ​​obtained during system initialization by performing principal component analysis or independent component analysis on the data records in the reference data buffer and is written to the third BRAM storage area through the configuration controller.

[0074] The product quantization and fine-ranking module performs matrix-vector multiplication on the query feature vector and the orthogonal rotation matrix to obtain the rotated query vector. Let the query feature vector be... The orthogonal rotation matrix is The query vector after rotation is ,in for orthogonal matrix, To query the dimension of the feature vectors. The property of orthogonal matrices guarantees that rotation transformations do not change the Euclidean distance between vectors, that is, for any two vectors... and ,have ,in This represents the Euclidean norm. This property ensures that nearest neighbor retrieval in the rotated feature space yields equivalent results to retrieval in the original feature space.

[0075] Matrix-vector multiplication is performed collaboratively by the arithmetic logic cores of four processing units in the nearest neighbor interpolation processing unit group using a systolic array. Each processing unit is responsible for calculating the values ​​of consecutive quarter-dimension elements in the rotated query vector. The systolic array is a highly efficient parallel computing architecture where data flows systematically between processing units. Each unit receives data, performs local computation, and passes a portion of the result to downstream processing units. In the systolic array implementation of matrix-vector multiplication, the rows of the orthogonal rotation matrix are pre-allocated to the local register files of each processing unit. The elements of the query feature vector are input sequentially from the first processing unit and passed down level by level. Each processing unit, upon receiving a vector element, multiplies it with the row elements of the matrix stored locally and accumulates the result in a partial sum register. Once all elements of the vector have been passed, the partial sum registers of each processing unit store the corresponding elements of the rotated query vector. This systolic array implementation fully utilizes the parallel computing capabilities of the coarse-grained reconfigurable array cleaning pipeline and the data channels between processing units, avoiding the communication bottlenecks associated with centralized computing.

[0076] The product quantization and ranking module uniformly divides the rotated query vector into multiple subspace segments along its dimensional direction. In one specific implementation, the rotated query vector is divided into 8 subspace segments. If the dimension of the rotated query vector is 32, each subspace segment contains 4 consecutive dimensional elements; if the dimension is 64, each subspace segment contains 8 dimensional elements. The number of subspace segments (8) is a trade-off between quantization accuracy and computational efficiency: more subspace segments result in lower dimensionality and smaller quantization error, but also require more codebooks and distance lookup tables; fewer subspace segments have the opposite effect. Practice shows that 8 subspace segments can provide sufficient approximate accuracy in most application scenarios.

[0077] The product quantization and ranking module also includes a fourth BRAM storage area, which stores eight subspace codebooks. Each subspace codebook corresponds to one subspace segment, and each subspace codebook contains 256 codeword vectors. The dimension of each codeword vector is the same as the dimension of the corresponding subspace segment. The subspace codebooks are obtained during system initialization by performing K-means clustering on the corresponding subspace segments of the data records in the reference data buffer. The 256 codeword vectors are the 256 cluster centers of the K-means clustering. The reason for choosing 256 codeword vectors is that the quantization codeword index of each subspace segment can be represented by an 8-bit binary number, which is convenient for storage and processing; at the same time, the 256 cluster centers can provide lower quantization error under most data distributions.

[0078] The product quantization fine-sorting module calculates the squared Euclidean distance between each subspace segment of the rotated query vector and all 256 codeword vectors in the corresponding subspace codebook. The squared Euclidean distance is calculated by subtracting each subspace segment from the codeword vector element-by-element, squaring the results, and then summing the sums. The subspace is segmented as follows: The first subspace codebook Each codeword vector is The squared value of the Euclidean distance is ,in The dimension for segmenting the subspace. The first segment of the subspace represents the... One element, The first codeword vector represents the... One element, This represents the squared Euclidean distance between the subspace segment and the codeword vector. After calculation, the product quantization and ranking module selects the index of the codeword vector with the smallest squared Euclidean distance as the quantized codeword index. The eight subspace segments generate a total of eight quantized codeword indices, which constitute the quantization encoding of the rotated query vector.

[0079] In another alternative implementation, to reduce the latency of quantized codeword index calculation, an approximate nearest neighbor search strategy can be used instead of an exact search. Specifically, the 256 codeword vectors in each subspace codebook are pre-organized into a KD-tree or ball tree structure and stored in the fourth BRAM storage area. When searching for the nearest codeword vector, it is not necessary to traverse all 256 codeword vectors; instead, a pruning search is performed along the tree structure, reducing the average search complexity to the logarithmic level.

[0080] The product quantization and fine-sorting module also includes a fifth BRAM storage area, which stores eight subspace distance lookup tables. Each subspace distance lookup table is a two-dimensional array of 256 rows and 256 columns, with row and column indices ranging from 0 to 255. Each element in the two-dimensional array stores the squared Euclidean distance between the corresponding row-indexed codeword vector and the column-indexed codeword vector. The subspace distance lookup tables are pre-calculated and generated during system initialization. The subspace codebook, and the corresponding subspace distance lookup table of the th subspace codebook. Line number Column elements are ,in and The first In the codebook of subspaces, the first The and the first Each subspace distance lookup table contains a codeword vector. The advantage of pre-computed subspace distance lookup tables is that they transform distance calculations during the online retrieval phase from floating-point vector operations to integer index lookup table operations, significantly reducing computational latency and power consumption. Each subspace distance lookup table contains 256 x 256, or 65536 elements. If each element is represented by a 16-bit fixed-point number, then each subspace distance lookup table has a storage capacity of 128 kilobytes, and eight subspace distance lookup tables require a total of 1 megabyte of BRAM storage space.

[0081] For each candidate data record in the candidate nearest neighbor set, the quantization code of that candidate data record is pre-stored in the reference data buffer. The quantization code of the candidate data record is calculated and generated during the system initialization phase, and the calculation process is the same as that for the quantization code of the rotated query vector. The product quantization and ranking module uses the quantization code of the rotated query vector as the first... The quantized codeword index is a row index, and the quantized codeword is the first quantized codeword in the candidate data record. The quantized codeword index is a column index, starting from the ... Read the corresponding element value from the subspace distance lookup table as the first... The approximate distance contribution value of each subspace. Let the quantization encoding of the query vector after rotation be... The quantization encoding of candidate data records is Then the first The approximate distance contribution value of each subspace is The approximate Euclidean distance squared between the rotated query vector and the candidate data record is obtained by summing the approximate distance contribution values ​​of the eight subspaces. ,in This represents the approximate squared Euclidean distance. This approximate distance calculation based on a subspace distance lookup table requires only 8 table lookups and 7 addition operations, which is a significant improvement in computational efficiency compared to the hundreds of multiplications and additions required to directly calculate high-dimensional Euclidean distances.

[0082] The product quantization fine-sorting module sorts all candidate data records in the candidate nearest neighbor set according to their approximate squared Euclidean distance values ​​from smallest to largest, and selects the top 8 candidate data records after sorting to form the fine-sorted nearest neighbor set. The number of 8 candidate data records can be adjusted according to the interpolation accuracy requirements; a larger number results in better statistical stability of the interpolation results, but also increases the computational cost. The sorting operation is implemented within the processing unit of the nearest neighbor interpolation processing unit group by executing comparison and exchange instructions through an arithmetic logic core. If the size of the candidate nearest neighbor set is small, a simple bubble sort or insertion sort algorithm can be used; if the size is large, a parallel sorting algorithm based on a comparison network can be used.

[0083] The imputation value generation module is used to generate imputation values ​​based on the fine-ranked nearest neighbor set. This module extracts the field values ​​from the eight candidate data records in the fine-ranked nearest neighbor set that correspond to the missing field positions in the data record to be imputed. Since all candidate data records in the fine-ranked nearest neighbor set are complete data records, they have valid values ​​at any field position; therefore, the values ​​at the corresponding field positions can be directly read as imputation reference values.

[0084] The interpolation value generation module stores the extracted values ​​of the eight fields into the first eight registers of the local register file of the first processing unit in the nearest interpolation processing unit group. The reason for storing the data centrally in a single processing unit is that subsequent sorting and median calculation operations require frequent comparisons and exchanges of values, and centralized storage can avoid the data transfer overhead between processing units.

[0085] The interpolation generation module uses a bitonic sorting network in the arithmetic logic unit of the first processing unit to compare the values ​​of the eight fields pairwise and rearrange them in ascending order. The bitonic sorting network is a fixed-structure comparison network whose sorting process does not depend on the initial order of the input data, has a definite number of comparisons and latency, and is very suitable for hardware implementation. For eight input elements, the bitonic sorting network includes three stages and a total of 24 comparison and exchange operations, which can be completed within 24 clock cycles of the arithmetic logic unit. The working process of the bitonic sorting network is as follows: In the first stage, the eight elements are divided into four pairs, and each pair is sorted internally to form four ordered sequences of length 2; in the second stage, the four ordered sequences of length 2 are merged into two bitonic sequences of length 4, and then sorted again to form two ordered sequences of length 4; in the third stage, the two ordered sequences of length 4 are merged into one bitonic sequence of length 8, and finally sorted to form one ordered sequence of length 8.

[0086] The interpolation value generation module selects the 4th and 5th values ​​from the sorted 8 field values. It then adds the 4th and 5th values ​​using an arithmetic logic operation core and shifts the result one bit to the right to obtain the median value. The median, rather than the mean, is chosen as the interpolation value because it is more robust to outliers. If there are a few outlier records in the finely sorted nearest neighbor set, their field values ​​may deviate significantly from the normal range. Using the mean would directly affect the final result, while using the median limits the impact of outliers to the two ends of the sorted sequence, not affecting the values ​​in the middle. For 8 ordered values, the median is defined as the average of the 4th and 5th values. A one-bit right shift is equivalent to division by 2. The reason for using a shift instead of division is that the shift operation can be implemented in hardware with only simple wiring rearrangement, eliminating the need for a divider circuit, thus significantly reducing computational latency and resource consumption.

[0087] The interpolation value generation module writes the median value as the interpolation value for the missing field into the corresponding field position of the data record to be interpolated, forming a data record with completed interpolation. The data record with completed interpolation is then transmitted to the deduplication output processing unit group through the crossbar router for further processing.

[0088] In another alternative implementation, the imputation value generation module can use a distance-weighted average method instead of the median method to generate the imputation value. Specifically, weight coefficients are calculated based on the approximate squared Euclidean distance between each candidate data record in the finely ranked nearest neighbor set and the data record to be imputed. Candidate data records that are closer in distance are assigned higher weights. The imputed value is obtained by multiplying the values ​​of each candidate data record at the missing field position by the corresponding weight coefficient and then summing them. The distance-weighted average method can make fuller use of nearest neighbor information and achieve higher imputation accuracy in scenarios with relatively smooth data distribution, but the computational complexity also increases accordingly.

[0089] The data interface receives the raw data stream to be cleaned and outputs the cleaned data records. The data interface is implemented internally within the FPGA chip using the AXI-Stream protocol. AXI-Stream is an on-chip bus protocol defined by ARM for streaming data transmission and is widely used in FPGA and system-on-chip designs. The AXI-Stream protocol uses a handshake mechanism for flow control, including signal lines such as TDATA, TVALID, TREADY, and TLAST. TDATA is the data signal line, TVALID is the data valid indicator signal on the transmitting side, TREADY is the receive ready indicator signal on the receiving side, and TLAST is the end-of-packet indicator signal. Data transmission only occurs when both TVALID and TREADY are high. This handshake mechanism ensures that data loss does not occur when the processing speeds of the transmitting and receiving sides are mismatched.

[0090] In the data receiving direction, the data interface connects to the high-speed serial transceiver or memory interface controller of the FPGA chip to receive raw data streams from the host system or external data sources. The raw data stream is encapsulated according to a predefined frame format. In one specific implementation, each data record is encapsulated as a fixed-length data frame of 128 bytes. The first 4 bytes of the data frame are the frame header, containing the record sequence number and checksum; the middle 120 bytes are the data field area, containing 30 32-bit field values; and the last 4 bytes are the frame trailer, containing an end marker. The data interface parses the received data frames, extracts the content of the data field area, and passes it to the format verification processing unit group.

[0091] In the data output direction, the data interface reads the cleaned data records from the output data buffer, encapsulates them according to the same frame format as the input, and then transmits them to the host system via the AXI-Stream protocol. The output data buffer adopts a first-in-first-out queue structure, implemented by the BRAM resources inside the FPGA chip. The queue depth can be configured from 256 to 4096 data records according to the system throughput requirements.

[0092] In another alternative implementation, the data interface supports multi-channel concurrent transmission, with each channel independently running the AXI-Stream protocol. Data from multiple channels is distributed to processing units in different columns within the coarse-grained reconfigurable array cleaning pipeline for parallel processing, thereby significantly increasing system throughput. For example, when configured with 4 input channels and 4 output channels, each column of the 4x4 processing unit array processes data from one channel, resulting in a 4x increase in system throughput compared to a single-channel configuration.

[0093] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A streaming data cleaning accelerator based on FPGA, characterized in that, include: The coarse-grained reconfigurable array cleaning pipeline is located within the programmable logic region of the FPGA chip and contains multiple processing units arranged in a two-dimensional grid topology. Adjacent processing units are interconnected through bidirectional data channels. Each processing unit contains an arithmetic logic operation core, a local register file, a crossbar switch router, and an instruction buffer. The configuration controller is used to issue function configuration instructions to each processing unit in the coarse-grained reconfigurable array cleaning pipeline, so as to configure different processing units as processing unit groups that perform different functions. The quantization-aware streaming nearest neighbor interpolation engine, deployed inside the FPGA chip, includes a locality-sensitive hash coarse screening module, a product quantization fine ranking module, and an interpolation value generation module. The locality-sensitive hash coarse screening module is used to generate a composite hash signature based on the query feature vector of the data record to be interpolated and retrieve the candidate nearest neighbor set. The product quantization fine ranking module is used to perform approximate distance calculation based on the subspace distance lookup table on the candidate data records in the candidate nearest neighbor set and generate a finely ranked nearest neighbor set. The interpolation value generation module is used to generate interpolation values ​​based on the finely sorted nearest neighbor set; The data interface is used to receive the raw data stream to be cleaned and to output the cleaned data records.

2. The FPGA-based streaming data cleaning accelerator according to claim 1, characterized in that, The coarse-grained reconfigurable array cleaning pipeline comprises 16 processing units arranged in a 4x4 two-dimensional grid topology. Adjacent processing units are interconnected via a 32-bit bidirectional data channel. Each processing unit contains an arithmetic logic unit, a local register file with 32 32-bit registers, a crossbar router with 4 input ports and 4 output ports, and a 512-byte instruction buffer. The four input ports of the crossbar router are connected to bidirectional data channels from the adjacent processing units above, below, left, and right, respectively. The four output ports of the crossbar router are also connected to bidirectional data channels leading to the adjacent processing units above, below, left, and right, respectively.

3. The FPGA-based streaming data cleaning accelerator according to claim 2, characterized in that, Configure the controller to set the four processing units in the first row as a format verification processing unit group, the four processing units in the second row as a missing detection processing unit group, the four processing units in the third row as a nearest neighbor interpolation processing unit group, and the four processing units in the fourth row as a deduplication output processing unit group.

4. The FPGA-based streaming data cleaning accelerator according to claim 3, characterized in that, The format validation processing unit group performs field boundary validation on the raw data stream input through the data interface, and passes the valid data records that pass the validation down to the missing data detection processing unit group through the cross switch router. The missing data detection processing unit group scans the valid data records field by field to identify missing fields, marks the data records containing missing fields as data records to be imputed and passes them to the nearest imputed processing unit group through the cross switch router, and marks the data records that do not contain missing fields as complete data records and passes them to the deduplication output processing unit group through the cross switch router.

5. The FPGA-based streaming data cleaning accelerator according to claim 1, characterized in that, The Locality Sensitive Hash (LSH) coarse screening module includes a first BRAM storage area, which stores eight sets of random projection vectors, each containing 16 random projection vectors. The LSH coarse screening module extracts the values ​​of all non-missing fields from the data records to be imputed, forming a query feature vector. It then performs element-wise multiplication of the query feature vector with each of the 16 random projection vectors in each set, accumulating the results to obtain 16 inner product results. Each inner product result is compared to zero; if the result is greater than or equal to zero, a binary bit of 1 is recorded; if less than zero, a binary bit of 0 is recorded. These 16 bits are concatenated to form a 16-bit local hash code. The eight sets of random projection vectors generate a total of eight 16-bit local hash codes, which are then concatenated to form a 128-bit composite hash signature.

6. The FPGA-based streaming data cleaning accelerator according to claim 5, characterized in that, The locality-sensitive hash coarse screening module also includes a second BRAM storage area, which is used to store the hash bucket index table. The hash bucket index table uses a 128-bit composite hash signature as the index key and the list of storage addresses of data records in the reference data buffer as the index value. The locality-sensitive hash coarse screening module retrieves the corresponding list of storage addresses in the hash bucket index table according to the 128-bit composite hash signature of the query feature vector, and reads all the data records pointed to by the list of storage addresses from the reference data buffer to form a candidate nearest neighbor set.

7. The FPGA-based streaming data cleaning accelerator according to claim 3, characterized in that, The product quantization and fine sorting module includes a third BRAM storage area, which is used to store orthogonal rotation matrices. The number of rows and columns of the orthogonal rotation matrix are equal to the dimension of the query feature vector. The product quantization and fine sorting module performs matrix-vector multiplication on the query feature vector and the orthogonal rotation matrix to obtain the rotated query vector. The matrix-vector multiplication is completed collaboratively by the arithmetic logic operation kernels of the four processing units in the nearest neighbor interpolation processing unit group in a pulsating array manner. Each processing unit is responsible for calculating the element values ​​of a consecutive quarter dimension in the rotated query vector. The product quantization fine sorting module divides the rotated query vector into 8 subspace segments along the dimensional direction.

8. The FPGA-based streaming data cleaning accelerator according to claim 7, characterized in that, The product quantization and fine-ranking module also includes a fourth BRAM storage area, which stores eight subspace codebooks. Each subspace codebook corresponds to one subspace segment, and each subspace codebook contains 256 codeword vectors. The dimension of each codeword vector is the same as the dimension of the corresponding subspace segment. For each subspace segment of the rotated query vector, the product quantization and fine-ranking module subtracts the subspace segment from the 256 codeword vectors in the corresponding subspace codebook element by element, squares the results, and sums them to calculate the squared Euclidean distance. The index of the codeword vector with the smallest squared Euclidean distance is selected as the quantization codeword index. A total of eight quantization codeword indices are generated from the eight subspace segments to form the quantization encoding of the rotated query vector.

9. The FPGA-based streaming data cleaning accelerator according to claim 8, characterized in that, The product quantization and fine sorting module also includes a fifth BRAM storage area, which is used to store eight subspace distance lookup tables. Each subspace distance lookup table is a two-dimensional array of 256 rows and 256 columns. The row index and column index are both in the range of 0 to 255. Each element in the two-dimensional array stores the squared Euclidean distance between the corresponding row index codeword vector and the column index codeword vector. The product quantization fine ranking module takes each candidate data record in the candidate nearest neighbor set as an example. It uses the i-th quantization codeword index in the quantization encoding of the rotated query vector as the row index and the i-th quantization codeword index in the quantization encoding of the candidate data record as the column index. It reads the corresponding element value from the distance lookup table of the i-th subspace as the approximate distance contribution value of the i-th subspace. It accumulates the approximate distance contribution values ​​of the 8 subspaces to obtain the approximate Euclidean distance squared value between the rotated query vector and the candidate data record. It sorts all candidate data records in the candidate nearest neighbor set in ascending order according to the approximate Euclidean distance squared value and selects the top 8 candidate data records after sorting to form the finely ranked nearest neighbor set.

Images