FPGA-based multi-modal perception data fusion method and system

Abstract

The application relates to the technical field of real-time fusion of vehicle-mounted heterogeneous sensors, and discloses a multi-modal sensing data fusion method and system based on FPGA, which comprises the following steps: form-unified multi-modal data output by different sensors is obtained through programmable logic units arranged in the internal field programmable gate array; the form-unified data is written into a double-ring buffer zone composed of random access memory; the form-unified data in the double-ring buffer zone is transmitted to a PCIe interface module through a high-level extensible interface bus; and the form-unified data is transmitted to the input cache of an artificial intelligence reasoning engine in a scattered and aggregated manner through the state machine control of the PCIe interface module, so that the data fusion of the form-unified data is realized. The application can realize extremely low-delay data unification and transmission with a single programmable gate array chip without external storage and without the intervention of a central processing unit or an operating system.

Description

Technical Field

[0001] This invention relates to a multimodal sensing data fusion method and system based on FPGA, belonging to the field of real-time fusion technology of vehicle-mounted heterogeneous sensors. Background Technology

[0002] In existing technologies, multimodal perception data fusion typically relies on a central processing unit (CPU) or graphics processing unit (GPU) for data preprocessing, coordination, and forwarding. A typical solution requires the raw data output from each sensor to be transmitted to the host memory via an interface. The CPU then performs preprocessing operations such as timestamp alignment, coordinate system transformation, and format unification. The processed data is then copied to a GPU or dedicated AI accelerator for fusion inference. This process involves multiple system bus transmissions, CPU interrupt handling, operating system scheduling, and memory copying, resulting in long data paths and large processing delays. This makes it difficult to meet the real-time requirements of scenarios such as autonomous driving and robotics. In particular, the core data coordination and preprocessing stages heavily rely on CPU software scheduling and computation, becoming a major bottleneck for further reducing system latency.

[0003] Therefore, there is an urgent need for a technical solution that can eliminate the intervention of the CPU and operating system, thereby achieving extremely low latency data unification and transmission at the hardware level. Summary of the Invention

[0004] This invention provides a multimodal sensing data fusion method and system based on FPGA. Its main purpose is to achieve extremely low latency data unification and transmission using a single programmable gate array chip without the need for external storage or the intervention of a central processing unit or operating system.

[0005] To achieve the above objectives, the present invention provides a multimodal sensing data fusion method based on FPGA, comprising:

[0006] The multimodal data output from different sensors is format-unified by programmable logic units deployed inside the field-programmable gate array (FPGA) to obtain format-unified data. The programmable logic unit integrates a time unification module, a spatial unification module, and a format unification module. The FPGA is deployed with programmable logic units, random access memory, advanced scalable interface bus, PCIe interface module, and state machine.

[0007] The unified data in the aforementioned form is written into a double-ring buffer formed by the random access memory;

[0008] The unified data in the dual-ring buffer is transmitted to the PCIe interface module via the Advanced Scalable Interface Bus.

[0009] The state machine controls the PCIe interface module to transmit the unified data in a distributed-aggregated manner to the input cache of the artificial intelligence inference engine, so as to achieve data fusion of the unified data in a distributed-aggregated manner.

[0010] Optionally, the multimodal data output from different sensors can be format-unified using programmable logic units deployed within the field-programmable gate array (FPGA) to obtain format-unified data, including:

[0011] The time unification module performs timestamp alignment on the multimodal data respectively;

[0012] The spatial unification module performs coordinate system transformation on the data that has completed timestamp alignment;

[0013] The format unification module unifies the data bit width and memory layout of the data after coordinate system transformation, resulting in data with a unified format.

[0014] The time unification module, the space unification module, and the format unification module are connected into the same hardware pipeline by the programmable logic unit.

[0015] Optionally, the multimodal data includes, but is not limited to, lidar point cloud data, camera image data, radar complex matrix data, and inertial measurement unit attitude data, and the multimodal data is processed in parallel in the same hardware pipeline.

[0016] Optionally, the random access memory is an on-chip BRAM of a field-programmable gate array, and the dual-ring buffer does not depend on a double data rate synchronous dynamic random access memory located outside the field-programmable gate array during the entire data writing and data reading process.

[0017] Optionally, the dual-ring buffer adopts a read-write pointer differential mechanism. When the difference between the write pointer and the read pointer is less than a preset threshold, the dual-ring buffer feeds back a back pressure signal to the programmable logic unit to pause subsequent data writing.

[0018] Optionally, the Advanced Scalable Interface Bus (ASME) transmits uniform data in the dual-ring buffer to the PCIe interface module using the AXI-Stream protocol. The transmission of the uniform data is completed within the field-programmable gate array (FPGA) without passing through the cache and copying of the central processing unit (CPU) and graphics processor (GPU).

[0019] Optionally, the state machine configures the scattered aggregation DMA engine of the PCIe interface module through a hardware descriptor queue, and the configuration process does not go through the interrupt service routine of the central processing unit.

[0020] Optionally, the DMA engine's transfer process does not go through the operating system's network protocol stack, in order to form a real-time hardware data transfer path between different sensors and the artificial intelligence inference engine.

[0021] Optionally, the PCIe interface module encapsulates the data in the above-mentioned form into fixed-length TLP packets using a distributed aggregation method, and writes them into the input buffer of the artificial intelligence inference engine via the state machine. The encapsulation and writing processes of the TLP packets do not go through the network protocol stack of the operating system and the interrupt service routine of the central processing unit.

[0022] To address the aforementioned problems, this invention also provides an FPGA-based multimodal sensing data fusion system, the system comprising:

[0023] The data unification module is used to unify the form of multimodal data output from different sensors through programmable logic units deployed inside the field programmable gate array (FPGA) to obtain format-unified data. The programmable logic unit integrates a time unification module, a spatial unification module, and a format unification module. The FPGA is deployed with programmable logic units, random access memory, advanced scalable interface bus, PCIe interface module, and state machine.

[0024] The data writing module is used to write the uniform data in the above form into a double-ring buffer composed of the random access memory;

[0025] The data transmission module is used to transmit unified data in the dual-ring buffer to the PCIe interface module through the Advanced Scalable Interface bus.

[0026] The data fusion module is used by the state machine to control the PCIe interface module to transmit the unified data in the form of a distributed-aggregated manner to the input cache of the artificial intelligence inference engine, so as to realize the data fusion of the unified data in the form of a distributed-aggregated manner.

[0027] Compared to the problems described in the background art, the embodiments of the present invention connect the time unification module, spatial unification module, and format unification module into the same hardware pipeline, so that timestamp alignment, coordinate system transformation, and format unification are completed sequentially in the same hard logic path, thereby reducing cross-clock and cross-bus waiting and ultimately reducing data unification latency. Furthermore, the embodiments of the present invention eliminate the need for external DDR read / write waiting by making the dual-ring buffer independent of the double data rate synchronous dynamic random access memory located outside the field-programmable gate array during the entire data write and read process, thereby shortening the data dwell time in the buffer stage and reducing end-to-end latency. Furthermore, the embodiments of the present invention employ a read / write pointer differential mechanism in the dual-ring buffer. When the difference between the write pointer and the read pointer is less than a preset threshold, the dual-ring buffer feeds back a backpressure signal to the programmable logic unit to immediately pause subsequent data writing, thereby avoiding retransmissions or additional arbitration caused by buffer overflow. The invention aims to reduce the additional latency introduced by retransmission and arbitration. Furthermore, by bypassing the caching and copying processes of the central processing unit (CPU) and graphics processing unit (GPU), the data path is shortened, thereby reducing processing latency. Further, by using a hardware descriptor queue maintained by a state machine, multiple discontinuous memory blocks in the on-chip BRAM are sequentially linked into a single bus transaction sequence. This merges previously multiple scattered bus requests into a single continuous bus transaction sequence, ultimately reducing bus arbitration and handshake operations, and minimizing waiting time caused by additional bus transactions. Furthermore, by using a distributed DMA engine, the transmission process bypasses the operating system's network protocol stack, eliminating the waiting time associated with protocol stack encapsulation, decapsulation, and context switching, thus reducing end-to-end latency. Finally, by bypassing the operating system's network protocol stack and CPU interrupt service routines, the invention achieves extremely low-latency data transmission. Therefore, this invention can achieve extremely low-latency data unification and transmission with a single programmable gate array (FPGA) chip without requiring external storage or CPU / operating system intervention. Attached Figure Description

[0028] Figure 1 This is a flowchart illustrating a multimodal sensing data fusion method based on FPGA provided in an embodiment of the present invention.

[0029] Figure 2 This is a flowchart illustrating the multimodal hardware data fusion process for implementing the FPGA-based multimodal sensing data fusion method according to an embodiment of the present invention.

[0030] Figure 3 This is a system architecture diagram of a conventional scheme for implementing the FPGA-based multimodal sensing data fusion method according to an embodiment of the present invention;

[0031] Figure 4 This is a schematic diagram of the modules implementing the FPGA-based multimodal sensing data fusion method according to an embodiment of the present invention;

[0032] Figure 5 A schematic diagram of a computer device for an FPGA-based multimodal sensing data fusion method according to an embodiment of the present invention;

[0033] The objectives, features, and advantages of this invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0034] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0035] This application provides an FPGA-based multimodal sensing data fusion method. The execution entity of the FPGA-based multimodal sensing data fusion method includes, but is not limited to, at least one of the following electronic devices that can be configured to execute the method provided in this application: a server, a terminal, etc. In other words, the FPGA-based multimodal sensing data fusion method can be executed by software or hardware installed on a terminal device or a server device. The server includes, but is not limited to, a single server, a server cluster, a cloud server, or a cloud server cluster.

[0036] Reference Figure 1 The diagram shown is a flowchart illustrating a multimodal sensing data fusion method based on FPGA provided in an embodiment of the present invention. In this embodiment, the multimodal sensing data fusion method based on FPGA includes:

[0037] See Figure 2 The diagram shown is a flowchart illustrating the multimodal hardware data fusion process for implementing the FPGA-based multimodal sensing data fusion method according to an embodiment of the present invention. Figure 2In this context, FPGA-BASED HARDWARE DATA FUSIONPROCESS refers to the hardware data fusion process based on FPGA; LIDAR refers to LiDAR; RADAR refers to radar; CAMERA refers to camera; IMU refers to inertial measurement unit; TIME SYNCHRONIZATION refers to time synchronization; GPS / PTP refers to Global Positioning System / Precise Time Protocol; SPATIAL ALIGNMENT & CALIBRATION refers to spatial alignment and calibration; Matrix Transform refers to matrix transformation; COORDINATE TRANSFORMATION refers to coordinate transformation; Ego to World refers to the transformation from the vehicle coordinate system to the world coordinate system; Quaternion / Euler refers to quaternions / Euler angles; FEATURE EXTRACTION refers to feature extraction; Edge, Point Cloud refers to edges and point clouds; DSP Blocks refers to DSP blocks; PRE-FUSION PROCESSING refers to pre-fusion processing; Filtering, Normalization refers to filtering and normalization; BRAMBuffers refers to block random access memory cache; AXI Interconnect refers to advanced scalable interface interconnect; and AIINFERENCE refers to... ENGINE represents the artificial intelligence inference engine; CNN / RNN Accelerator represents the convolutional neural network / recurrent neural network accelerator; DATA FORMAT represents the data format; Radar represents radar; Accel / Gyro, FP32 represents accelerometer / gyroscope, 32-bit floating-point number; DATA FORMAT SPECS represents the data format specification; LiDAR (XYZI, FP32) represents LiDAR (XYZI, FP32); Radar (Range-Doppler, Int16) represents radar (range-Doppler, 16-bit signed integer); Camera (RGB, uint8) represents camera (red, green, and blue channels, 8-bit unsigned integer); IMU (Accel / Gyro, FP32) represents IMU (accelerometer / gyroscope, FP32); PROCESSING LATENCY ANNOTATIONS represents processing latency annotations; Time Sync (<2μs) represents time synchronization (<2μs); Spatial Align (<10μs) indicates spatial alignment (<10μs), Feature Extr.(<50μs) indicates feature extraction (<50μs), Pre-Fusion (<20μs) indicates pre-fusion (<20μs), Total Data Fusion Path (<100μs) indicates the total data fusion path (<100μs), PCIe ssBuffers indicates PCIe slave cache, PCIe Gen4 indicates fourth-generation peripheral interconnect fast channel, and the following are the corresponding values. Figure 2 The implementation process shown is as follows: The LiDAR, camera, radar, and inertial measurement unit input raw data to the Field Programmable Gate Array (FPGA); the FPGA sequentially runs a time synchronization module, a spatial alignment and calibration module, a coordinate transformation module, a feature extraction module, and a pre-fusion processing module within its programmable logic unit. The time synchronization module is used for timestamp alignment; the spatial alignment and calibration module and the coordinate transformation module are used for coordinate system transformation; the feature extraction module is used for feature extraction; and the pre-fusion processing module is used for unifying bit width and memory layout. The FPGA stores the pre-fused data in its on-chip BRAM cache; the FPGA sends the pre-fused data to the PCIe Gen4 interface via AXI interconnect; the PCIe Gen4 interface encapsulates the pre-fused data into fixed-length TLP packets using a distributed-aggregate DMA method and writes them directly to the input cache of the AI ​​inference engine to complete extremely low-latency hardware data transmission.

[0038] See Figure 3 The diagram shown is a system architecture diagram of a conventional scheme for implementing the FPGA-based multimodal sensing data fusion method according to an embodiment of the present invention. Figure 3In this context, TECHNICAL SYSTEM ARCHITECTURE represents the technical system architecture; MULTI-MODAL ENVIRONMENTAL PERCEPTION FUSION PROCESSOR FOR AUTONOMOUSVEHICLES represents the multimodal environmental perception fusion processor for autonomous driving; SENSOR INPUT INTERFACES represents sensor input interfaces; HIGH-SPEED DATA BUS represents high-speed data bus; FPGA represents Field Programmable Gate Array; PREPROCESSING represents preprocessing; UNIT represents a cell; RAW Data Ingestion represents raw data ingestion; PointCloud Filtering represents point cloud filtering; Image De-warping represents image distortion correction; Signal Conditioning represents signal conditioning; SHARED MEMORY POOLS represents shared memory pools; DDR4 represents fourth-generation double data rate synchronous dynamic random access memory; LPDDRS represents low-power double data rate synchronous dynamic random access memory; AI SOC INFERENCE ENGINE represents the artificial intelligence system-on-chip inference engine; Transformer Core represents the transformer core; DSP represents the digital signal processor; and CPU represents the CPU. Cluster represents the central processing unit cluster, REAL-TIMEFUSION ALGORITHMS represents real-time fusion algorithms, Kalman Filter represents a Kalman filter, Occupancy Grid represents an occupancy grid, Object Tracking represents target tracking, Sensor Fusion Logic represents sensor fusion logic, and PERCEPTION OUTPUT represents the sensing output. The following is a description of... Figure 3 The implementation process shown is as follows: LiDAR, camera, radar, and IMU input raw data to the Field-Programmable Gate Array (FPGA) through sensor input interfaces; the FPGA sequentially performs signal conditioning, image distortion correction, and point cloud filtering within its preprocessing unit to obtain preprocessed sensor data; the FPGA stores the preprocessed data in a shared storage pool (DDR4 / LPDDR5) and simultaneously sends it to the AI ​​SOC inference engine; the AI ​​SOC inference engine runs a CNN / Transformer accelerator to extract features and infer from the preprocessed data, obtaining perception results such as target lists and drivable areas; the FPGA's CPU cluster runs a real-time fusion algorithm to fuse and track the inference results, ultimately outputting the perception results. Finally, it should be noted that... Figure 3The process shown is not the process of this invention, but a traditional data fusion process, used for comparison with this invention. Figure 3 The proposed solution relies on external storage and operating system scheduling, resulting in higher end-to-end latency compared to this invention. It is only used for comparison with this invention to highlight the contribution of this invention in addressing the latency bottleneck.

[0039] S1. The multimodal data output by different sensors is format-unified by the programmable logic unit deployed inside the field programmable gate array (FPGA) to obtain format-unified data. The programmable logic unit integrates a time unification module, a spatial unification module, and a format unification module. The FPGA is equipped with a programmable logic unit, a random access memory, an advanced scalable interface bus, a PCIe interface module, and a state machine.

[0040] This invention connects the time unification module, spatial unification module, and format unification module into the same hardware pipeline, so that timestamp alignment, coordinate system transformation, and format unification are completed sequentially in the same hard logic path, thereby reducing waiting time across clocks and buses, and ultimately reducing the latency of data unification.

[0041] The field-programmable gate array (FPGA) is a semi-custom digital integrated circuit chip that can be reconfigured in the user's field using a hardware description language to achieve specific logic functions. The programmable logic unit (PLU) is the smallest configurable unit within the FPGA chip, consisting of a lookup table, registers, carry chains, and selectors, used to implement combinational logic, sequential logic, and arithmetic operations. The multimodal data refers to vehicle external environment data collected by different sensors.

[0042] In one embodiment of the present invention, the process of unifying the form of multimodal data output from different sensors by a programmable logic unit deployed inside a field-programmable gate array (FPGA) to obtain form-unified data includes: a time unification module performing timestamp alignment on the multimodal data; a spatial unification module performing coordinate system transformation on the timestamp-aligned data; and a format unification module unifying the data bit width and memory layout of the coordinate system-transformed data to obtain form-unified data. The programmable logic unit connects the time unification module, the spatial unification module, and the format unification module into a single hardware pipeline.

[0043] The time unification module is a hard logic circuit composed of on-chip registers and counters in the FPGA, used to store and compare the PTP timestamps of each sensor and output aligned data frames. The spatial unification module is a 4*4 matrix multiplication circuit implemented by the on-chip DSP block of the FPGA, used to convert the sensor coordinate system to the vehicle / world coordinate system. The format unification module is a bit-width conversion circuit composed of on-chip registers and shifters in the FPGA, used to normalize floating-point / integer / different channel sequences to the fixed tensor format required by AI. The same hardware pipeline is formed by hard-wired lookup tables, registers, DSP blocks and interconnect resources within the programmable logic unit. The programmable logic unit is the smallest physical carrier that constitutes the same hardware pipeline. The same hardware pipeline is a sequential data processing path without CPU and DDR formed by connecting these physical units in the time-space-format unification order.

[0044] It should be noted that after the coordinate system transformation is completed, the programmable logic unit can further call the conventional feature extraction module to perform geometric feature calculations on the point cloud edges, image gradients, or radar scattering points. The resulting feature data and the original coordinate data are sent to the format unification module to unify the dimensions, bit width, and memory layout. Since feature extraction is only a geometric operation known in the field and does not change the data flow and time order, it is not necessary to limit this step to achieve extremely low latency hardware fusion. Therefore, it is not written into the technical features.

[0045] In another embodiment of the present invention, the multimodal data includes, but is not limited to, lidar point cloud data, camera image data, radar complex matrix data, and inertial measurement unit attitude data, and the multimodal data is processed in parallel in the same hardware pipeline.

[0046] The lidar point cloud data consists of a three-dimensional spatial coordinate set and reflection intensity generated by the lidar after emitting and receiving reflected laser pulses, used to describe the geometric contours of the vehicle's external environment. The camera image data is a two-dimensional pixel array output by the camera sensor, with each pixel containing red, green, and blue channel intensity values, used to provide texture and color information of the scene around the vehicle. The radar complex matrix data is a complex echo signal matrix output by the radar receiving antenna, with each element containing amplitude and phase information, used to calculate the target's distance, velocity, and scattering characteristics. The inertial measurement unit attitude data consists of three-axis acceleration and three-axis angular velocity values ​​output by the inertial measurement unit, used to infer the vehicle's attitude changes and motion state in three-dimensional space.

[0047] S2. Write the unified data in the form into a double-ring buffer composed of the random access memory.

[0048] This invention eliminates read / write wait times for external DDR by making the dual-ring buffer independent of the double data rate synchronous dynamic random access memory located outside the field-programmable gate array during the entire data write and read process, thereby shortening the data dwell time in the buffer stage and reducing end-to-end latency.

[0049] The external DDR refers to a double data rate synchronous dynamic random access memory outside the field programmable gate array.

[0050] In one embodiment of the present invention, the random access memory is an on-chip BRAM of a field-programmable gate array, and the dual ring buffer does not depend on a double data rate synchronous dynamic random access memory located outside the field-programmable gate array during the entire data writing and data reading process.

[0051] The on-chip BRAM refers to the block random access memory integrated inside the field-programmable gate array chip, which is composed of a fixed-size array of memory blocks. The dual-ring buffer is two independent memory rings divided on the same on-chip BRAM, connected end to end to form a ping-pong structure. The write pointer and the read pointer each revolve around two rings. When the write pointer approaches the read pointer, it means that the difference between the write pointer and the read pointer is less than a preset threshold. The difference between the write pointer and the read pointer represents the number of bytes that have been written but not read. When the difference is less than the preset threshold, it is determined that an overflow is imminent, and at this time, a back pressure signal is generated by the hardware.

[0052] Furthermore, this embodiment of the invention employs a read-write pointer differential mechanism through a dual-ring buffer. When the difference between the write pointer and the read pointer is less than a preset threshold, the dual-ring buffer feeds back a backpressure signal to the programmable logic unit to immediately pause subsequent data writing, thereby avoiding retransmission or additional arbitration cycles caused by buffer overflow, and ultimately reducing the additional latency introduced by retransmission and arbitration.

[0053] In another embodiment of the present invention, the dual-ring buffer adopts a read-write pointer differential mechanism. When the difference between the write pointer and the read pointer is less than a preset threshold, the dual-ring buffer feeds back a back pressure signal to the programmable logic unit to pause subsequent data writing.

[0054] The back pressure signal is a signal composed of high / low level flags, which is used to immediately pause subsequent data writing, prevent on-chip BRAM overflow, and ensure a continuous, low-latency data flow from the sensor to the AI ​​engine.

[0055] S3. Transmit the unified data in the dual-ring buffer to the PCIe interface module through the Advanced Scalable Interface Bus.

[0056] This invention reduces processing latency by shortening the data path and avoiding caching and copying by the central processing unit and graphics processing unit.

[0057] The advanced extensible interface bus is as described above. Figure 2 The AXI Interconnect mentioned above refers to the Advanced Scalable Interface Bus, a high-speed, synchronous, point-to-point interconnect bus specification within the FPGA chip. It is used to transmit address, data, and handshake signals between programmable logic units and memory and peripheral interfaces. The PCIe interface module is described above. Figure 2 The winning bid includes a PCIe Gen4 hard core block (including its associated PCIe ssBuffers). The PCIe interface module is a hardware hard core that implements the PCIe Gen4 protocol. The PCIe interface module is used to complete TLP encapsulation and DMA transfer.

[0058] In one embodiment of the present invention, the Advanced Scalable Interface Bus transmits the uniform data in the dual-ring buffer to the PCIe interface module using the AXI-Stream protocol. The transmission of the uniform data is completed within the field-programmable gate array and does not involve caching or copying by the central processing unit and the graphics processing unit.

[0059] The AXI-Stream protocol is an on-chip point-to-point handshake protocol for continuous data streams. It has no address phase and only uses the VALID / READY signal pair to complete data clock transmission. It is used to realize low-latency real-time data stream transmission between FPGA programmable logic units and PCIe interface modules.

[0060] S4. The state machine controls the PCIe interface module to transmit the unified data in a distributed-aggregated manner to the input cache of the artificial intelligence inference engine, so as to realize the data fusion of the unified data in a distributed-aggregated manner.

[0061] This invention uses a hardware descriptor queue maintained by a state machine to sequentially link multiple non-contiguous memory blocks in the on-chip BRAM into a single bus transaction sequence. This merges the previously scattered bus requests that needed to be initiated multiple times into a single continuous bus transaction sequence, ultimately reducing the number of bus arbitrations and handshakes, and reducing the waiting time caused by additional bus transactions.

[0062] The state machine is a sequential circuit composed of on-chip registers and combinational logic of the FPGA. It is used to generate the next state and control signals based on the current state and input signals, and to schedule the DMA descriptor and data transfer process in a CPU-free manner. The hardware descriptor queue is a circular table composed of on-chip BRAMs of the FPGA. Each entry contains a data address, length, and completion flag. It is read and written cyclically by the state machine and is used to provide the discontinuous storage block transfer parameters to the scattered aggregation DMA engine. The scattered aggregation DMA engine is a DMA controller implemented by the on-chip hard core or soft core of the FPGA. By reading the hardware descriptor queue, it links multiple discontinuous BRAM blocks into a single bus transaction sequence to complete the direct data transfer to the PCIe interface module. The scattered aggregation method is a DMA transfer mode. The scattered aggregation DMA engine is a hard logic circuit that implements the DMA transfer mode on-chip in the field-programmable gate array. It is used to link multiple BRAM blocks into a single bus transaction sequence through discontinuous address descriptors to complete the direct data transfer to the PCIe interface module.

[0063] In one embodiment of the present invention, the state machine configures the scattered aggregation DMA engine of the PCIe interface module through a hardware descriptor queue, and the configuration process does not go through the interrupt service routine of the central processing unit.

[0064] Optionally, the process of configuring the scattered-collection DMA engine of the PCIe interface module through the hardware descriptor queue is as follows: the state machine writes the non-contiguous BRAM address and length of the data to be transferred into the circular entry of the hardware descriptor queue; the state machine writes the queue start address into the DMA descriptor pointer register of the PCIe interface module; the state machine sets the direction, length and enable bit of the DMA control register, etc., to complete the configuration of the scattered-collection DMA engine, without the intervention of CPU interrupts.

[0065] Furthermore, in this embodiment of the invention, the transmission process of the distributed aggregation DMA engine does not go through the network protocol stack of the operating system, thereby saving the waiting time caused by protocol stack encapsulation, decapsulation and context switching, and thus reducing end-to-end latency.

[0066] In another embodiment of the present invention, the transmission process of the scattered aggregation DMA engine does not go through the network protocol stack of the operating system, so as to form a real-time hardware data transmission path between different sensors and artificial intelligence inference engines.

[0067] The network protocol stack is a collection of software modules in the operating system kernel that implement communication protocols such as TCP / IP and UDP. It is responsible for data encapsulation, decapsulation, routing, and flow control. In this embodiment, the PCIe scattered collection-aggregated DMA transfer process does not call this module to save context switching and copy waiting, thereby reducing end-to-end latency.

[0068] Optionally, the transfer process of the scattered aggregation DMA engine specifically includes: the scattered aggregation DMA engine reads the hardware descriptor queue based on the queue start address of the write PCIe interface module, and sequentially obtains the address and length of non-contiguous BRAM blocks; the scattered aggregation DMA engine encapsulates each block of data into a continuous TLP packet, forming a single bus transaction sequence; the scattered aggregation DMA engine sequentially sends the TLP packets to the input buffer of the artificial intelligence inference engine through the PCIe interface module; the scattered aggregation DMA engine sets the completion flag after the last TLP is completed and returns a completion signal to the state machine.

[0069] Furthermore, embodiments of the present invention achieve extremely low latency data transmission by bypassing the network protocol stack of the operating system and the interrupt service routine of the central processing unit.

[0070] In one embodiment of the present invention, the PCIe interface module encapsulates the uniform data in the form of TLP packets in a scattered-set aggregation manner, and writes them into the input cache of the artificial intelligence inference engine via the state machine. The encapsulation and writing processes of the TLP packets do not go through the network protocol stack of the operating system and the interrupt service routine of the central processing unit.

[0071] It should be noted that the input cache written to the AI ​​inference engine via the state machine is intended to express the need to return a completion signal to the state machine.

[0072] Optionally, the data fusion process for achieving the unified data in form specifically includes: after receiving the unified data in form, the artificial intelligence inference engine can immediately read and execute subsequent fusion inference.

[0073] like Figure 4 The diagram shown is a functional block diagram of the FPGA-based multimodal sensing data fusion system of the present invention.

[0074] The FPGA-based multimodal sensing data fusion system 400 described in this invention can be installed in an electronic device. Depending on the functions implemented, the FPGA-based multimodal sensing data fusion system includes a data unification module 401, a data writing module 402, a data transmission module 403, and a data fusion module 404. The module described in this invention can also be called a unit, which refers to a series of computer program segments that can be executed by the processor of an electronic device and can perform a fixed function, and are stored in the memory of the electronic device.

[0075] In this embodiment of the invention, the functions of each module / unit are as follows:

[0076] The data unification module 401 is used to unify the form of multimodal data output by different sensors through programmable logic units deployed inside the field programmable gate array to obtain format-unified data. The programmable logic unit integrates a time unification module, a spatial unification module and a format unification module. The field programmable gate array deploys programmable logic units, random access memory, advanced scalable interface bus, PCIe interface module and state machine.

[0077] The data writing module 402 is used to write the uniform data in the form into a double-ring buffer composed of the random access memory;

[0078] The data transmission module 403 is used to transmit unified data in the dual-ring buffer to the PCIe interface module through the Advanced Scalable Interface bus.

[0079] The data fusion module 404 is used by the state machine to control the PCIe interface module to transmit the unified data in a distributed-aggregated manner to the input cache of the artificial intelligence inference engine, so as to realize the data fusion of the unified data in a distributed-aggregated manner.

[0080] In detail, the modules in the FPGA-based multimodal sensing data fusion system 400 described in this embodiment of the invention employ the same methods as described above. Figure 1 The method uses the same techniques as the FPGA-based multimodal sensing data fusion method described above and can produce the same technical effects, so it will not be repeated here.

[0081] In one embodiment, a computer device is provided, which may be a server or a client, and its internal structure diagram may be as follows: Figure 5 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a multimodal sensing data fusion method based on FPGA on the server or client side.

[0082] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps:

[0083] The data unification module is used to unify the form of multimodal data output from different sensors through programmable logic units deployed inside the field programmable gate array (FPGA) to obtain format-unified data. The programmable logic unit integrates a time unification module, a spatial unification module, and a format unification module. The FPGA is deployed with programmable logic units, random access memory, advanced scalable interface bus, PCIe interface module, and state machine.

[0084] The data writing module is used to write the uniform data in the above form into a double-ring buffer composed of the random access memory;

[0085] The data transmission module is used to transmit unified data in the dual-ring buffer to the PCIe interface module through the Advanced Scalable Interface bus.

[0086] The data fusion module is used by the state machine to control the PCIe interface module to transmit the unified data in the form of a distributed-aggregated manner to the input cache of the artificial intelligence inference engine, so as to realize the data fusion of the unified data in the form of a distributed-aggregated manner.

[0087] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0088] The data unification module is used to unify the form of multimodal data output from different sensors through programmable logic units deployed inside the field programmable gate array (FPGA) to obtain format-unified data. The programmable logic unit integrates a time unification module, a spatial unification module, and a format unification module. The FPGA is deployed with programmable logic units, random access memory, advanced scalable interface bus, PCIe interface module, and state machine.

[0089] The data writing module is used to write the uniform data in the above form into a double-ring buffer composed of the random access memory;

[0090] The data transmission module is used to transmit unified data in the dual-ring buffer to the PCIe interface module through the Advanced Scalable Interface bus.

[0091] The data fusion module is used by the state machine to control the PCIe interface module to transmit the unified data in the form of a distributed-aggregated manner to the input cache of the artificial intelligence inference engine, so as to realize the data fusion of the unified data in the form of a distributed-aggregated manner.

[0092] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0093] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0094] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0095] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0096] Finally, it should be noted that in the above embodiments, each embodiment can be combined with each other or independent. Deleting any one of them will not affect the technical implementation of other embodiments. The above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A multimodal sensing data fusion method based on FPGA, characterized in that, The method includes: By using programmable logic units deployed within a field-programmable gate array (FPGA), multimodal data output from different sensors is format-unified to obtain format-unified data, including: The time unification module performs timestamp alignment on the multimodal data respectively; The spatial unification module performs coordinate system transformation on the data that has completed timestamp alignment; The format unification module unifies the data bit width and memory layout of the data after coordinate system transformation, resulting in data with a unified format. The programmable logic unit connects the time unification module, the space unification module, and the format unification module into the same hardware pipeline. The programmable logic unit integrates a time unification module, a space unification module, and a format unification module. The field-programmable gate array (FPGA) deploys a programmable logic unit, random access memory (RAM), an advanced scalable interface bus (ASM), a PCIe interface module, and a state machine. The random access memory is the on-chip BRAM of the FPGA, and the dual-ring buffer does not rely on the double data rate synchronous dynamic random access memory located outside the FPGA during the entire data writing and reading process. The unified data in the above form is written into a dual-ring buffer composed of the random access memory. The dual-ring buffer adopts a read-write pointer differential mechanism. When the difference between the write pointer and the read pointer is less than a preset threshold, the dual-ring buffer feeds back a back pressure signal to the programmable logic unit to pause subsequent data writing. The unified data in the dual-ring buffer is transmitted to the PCIe interface module via the Advanced Scalable Interface Bus. The state machine controls the PCIe interface module to transmit the unified data in a distributed-aggregated manner to the input cache of the artificial intelligence inference engine, so as to achieve data fusion of the unified data in a distributed-aggregated manner.

2. The FPGA-based multimodal sensing data fusion method as described in claim 1, characterized in that, Multimodal data includes, but is not limited to, lidar point cloud data, camera image data, radar complex matrix data, and inertial measurement unit attitude data, and the multimodal data is processed in parallel within the same hardware pipeline.

3. The FPGA-based multimodal sensing data fusion method as described in claim 1, characterized in that, The Advanced Scalable Interface Bus (ASME) transmits uniform data in the dual-ring buffer to the PCIe interface module using the AXI-Stream protocol. The transmission of the uniform data is completed within the field-programmable gate array (FPGA) without passing through the cache and copying of the central processing unit (CPU) and graphics processor (GPU).

4. The FPGA-based multimodal sensing data fusion method as described in claim 1, characterized in that, The state machine configures the scattered aggregation DMA engine of the PCIe interface module through the hardware descriptor queue, and the configuration process does not go through the interrupt service routine of the central processing unit.

5. The FPGA-based multimodal sensing data fusion method as described in claim 4, characterized in that, The DMA engine's data transfer process bypasses the operating system's network protocol stack, enabling real-time hardware data transfer paths between different sensors and AI inference engines.

6. The FPGA-based multimodal sensing data fusion method as described in claim 1, characterized in that, The PCIe interface module encapsulates the data in the above-mentioned form into fixed-length TLP packets using a distributed aggregation method, and writes them into the input buffer of the artificial intelligence inference engine via the state machine. The encapsulation and writing process of the TLP packets does not go through the network protocol stack of the operating system or the interrupt service routine of the central processing unit.

7. A multimodal sensing data fusion system based on FPGA, characterized in that, The system includes: The data unification module is used to unify the format of multimodal data output from different sensors through programmable logic units deployed inside the field-programmable gate array (FPGA), resulting in format-unified data, including: The time unification module performs timestamp alignment on the multimodal data respectively; The spatial unification module performs coordinate system transformation on the data that has completed timestamp alignment; The format unification module unifies the data bit width and memory layout of the data after coordinate system transformation, resulting in data with a unified format. The programmable logic unit connects the time unification module, the space unification module, and the format unification module into the same hardware pipeline. The programmable logic unit integrates a time unification module, a space unification module, and a format unification module. The field-programmable gate array (FPGA) deploys a programmable logic unit, random access memory (RAM), an advanced scalable interface bus (ASM), a PCIe interface module, and a state machine. The random access memory is the on-chip BRAM of the FPGA, and the dual-ring buffer does not rely on the double data rate synchronous dynamic random access memory located outside the FPGA during the entire data writing and reading process. The data writing module is used to write the uniform data in the form described above into a dual-ring buffer composed of the random access memory. The dual-ring buffer adopts a read-write pointer differential mechanism. When the difference between the write pointer and the read pointer is less than a preset threshold, the dual-ring buffer feeds back a back pressure signal to the programmable logic unit to pause subsequent data writing. The data transmission module is used to transmit unified data in the dual-ring buffer to the PCIe interface module through the Advanced Scalable Interface bus. The data fusion module is used by the state machine to control the PCIe interface module to transmit the unified data in the form of a distributed-aggregated manner to the input cache of the artificial intelligence inference engine, so as to realize the data fusion of the unified data in the form of a distributed-aggregated manner.

Images