Data processing method, device, distributed data flow programming framework and related components

Abstract

The application discloses a data processing method, which is applied to the host end, the host end is connected to the FPGA acceleration end, the FPGA acceleration end includes a plurality of FPGA acceleration boards, and the FPGA acceleration boards are connected through a network. The data processing method includes The data processing task is divided into multiple data processing subtasks, and the target FPGA accelerator board corresponding to each data processing subtask is determined in the FPGA acceleration terminal; the data to be calculated is transmitted to the target FPGA accelerator board, and each target FPGA is used to The accelerator board executes the corresponding data processing subtask to obtain the data processing result. The application can avoid the physical limitation of the number of FPGA acceleration boards in the FPGA acceleration terminal by the host interface, and improve the data processing efficiency. The application also discloses a data processing device, a distributed data stream programming framework, an electronic device and a storage medium, which have the above beneficial effects.

Description

technical field [0001] The present application relates to the technical field of cloud computing, and in particular to a data processing method and device, a distributed data flow programming framework, an electronic device, and a storage medium. Background technique [0002] With the rapid development of artificial intelligence and big data, the data in the data center has doubled. At the same time, the rapid development of AI (Artificial Intelligence, artificial intelligence) has also brought the requirements for high-performance computing of data. On the one hand, this leads to the exponential growth and development of data, and on the other hand, the computing resources required for these data processing also increase accordingly. Traditional computing is handled by CPU. With the continuous development of technology, the performance of CPU can no longer be doubled. Reconfigurable chips such as Field Programmable Gate Array (FPGA, Field Programmable Gate Array) have the ...

AI Technical Summary

Problems solved by technology

However, when data interaction is required between two FPGA accelerator boards in the FPGA accelerator in the related art, the FPGA accelerator board needs to transmit data to the host through PCIE, and the host then transmits the data to other boards. The efficiency is low, and the number of supported multi-board devices in the FPGA cloud server is limited by the physical limit of the number of PCIE boards that the host can support.

[0003] Therefore, how to avoid the physical limitation of the number of FPGA accelerator boards in the FPGA acceleration terminal by the host interface and improve data processing efficiency is a technical problem that those skilled in the art need to solve at present

Images