Data processing method and device, distributed data flow programming framework and related components

Abstract

The invention discloses a data processing method, which is applied to a host end, the host end is connected with the FPGA acceleration end. The FPGA acceleration end comprises a plurality of FPGA acceleration board cards, the FPGA acceleration board cards are connected through a network, and the data processing method comprises the following steps: dividing a data processing task into a pluralityof data processing sub-tasks, and determining a target FPGA acceleration board card corresponding to each data processing sub-task in the FPGA acceleration end; and transmitting the to-be-calculated data to the target FPGA acceleration board cards, and executing the corresponding data processing sub-tasks by utilizing each target FPGA acceleration board card to obtain a data processing result. According to the invention, the number of FPGA acceleration board cards in the FPGA acceleration end can be prevented from being physically limited by the host interface, and the data processing efficiency is improved. The invention further discloses a data processing device, a distributed data flow programming framework, electronic equipment and a storage medium, which have the above beneficial effects.

Description

technical field [0001] The present application relates to the technical field of cloud computing, and in particular to a data processing method and device, a distributed data flow programming framework, an electronic device, and a storage medium. Background technique [0002] With the rapid development of artificial intelligence and big data, the data in the data center has doubled. At the same time, the rapid development of AI (Artificial Intelligence, artificial intelligence) has also brought the requirements for high-performance computing of data. On the one hand, this leads to the exponential growth and development of data, and on the other hand, the computing resources required for these data processing also increase accordingly. Traditional computing is handled by CPU. With the continuous development of technology, the performance of CPU can no longer be doubled. Reconfigurable chips such as Field Programmable Gate Array (FPGA, Field Programmable Gate Array) have the ...

AI Technical Summary

Problems solved by technology

However, when data interaction is required between two FPGA accelerator boards in the FPGA accelerator in the related art, the FPGA accelerator board needs to transmit data to the host through PCIE, and the host then transmits the data to other boards. The efficiency is low, and the number of supported multi-board devices in the FPGA cloud server is limited by the physical limit of the number of PCIE boards that the host can support.

[0003] Therefore, how to avoid the physical limitation of the number of FPGA accelerator boards in the FPGA acceleration terminal by the host interface and improve data processing efficiency is a technical problem that those skilled in the art need to solve at present

Images