Data parallel processing method and device based on many-core processor

Abstract

The embodiment of the invention provides a data parallel processing method and device based on a many-core processor, which are applied to a stand-alone database. The method includes: dividing a to-be-operated data table to obtain a plurality of operation tasks after the to-be-operated data table is determined each time; and distributing the divided operation tasks to the slave cores of the many-core processor so as to process the operation tasks in parallel through different slave cores. On one hand, the data tables to be processed are sorted and divided to obtain the operation tasks suitablefor parallel processing, and the operation tasks are distributed to the many-core processor of the single-machine database, so that parallel operation when the single-machine database processes big data tasks is realized, and the operational capability of the many-core processor is fully exerted; and on the other hand, parallel processing of tasks is realized among many cores, the bottleneck of network data transmission speed among parallel computing nodes is eliminated, the operation process is accelerated, and the task processing efficiency is improved.

Description

technical field [0001] The invention relates to the field of analysis in a distributed database, in particular to a data parallel processing method and device based on many-core processors. Background technique [0002] In-library analysis refers to performing scientific computing, machine learning, regression analysis, and cluster analysis within the database, and implementing related algorithms through database-defined functions. Users can directly obtain analysis results by calling database-defined functions. The syntax of the database custom function is based on SQL, and all data calls and calculations can be completed in the database without importing and / or exporting data in multiple databases. The user calls the built-in function of the database through the database interface, and the built-in function of the database obtains the data by accessing the underlying storage, performs calculation, and returns the result. As more researchers in business fields get involved...

AI Technical Summary

Problems solved by technology

[0005] Embodiments of the present invention provide a data parallel processing method and device based on many-core processors, which are used to solve the problem of low processing speed of single-machine databases and distributed The database is easy to reach the bottleneck of network data transmission speed between parallel computing nodes, which affects the speed of parallel computing and reduces the efficiency of task processing

Images