Data clustering method, device and system, server and storage medium

Abstract

The invention relates to a data clustering method, device and system, a server and a storage medium. The invention belongs to the technical field of data processing, the method comprises the steps ofdetermining a resource demand of a target data set, responding to the resource demand, determining a plurality of servers and GPU resources on the plurality of servers, and performing data clusteringbased on each sub-data set in the target data set through the GPU resources on the plurality of servers to obtain a data clustering result of the target data set. According to the embodiment of the invention, the control server performs data clustering on a part of data in the target data set through the available GPU resources on the plurality of servers according to the resource requirements ofthe target data set so as to obtain the data clustering result of the target data set, and distributed data clustering can be realized, so that data clustering of super-large-scale data is realized.

Description

technical field [0001] The present disclosure relates to the technical field of data processing, and in particular to a data clustering method, device, system, server and storage medium. Background technique [0002] The rapid development of computer technology and mobile Internet has promoted the arrival of the era of big data. The large-scale data generated in modern society has put enormous pressure on the existing data processing methods. The management and utilization of large-scale data has become an inevitable trend. . As the most commonly used data processing method, data clustering has become very common in large-scale data. [0003] At present, the data clustering method is usually: obtain a plurality of data to be clustered, estimate the resource requirements of the plurality of data and the remaining GPU (Graphics Processing Unit, Graphics Processing Unit) resources on the server, from the remaining GPU resources Allocate GPU resources that meet resource requir...

AI Technical Summary

Problems solved by technology

[0004] In the above technology, when clustering ordinary large-scale data, GPU resources that meet the resource requirements can be allocated to it. However, when clustering tens of billions of ultra-large-scale data, due to the GPU resources are limited, which will cause the server's memory to be unable to load the data to be clustered at one time, and clustering of ultra-large-scale data cannot be realized

Images