Data clustering method, device, system, server and storage medium

Abstract

The disclosure relates to a data clustering method, device, system, server, and storage medium, and belongs to the technical field of data processing. The method includes: determining the resource requirements of the target data set, and determining multiple servers and multiple servers in response to the resource requirements The GPU resources on the server, through the GPU resources on multiple servers, perform data clustering based on each sub-dataset in the target data set, and obtain the data clustering results of the target data set. In the embodiment of the present disclosure, according to the resource requirements of the target data set, the control server performs data clustering on a part of the data in the target data set through the available GPU resources on multiple servers, and then obtains the data clustering results of the target data set , can realize distributed data clustering, so as to realize data clustering of ultra-large-scale data.

Description

technical field [0001] The present disclosure relates to the technical field of data processing, and in particular, to a data clustering method, apparatus, system, server, and storage medium. Background technique [0002] The rapid development of computer technology and mobile Internet has promoted the arrival of the era of big data. The large-scale data generated in modern society has created enormous pressure on the existing data processing methods. The management and utilization of large-scale data has become an inevitable trend. . As the most commonly used data processing method, data clustering has also become very common in large-scale data applications. [0003] At present, the data clustering method is usually as follows: acquiring multiple pieces of data to be clustered, estimating the resource requirements of the multiple pieces of data and the remaining GPU (Graphics Processing Unit, graphics processor) resources on the server, from the remaining GPU resources G...

AI Technical Summary

Problems solved by technology

[0004] In the above technology, when clustering ordinary large-scale data, GPU resources that meet the resource requirements can be allocated to it. However, when clustering tens of billions of ultra-large-scale data, due to the GPU resources are limited, which will cause the server's memory to be unable to load the data to be clustered at one time, and clustering of ultra-large-scale data cannot be realized

Images