Distributed comparison clustering method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention discloses a distributed comparison clustering method and a device, electronic equipment and a storage medium. The method according to an embodiment of the invention comprises the following steps: receiving sub-tasks equally divided by clustering tasks in a task queue through each computing node; processing the received sub-tasks by the computing node; and sending acomparison clustering result generated by processing to a result queue, continuously getting new sub-tasks from the task queue for processing until all the sub-tasks equally divided corresponding to one clustering task are processed, and finally summarizing the comparison clustering result of clustering each sub-task by the management node to finish the comparison clustering task. And the computing nodes process one subtask according to the processing process of the subtask and then get a new subtask until the comparison clustering task is completed, so that the computing power of each computing node is balanced and fully utilized, and the data comparison clustering is more efficient.

Description

technical field [0001] The embodiments of the present application relate to the technical field of data clustering, and in particular, to a distributed comparison and clustering method, device, electronic equipment, and storage medium. Background technique [0002] In the security tracking and monitoring scenario, it is necessary to compare and cluster the video images monitored by the camera, and use the image feature data after comparison and clustering for subsequent security data analysis. At present, traditional data comparison and analysis mostly use a single node to perform comparison and clustering tasks. However, with the increase in the size of feature data, single-node comparison and clustering can no longer meet the computing needs of products and projects. For urban security monitoring, the amount of monitoring image data is huge, and only a single node is used to perform comparison and clustering tasks. The CPU load is too large and the data calculation efficie...

AI Technical Summary

Problems solved by technology

At present, traditional data comparison and analysis mostly use a single node for the calculation of comparison and clustering tasks, but with the increase in the size of feature data, single-node comparison and clustering can no longer meet the computing needs of products and projects

For urban security monitoring, the amount of monitoring image data is huge, and only a single node is used to perform comparison and clustering tasks, the CPU load is too large, and the data calculation efficiency is low

In order to improve the efficiency of data computing, distributed computing is usually used to relieve the pressure of single-node processing tasks. However, simply distributing tasks to multiple nodes for processing, due to the different computing power of each node, processing the same amount of data The timeliness of tasks is also different. Simply dividing the computing tasks equally cannot make full use of the computing power of each node, which in turn affects the efficiency of data clustering

Images