Distributed data processing method and apparatus

Abstract

The embodiment of the invention provides a distributed data processing method and an apparatus. The distributed data processing method comprises that a cluster client processing apparatus receives storage information of logic roll used by users at present transmitted by a management apparatus, the cluster client processing apparatus receives read or write data instructions transmitted by users, first storage information is determined based on the read or write data instructions and the storage information of the logic roll, the cluster client processing apparatus determines data processing nodes corresponding to every set of four-element set storage information based on tree identification and data block identification in every set of four-element set information, and the cluster client processing apparatus transmits data processing instructions to the data processing nodes corresponding to every set of four-element set information. According to the distributed data processing method and the apparatus provided by the embodiment of the invention, a resource consumption problem because a distributed storage system needs overall setting or read and write Bitmap is avoided, and a data processing effect is raised.

Description

technical field [0001] Embodiments of the present invention relate to data storage technologies, and in particular, to a distributed data processing method and device. Background technique [0002] In distributed data storage technology, compared with physical volumes such as disks and partitions, after the space of physical volumes can be managed, logical volumes (Logical Volumes) can be virtualized. Logical volumes will virtualize storage, and storage is no longer limited by physical disks. the size of. Using logical volumes, the file system can be extended to multiple disks, and multiple disks or disk partitions can be aggregated into a logical volume. When the user does not write data, the created logical volume does not need to actually allocate physical storage space, but When it comes time to write, the physical storage space is allocated dynamically, so as to achieve the purpose of saving physical storage space and delaying cost investment. [0003] A snapshot (Sna...

AI Technical Summary

Problems solved by technology

If the Bitmap is cached in the memory, it will consume a lot of memory resources. If it is not cached in the memory, it will increase the resource consumption of the storage device itself and affect the efficiency of data processing.

Images