Distributed storage data consistency processing method and device

Abstract

The invention discloses a distributed storage data consistency processing method and device.The method comprises the steps that according to operation triggered by a user in a user space file system, the operation is executed for bottom-layer file storage; determining to-be-synchronized information according to the execution result, and storing the to-be-synchronized information in a distributed storage cluster; and triggering synchronization processing according to a preset synchronization time interval, and performing data synchronization on the distributed storage cluster and the underlying file storage according to the to-be-synchronized information. According to the operation triggered by the user, the operation can be given to the bottom-layer file storage, the operation is directly executed by the bottom-layer file storage, the synchronization processing is triggered when the preset synchronization time interval is reached, and the distributed storage cluster is subjected to data synchronization, so that the problems of distributed storage cluster jitter and the like caused by a large amount of synchronization due to the operation are reduced; and the recovery pressure of a large number of synchronous process tasks is also relieved, and the stability of data access is also ensured while the data consistency is ensured.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data storage, in particular to a distributed storage data consistency processing method and device. BACKGROUND

[0002] Large model training has higher requirements for data, and needs to provide a data storage medium with high throughput and large capacity. In the data access processing process, data cleaning and processing will access the underlying data access interface, and unified processing of data access is also needed to shield the underlying heterogeneous storage, simplify the upper-layer development access and operation and maintenance management. Furthermore, data storage also needs to guarantee high availability and stable read-write performance to avoid abnormal influences on training task progress and computing resource utilization rate caused by, for example, fluctuations in I / O operations and node faults. Based on the above requirements, using only basic storage, such as a single distributed file system, cannot meet the needs of large model training. Therefore, data storage is constructed using, for example, underlying file storage (including, for example, a distributed file system), a distributed storage cluster, and a user space file system. The user space file system can map network resources as a local file system, which is convenient for users to directly operate. The distributed storage cluster can be directly accessed through the user space file system, and the distributed storage cluster can be synchronized with the distributed file system, preloaded into the distributed storage cluster, and cache acceleration is achieved to guarantee the stability of data access and the data consistency between the distributed file system and the distributed storage cluster.

[0003] However, if the data strong consistency between the distributed file system and the distributed storage cluster is ensured, the cluster jitter of the distributed storage cluster may be caused by a large number of synchronizations and the like caused by operations, the task amount of the synchronization thread processing is greatly increased, and the process garbage collection pressure is also increased. Therefore, a distributed storage data consistency processing method is needed to improve the data consistency processing. SUMMARY

[0004] In view of the above problems, the embodiments of the present application are proposed to provide a distributed storage data consistency processing method and device that can overcome the above problems or at least partially solve the above problems.

[0005] According to a first aspect of the embodiments of the present application, a distributed storage data consistency processing method is provided, which includes:

[0006] According to the operation triggered by the user in the user space file system, the operation is executed on the underlying file storage;

[0007] According to the execution result, the to-be-synchronized information is determined and stored in the distributed storage cluster;

[0008] Trigger the synchronization processing according to the preset synchronization time interval, and synchronize the data between the distributed storage cluster and the underlying file storage according to the to-be-synchronized information.

[0009] Optionally, the operation is further executed on the underlying file storage according to the operation triggered by the user in the user space file system, and the operation further comprises:

[0010] According to the operation triggered by the user in the user space file system, the operation type of the operation is determined.

[0011] If the operation type of the operation is a preset type, the operation is executed on the underlying file storage; the underlying file storage comprises a distributed file system; and the preset type comprises a non-reading operation type.

[0012] Optionally, the to-be-synchronized information is determined according to the execution result, and the to-be-synchronized information is stored in the distributed storage cluster, and the operation further comprises:

[0013] The execution result of the operation is determined as the to-be-synchronized information, and is stored in the management node of the distributed storage cluster, so as to synchronize the data according to the to-be-synchronized information; the storage time limit of the to-be-synchronized information is the same as the preset synchronization time interval; and the preset synchronization time interval is set based on minutes.

[0014] Optionally, the to-be-synchronized information is determined according to the execution result, and the to-be-synchronized information is stored in the distributed storage cluster, and the operation further comprises:

[0015] According to the execution result of the operation, it is determined whether there is to-be-synchronized information.

[0016] If yes, the to-be-synchronized information is determined according to the execution result of the operation, and is stored in the management node of the distributed storage cluster.

[0017] Optionally, the operation is further executed on the underlying file storage according to the operation triggered by the user in the user space file system, and the operation further comprises:

[0018] According to the operation triggered by the user in the user space file system, the operation is routed to the master node of the distributed file system of the underlying file storage by the distributed storage cluster, and the operation is executed by the master node of the distributed file system.

[0019] Optionally, the synchronization processing is triggered according to the preset synchronization time interval, and the data between the distributed storage cluster and the underlying file storage is synchronized according to the to-be-synchronized information, and the operation further comprises:

[0020] It is judged whether the interval between the current time and the previous synchronization time reaches the preset synchronization time interval.

[0021] If yes, the update operation is executed according to the to-be-synchronized information stored in the management node of the distributed storage cluster, and the data between the distributed storage cluster and the underlying file storage is synchronized.

[0022] Optionally, the distributed storage cluster comprises a management node and a worker node.

[0023] The updating operation is performed according to the to-be-synchronized information stored in the management node of the distributed storage cluster, and the data synchronization between the distributed storage cluster and the underlying file storage further comprises:

[0024] The to-be-synchronized information is read by the management node of the distributed storage cluster, and the data of the corresponding worker node is synchronized with the underlying file storage according to the to-be-synchronized information, and the metadata in the management node is updated.

[0025] Optionally, the updating operation is performed according to the to-be-synchronized information stored in the management node of the distributed storage cluster, and the data synchronization between the distributed storage cluster and the underlying file storage further comprises:

[0026] According to the to-be-synchronized information stored in the management node of the distributed storage cluster, the management node of the distributed storage cluster constructs client objects of a plurality of distributed file systems of the underlying file storage by using a thread pool, so as to perform data synchronization by using the client objects of the plurality of distributed file systems.

[0027] Optionally, the distributed storage cluster comprises a master-slave structure.

[0028] The updating operation is performed according to the to-be-synchronized information stored in the management node of the distributed storage cluster, and the data synchronization between the distributed storage cluster and the underlying file storage further comprises:

[0029] The master management node and the standby management node of the distributed storage cluster are synchronized and updated in state.

[0030] When the master-slave switching is performed, the standby management node re-executes the failed updating operation according to the updating state.

[0031] According to a second aspect of the embodiment of the application, a distributed storage data consistency processing device is provided, which comprises:

[0032] An operation triggering module is adapted to execute an operation triggered by a user in a user space file system to the underlying file storage.

[0033] A storage module is adapted to determine to-be-synchronized information according to an execution result, and store the to-be-synchronized information into a distributed storage cluster.

[0034] A synchronization module is adapted to trigger a synchronization processing according to a preset synchronization time interval, and synchronize the distributed storage cluster with the underlying file storage according to the to-be-synchronized information.

[0035] According to a third aspect of the embodiments of the present application, a computing device is provided, comprising a processor, a memory, a communication interface and a communication bus, the processor, the memory and the communication interface complete communication with each other through the communication bus;

[0036] The memory is configured to store at least one executable instruction, and the executable instruction causes the processor to perform the operations corresponding to the distributed storage data consistency processing method.

[0037] According to a fourth aspect of the embodiments of the present application, a computer storage medium is provided, and the storage medium stores at least one executable instruction, and the executable instruction causes the processor to perform the operations corresponding to the distributed storage data consistency processing method.

[0038] According to a fifth aspect of the embodiments of the present application, a computer program product is provided, comprising at least one executable instruction, and the executable instruction causes the processor to perform the operations corresponding to the distributed storage data consistency processing method.

[0039] According to the distributed storage data consistency processing method and device provided by the present application, according to the operation triggered by the user, the operation is given to the underlying file storage, and the underlying file storage directly executes the operation, and when the preset synchronization time interval is reached, the synchronization processing is triggered again, the data synchronization of the distributed storage cluster is performed, the problem of jitter of the distributed storage cluster caused by a large number of synchronizations caused by operations is reduced, and the recovery pressure of a large number of synchronization process tasks is also reduced, the data consistency is guaranteed, and the stability of data access is also guaranteed.

[0040] The above description is only a summary of the technical solutions of the present application, in order to more clearly understand the technical means of the present application, the content of the specification can be implemented, and in order to make the above and other purposes, characteristics and advantages of the present application more obvious and easy to understand, the following specific embodiments of the present application are described. BRIEF DESCRIPTION OF DRAWINGS

[0041] By reading the detailed description of the preferred embodiments below, various other advantages and benefits will become clear to those of ordinary skill in the art. The drawings are only for the purpose of illustrating the preferred embodiments and are not considered as limiting the present application. Moreover, the same reference symbols are used to represent the same components throughout the drawings. In the drawings:

[0042] Figure 1 A flowchart of a distributed storage data consistency processing method according to an embodiment of the present application is shown;

[0043] Figure 2 A structure diagram among the user space file system, the distributed storage cluster and the underlying file storage is shown;

[0044] Figure 3 A flow chart of a method for processing consistency of distributed storage data according to another embodiment of the present application is shown;

[0045] Figure 4 A structural schematic diagram of an apparatus for processing consistency of distributed storage data according to an embodiment of the present application is shown;

[0046] Figure 5 A structural schematic diagram of a computing device according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0047] Exemplary embodiments of the present application will be described herein below with reference to the accompanying drawings. Although exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided so that this application will be thoroughly and completely understood, and so that the scope of the present application will be conveyed completely to those skilled in the art.

[0048] First, the nomenclature related to one or more embodiments of the present application is explained.

[0049] User space file system: by providing a simplified API library, it supports non-privileged users to safely create and mount file systems, without deep kernel module programming knowledge, significantly reducing the development threshold, mapping network resources as local file systems, allowing users to treat remote files as local files.

[0050] Underlying file storage: the storage space of the underlying file storage is not managed by the distributed storage cluster, and can store data from external file systems, including distributed file systems, etc. The distributed storage cluster can be connected to one or more underlying file storages, and present such underlying storages uniformly in a namespace. The underlying file storage is intended for long-term storage of large amounts of data.

[0051] Distributed file system: the physical storage resources managed by the file system are not necessarily directly connected to the local node, but are connected to the node through a computer network; or a complete hierarchical file system formed by combining several different logical disk partitions or volume labels; a highly fault-tolerant system suitable for deployment on inexpensive machines, providing high-throughput data access, and very suitable for large-scale data sets.

[0052] Distributed storage cluster: a bridge between computing framework and storage system, which makes the computing framework connect to multiple independent storage systems through a public interface, and isolates the computing and storage. The distributed storage cluster unifies the data of different storage systems and provides a unified interface for the upper data framework. The distributed storage cluster can use multiple different storage resources such as memory, solid state disk (SSD), and disk, provide distributed shared cache service, facilitate the frequently accessed data, and provide approximate memory-level I / O throughput to improve stability.

[0053] Management node: a node of the distributed storage cluster, which stores and manages the metadata of all distributed file systems, including the structure information of files and directories, permission settings, timestamps, etc., to facilitate quick positioning to the required data block.

[0054] Worker node: a node of the distributed storage cluster, which manages the local storage space and stores the cached data blocks from the underlying distributed storage system, including files or objects in the distributed file system, etc., to reduce direct access to the underlying storage and improve access speed. When the computing framework needs to access data, the worker node transmits the cached data to the node of the computing framework.

[0055] Master-slave structure: the distributed storage cluster can include multiple management nodes, including a master management node and a standby management node, and the master and standby management nodes synchronize the metadata information through, for example, Journal log.

[0056] Master node of the distributed file system: manages the namespace of the file system, maintains file metadata, and processes client requests.

[0057] Figure 1 A flowchart of a method for processing consistency of distributed storage data according to an embodiment of the present application is shown, as shown in Figure 1 The method includes the following steps:

[0058] Step S101: According to the operation triggered by the user in the user space file system, the operation is executed to the underlying file storage.

[0059] In this embodiment, when the user accesses data, as shown in Figure 2As shown, the user can access the distributed storage cluster through the user space file system, there is data exchange between the distributed storage cluster and the underlying file storage, the data such as files and objects in the underlying file storage can be pre-cached to the distributed storage cluster, when the user accesses the data, the user can directly operate the data from the distributed storage cluster, just like accessing local data, which is convenient and fast. When the user operates through the user space file system, in order to ensure the accuracy of the data obtained by the user, the consistency of the distributed storage data needs to be maintained between the distributed storage cluster and the underlying file storage. The consistency requires data synchronization between the distributed storage cluster and the underlying file storage, and the distributed storage cluster will generate corresponding synchronization process tasks according to the operation, and synchronize data with the underlying file storage. However, when the user performs operations such as viewing the directory size, counting the directory content, and recursively querying the directory on multiple directories, a large number of synchronization operations will be recursively triggered, which will cause the number of tasks in the synchronization process pool to increase sharply, affecting the stability of the distributed storage cluster and causing problems such as cluster jitter. For the underlying file storage, high-concurrency synchronization process tasks will increase the time consumption and the pressure of process recycling and destruction.

[0060] In order to ensure the stability of the user space file system, the distributed storage cluster, and the underlying file storage, and provide stable and fast data access for large model training, the data synchronization between the distributed storage cluster and the underlying file storage is adjusted. When the user triggers an operation through the user space file system, the original operation will be executed by the distributed storage cluster. When the distributed storage cluster executes the operation, it will generate synchronization process tasks for various operations, and the number of tasks in the thread pool of the distributed storage cluster will increase sharply, affecting the operation of the distributed storage cluster and causing cluster jitter. In this embodiment, when the user triggers an operation through the user space file system, the distributed storage cluster can directly route the operation to the underlying file storage for execution, that is, the operation is directly executed by the underlying file storage, and the operation is directly performed on the underlying file storage, without affecting the operation of the distributed storage cluster.

[0061] Further, different operations will produce different execution results, and different execution results may have different effects on the synchronization of the distributed storage cluster. First, the operation is executed by the underlying file storage, and then the to-be-synchronized information is determined according to the execution result, so that the distributed storage cluster does not need to immediately construct a large number of synchronization process tasks according to the files, data, and the like involved in the operation, but determines the to-be-synchronized information according to the execution result. The to-be-synchronized information obtained may or may not need to be synchronized by the distributed storage cluster, and the distributed storage cluster does not need to construct a large number of synchronization process tasks in the synchronization thread pool with high concurrency, thereby reducing the impact on the distributed storage cluster.

[0062] In step S102, the to-be-synchronized information is determined according to the execution result, and the to-be-synchronized information is stored in the distributed storage cluster.

[0063] After the underlying file storage performs an operation, a corresponding execution result can be obtained. The execution result includes, for example, modification, deletion, and addition of files and directories, which actually affect the data of the underlying file storage, and also includes, for example, statistics and display of files and directories, which do not change the data of the underlying file storage itself.

[0064] According to the execution result, the to-be-synchronized information can be determined. For example, if the data of the underlying file storage changes, the to-be-synchronized information can be determined by comparing the distributed storage cluster and the underlying file storage to determine the changed files, directories, and other data as the to-be-synchronized information. The to-be-synchronized information can record the changed data one by one, and the to-be-synchronized information is stored in the distributed storage cluster, so that the distributed storage cluster can synchronize the data with the underlying file storage according to the to-be-synchronized information. For example, when a user performs a delete directory operation through a user space file system, the user will delete the directory and all subdirectories, files, and files in the directory. If the delete operation is performed by the distributed storage cluster, a plurality of synchronization process tasks will be triggered to delete the underlying file storage, which will affect the jitter of the distributed storage cluster and cause the underlying file storage to execute a large number of synchronization process tasks with high concurrency, resulting in high pressure. The underlying file storage can execute the delete operation without handling the synchronization process task in the high concurrency scenario. After the underlying file storage performs the operation, the execution result of the operation can be provided to the distributed storage cluster, so as to determine the to-be-synchronized information required by the distributed storage cluster. For example, according to the execution result of the delete operation, the XX directory and all files in the subdirectories are deleted, and the to-be-synchronized information includes the deletion of the XX directory and all files in the subdirectories. The to-be-synchronized information can be stored in the distributed storage cluster, and the distributed storage cluster can synchronize step by step according to the to-be-synchronized information.

[0065] In step S103, the synchronization processing is triggered according to a preset synchronization time interval, and the distributed storage cluster and the underlying file storage are synchronized according to the to-be-synchronized information.

[0066] To facilitate users to quickly obtain data, a synchronization processing is set between the distributed storage cluster and the underlying file storage. The distributed storage cluster compares and synchronizes with the data in the underlying file storage to ensure the consistency of the distributed storage data, so that the data cached in the distributed storage cluster is consistent with the underlying file storage. Users can quickly obtain data consistent with the underlying file storage by reading the data in the distributed storage cluster, thereby improving the reading efficiency. The synchronization processing can be used to establish synchronization between the distributed storage cluster and the underlying file storage.

[0067] For synchronous processing, if the preset synchronization time interval is set too small, such as 0, 1, etc., the synchronous processing will be frequently executed, especially when the operation involves a large amount of data, such as a statistical directory information operation, which recursively counts all subdirectories and files under the XX directory, and constructs a large number of synchronization process tasks, the number of tasks of the thread pool of the distributed storage cluster increases sharply, occupies a large amount of resources of the distributed storage cluster, affects the use of the distributed storage cluster, and causes cluster jitter. The average synchronization processing time of the distributed storage cluster is about 30ms, and for small file reading, such as files smaller than 1MB, the proportion of overhead is too high, and the benefit of data caching of the distributed storage cluster is low.

[0068] In view of the above problems, the preset synchronization time interval of the embodiment can be set based on minutes, such as 10 minutes, that is, every 10 minutes, trigger a synchronization processing. The preset synchronization time interval of minutes can distribute the original synchronization process tasks that need to be processed at the same time in multiple minutes for processing in turn. For example, the original distributed storage cluster executes operations and constructs a large number of concurrent synchronization process tasks for simultaneous execution. After setting the preset synchronization time interval of minutes, a large number of synchronization process tasks that need to be processed at the same time can be distributed within the preset synchronization time interval for execution, reducing the pressure of high concurrency execution and avoiding bottleneck problems.

[0069] The storage time limit of the to-be-synchronized information is the same as the preset synchronization time interval, such as 10 minutes for the preset synchronization time interval, and the storage time limit is also 10 minutes, which ensures that the to-be-synchronized information is valid during synchronization processing, avoids reading expired to-be-synchronized information, and causes inconsistency of the data after synchronization.

[0070] During synchronization processing, the distributed storage cluster obtains the to-be-synchronized information, which can generate synchronization process tasks one by one and execute them one by one, thereby avoiding the pressure of high concurrency execution and the bottleneck problem of execution. According to the to-be-synchronized information, the distributed storage cluster and the underlying file storage are synchronized, so that the data of the distributed storage cluster and the underlying file storage are consistent.

[0071] According to the distributed storage data consistency processing method provided by the application, according to the operation triggered by the user, the operation is given to the underlying file storage, and the underlying file storage directly executes the operation. When the preset synchronization time interval is reached, the synchronization processing is triggered again to synchronize the data of the distributed storage cluster, which reduces the problem of jitter of the distributed storage cluster caused by a large number of synchronizations caused by operations, and also reduces the recovery pressure of a large number of synchronization process tasks, which guarantees the stability of data access while guaranteeing the consistency of data.

[0072] Figure 3A flowchart of a distributed storage data consistency processing method according to an embodiment of the present application is shown in FIG. 1. As shown in FIG. 1, the method comprises the following steps: Figure 3

[0073] In step S301, according to an operation triggered by a user in a user space file system, the operation type of the operation is determined. If the operation type of the operation is a preset type, the operation is executed by a bottom layer file storage.

[0074] The user can trigger operations through the user space file system, such as file statistics operations, directory list viewing operations, directory renaming operations, deleting directory operations, deleting file operations, reading file operations, and various operations. Through the user space file system, the user's operation is like accessing a local file. To improve access efficiency, the distributed storage cluster can pre-cache the data of the bottom layer file storage for reading, and the user can directly read the corresponding data from the distributed storage cluster. For example, the distributed storage cluster includes a management node and a worker node, the management node stores the metadata of the file, such as directory structure information, etc., to facilitate quick positioning of the file location according to the management node, and directly obtaining the corresponding file from the worker node. To ensure the accuracy of the data obtained by the user, a synchronization processing is set between the distributed storage cluster and the bottom layer file storage to ensure the data consistency of the distributed storage cluster and the bottom layer file storage.

[0075] The operations triggered by the user in the user space file system are various, and the synchronization processing set between the distributed storage cluster and the bottom layer file storage will generate synchronization process tasks due to the operations. When the operation involves a large number of files and directories, a large number of synchronization process tasks will also be created in the synchronization thread pool, which makes the running pressure of the distributed storage cluster larger and also easily leads to the execution bottleneck problem of the synchronization process tasks. When the operation involves partial file reading, the file can be directly read from the distributed storage cluster, and the synchronization operation will not have a great impact on the distributed storage cluster. Considering the above problems, for the operation triggered by the user in the user space file system, the operation type of the operation can be determined first. The operation type can include operation types such as read operation, write operation, list operation, and statistics operation. When the operation type of the operation is a preset type, the preset type includes a non-reading operation type, i.e., the operation type is an operation type other than the read operation, which may involve a large number of file and directory synchronization operations. If it is processed by the distributed storage cluster, a large number of synchronization process tasks may be generated at the same time, and the synchronization thread pool task will increase sharply. However, the bottom layer file storage executes the operation, which will not have an impact on the distributed storage cluster. Therefore, the operation can be directly routed to the bottom layer file storage for execution by the distributed storage cluster, and the bottom layer file storage can directly execute the operation.

[0076] For example, Figure 2 ​As shown, the underlying file storage includes, for example, a distributed file system. Further, the underlying file storage can also include other systems, such as a distributed object system, etc., which are not limited herein. The distributed storage cluster can provide a unified access interface for the user space file system. The distributed storage cluster can adapt to multiple data storage systems. For ease of understanding, the distributed file system is taken as an example for description, but is not limited thereto. The distributed file system can adopt a master-slave structure. The distributed storage cluster can route the operation directly to the master node of the distributed file system, and the master node of the distributed file system executes the operation. The slave node can synchronize with the master node in the distributed file system, which is not described herein. The distributed storage cluster can route the operation directly to the master node of the distributed file system. When the master node executes the operation, there is no read-write delay. The synchronization, load, etc. of the slave node can not be concerned, the operation efficiency of the distributed file system is improved, and the efficiency of the synchronization operation is also improved.

[0077] Further, if the operation type of the operation is not the preset type, for example, the operation type of the operation is a read operation type, that is, the user sends a read file operation or a read directory operation to the distributed storage cluster through the user space file system. When the distributed storage cluster executes the operation of the read operation type, the comparison with the underlying file storage is only for the target file or directory, and a large number of synchronization process tasks are not generated. The distributed storage cluster can directly execute the operation of the read operation type. Further, if the user triggers the operation in the user space file system, for example, reads the file, and the distributed storage cluster has an abnormal condition, the read fails. A degraded read mode can be used, for example, the underlying file storage executes the read operation when the read is abnormal, which is not limited herein.

[0078] In step S302, whether there is to-be-synchronized information is determined according to the execution result of the operation.

[0079] After the operation is executed on the underlying file storage, a corresponding execution result is generated, and the execution result generated according to different operations is also different. For example, when the operation is a write operation type, the data of the underlying file storage is affected, and some data such as new files or directories may be added, some data such as files or directories may be modified, and some data such as files or directories may be deleted. After the underlying file storage executes the write operation type operation, the data cached in the underlying file storage and the distributed storage cluster is inconsistent. When the operation is a list operation type or a statistical operation type, the underlying file storage displays or counts the directory files, and does not cause the change of the data such as files or directories, and does not affect the data cached in the underlying file storage and the distributed storage cluster. According to the execution result of the above different operations, it can be determined whether there is to-be-synchronized information. If the data cached in the underlying file storage and the distributed storage cluster is inconsistent, it is necessary to synchronize the underlying file storage and the distributed storage cluster for the convenience of users, that is, it is necessary to determine the to-be-synchronized information between the underlying file storage and the distributed storage cluster, so as to synchronize it, that is, to execute step S303. If the data cached in the underlying file storage and the distributed storage cluster is not inconsistent, the execution result of the underlying file storage is returned to the user, and there is no to-be-synchronized information between the underlying file storage and the distributed storage cluster.

[0080] Further, when the operation is a read operation type, the distributed storage cluster directly executes the read operation type operation or the underlying file storage executes the read operation, which does not cause the data cached in the underlying file storage and the distributed storage cluster to be inconsistent, and the to-be-synchronized information can not be considered or can be compared to avoid other operations within the preset synchronization time interval, which is not limited here.

[0081] Step S303: Determine the to-be-synchronized information according to the execution result of the operation, and store it in the management node of the distributed storage cluster.

[0082] According to the execution result of the operation, the inconsistent part of the data cached in the underlying file storage and the distributed storage cluster can be determined, and then the to-be-synchronized information is determined. The inconsistent part of the data can be determined according to the directory file involved in the execution result of the underlying file storage, or the underlying file storage and the distributed storage cluster can be compared to determine the to-be-synchronized information, considering that the distributed storage cluster has not been synchronized with the underlying file storage for a period of time, which is not limited here.

[0083] The distributed storage cluster includes a management node and a working node. For the to-be-synchronized information, it can be stored in the management node of the distributed storage cluster. The management node can read the to-be-synchronized information in sequence and synchronize with the underlying file storage.

[0084] In step S304, the synchronization processing is triggered according to the preset synchronization time interval, and the distributed storage cluster and the underlying file storage are synchronized according to the to-be-synchronized information.

[0085] In the embodiment, when processing the synchronization processing, the preset synchronization time interval can be set, which is a minute level. The synchronization processing can be distributed in multiple minutes, the pressure of high concurrency processing of the distributed storage cluster is reduced, and the performance bottleneck is avoided.

[0086] When performing the synchronization processing, whether the interval between the current time and the previous synchronization time reaches the preset synchronization time interval can be judged. If not, the judgment is continued until the interval between the current time and the previous synchronization time reaches the preset synchronization time interval. If yes, the synchronization processing is triggered, and the management node of the distributed storage cluster can perform the update operation according to the to-be-synchronized information stored in the management node. Specifically, the to-be-synchronized information stored by the management node of the distributed storage cluster is read, and the update processing is performed according to the to-be-synchronized information, the data of the corresponding worker node is synchronized with the underlying file storage, that is, the files stored in the worker node are synchronized with the files in the underlying file storage, and the metadata in the management node is updated, such as the directory structure information and file size information of the file in the metadata, so that the data of the distributed storage cluster and the underlying file storage are kept synchronized. Further, if the underlying file storage includes different types of data storage media, such as different distributed file systems and object systems, when performing data synchronization, the data of each data storage medium in the distributed storage cluster and the underlying file storage needs to be synchronized.

[0087] When synchronizing data, considering that there may be problems such as performance bottleneck and lock mechanism when synchronizing the data storage medium of the underlying file storage and the distributed storage cluster, which affects the efficiency of the synchronization processing, when performing the synchronization processing, multiple stored client objects of the distributed storage cluster can be created in the thread pool of the distributed storage cluster. Taking the distributed file system as an example, when the synchronization processing is performed by the management node of the distributed storage cluster according to the to-be-synchronized information stored in the management node of the distributed storage cluster, the client objects of multiple distributed file systems of the underlying file storage can be constructed by the distributed storage cluster using the thread pool. The multiple client objects of the distributed file systems can simultaneously provide services for the distributed storage cluster, so that the data synchronization can be performed by using the multiple client objects of the distributed file systems, the efficiency of the synchronization is improved, and the lock consumption is avoided.

[0088] For a distributed storage cluster, it can include a master-standby structure, such as including a master management node and a standby management node. Considering master-standby switching, data loss and other problems can be caused, and the master management node and the standby management node can be synchronized through, for example, a log Journal. When the working node reports information to the management node during execution, to avoid time sequence competition between the channel for reporting information and the log Journal playback during master-standby switching, loss or false labeling of the reported information as abnormal information, the channel for reporting information of the working node can be closed before the master management node and the standby management node are synchronized, and the channel for reporting information of the working node can be opened after the master management node and the standby management node are synchronized. When the master management node and the standby management node are synchronized, the master management node and the standby management node of the distributed storage cluster are synchronized and updated for each state, such as the state of each task during execution of a synchronization process task. When master-standby switching occurs, the standby management node can receive the reported information of the working node, determine the execution according to the reported information, and re-execute the failed update operation according to the situation in the updated state, so as to ensure that master-standby switching does not affect the execution integrity of the synchronized data.

[0089] According to the distributed storage data consistency processing method provided in the present application, the user can perform a preset type of operation through a user space file system, and the operation can be directly routed to the underlying file storage by the distributed storage cluster for execution, avoiding the generation of a large number of synchronization process tasks by the distributed storage cluster due to the operation, blocking the synchronization thread pool, and affecting the execution efficiency. The operation is directly executed by the underlying file storage, which does not affect the operation of the distributed storage cluster. According to the execution result, it is determined whether there is to-be-synchronized information. The management node of the distributed storage cluster stores the to-be-synchronized information, which can trigger synchronization processing when a preset synchronization time interval is reached, and synchronizes the data of the working node and the management node of the distributed storage cluster, thereby reducing the problem of jitter of the distributed storage cluster caused by a large number of synchronizations due to operations. During data synchronization, the management node of the distributed storage cluster uses a thread pool to build client objects of a plurality of distributed file systems of the underlying file storage for data synchronization, thereby improving synchronization efficiency, reducing lock holding consumption of the distributed file system, and ensuring the stability of user data access.

[0090] Figure 4 A structure diagram of a distributed storage data consistency processing apparatus provided by an embodiment of the present application is shown. As shown in the figure, Figure 4 The apparatus includes:

[0091] The operation triggering module 410 is adapted to execute the operation on the underlying file storage according to the operation triggered by the user on the user space file system.

[0092] The storage module 420 is adapted to determine to-be-synchronized information according to the execution result, and store the to-be-synchronized information in the distributed storage cluster.

[0093] The synchronization module 430 is adapted to trigger the synchronization process according to the preset synchronization time interval, and synchronize the distributed storage cluster with the underlying file storage according to the to-be-synchronized information.

[0094] Optionally, the operation triggering module 410 is further adapted to:

[0095] determining the operation type of the operation according to the operation triggered by the user in the user space file system;

[0096] if the operation type of the operation is a preset type, executing the operation on the underlying file storage; the underlying file storage comprises a distributed file system; and the preset type comprises a non-reading operation type.

[0097] Optionally, the storage module 420 is further adapted to:

[0098] determining the execution result of the operation as the to-be-synchronized information, and storing the to-be-synchronized information to the management node of the distributed storage cluster for data synchronization according to the to-be-synchronized information; the storage time limit of the to-be-synchronized information is the same as the preset synchronization time interval; and the preset synchronization time interval is set based on a minute level.

[0099] Optionally, the storage module 420 is further adapted to:

[0100] determining whether there is to-be-synchronized information according to the execution result of the operation;

[0101] if yes, determining the to-be-synchronized information according to the execution result of the operation, and storing the to-be-synchronized information to the management node of the distributed storage cluster.

[0102] Optionally, the synchronization module 430 is further adapted to:

[0103] routing the operation triggered by the user in the user space file system to the master node of the distributed file system of the underlying file storage by the distributed storage cluster, and executing the operation by the master node of the distributed file system.

[0104] Optionally, the synchronization module 430 is further adapted to:

[0105] judging whether the interval between the current time and the previous synchronization time reaches the preset synchronization time interval;

[0106] if yes, performing an update operation according to the to-be-synchronized information stored in the management node of the distributed storage cluster, and synchronizing the distributed storage cluster with the underlying file storage.

[0107] Optionally, the distributed storage cluster comprises a management node and a working node.

[0108] The synchronization module 430 is further adapted to:

[0109] The stored to-be-synchronized information is read by a management node of the distributed storage cluster, and according to the to-be-synchronized information, data of a corresponding worker node is synchronized with the underlying file storage, and metadata in the management node is updated.

[0110] Optionally, the synchronization module 430 is further adapted to:

[0111] According to the to-be-synchronized information stored in the management node of the distributed storage cluster, the management node of the distributed storage cluster constructs client objects of a plurality of distributed file systems of the underlying file storage by using a thread pool, so as to perform data synchronization by using the client objects of the plurality of distributed file systems.

[0112] Optionally, the distributed storage cluster comprises a master-backup structure.

[0113] The synchronization module 430 is further adapted to:

[0114] Synchronize and update states of a master management node and a backup management node of the distributed storage cluster.

[0115] When the master-backup switches, the backup management node re-executes a failed update operation according to the update state.

[0116] The above description of each module refers to the corresponding description in the method embodiment, and will not be repeated here.

[0117] According to the distributed storage data consistency processing apparatus provided in the present application, according to a user-triggered operation, the operation can be performed on the underlying file storage, and the underlying file storage directly executes the operation, and when a preset synchronization time interval is reached, a synchronization processing is triggered again, and the distributed storage cluster is synchronized, which reduces problems such as jitter of the distributed storage cluster caused by a large amount of synchronization caused by the operation, and also reduces the recovery pressure of a large amount of synchronization process tasks, guarantees data consistency, and also guarantees the stability of data access.

[0118] The present application also provides a non-volatile computer storage medium, and the computer storage medium stores at least one executable instruction, and the executable instruction can execute the operation corresponding to the distributed storage data consistency processing method in any method embodiment.

[0119] The present application also provides a computer program product, and the computer program product comprises at least one executable instruction or computer program, and the executable instruction or computer program can make a processor execute the operation corresponding to the distributed storage data consistency processing method in any method embodiment.

[0120] Figure 5 A structural schematic diagram of a computing device according to an embodiment of the present application is shown, and specific embodiments of the present application do not limit the specific implementation of the computing device.

[0121] As shown in Figure 5 The computing device can include a processor 502, a communications interface 504, a memory 506, and a communications bus 508.

[0122] Wherein:

[0123] The processor 502, the communications interface 504, and the memory 506 complete the communication with each other through the communications bus 508.

[0124] The communications interface 504 is configured to communicate with network elements such as clients or other servers.

[0125] The processor 502 is configured to execute the program 510, and specifically can execute the related steps in the distributed storage data consistency processing method embodiments described above.

[0126] Specifically, the program 510 can include program code, which includes computer operation instructions.

[0127] The processor 502 can be a central processing unit CPU, or an application specific integrated circuit ASIC, or one or more integrated circuits configured to implement the present application. The one or more processors included in the computing device can be the same type of processor, such as one or more CPUs; or can be different types of processors, such as one or more CPUs and one or more ASICs.

[0128] The memory 506 is configured to store the program 510. The memory 506 can include a high-speed RAM memory, and can also include a non-volatile memory, for example at least one disk memory.

[0129] The program 510 can be specifically used to cause the processor 502 to execute the distributed storage data consistency processing method in any of the method embodiments described above. The specific implementation of each step in the program 510 can refer to the corresponding description in the corresponding steps and units in the distributed storage data consistency processing embodiments described above, and will not be described here. Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the device and the module described above can refer to the corresponding process description in the foregoing method embodiments, and will not be described here.

[0130] The algorithms and displays presented herein are not inherently related to any particular computer, virtual system, or other apparatus. Various general purpose systems can be used with programs in accordance with the teachings herein, or it can prove convenient to construct more specialized apparatus to perform the required method steps. The required structure for a variety of these systems will be apparent from the description above. In addition, the present application is not intended to be limited to a particular programming language. It will be appreciated that a variety of programming languages can be used to implement the teachings of the application as described herein, and any references below to specific languages are provided for disclosure of enablement only.

[0131] In the description provided herein, numerous specific details are set forth. However, it is understood that embodiments of the application can be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been described in detail in order to avoid obscuring the understanding of this description.

[0132] Similarly, it is to be understood that the mechanical details of the application sometimes are grouped into single embodiments, figures or descriptions of embodiments for purposes of brevity. However, this method of disclosure is not to be interpreted as reflecting an intention that the application requires more features than are explicitly recited in each claim. Rather, inventive aspects lie in less than all features of a single foregoing disclosed embodiment. Accordingly, the claims are to be interpreted not as admitting only embodiments that literally encompass all features of the description, but rather as admitting embodiments that encompass less than all features of the description. Thus, the claims following, when interpreted as including what is equivalent to what is explicitly recited, are to be understood to embrace all embodiments of the application.

[0133] Those skilled in the art will appreciate that the modules in the apparatuses in the embodiments can be adapted and placed in one or more apparatuses other than the embodiments. The modules or units or components in the embodiments can be combined into one module or unit or component, and further can be divided into more sub-modules or sub-units or sub-components. Any combination of all the features disclosed in the specification (including the accompanying claims, abstract and drawings), and any method or apparatus otherwise disclosed in the specification, can be used in any combination, except that at least some of such features and / or processes or units are mutually exclusive. Unless explicitly stated otherwise, each feature disclosed in the specification (including the accompanying claims, abstract and drawings) can be replaced by alternative features that serve the same, equivalent or similar purpose.

[0134] Furthermore, those skilled in the art will recognize that, while certain embodiments described herein include certain features that are not included in other embodiments, combinations of features of the different embodiments are meant to be within the scope of the application and form different embodiments. For example, in the claims below, any of the claimed embodiments can be used in any combination.

[0135] Various component embodiments of the present application can be implemented in hardware, or as software modules running in one or more processors, or in combinations thereof. Those skilled in the art will appreciate that a microprocessor or digital signal processor (DSP) can be used in practice to implement some or all of the functionality of some or all of the components according to the present application. The present application can also be implemented as a program of instructions for performing part or all of the methods described herein, e.g., a computer program and a computer program product. Such a program of instructions of the present application can be stored on a computer readable medium, or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, or provided on a carrier signal, or in any other form.

[0136] It is noted that the foregoing examples have been provided merely for the purpose of explanation and are in no way to be construed as limiting of the present application. While the application has been described with reference to preferred embodiments, it is understood that the words which have been used herein are words of description, and that changes can be made within the scope and spirit of the application. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word comprising does not exclude the presence of elements or steps not listed in a claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The application can be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer. In the system claims enumerating several means, several of these means can be embodied by one and the same item of hardware. The use of the words first, second, third, etc. do not imply any order. The terms first, second and third are to be interpreted as names. The steps of any of the methods disclosed herein do not have to be performed in the exact order disclosed.

Claims

1. A method for processing consistency of data stored in a distributed storage, comprising: executing an operation triggered by a user in a user space file system on an underlying file storage according to the operation; determining to-be-synchronized information according to an execution result of the operation, and storing the to-be-synchronized information in a distributed storage cluster; triggering a synchronization process according to a preset synchronization time interval, and synchronizing data between the distributed storage cluster and the underlying file storage according to the to-be-synchronized information.

2. The method of claim 1, wherein, The step of executing the operation triggered by the user in the user space file system on the underlying file storage according to the operation further comprises: determining an operation type of the operation triggered by the user in the user space file system; if the operation type of the operation is a preset type, executing the operation on the underlying file storage; the underlying file storage comprises a distributed file system; and the preset type comprises a non-reading operation type.

3. The method of claim 1, wherein, The step of determining to-be-synchronized information according to an execution result of the operation, and storing the to-be-synchronized information in a distributed storage cluster further comprises: determining to-be-synchronized information according to the execution result of the operation, and storing the to-be-synchronized information in a management node of the distributed storage cluster for data synchronization according to the to-be-synchronized information; the storage time limit of the to-be-synchronized information is the same as the preset synchronization time interval; and the preset synchronization time interval is set based on a minute level.

4. The method of claim 3, wherein, The step of determining to-be-synchronized information according to an execution result of the operation, and storing the to-be-synchronized information in a distributed storage cluster further comprises: determining whether there is to-be-synchronized information according to the execution result of the operation; if yes, determining to-be-synchronized information according to the execution result of the operation, and storing the to-be-synchronized information in a management node of the distributed storage cluster.

5. The method of any one of claims 1-4, wherein, The step of executing the operation triggered by the user in the user space file system on the underlying file storage according to the operation further comprises: executing the operation triggered by the user in the user space file system on a master node of a distributed file system of the underlying file storage by the distributed storage cluster, and executing the operation by the master node of the distributed file system.

6. The method of claim 1, wherein, The step of triggering a synchronization process according to a preset synchronization time interval, and synchronizing data between the distributed storage cluster and the underlying file storage according to the to-be-synchronized information further comprises: determining whether an interval between a current time and a previous synchronization time reaches a preset synchronization time interval; if yes, performing an update operation according to to-be-synchronized information stored in a management node of the distributed storage cluster, and synchronizing data between the distributed storage cluster and the underlying file storage.

7. The method of claim 6, wherein, The distributed storage cluster comprises a management node and a working node. The step of performing an update operation according to to-be-synchronized information stored in a management node of the distributed storage cluster, and synchronizing data between the distributed storage cluster and the underlying file storage further comprises: reading the stored to-be-synchronized information by the management node of the distributed storage cluster, synchronizing data between a corresponding working node and the underlying file storage according to the to-be-synchronized information, and updating metadata in the management node.

8. The method of claim 6 or 7, wherein, The updating operation is performed according to the to-be-synchronized information stored in the management node of the distributed storage cluster, and the data synchronization between the distributed storage cluster and the underlying file storage further includes: According to the to-be-synchronized information stored in the management node of the distributed storage cluster, the management node of the distributed storage cluster constructs client objects of a plurality of distributed file systems of the underlying file storage by using a thread pool, so as to perform data synchronization by using the client objects of the plurality of distributed file systems.

9. The method of any one of claims 6-8, wherein, The distributed storage cluster includes a master-slave structure. The updating operation is performed according to the to-be-synchronized information stored in the management node of the distributed storage cluster, and the data synchronization between the distributed storage cluster and the underlying file storage further includes: The master management node and the standby management node of the distributed storage cluster are synchronized and updated in a state; When the master-slave switching is performed, the standby management node re-executes the failed updating operation according to the updating state.

10. A distributed storage data consistency processing apparatus, comprising: An operation triggering module adapted to execute an operation triggered by a user in a user space file system to an underlying file storage according to the operation; A storage module adapted to determine to-be-synchronized information according to an execution result and store the to-be-synchronized information into a distributed storage cluster; A synchronization module adapted to trigger a synchronization processing according to a preset synchronization time interval and perform data synchronization between the distributed storage cluster and the underlying file storage according to the to-be-synchronized information.

11. A computing device comprising: A processor, a memory, a communication interface and a communication bus, the processor, the memory and the communication interface complete communication with each other through the communication bus; The memory is used to store at least one executable instruction, and the executable instruction causes the processor to perform the operation corresponding to the distributed storage data consistency processing method in any one of claims 1-9.

12. A computer storage medium, the storage medium stores at least one executable instruction, and the executable instruction causes a processor to perform the operation corresponding to the distributed storage data consistency processing method in any one of claims 1-9.

13. A computer program product, comprising at least one executable instruction, and the executable instruction causes a processor to perform the operation corresponding to the distributed storage data consistency processing method in any one of claims 1-9.

Images