Data management method and device of distributed storage system, medium and product

Abstract

The invention discloses a data management method and device for a distributed storage system, a medium and a product, and relates to the technical field of storage, and the method comprises the steps: analyzing a stored first specified file on the basis of presetting storage pools of different fault-tolerant types in the distributed storage system, and storing the first specified file in the distributed storage system; and when the data attributes change, the optimal fault-tolerant type of the changed specified file is obtained in time, and the file is carried according to the fault-tolerant type so as to be reasonably stored in the storage pool under the matched fault-tolerant type, so that the performance of the storage system is improved.

Description

Technical Field

[0001] The present application relates to the field of storage technology, and in particular to data management methods, devices, media and products of distributed storage systems. Background Art

[0002] With the rapid development of big data and artificial intelligence technology, the total amount of global data is growing exponentially. In order to achieve the storage and management of big data, distributed storage systems are widely used.

[0003] Distributed storage systems are based on network technology and effectively share storage loads by distributing data across multiple independent physical storage devices. As the storage scale of large-scale distributed storage systems grows and data attributes continue to change, the performance of existing distributed storage systems with a single fault-tolerant mode continues to decline. Summary of the invention

[0004] The present application provides a data management method, device, medium and product for a distributed storage system to at least solve related technical problems.

[0005] The present application provides a data management method for a distributed storage system, wherein the distributed storage system includes a management layer and an application layer, and a plurality of storage pools of different fault-tolerant types are arranged in the distributed storage system, and the method includes: In response to a preset condition trigger, the management layer obtains first metadata of the first designated file to determine current data attributes of the first designated file; In response to detecting one or more attribute changes in the current data attributes of the first designated file, the management layer determines a target fault tolerance type of the first designated file according to access records in the current data attributes and preset rules; In response to detecting a change in the target fault tolerance type, the management layer performs a data migration operation on the first designated file to migrate the first designated file to a matching target storage pool.

[0006] The present application also provides a computer-readable storage medium, in which a computer program is stored, wherein when the computer program is executed by a processor, the steps of any of the above-mentioned distributed storage system data management methods are implemented.

[0007] The present application also provides a computer program product, including a computer program, which implements the steps of any of the above-mentioned distributed storage system data management methods when executed by a processor.

[0008] The present application analyzes the first designated file that has been stored on the basis of presetting storage pools of different fault tolerance types in a distributed storage system, promptly obtains the optimal fault tolerance type of the designated file after the change when the data attribute changes, and further moves the file according to the fault tolerance type to reasonably store the file in a storage pool with a matching fault tolerance type, thereby reducing the storage cost of the storage system and improving the performance of the storage system. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0010] Figure 1 A schematic diagram of a distributed storage system architecture provided in an embodiment of the present application; Figure 2 A schematic diagram of a data management method provided in an embodiment of the present application; Figure 3 A schematic diagram of another data management method provided in an embodiment of the present application; Figure 4 A file concept diagram provided for an embodiment of the present application; Figure 5 A schematic diagram of a data fingerprint array provided in an embodiment of the present application; Figure 6 A schematic diagram of a data management device provided in an embodiment of the present application; Figure 7 A schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0011] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. According to the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0012] It should be noted that, in the description of this application, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. The terms "first", "second", etc. in this application are used to distinguish similar objects, and are not used to describe a specific order or sequence.

[0013] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0014] With respect to the data management method of the distributed storage system disclosed in this application, Figure 1 A typical distributed storage system architecture is shown. The distributed storage system usually includes an application layer, a protocol layer, a management layer and a hardware layer. The application layer is usually a user-facing program, which can be a client, an application or a virtual machine. The specific form of the application layer is not limited in this application. The protocol layer mainly meets the protocol access requirements of different users. A protocol library designed by a person skilled in the art is deployed therein. The protocol library provides one or more sets of API interfaces to support different interface protocols and allows the application layer to interact with the distributed storage system through the protocol library. The management layer is the core of the distributed storage system, and mainly includes a monitor service module, a management service module, an object storage service module and a metadata service module. Among them, the monitor service module is mainly used to maintain the storage cluster status, such as how many storage pools there are in the cluster, how many data groups there are in each storage pool, and the mapping relationship between the storage pool and the data group. The management service module is responsible for managing the cluster information and tracking the runtime indicators and current status. There are multiple object storage service nodes in the object storage service module. Each object storage service node manages a disk and is responsible for processing tasks such as cluster data replication, recovery and rebalancing; and provides information to the monitor service module and the management service module by checking the daemon mentality of other object storage service nodes. The metadata service is mainly responsible for managing the metadata of the cluster. The storage pool is a logical storage unit in the distributed storage system, which is used to organize and manage data and is mapped to specific data groups through the map layer. The data group is a physical data distribution unit, which is used to disperse data to different object storage service nodes. By dividing the data into multiple parts and distributing them on different object storage service nodes, load balancing is achieved and data reliability and access efficiency are improved. The hardware layer is composed of physical disks that actually provide storage services.

[0015] It is understandable that in some specific application scenarios, the data management method disclosed in the present application can also be adaptively adjusted and applied to a storage cluster. It only requires deploying a corresponding management module in the storage system to execute the data management method disclosed in the present application.

[0016] An embodiment of the present application provides a data management method for a distributed storage system. It is worth noting that the distributed storage system disclosed in the present application is pre-set with storage pools of different fault-tolerant types. In a specific implementation scenario, it is preferably set to two types of storage pools of fault-tolerant types, such as a multi-copy storage pool and an erasure code storage pool. Of course, in other implementation scenarios, other types of storage pools of fault-tolerant types can also be set, such as a data sharding fault-tolerant storage pool. The present application does not limit the specific number and types of fault-tolerant types.

[0017] like Figure 2 As shown, the above data management method specifically includes the following steps: S1. In response to a preset condition trigger, the management layer obtains first metadata of a first designated file to determine current data attributes of the first designated file.

[0018] In a specific implementation scenario, the preset conditions of the above response include detecting a file read request sent by the application layer and reaching a certain period; specifically, in response to detecting a file read request sent by the application layer, the management layer obtains the first metadata of the first designated file to determine the current data attributes of the first designated file; and / or, in response to the time difference between the current time node collected and the previous historical time node collected being greater than or equal to the preset time difference, the management layer obtains the first metadata of the first designated file to determine the current data attributes of the first designated file. Among them, the above determination of the current data attributes of the first designated file based on the first metadata of the first designated file means that the object information, access records, file identifiers (including file paths, file names) and any other information used to describe file attributes contained in the metadata of the file are all identified as current data attributes.

[0019] Optionally, the above-mentioned preset time difference can be set to 30 minutes, 2 hours, etc. The specific preset time difference is set by a person skilled in the art according to the scale and performance of the storage system, and the present application does not limit this. It is worth noting that the above-mentioned two preset conditions can be adopted in the storage system either one or both. Based on the different trigger mechanisms set up above, it is possible to achieve a comprehensive analysis of the files in the storage system, thereby achieving reasonable storage of all files to improve system stability; it is also possible to conduct targeted analysis of specific files to save storage system computing resources. Therefore, it can be understood that the above-mentioned monitoring of current data attributes can occur at any time when the distributed storage system is working.

[0020] The definition of the first designated file is explained as follows: when responding to a file read request, the first designated file is a file that matches the file identifier such as the file name or file path contained in the file read request sent by the application layer; when the response time difference is greater than or equal to the preset time difference, the first designated file can be set to all files in the storage system.

[0021] Preferably, the first metadata information includes object information, fault tolerance type, and the most recent ten access records constituting the first designated file; the object information includes a first object identifier, segmentation information, and a hash value combination of the object data. It is understandable that in this application, "first" and "second" are only used for distinction, such as the first designated file and the second designated file, both of which belong to files; as well as the first metadata and the second metadata.

[0022] S2. In response to detecting one or more attribute changes in the current data attributes of the first designated file, the management layer determines a target fault tolerance type of the first designated file according to access records in the current data attributes and preset rules.

[0023] It is understandable that as long as any change occurs to the file, that is, any attribute of the data attributes changes, the next step of determining the target fault tolerance type will be triggered. In the method disclosed in the embodiment of the present application, the management layer determines whether the current data attributes and the historical data attributes are consistent based on the current data attributes of the first specified file and the historical data attributes obtained when responding to the preset conditions last time. If they are consistent, it is determined that the current data attributes have not changed. If there is a data attribute change, it triggers the next step of determining the target fault tolerance type of the first specified file.

[0024] In the specific implementation scenario of setting up an erasure code fault-tolerant storage pool and multi-copy fault-tolerant storage, the above-mentioned determination of the target fault tolerance type of the first designated file based on the access records and preset rules can be achieved by calculating the file heat value. The specific management layer calculates the file heat value based on the access records and the preset formula contained in the first metadata corresponding to the first designated file; the management layer determines the target fault tolerance type of the first designated file based on the file heat value.

[0025] The above preset formula is expressed as: ; in, Hot is the file heat value, i is the serial number of the access record, n represents the maximum number of access records matched by the first specified file, T cur Indicates the current time. T i For the most recent i The access time, αi Indicates i The weight coefficient of the access record. The weight coefficient can be generated by training a weight coefficient determination model based on historical access data or by a trained model; it can also be determined by those skilled in the art based on empirical values, which is not limited in this application.

[0026] Among them, the above-mentioned management layer determines the target fault tolerance type of the first designated file according to the file heat value, including: the management layer responds to detecting that the file heat value is less than or equal to the preset threshold, that is, the thermal attribute of the file is hot data, and determines that the target fault tolerance type of the first designated file is the first fault tolerance type (i.e., multi-copy fault tolerance); the management layer responds to detecting that the file heat value is greater than the preset threshold, that is, the thermal attribute of the file is cold data, and determines that the target fault tolerance type of the first designated file is the second fault tolerance type (i.e., erasure code fault tolerance). That is, it is defined that the activity of files stored in the storage pool matching the first fault tolerance type is higher than that of files stored in the storage pool matching the second fault tolerance type. Among them, the preset threshold is set by those skilled in the art according to actual conditions, and this application does not limit this. Of course, if more than two fault tolerance types are set, it can be adaptively adjusted to add a preset threshold, which is divided into the first threshold and the second threshold. Determine that the target fault tolerance type of the first designated file whose file heat value is greater than the first threshold is the first fault tolerance type, the target fault tolerance type of the first designated file whose file heat value is less than the first threshold and greater than the second threshold is the second fault tolerance type, and the target fault tolerance type of the first designated file whose file heat value is less than the second threshold is the third fault tolerance type.

[0027] The present application discloses a specific method for determining the target fault tolerance type of a first designated file, namely, by setting a calculation formula for the heat value, determining the file heat values ​​of different files according to access records, that is, using the heat value to distinguish the access busyness of different files, thereby further improving the accuracy of confirming the optimal target fault tolerance type of the file.

[0028] Of course, in the implementation scenario of setting up storage pools of other fault-tolerant types, different rules can also be designed to determine the optimal target fault-tolerance type for each first designated file. For example, by combining the file heat value and storage cost, the index scores of different files are calculated by setting different weights, and the corresponding thresholds are further set to determine that the target fault-tolerance type of files whose index scores exceed a specific threshold is the first fault-tolerance type, and the target fault-tolerance type of files that do not exceed the specific threshold is the second fault-tolerance type; of course, the index scores of different files can also be calculated in combination with factors such as data sensitivity and network delay. This application does not impose constraints on specific preset rules.

[0029] S3. In response to detecting a change in the target fault tolerance type, the management layer performs a data transfer operation on the first designated file to transfer the first designated file to a matching target storage pool.

[0030] Whether the target fault tolerance type has changed is determined by comparing the target fault tolerance type with the latest fault tolerance type recorded in the first metadata matching the first designated file. If the target fault tolerance type is the same as the latest fault tolerance type, it is determined that the target fault tolerance type has not changed and no data transfer is required. If the target fault tolerance type is different from the latest fault tolerance type, the management layer determines that the target fault tolerance type has changed. In response to detecting a change in the target fault tolerance type, the management layer performs a data transfer operation on the first designated file to transfer the first designated file to a matching target storage pool, including: In response to detecting that the target fault tolerance type of the first designated file changes from the second fault tolerance type to the first fault tolerance type, the management layer performs a data transfer operation on the first designated file to transfer the first designated file to a storage pool that matches the first fault tolerance type; in response to detecting that the fault tolerance type of the first designated file changes from the first fault tolerance type to the second fault tolerance type, the management layer obtains the object distribution of the first object contained in the first designated file; the management layer determines whether to perform a data transfer operation on the first designated file based on the object distribution, and the object distribution includes single-file distribution and multi-file distribution. That is, when the fault tolerance type changes, the fault tolerance types of other files in the object distribution need to be considered. Only when the fault tolerance types of the first designated file and all files in the object distribution contained in the first designated file are consistent with the target fault tolerance type of the first designated file, the first designated file will be transferred to a storage pool that matches the target fault tolerance type (i.e., the target storage pool). As long as there is a file in the object distribution whose fault tolerance type is not uniform, no data transfer will be performed.

[0031] The above-mentioned management layer determines whether to perform a data transfer operation on the first designated file according to the object distribution situation, including: in response to detecting that the object distribution situation is a single-file distribution, the management layer transfers the first designated file to a storage pool that matches the second fault tolerance type; in response to detecting that the object distribution situation is a multi-file distribution, the management layer obtains the target fault tolerance type of the remaining distributed files of the first object distribution except the first designated file; if the target fault tolerance type of the remaining distributed files is the second fault tolerance type, the management layer performs a data transfer operation on the first designated file to transfer the first designated file to a storage pool that matches the second fault tolerance type; if there are remaining distributed files with a target fault tolerance type of the first fault tolerance type, the management layer does not perform a data transfer operation on the first designated file.

[0032] In a specific scenario of setting up a multi-copy fault-tolerant storage pool and an erasure code fault-tolerant storage pool, the above scheme specifically includes: in response to detecting that the target fault tolerance type of the first designated file is changed from erasure code fault tolerance to multi-copy fault tolerance, the management layer performs a data transfer operation on the first designated file to transfer the first designated file to the multi-copy storage pool; in response to detecting that the fault tolerance type of the first designated file is changed from multi-copy fault tolerance to erasure code fault tolerance, the management layer obtains the object distribution of the first object contained in the first designated file; the management layer determines whether to perform a data transfer operation on the first designated file based on the object distribution, and the object distribution includes single-file distribution and multi-file distribution. In response to detecting that the object distribution is a single-file distribution, the management layer moves the first designated file to the erasure code storage pool; in response to detecting that the object distribution is a multi-file distribution, the management layer obtains the target fault tolerance type of the remaining distributed files of the first object distribution except the first designated file; if the target fault tolerance type of the remaining distributed files is erasure code fault tolerance, the management layer performs a data moving operation on the first designated file to move the first designated file to the erasure code storage pool; if the target fault tolerance type of the remaining distributed files is multi-copy fault tolerance, the management layer does not perform a data moving operation on the first designated file.

[0033] The embodiment of the present application timely performs a fault tolerance type analysis on the first specified file that has been stored based on storage pools with different fault tolerance types preset in a distributed storage system, and reasonably stores the file in a storage pool with a matching fault tolerance type, thereby reducing the storage cost of the storage system and improving the performance of the storage system.

[0034] Further, if Figure 3 As shown, a data management method disclosed in an embodiment of the present application also includes the following steps: X1. In response to detecting a file write request sent by the application layer, the management layer feeds back second metadata of a second designated file included in the file write request to the application layer.

[0035] For ease of understanding, some concepts involved in this application are briefly explained first, such as Figure 4As shown, a file is a file that a user needs to store or access; an object is a unit managed internally by a distributed storage system, and the maximum disk space occupied by each object is set by the system; when a file exceeds the preset disk space occupied (usually set to the maximum disk space occupied), the storage system will divide the file into multiple objects for storage. Objects are mapped to different object storage services through data groups. Files are usually divided according to the preset disk space occupied, and the first mapping relationship between the file and the divided object is stored in the metadata management service module; each object is independently mapped to a unique data group, and there is also a second mapping relationship between the data group and the object storage service module. A data group will be mapped to multiple object storage service modules, and the second mapping relationship will also be stored in the metadata service module. Preferably, the above-mentioned preset occupied disk space can be the maximum occupied disk space, and of course it can also be set by a technician in this field according to actual conditions.

[0036] Specifically, the composition of the above-mentioned second metadata is the same as that of the above-mentioned first metadata, both of which include object information, fault tolerance type and recent ten access records; for the convenience of explanation, the object contained in the second metadata is defined as a second object. It can be determined that a second designated file includes at least one second object, then the second object information in the second metadata includes at least one; the second object information includes segmentation information (that is, the starting position and ending position of each second object in the file), a second object identifier and a corresponding target hash value combination of the second object data.

[0037] The above-mentioned generating the matching second metadata according to the second specified file included in the file writing request specifically includes: The management layer segments the second designated file according to the preset occupied disk space to generate segmentation information and multiple second objects and assign a unique second object identifier to the second object; consistent with the embodiment disclosed above, in this embodiment, the preset occupied disk space is preferably set to the maximum occupied disk space, and can also be set by those skilled in the art according to actual conditions, and this application does not limit this. The management layer determines the fault tolerance type according to the access type of the file write request, and the access type includes modifying the file type and creating the file type.

[0038] Among them, the above-mentioned determination of the fault tolerance type according to the access type of the file write request includes, in response to detecting that the access type is a file creation type, determining the fault tolerance type to be the first fault tolerance type; if the file write request is to create a new file and the file activity is high, the created second specified file can be allocated to a storage pool that matches the first fault tolerance type. For example, in a scenario where the first fault tolerance type is multi-copy fault tolerance and the second fault tolerance type is erasure code fault tolerance, the fault tolerance type of the second specified file is determined to be multi-copy fault tolerance and allocated to the multi-copy storage pool.

[0039] In response to detecting that the access type is a file modification type, the historical storage pool address of the file to be modified that matches the second specified file is obtained; if the historical storage pool address is a first preset value, the fault tolerance type is determined to be the first fault tolerance type; if the historical storage pool address is a second preset value, the fault tolerance type is determined to be the second fault tolerance type. The mapping relationship between the preset value and the fault tolerance type is pre-set and stored in the storage. In this embodiment, the first preset value is bound to the first fault tolerance type, and the second preset value is bound to the second fault tolerance type. In other embodiments, they can also be replaced. In embodiments with more than two fault tolerance types, preset values ​​can also be added. This application will not explain them one by one here.

[0040] That is, if the file write request is to modify an existing file (the file to be modified is the second specified file), then it is necessary to verify the historical storage pool address of the file to be modified; keep the second storage pool address corresponding to the second specified file consistent with the historical storage pool address of the file to be modified, that is, keep the fault tolerance type of the second specified file consistent with the fault tolerance type of the file to be modified. The present application determines the different fault tolerance types corresponding to the file according to the access types of different file write requests, ensures the consistency of the file fault tolerance type when responding to a request to modify the file, and reduces the possibility of file abnormality.

[0041] X2. The application layer performs a data duplication check on the second designated file based on the second metadata and a preset data fingerprint array, and generates a data write request after the data duplication check passes, and sends it to the management layer.

[0042] In order to avoid writing duplicate data in the storage system, this embodiment further proposes to perform data duplication verification using a second designated file, including: The application layer obtains and determines multiple second object data contained in the second designated file according to the segmentation information of the second designated file; the application layer calculates the target hash value combination of the second object data; the application layer searches in a preset data fingerprint array whether there is an existing hash value combination that matches the target hash value combination; if not, the application layer determines that the data repeatability check passes; if so, the application layer performs a secondary data repeatability check based on the second object information and the existing object information that matches the existing hash value combination.

[0043] The application layer performs a secondary data duplication check based on the second object information and the existing object information matched by the existing hash value combination, including: the application layer determines that the data duplication check has passed in response to detecting that the segmentation information in the second object information and the segmentation information in the existing object information matched by the existing hash value combination are inconsistent; the application layer determines that the data duplication check has not passed in response to detecting that the segmentation information in the second object information and the segmentation information in the existing object information matched by the existing hash value combination are consistent, and updates the second object identifier matched by the second object according to the target object identifier contained in the object information matched by the existing hash value combination. The segmentation information is obtained by the management layer from the second metadata matched by the second designated file obtained by querying the metadata service module and sent to the application layer.

[0044] That is, if there is no existing hash value combination matching the target hash value combination in the preset data fingerprint array, it is determined that the object data of the second designated file is not stored in the storage system. At this time, the application layer generates a data write request to the second designated object storage service node according to the second metadata and sends it to the management layer. If there is an existing hash value combination matching the target hash value combination in the preset data fingerprint array, it is further determined whether the object data is stored in the storage system according to the segmentation information contained in the second metadata matching the second designated file. If it is inconsistent, it is determined that the object data is not stored in the storage system. At this time, it is still necessary to determine the second designated object storage service node matching the second designated file, generate a data write request and send it to the second designated object storage service node. If it is consistent, the application layer sends the object information corresponding to the existing hash value matching the target hash value combination to the metadata service module of the management layer through the protocol, updates the second object identifier in the second metadata of the second designated file to the target object identifier, and determines that the data of the object has been written to the storage system. At this time, there is no need to perform repeated storage, and no write storage request is initiated to the object storage service.

[0045] On the basis of the pre-set data fingerprint array, it is verified whether the object data in the file to be written is repeated according to the pre-set data fingerprint array. For the repeated object data, it is only necessary to update the object identifier in the object information in the file metadata, so that the address where the repeated object data has been stored can be obtained according to the object data, thereby reducing the storage amount of repeated data in the distributed storage system and reducing the interactive operations of file storage, greatly improving the performance of the distributed storage system and reducing storage pressure.

[0046] The above application layer calculates the target hash value combination of the second object data; specifically, the hash values ​​of the same second object data are calculated using the preset first hash algorithm and the preset second hash algorithm respectively to obtain the first hash value and the second hash value with different values, and then the first hash value and the second hash value are combined to generate the target hash value combination and then saved in the metadata information. It can be determined that one object data corresponds to one hash value combination, and the present application does not limit the specific types of the first hash algorithm and the second hash algorithm, which can be MD5 (Message Digest Algorithm 5, information digest algorithm), MurmurHash (non-encrypted hash function), etc.

[0047] The application layer searches the preset data fingerprint array for an existing hash value combination that matches the target hash value combination. The above data fingerprint array is a two-dimensional data fingerprint array set by the distributed storage system for each fault-tolerant storage pool; Figure 5 As shown, a data unit corresponds to an object, and its row index and column index are respectively the first hash value and the second hash value generated according to the object data. The first hash value and the second hash value corresponding to a data unit in the data fingerprint array are called the existing hash value combination; the data unit at least stores the object information. Of course, in order to facilitate other subsequent operations, other information can also be stored in the data unit of the data fingerprint array. It can be understood that since the first hash algorithm and the second hash algorithm are different, the possibility of different object data generating the same first hash value and second hash value is very low. In this application, it is recognized that only the same object data can generate the same first hash value and second hash value.

[0048] In addition, the application layer determines the target object storage service node based on the second metadata of the received second specified file, and then generates a data write request and sends it to the target object storage service node in the management layer. That is, the present application further determines the corresponding object storage service by searching for the metadata of the file to be written, and then the application layer directly initiates a data write request to the object storage service, which reduces the data access time and realizes the rapid writing of the stored data. Specifically, the determination of the target storage service node includes: Determine at least one second data address that matches the second designated file based on the second object identifier and the fault tolerance type included in the second metadata. Determine the second designated object storage service node that matches the second data address based on the data address and the preset data distribution algorithm; generate a data write request and send it to the second designated object storage service node. That is, after obtaining the second data address, the application layer obtains the second designated object storage service node that matches the second data address based on the preset data distribution algorithm, then sends a data write request to the second object storage service node, and divides the data of the second designated file into data blocks, and then transmits them one by one to the corresponding second designated object storage service nodes.

[0049] Specifically, the above-mentioned determination of at least one second data address matching the second designated file according to the object identifier and fault tolerance type included in the second metadata includes: calling a preset hash function, calculating a transition hash value corresponding to the second object identifier, and converting the object identifier into a fixed-length value. The hash function includes but is not limited to a non-encrypted hash function, an encrypted hash function, and a distributed storage-specific hash algorithm. Specific hash functions that can be used include MurmurHash (non-encrypted hash function) and Jenkins hash (Jenkins hash function), etc.; this application does not limit the selection of a specific hash function. The transition hash value and the bit mask are bitwise ANDed to generate a second data group address corresponding to the second object identifier, that is, the second data group address PGid = Hash (Oid) & mask, that is, a bitwise AND operation is used on the basis of the hash value in combination with a bit mask (mask) to limit the range of the hash value. The application layer can also use heterogeneous acceleration and pipeline processing to reduce the time of hash calculation and improve system response performance.

[0050] Then, the storage pool address is determined according to the fault tolerance type. In the scenario of setting a storage pool of a multi-copy fault tolerance type and an erasure code fault tolerance type disclosed in the aforementioned content, in response to detecting that the fault tolerance type is multi-copy fault tolerance, the first storage pool address is determined to be a first preset value; in response to detecting that the fault tolerance type is erasure code fault tolerance, the first storage pool address is determined to be a second preset value. Among them, the preferred first preset value can be set to 1, and the second preset value can be set to 2. Of course, it can also be set to other values ​​by a technician in this field. It should be noted that the first preset value is bound to the multi-copy fault tolerance type, and the second preset value is bound to the erasure code fault tolerance type. Of course, in the case of integrating a storage pool of more than two fault tolerance types, a third preset value can be further set to bind to the third fault tolerance type. This type is not expanded in this application.

[0051] Finally, the second data address is generated according to the second data group address and the second storage pool address, that is, a complete second data address can be expressed as "first storage pool address: PGid". This application limits the storage address corresponding to the object after the file division to a specific range through the above steps, thereby improving the stability of the system storage; on this basis, the final data address of a specific file object is directly generated in one step according to the object identifier and the fault tolerance type, so that the corresponding data addresses can be generated for multiple fault tolerance types, which makes it easier for the subsequent application layer to issue a data read request according to the data address, thereby realizing fast data reading.

[0052] The above data distribution algorithm is used to establish a mapping relationship between data addresses and object storage service nodes to ensure data reliability and efficient access. In this embodiment, the data distribution algorithm includes but is not limited to the consistent hashing algorithm, the CRUSH algorithm, and the random allocation method. The above consistent hashing algorithm forms a virtual ring with all possible hash values, and maps each object storage service node to a certain position on the ring; when a new data group needs to be allocated, a hash value is first calculated based on the data group address, and then the corresponding point is found on the hash ring; the first object storage service node found clockwise from this point is the object storage service node corresponding to the data group. The above CRUSH (Controlled Replication Under Scalable Hashing) algorithm is a data distribution algorithm designed specifically for distributed storage systems such as Ceph; it uses the physical and logical hierarchical structure of the storage cluster to allow administrators to define complex data distribution strategies to optimize performance and reliability; the algorithm specifically defines how to select an object storage service to store data in a specific data group. The CRUSH Map contains information about all storage devices in the storage cluster and a set of rules that describe the relationship between these storage devices. It can be determined that the mapping relationship between the above data addresses and object storage services is usually stored in the data group for easy calling; of course, in some special scenarios, it can also be stored separately in other modules in the storage system; this application does not limit this.

[0053] X3. In response to the received data write request, the management layer writes the second designated file to the storage device on the second designated object storage service node that matches the data write request.

[0054] Specifically, after the second designated object storage service node in the management layer receives the data of the second designated file, the object service storage node in the management layer will write the data block to the underlying storage device and return a successful write response to the application layer. After the application layer receives the response, the file writing process is completed. This application implements the repeatability check of the data to be written based on the object data fingerprint array set for each storage pool, prevents duplicate data storage, and further reduces storage costs.

[0055] It is understandable that while the distributed storage system executes the data management method disclosed in the above embodiment, in order to improve the performance and stability of the storage system and avoid data reading and writing anomalies, the present application also proposes to monitor the status of each object storage service node and manage and repair abnormal nodes in a timely manner. Specifically, each object storage service node corresponding to the same data group monitors the remaining object storage service nodes corresponding to the data group in real time; each object storage service node reports the collected health information of the abnormal node to be determined to the monitor service; the monitor service summarizes the health information of multiple abnormal nodes to be determined; for an abnormal node to be determined, when the number of object storage service nodes reporting the abnormal node to be determined exceeds the preset number, the abnormal node to be determined is determined to be an abnormal node. Among them, the preset number is set according to the total number of object storage service nodes contained in a data group, such as setting 80% of the total number as the preset number, which can be set by a person skilled in the art. Then, the monitor service analyzes the failure risk probability of the abnormal node according to the health information of the abnormal node, and further determines the corresponding repair operation according to the failure risk probability. Specifically, if the risk probability of the abnormal node is high but it has not failed, any idle object storage service node is selected to replace the abnormal object storage service node to store the subsequently written data; if the abnormal node has failed, an idle object storage service node with a different fault tolerance type from the current fault tolerance type of the abnormal node is selected to perform data recovery.

[0056] In addition, in the scenario where the application layer initiates a file read request to the management layer, the embodiment of the present application also discloses that: in response to detecting the file read request sent by the application layer, the management layer searches for the first designated file according to the file identifier in the file read request, and feeds back the first metadata corresponding to the first designated file to the application layer. In response to receiving the data read request generated by the application layer to the first designated data storage service node according to the first metadata, the target data in the storage device matching the first designated data storage server is read and returned to the application layer.

[0057] The above-mentioned file read request at least includes a file identifier for searching the first designated file, such as a file name and a file path. Specifically, after receiving the file read request initiated by the application layer, the metadata service module of the management layer searches for the first designated file according to the file identifier, and returns the first metadata corresponding to the first designated file to the application layer. In addition, the metadata service module will also update the access record in the first metadata of the first designated file to save the access request (i.e., file read request) issued by the application layer this time. After the management layer feeds back the first metadata to the application layer, the application layer will calculate one or more first designated object storage service nodes that match the first designated file according to the first metadata, that is, the object storage service node that stores the first designated file can be calculated through the first metadata. At this time, the application layer generates a data read request and directly initiates it to the determined first designated object storage service node; after receiving the data read request, the first designated object storage service node in the management layer immediately reads the corresponding data block from the corresponding underlying storage device, and returns the target data stored in the data block to the application layer; thus, the data reading of the first designated file is completed.

[0058] The management layer interacts with the application layer when responding to a file read request. After feeding back the first metadata to the application layer, the management layer generates a data read request for the first designated object storage service node based on the first metadata, and specifically determines at least one first data address that matches the first designated file based on the first object identifier and fault tolerance type included in the first metadata. According to the data address and a preset data distribution algorithm, the first designated object storage service node that matches the first data address is determined; a data read request is generated and sent to the first designated object storage service node. This is consistent with the above-mentioned method for determining the second designated object storage service node, and the present application will not repeat it here. The present application determines the corresponding object storage service by searching for the metadata of the file to be read, and then the application layer directly initiates a data read request to the object storage service, thereby reducing data access time and achieving fast reading of stored data.

[0059] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method.

[0060] The embodiment of the present application also provides a data management device, such as Figure 6 As shown, it includes a file analysis module 610 and a file management module 620; The file analysis module 610 is used to trigger the management layer to obtain the first metadata of the first designated file in response to a preset condition trigger to determine the current data attribute of the first designated file; The file analysis module 610 is used to trigger the management layer to determine the target fault tolerance type of the first designated file according to the access records in the current data attributes and the preset rules in response to detecting one or more attribute changes in the current data attributes of the first designated file; The file analysis module 610 is used to trigger the management layer to determine the target fault tolerance type of the first designated file according to a preset rule in response to detecting that the current data attribute and the historical data attribute are inconsistent; The file management module 620 is used to trigger the management layer to perform a data transfer operation on the first designated file in response to detecting a change in the target fault tolerance type, so as to transfer the first designated file to a matching target storage pool.

[0061] The above-mentioned data management device also includes a file interaction module 630 and a duplication checking module 640; The file interaction module 630 is used to trigger the management layer to respond to detecting a file write request sent by the application layer, and to feed back the second metadata of the second specified file included in the file write request to the application layer; The above-mentioned duplication check module 640 is used to trigger the application layer to perform data duplication check on the second designated file based on the second metadata and the preset data fingerprint array, and generate a data write request after the data duplication check passes and send it to the management layer; The file interaction module 630 is also used for the management layer to respond to the received data write request and write the second specified file to the storage device on the second specified object storage service node that matches the data write request.

[0062] For the description of the features in the embodiments corresponding to the above data management device, reference can be made to the relevant description of the embodiments corresponding to the data management method of the distributed storage system, which will not be described in detail here.

[0063] An embodiment of the present application further provides an electronic device, comprising: one or more processors; and a memory associated with the one or more processors, the memory being used to store program instructions, and the program instructions, when read and executed by the one or more processors, perform the following operations: The trigger management layer acquires first metadata of the first designated file in response to a preset condition trigger to determine current data attributes of the first designated file; The triggering management layer determines a target fault tolerance type of the first designated file according to access records in the current data attributes and preset rules in response to detecting one or more attribute changes in the current data attributes of the first designated file; In response to detecting a change in the target fault tolerance type, the trigger management layer performs a data migration operation on the first designated file to migrate the first designated file to a matching target storage pool.

[0064] In some implementation scenarios, the memory is used to store program instructions, and when the program instructions are read and executed by one or more processors, the following operations are also performed: The trigger management layer feeds back second metadata of the second designated file included in the file write request to the application layer in response to detecting the file write request sent by the application layer; The application layer is triggered to perform a data duplication check on the second designated file based on the second metadata and a preset data fingerprint array, and after the data duplication check passes, a data write request is generated and sent to the management layer; The trigger management layer writes the second designated file to a storage device on a second designated object storage service node that matches the data write request in response to the received data write request.

[0065] in, Figure 7 The architecture of the electronic device is shown as an example, which may include a processor 710, a video display adapter 711, a disk drive 712, an input / output interface 713, a network interface 714, and a memory 720. The processor 710, the video display adapter 711, the disk drive 712, the input / output interface 713, the network interface 714, and the memory 720 may be communicatively connected via a bus 730.

[0066] Among them, the processor 710 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, etc., to execute relevant programs to implement the technical solution provided in this application.

[0067] The memory 720 can be implemented in the form of ROM (Read Only Memory, writable memory), RAM (Random Access Memory, random access memory), static storage device, dynamic storage device, etc. The memory 720 can store an operating system 721 for controlling the execution of the electronic device 700, and a basic input and output system (BIOS) 722 for controlling the low-level operations of the electronic device 700. In addition, a web browser 723, a data storage management system 724, and an icon font processing system 725, etc. can also be stored. The above-mentioned icon font processing system 725 can be an application program that specifically implements the operations of the aforementioned steps in the embodiment of the present application. In short, when the technical solution provided in the present application is implemented by software or firmware, the relevant program code is stored in the memory 720 and is called and executed by the processor 710.

[0068] The input / output interface 713 is used to connect the input / output module to realize information input and output. The input / output module can be configured as a component in the device (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0069] The network interface 714 is used to connect to a communication module (not shown) to achieve communication interaction between the device and other devices. The communication module can achieve communication through a wired method (such as USB, network cable, etc.) or a wireless method (such as mobile network, WIFI, Bluetooth, etc.).

[0070] The bus 730 comprises a pathway for transmitting information between the various components of the device (eg, the processor 710 , the video display adapter 711 , the disk drive 712 , the input / output interface 713 , the network interface 714 , and the memory 720 ).

[0071] In addition, the electronic device 700 can also obtain information on specific collection conditions from the virtual resource object collection condition information database for use in condition judgment, etc.

[0072] It should be noted that, although the above device only shows a processor 710, a video display adapter 711, a disk drive 712, an input / output interface 713, a network interface 714, a memory 720, a bus 730, etc., in the specific implementation process, the device may also include other components necessary for normal execution. In addition, it can be understood by those skilled in the art that the above device may also only include components necessary for implementing the solution of the present application, and does not necessarily include all the components shown in the figure.

[0073] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above-mentioned distributed storage system data management method embodiments when running.

[0074] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0075] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in any of the above-mentioned data management method embodiments of the distributed storage system are implemented.

[0076] An embodiment of the present application also provides another computer program product, including a non-volatile computer-readable storage medium, the non-volatile computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, implementing the steps in any of the above-mentioned distributed storage system data management method embodiments.

[0077] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0078] The above is a detailed introduction to a data management method for a distributed storage system provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and its core idea of ​​the present application. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of the present application.

Claims

1. A data management method for a distributed storage system, wherein the distributed storage system comprises a management layer and an application layer, characterized in that: The distributed storage system is provided with a plurality of storage pools of different fault-tolerant types, and the method comprises: The management layer acquires first metadata of a first designated file in response to a preset condition trigger to determine a current data attribute of the first designated file; In response to detecting one or more attribute changes in current data attributes of the first designated file, the management layer determines a target fault tolerance type of the first designated file according to access records in the current data attributes and preset rules; In response to detecting a change in the target fault tolerance type, the management layer performs a data transfer operation on the first designated file to transfer the first designated file to a matching target storage pool.

2. The data management method according to claim 1, characterized in that: In response to detecting that one or more attribute changes exist in the current data attributes of the first designated file, the management layer determines the target fault tolerance type of the first designated file according to the access records in the current data attributes and the preset rules, including: The management layer calculates the file heat value according to the access record corresponding to the first designated file and a preset formula; The management layer determines a target fault tolerance type of the first designated file according to the file heat value.

3. The data management method according to claim 2, characterized in that: The preset formula is: ； in, Hot is the file heat value, i is the serial number of the access record, n represents the maximum number of access records matched by the first specified file, T cur Indicates the current time. T i For the most recent i The access time, α i Indicates i The weight coefficient of the access record.

4. The data management method according to claim 2, characterized in that: The fault tolerance type includes a first fault tolerance type and a second fault tolerance type, and the management layer determines the target fault tolerance type of the first designated file according to the file heat value, including: In response to detecting that the file heat value is less than or equal to a preset threshold, the management layer determines that the target fault tolerance type of the first designated file is a first fault tolerance type; In response to detecting that the file heat value is greater than the preset threshold, the management layer determines that the target fault tolerance type of the first designated file is the second fault tolerance type.

5. The data management method according to claim 2, characterized in that: In response to detecting a change in the target fault tolerance type, the management layer performs a data transfer operation on the first designated file to transfer the first designated file to a matching target storage pool, including: In response to detecting that the target fault tolerance type of the first designated file is changed from the second fault tolerance type to the first fault tolerance type, the management layer performs a data transfer operation on the first designated file to transfer the first designated file to a storage pool matching the first fault tolerance type; In response to detecting that the fault tolerance type of the first designated file is changed from a first fault tolerance type to a second fault tolerance type, the management layer obtains the object distribution of the first object included in the first designated file; The management layer determines whether to perform a data transfer operation on the first designated file according to the object distribution situation, where the object distribution situation includes single-file distribution and multi-file distribution.

6. The data management method according to claim 5, characterized in that: The management layer determines whether to perform a data transfer operation on the first designated file according to the object distribution, including: In response to detecting that the object distribution is a single-file distribution, the management layer moves the first designated file to a storage pool matching a second fault tolerance type; In response to detecting that the object distribution is a multi-file distribution, the management layer obtains a target fault tolerance type of remaining distribution files of the first object distribution except the first designated file; If the target fault tolerance type of the remaining distributed files is the second fault tolerance type, the management layer performs a data transfer operation on the first designated file to transfer the first designated file to a storage pool matching the second fault tolerance type; If the target fault tolerance type of the remaining distributed files is the first fault tolerance type, the management layer does not perform a data transfer operation on the first designated file.

7. The data management method according to claim 1, characterized in that: The management layer acquires first metadata of a first designated file in response to a preset condition trigger to determine a current data attribute of the first designated file, including: In response to detecting the file read request sent by the application layer, the management layer obtains first metadata of the first specified file to determine current data attributes of the first specified file; And / or, in response to the time difference between the collected current time node and the previously collected historical time node being greater than or equal to a preset time difference, the management layer obtains first metadata of the first designated file to determine the current data attribute of the first designated file.

8. The data management method according to claim 1, characterized in that: The method further comprises: In response to detecting the file write request sent by the application layer, the management layer feeds back second metadata of the second designated file included in the file write request to the application layer; The application layer performs a data duplication check on the second designated file based on the second metadata and the preset data fingerprint array, generates a data write request after the data duplication check passes, and sends the request to the management layer; In response to the received data write request, the management layer writes the second designated file to a storage device on a second designated object storage service node that matches the data write request.

9. The data management method according to claim 8, characterized in that: The second metadata includes second object information and a fault tolerance type, the second object information includes segmentation information and a second object identifier, and the generation of the second metadata includes: The management layer divides the second designated file according to the preset occupied disk space to generate division information and a plurality of second objects, and assigns a unique second object identifier to the second object; The management layer determines the fault tolerance type according to the access type of the file write request, where the access type includes a file modification type and a file creation type.

10. The data management method according to claim 9, characterized in that: The application layer performs data duplication verification on the second designated file based on the second metadata and a preset data fingerprint array, including: The application layer obtains and determines a plurality of second object data included in the second designated file according to the segmentation information of the second designated file; The application layer calculates a target hash value combination of the second object data; The application layer searches a preset data fingerprint array for an existing hash value combination that matches the target hash value combination; If not, the application layer determines that the data repeatability check passes; If so, the application layer performs a secondary data duplication check based on the second object information and the existing object information that matches the existing hash value combination.

11. The data management method according to claim 10, characterized in that: The application layer performs a secondary data duplication check based on the second object information and the existing object information matched by the existing hash value combination, including: The application layer determines that the data duplication check passes in response to detecting that the segmentation information in the second object information and the segmentation information in the existing object information that matches the existing hash value combination are inconsistent; In response to detecting that the segmentation information in the second object information is consistent with the segmentation information in the existing object information that matches the existing hash value combination, the application layer determines that the data duplication check has failed, and updates the second object identifier that matches the second object according to the target object identifier contained in the object information that matches the existing hash value combination.

12. The data management method according to claim 9, characterized in that: The management layer determines the fault tolerance type according to the access type of the file write request, including: In response to detecting that the access type is a create file type, the management layer determines that the fault tolerance type is a first fault tolerance type; In response to detecting that the access type is a file modification type, the management layer obtains a historical storage pool address of a file to be modified that matches the second designated file; If the historical storage pool address is a first preset value, the application layer determines that the fault tolerance type is a first fault tolerance type; If the historical storage pool address is a second preset value, the application layer determines that the fault tolerance type is a second fault tolerance type.

13. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the data management method according to any one of claims 1 to 12 when executing the computer program.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the data management method according to any one of claims 1 to 12.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the data management method according to any one of claims 1 to 12 are implemented.

Images