Data processing method and device, equipment, program product and storage medium

Abstract

The invention discloses a data processing method and device, equipment, a program product and a storage medium. The method comprises the following steps: storing data into a first storage pool set according to a first mode; the first mode is to split the data into N data fragments, and the N data fragments correspond to the first storage pool set; wherein N is a positive integer; acquiring state information of the N data fragments; and processing the N data fragments according to the state information of the N data fragments.

Description

Technical Field

[0001] The present application relates to the field of big data technology, and in particular to a data processing method and apparatus, equipment, program product, and storage medium. Background Art

[0002] In the existing distributed storage system (ceph), the scrub strategy is used to scan the cluster data to detect the consistency between replicas. Scrub is a thread started by the Object Storage Device (OSD) itself, which mainly scans metadata and data and regularly compares the consistency of the metadata of three replicas. Deep-scrub checks whether the data content of the three replica objects is consistent. However, due to changes in the fourth-generation object storage architecture and different writing methods, scrub cannot fully meet the corresponding needs. Summary of the invention

[0003] In order to solve the above technical problems, the present application provides a data processing method and device, equipment, program product, and storage medium.

[0004] The data processing method provided by the present application comprises:

[0005] The data is stored in the first storage pool set in a first manner; the first manner is to split the data into N data slices, and the N data slices correspond to the first storage pool set; wherein N is a positive integer;

[0006] Obtain status information of N data shards; process the N data shards according to the status information of the N data shards.

[0007] The data processing device provided by the present application comprises:

[0008] A storage unit, used to store data in a first storage pool set according to a first method; the first method is to split the data into N data slices, and the N data slices correspond to the first storage pool set; wherein N is a positive integer;

[0009] An acquisition unit, used to acquire status information of N data shards;

[0010] The processing unit is used to process the N data shards according to the status information of the N data shards.

[0011] The data processing device provided in the present application includes: a processor and a memory, the memory is used to store a computer program, and the processor is used to call and run the computer program stored in the memory to execute the above method.

[0012] A computer program product provided by the present application includes: a computer program, and the computer program implements the above method when executed by a processor.

[0013] The computer-readable storage medium provided in the present application is used to store a computer program, and the computer program enables a computer to execute the above method.

[0014] In the technical solution of the present application, data is stored in the first storage pool set in a first manner; the first manner is to split the data into N data shards, and the N data shards correspond to the first storage pool set; wherein N is a positive integer; obtain the status information of the N data shards; and process the N data shards according to the status information of the N data shards. In this way, relevant data can be restored or migrated according to the status information of the data shards, thereby ensuring the security of the data. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a flowchart of a data processing method provided in an embodiment of the present application;

[0016] Figure 2 It is a schematic diagram of a method for storing objects provided in an embodiment of the present application;

[0017] Figure 3 It is a schematic diagram of the architecture of the manager execution scheduling program provided in the embodiment of the present application;

[0018] Figure 4 It is a schematic diagram of the structure of a data processing device provided in an embodiment of the present application;

[0019] Figure 5 is a schematic structural diagram of a data processing device provided in an embodiment of the present application;

[0020] Figure 6 It is a schematic structural diagram of the chip of an embodiment of the present application. DETAILED DESCRIPTION

[0021] The following will describe the technical solutions in the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0022] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0023] It should also be pointed out that the terms "first\second\third" involved in the embodiments of the present application are only used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged in a specific order or sequence where permitted, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.

[0024] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there may be three kinds of relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the objects associated before and after are in an "or" relationship. It should also be understood that the "indication" mentioned in the embodiments of the present application can be a direct indication, an indirect indication, or an indication of an association relationship. For example, A indicates B, which can mean that A directly indicates B, for example, B can be obtained through A; it can also mean that A indirectly indicates B, for example, A indicates C, and B can be obtained through C; it can also mean that there is an association relationship between A and B. It should also be understood that the "correspondence" mentioned in the embodiments of the present application can mean that there is a direct or indirect correspondence relationship between the two, or it can mean that there is an association relationship between the two, or it can mean the relationship between indication and being indicated, configuration and being configured, etc.

[0025] To facilitate understanding of the technical solutions of the embodiments of the present application, the relevant technologies of the embodiments of the present application are described below. The following related technologies can be arbitrarily combined with the technical solutions of the embodiments of the present application as optional solutions, and they all belong to the protection scope of the embodiments of the present application.

[0026] Scrub is developed based on the ceph architecture. It starts threads on its own osd, which puts a certain amount of pressure on the osd itself. In the case of business writing, its own pressure is more obvious. However, the fourth-generation object storage architecture (object storage gateway, metadata cluster (indexset), data cluster (dataset) and other components) is different from the ceph architecture. Scrub cannot operate the indexset component of the fourth-generation object storage architecture and cannot repair data. Erasure Coding (EC) storage is the storage method of the fourth-generation object storage architecture. Unlike Ceph's three copies and erasure coding storage methods, the fourth-generation object storage can write across the availability zone (AZ) level. The object writing method is to store complete objects in each group of storage pools (pools) (called poolsets), and each pool only stores one shard of the complete object. The number of pools in poolset is k+m. When the number of pools successfully read meets k, or the number of shards lost does not exceed m, EC can be successfully encoded and decoded, and one of the pools mounts a disk to store real data. Although data can be found to be written successfully during the upload process, it cannot be guaranteed that all shards exist. In order to ensure that data is not lost, it is necessary to restore the shards that have not been written successfully, or in some cases, it is necessary to migrate the data to another poolset. Therefore, how to restore the lost data and migrate related data is a problem that needs to be considered. To this end, the following technical solution of the embodiment of the present application is proposed.

[0027] To facilitate understanding of the technical solutions of the embodiments of the present application, the technical solutions of the present application are described in detail below through specific embodiments. The above related technologies can be combined arbitrarily with the technical solutions of the embodiments of the present application as optional solutions, and they all belong to the protection scope of the embodiments of the present application. The embodiments of the present application include at least part of the following contents.

[0028] Figure 1 is a flow chart of a data processing method provided in an embodiment of the present application, such as Figure 1 As shown, the data processing method comprises the following steps:

[0029] Step 101: Store data in a first storage pool set in a first manner; the first manner is to split the data into N data slices, and the N data slices correspond to the first storage pool set; wherein N is a positive integer.

[0030] In some implementations, the EC storage method is adopted, the data to be stored is split into N data slices, each data slice is written into a storage pool, and the storage pool corresponding to the N data slices is the first storage pool set. Wherein, N = M + K, M and K are both positive integers, and only any K data slices need to be obtained to read the complete data.

[0031] Step 102: Obtain status information of N data shards; and process the N data shards according to the status information of the N data shards.

[0032] In some implementations, during the upload process, it is found that the data is written successfully, but it cannot be guaranteed that all data shards exist. In order to ensure that the data is not lost, the data shards that have not been written successfully need to be restored. Therefore, the data is divided into the first type of data and the second type of data, and different operations are performed for different types of data to obtain the status information of N data shards. Among them, the first type of data can be active data, that is, data that users often operate and / or data that has just been uploaded; the second type of data is inactive data, that is, data that users do not often operate.

[0033] In some embodiments, when the data is of the first category, obtaining status information of N data shards includes: obtaining a first operation process of the user, and determining status information of each of the N data shards based on the first operation process; the first operation process is used to obtain N data shards.

[0034] In some embodiments, when the data is of the first type, the status information of the data slice corresponding to the data can be found according to the user's first operation process. Exemplarily, the first operation process can be a data upload process and / or a data download process. Specifically, when uploading data and / or downloading data, the data slice corresponding to the data will be sent. In this process, the status information of each data slice is read, and it is determined whether the data slice has been deleted according to the status information. At this time, it is also necessary to determine whether the metadata corresponding to the data slice exists. If the metadata does not exist, it means that the data has been deleted, and therefore, it does not need to be restored.

[0035] In some embodiments, during the first operation, when each data shard is sent, the status information of the data shard is read. Exemplarily, the status information of the data shard can be obtained based on the data sent by the data shard. Exemplarily, when the return code data sent by the data shard is -2, it means that the data shard does not exist, but the disk can be read normally, and the data is lost due to some reasons of the disk; when the return code data sent by the data shard is -110, it means that the disk is busy or there is a network problem, resulting in a timeout in reading the data shard, and it is impossible to determine whether the current shard exists at this time; the return code data sent by the data shard is -112, indicating that there is a problem with the cluster, the osd status is down, and it cannot be read, and it is impossible to determine whether the shard exists; some data cannot be read if at least K shards cannot be read. When the return code data sent by the data shard is less than 0, it is considered that the data shard has an abnormality and needs to be restored.

[0036] In some implementations, based on the return code data sent by the data segment, it is necessary to determine whether the data has actually been deleted, that is, to determine whether the metadata of the data exists. If the metadata of the data does not exist, the return code data of the request to obtain the metadata is -2.

[0037] In some embodiments, when the data is the second type of data; obtaining status information of N data slices includes: scanning the N data slices according to the first time period to generate scanning results; the scanning results are used to determine the status information of each data slice in the N data slices.

[0038] In some embodiments, when the data is the second type of data, the data slices are scanned according to the set first time period, and the status information of each data slice in the N data slices is determined according to the scanning result. Among them, the corresponding first time period can be set according to the size of the storage pool set, and this application does not make specific limitations. In this way, all data slices are scanned through a periodic scheduling cycle to prevent the loss of data slices caused by disk silence or data changes.

[0039] For example, the manager schedules the poolset scour program to scan the data in the poolset at the poolset granularity. A finer granularity method may also be used to perform the scan, and the data on different pgs of each pool may be scanned concurrently according to the range of pgid.

[0040] For example, in a 300pb resource pool, Scour can be scheduled to scan the entire resource pool in 3-4 days, so the cycle can be set to 4 days, with one round scheduled every 4 days. For deep-scour, its scanning data occupies disk utilization. In order not to affect the business, its execution rate needs to be reduced. It can be scheduled to execute one round in a week or half a month.

[0041] In some implementations, before scanning the N data shards according to the first time period, the method further includes: freezing the first storage pool set.

[0042] In some implementations, users cannot read data shards in the frozen first storage pool set and / or write the remaining data into the frozen first storage pool set. Because the second type of data will not be accessed by the scene, the scanning process and the business reading process will increase the pressure on the disk. In order to solve the problem of excessive disk pressure and not affect the actual business, the first storage pool set to be scanned will be frozen before the first storage pool set is required to be scanned. This can speed up the scanning speed and not affect the problem of write failure in the business, reduce IO resource usage, and unfreeze the first storage pool set after the repair.

[0043] In some embodiments, processing N data slices according to status information includes: determining whether to store M first information corresponding to M data slices among the N data slices into a first table according to the status information; the M first information includes information of the M data slices and object information corresponding to the M data slices respectively; wherein M is a positive integer and M is less than N; and repairing the M data slices according to the first table.

[0044] In some embodiments, after obtaining the status information of each data shard among the N data shards, if the status information of the data shard shows that the data shard is lost or the data in the data shard is incomplete, these data shards are collectively referred to as lost data shards, and the first information of these lost data shards is stored in a first table, wherein the first table is a table to be repaired, and the first information includes the information of the lost data shards and the information of the objects corresponding to the data shards. The objects corresponding to the data shards are the data before the split.

[0045] In some implementations, the first information includes the data shard name, the number of lost data shards, the number of data shards from which the object is split, the lost data shard set, the data name, the bucket name, and related information of the first storage pool set, etc. Exemplarily, the lost shard set is [0,3,5], and the 0th, 3rd, and 5th shards are lost.

[0046] In some embodiments, according to the information recorded in the first table, the data shards are sorted according to the number of lost data shards. First, the information of the data record with the largest number of lost data shards is obtained. After obtaining the attribute information that needs to be repaired, by utilizing the characteristics of the erasure code, only the K data shards of the data to be repaired need to be read to obtain the complete data. Then, based on the complete data, the content in the lost data shards can be obtained, and the relevant content can be written into the lost data shards, and also written into the first storage pool set, until all the lost data shards recorded in the first table are repaired.

[0047] In some implementations, for the second type of data, the data may be repaired at the granularity of the storage pool set according to the different degrees of data loss in different disks and different storage pool sets, thereby ensuring that disks prone to data loss are repaired first.

[0048] In some embodiments, after the first information of the data shards of the lost data is written into the first table, each time the information is written into the first table, it is detected whether the number of lost data shards is close to the configured number of recoverable data shards. Once detected, the first storage pool set is frozen, and then the lost data shards are recovered after reading the complete data based on the non-lost data shards. It can be understood that the number of recoverable shards can correspond to the K described above, and the number of loseable shards can correspond to the M mentioned above, and generally K>M. Therefore, when it is detected that M is about to approach K, it means that there are many lost data shards and these data shards need to be recovered. Therefore, the first storage pool set is frozen to avoid continuing to read and / or write data from the storage pool set.

[0049] In some implementations, for the restored data slices, the first information corresponding to the data slices is deleted from the first table.

[0050] In some embodiments, when M first information corresponding to M data shards are stored in a first table, the method also includes: obtaining second information of the M data shards stored in the first table; the second information includes storage configuration information of the M data shards; and adjusting the storage configuration of the M data shards according to the second information.

[0051] In some implementations, for data shards that have lost data, it is also necessary to obtain second information about these data shards, where the second information includes storage configuration information of these data shards. The storage configuration information is analyzed and checked to adjust the corresponding storage configuration.

[0052] Specifically, by analyzing the second information, we can determine which storage pools are prone to data loss, which disks are prone to data loss, and which machines are prone to data loss. We can compare and analyze the network, resource pool configuration, different disks, and different systems, and then adjust the corresponding storage configuration.

[0053] Specifically, obtain the location information, size information, shard name information, storage pool information, etc. corresponding to the data shards, and write this information into the archive table for analyzing disks prone to data loss, analyzing storage pools and other characteristics, and finding out the cause of data loss. At the same time, it can also better set the scanning cycle to reduce data loss.

[0054] In some implementations, processing the N data shards according to the state information includes: migrating the N data shards from a first set of storage pools to a second set of storage pools.

[0055] In some implementations, for data shards that are prone to data loss, after analysis, if a large number of disks in this group of disks are inaccessible due to external factors and the problem cannot be solved by replacing the disks, N data shards can be migrated from the first storage pool set to the second storage pool set to ensure data security.

[0056] In some embodiments, the readable data in the current first storage pool set is compared with the corresponding metadata by scanning the data shards stored in each storage pool to obtain S3 object information, that is, all the information of the data corresponding to the data shards, and writing this information into the table to be migrated. At this time, the unavailable first storage pool set is changed to a frozen state, so that the user's data will no longer be written to this first storage pool set. For example, at this time, the weight of the first storage pool set can be temporarily adjusted to 0, and no writing is performed; after the physical cause is processed, the new storage pool can be added to the first storage pool set, or the first storage pool set can be removed, and the new storage pool set can be re-added after the problem is resolved. When all the data to be migrated are successfully obtained, the first program can be executed to migrate the data shards, and the unavailable data shards can be migrated to the available second storage pool set. For example, the first program can be poolset migrate. The specific second storage pool set to be migrated can be determined according to the size of the data to be migrated.

[0057] In some implementations, the migration progress can be determined through the records in the migration table. If all the migration data in the records are deleted, all objects are successfully migrated.

[0058] The technical solution of the embodiment of the present application stores data in a first storage pool set in a first manner; the first manner is to split the data into N data shards, and the N data shards correspond to the first storage pool set; wherein N is a positive integer; obtain status information of the N data shards; process the N data shards according to the status information of the N data shards, so that related data can be restored or migrated according to the status of the data shards, thereby ensuring data security.

[0059] The technical solution of the embodiment of the present application is further described below in conjunction with specific implementation methods.

[0060] In the existing ceph storage technology, the scrub strategy is used to scan the data of the cluster to detect the consistency between the replicas. Scrub is a thread started by the osd module itself, which mainly scans the metadata and data, and regularly compares the consistency of the metadata of the three replicas. Deep-scrub checks whether the data content of the three replica objects is consistent. However, due to changes in the fourth-generation object storage architecture and different writing methods (online EC storage across az levels), scrub cannot fully meet the corresponding requirements. In order to ensure the security of the data, the embodiment of the present application is based on the EC storage method, combined with the characteristics of its own architecture, and has developed a data repair method for the object storage system in the event of data shard loss, disk damage, data shard content changes, etc.

[0061] Scrub is developed based on the Ceph architecture. It starts threads on its own OSD, which puts a certain amount of pressure on the OSD itself. In the case of business writing, its own pressure is more obvious. However, the fourth-generation object storage architecture (object storage gateway, metadata cluster (indexset), data cluster (dataset) and other components) is different from the Ceph architecture. Scrub cannot operate the indexset component of the fourth-generation object storage architecture and cannot repair data. The cross-AZ online EC storage method is a storage method of the fourth-generation object storage architecture. Different from the three-copy storage or EC storage method of Ceph, it stores complete objects in each group of pools (called poolset), and each pool only stores one shard of the complete object. The number of pools in the poolset is k+m. When the number of pools successfully read meets k, or the number of shards lost does not exceed m, EC can be successfully encoded and decoded. The scanning, repair, and migration tasks of the data in the dataset are completed through manager scheduling, which reduces the thread overhead of scrub in the OSD module. Before scanning the data in the poolset, the poolset will be frozen first. This will not affect the business and speed up the scanning speed of the current poolset. The manager scheduling is used more intelligently to ensure that the scheduling tasks are more reasonable and the data is more secure.

[0062] Figure 2 Schematic diagram of the storage object method provided by the embodiment of the present application. Figure 2 As shown in part (a), the gateway access layer splits the S3 object into segment object 1, segment object 2, and segment object 3. Each segment is split into n shards, where n is the number of pools in a poolset. Figure 2 As shown in part (b), the client divides the S3 large object into part object 1, part object 2 and part object 3, where part object is a block object, and each part object is divided into segment objects. For example, it can be divided into two segment objects, or it can be divided into three or more segment objects. Each segment is divided into n shards, where n is the number of pools in a poolset. It should be noted that Figure 2 This is a method of storing objects provided in an embodiment of the present application. The specific storage method can be determined based on actual conditions.

[0063] According to the technical architecture and characteristics of object storage, multiple methods are used to ensure data security. First of all, the object storage architecture is divided into three major components in data storage, namely object storage gateway, metadata cluster (indexset), and data cluster (dataset). The object storage gateway supports the S3 interface. The metadata cluster stores some information about users, buckets, and objects, and the data cluster is used to store object data. When data is written through the object storage gateway, it will be divided into k+m shards in the EC storage method. These k+m shards are written into k+m pools respectively. These k+m pools form a poolset, and one of the pools mounts a disk to store real data. Since when uploading data, as long as k data shards are met, the data can be downloaded. In order to ensure that the data is written successfully and the data shards are not lost and cannot be read, at least k+2 data shards corresponding to the data are written to indicate a successful write.

[0064] Although data can be found to be written successfully during the upload process, it cannot be guaranteed that all data shards exist. In order to ensure that data is not lost, it is necessary to recover the data shards that have not been written successfully. In this case, the data to be recovered is divided into two categories: active data and inactive data. The data that users frequently operate and the data just uploaded are set as active data. For this kind of active data, it can be found whether k+m data shards exist during the upload and download process. If the number of data shards lost by the data that can be read normally is large and less than m, in order to prevent the data shards from being lost and causing the data to be unreadable, the number of lost data shards will be used as the priority for repair; for inactive data, that is, data that is not frequently operated. This kind of data cannot actively discover its integrity on the object storage gateway side. Therefore, the manager schedules the poolset scour program to scan the data in the poolset at the poolset granularity, and compares each shard from the k+m data pools to determine whether the data shards in the poolset are lost and whether the data shards are complete. With the poolset as the granularity, the data of the entire object can be completely scanned to see if it is complete. Specifically, in a poolset, each pool will store objects with the same name. When data is lost in some pools, the number of objects in each pool is not complete. Therefore, the set of objects stored in the poolset can be obtained based on the objects in all pools. By comparing this complete set with each pool, the lost objects in the current pool can be obtained, that is, the data in the current pool is lost and needs to be restored. If the amount of data in the current poolset is large, and in order to list the data on the pool faster, a more fine-grained method can be used for scanning. The data on different pgs of each pool can be scanned concurrently according to the range of pgid. By comparing the data on pgs, the lost shards or tampered shards can be found. Specifically, there will be multiple pgs in a pool. When objects are written into the pool, they will fall into different pgs. It is the same as the previous comparison description, but the granularity is smaller, the data in the memory is smaller, and it is not easy to overflow. The lost data shards found are written into the table to be repaired, and then the system concurrently schedules the repair program to repair the data. Multiple poolsets do not affect each other, which ensures that the user's business is written normally and that the pressure of each poolset is within an acceptable range.

[0065] For active data during upload and download, data shards that are not written or fail to read in the middle may be due to abnormal network or cluster status, busy disk, or disk damage, etc., which can be recorded and marked as data to be repaired. For data upload and download, each data shard will be read when sending each data shard. For some data shards, when the return code data sent is -2, it means that the data shard does not exist, but the disk can be read normally, which is caused by data loss due to some reasons of the disk; when the return code data sent by some data shards is -110, it means that the disk is busy or the network problem causes the reading of the data shard to time out. At this time, it is impossible to determine whether the current shard exists, and no pool comparison is performed; the return code data sent by some data shards is -112, which means that there is a problem with the cluster, the osd status is down, the data may still be on the disk, but it cannot be read, and it is impossible to determine whether the data shard exists; some data shards cannot reach k, and the data cannot be read. Therefore, it is necessary to determine whether the data shard has really been deleted based on the return status, and also to determine whether the metadata corresponding to the data shard exists. When the metadata does not exist, it means that the data has been deleted and the data shard corresponding to the data does not need to be restored. When the metadata does not exist, the return code data of the metadata request is -2. According to the data returned by the previous data shards, except for -2, -110 and -112, the return code data less than 0 is considered to be an abnormal situation, that is, these data shards need to be restored, and the lost data shards need to be recorded in the table to be repaired. At the same time, the table to be repaired will record the attribute information of this data shard and the attribute information of the data corresponding to the data shard. In this way, a complete data will record multiple lost data shards. When repairing the data, it will be sorted according to the number of lost data shards. First, the attribute information of the data records with more lost data shards is obtained, including the shard name, the number of lost data shards, the set of the number of lost shards, the data name corresponding to the data shard, the bucket name, poolset and other information. After obtaining the attribute information to be repaired, the characteristics of erasure codes are used to divide the data into k+m shards, including k original data and m redundant data. When reading data, you can successfully read the complete data by arbitrarily obtaining k data shards. You only need to read k data shards of the data you want to repair to obtain the complete data, and then obtain the content of the lost data shard based on the complete data, write these contents into the lost data shard, and also write the data shard into the data pool until all the lost data shards recorded in the table to be repaired are repaired.In this way, all lost data shards are rewritten, and the recovered data shards will also be deleted from the table to be repaired. At the same time, the data information corresponding to the lost data shards is written into the archive table. By analyzing the archive table, it can be determined which storage pools are prone to data loss, which disks are prone to data loss, and which machines are prone to data loss. This can be done by comparing and analyzing the network, resource pool configuration, different disks, and different systems.

[0066] For inactive data, all data shards will be scanned through the manager's periodic scheduling cycle to prevent data loss caused by disk silence or data change. The manager's scheduling time is set by different resource pool sizes. Generally, scour will be set to execute once every 2 to 3 days. Since these data are not frequently accessed, manager scanning plus business reading and writing will increase the pressure on the disk. In order to solve the problem of excessive disk pressure without affecting the actual business. Therefore, before the poolset required to be scanned, the poolset to be scanned will be frozen first, which can speed up the scanning speed while not affecting the problem of business write failure, reducing IO resource usage, and unfreezing the poolset after repair. For the operation of scour scanning lost shards, it mainly checks whether the data shards are lost, the utilization of the disk is relatively low, and the execution speed is relatively fast. Therefore, the execution cycle can be relatively short. For deep-scour, it is necessary to read the data shards to determine whether the shards are modified. In this way, the utilization of the disk will be relatively high, and the execution speed will be relatively slow. Therefore, the execution cycle can be relatively long. By freezing the poolset, the impact on the business is also reduced, and the execution speed is improved. By reasonably configuring various parameters in different environments and scheduling different functions, for example, in a 300pb resource pool, scour can be scheduled to scan the entire resource pool in 3 to 4 days, so the cycle can be set to 4 days, and one round is scheduled every 4 days. For deep-scour, its scanning data occupies disk utilization. In order not to affect the business, it is necessary to reduce its execution rate. It can be scheduled to execute one round in a week or half a month to deal with business problems caused by system resources. Through the scanning results, the lost data shards are obtained, and the attribute information of the lost data shards is written into the table to be repaired. Each time the attribute information is written into the table, it will detect whether the number of lost data shards is close to the configured number of recoverable shards. Among them, the number of recoverable shards can correspond to the k described above, and the number of lost shards can correspond to the m mentioned above, generally k>m. Once it is detected that the number of lost data shards is close to the configured number of recoverable shards, the manager scheduling will be started immediately. By freezing the poolset, the impact on the business will not be affected. In the early morning when there is less business, the number of scanning tasks for inactive data will increase, which can increase the scanning cycle of the entire dataset, schedule more poolsets for scanning, and timely discover lost data shards. Data will be repaired at the granularity of poolset according to the different degrees of data loss of different disks and different poolsets, ensuring that disks that are prone to data loss are repaired first.The data of all lost data shards will have their location information, data size, lost data shard location and other attributes written into the archive table for analyzing the characteristics of disks and storage pools that are prone to data loss, to see the causes and better optimize manager scheduling to reduce data loss.

[0067] For poolsets with large data volumes and many disk damages, due to some reasons, even after replacing the disk, shards are often lost. It is also possible that all the disks on a rack are damaged and the disk cannot be replaced due to physical factors. In this case, it is necessary to migrate this data to other poolsets to ensure data security. Therefore, a cross-poolset migration solution is designed. The readable data in the current poolset is compared with the metadata by scanning the underlying shards, that is, the objects recorded in each pool, to obtain the s3 object information. This information is written to the table to be migrated. In this case, the unavailable poolset will be changed to the freeze state, so that the user's data will no longer be written to this poolset. At this time, these poolsets can temporarily adjust the weight to 0 and not write. After the physical reasons of the region are handled, the new pool can be added to this poolset, or the poolset can be removed and re-added to the new poolset after the handling is completed. This can prevent data from being continuously written to the bad poolset, causing migration failure or incomplete migration. When all the data to be migrated are successfully acquired, you can execute poolset migrate to migrate the data and migrate the unavailable data to the available poolset. Here, you can migrate according to the capacity of the specific poolset. You can migrate to a random selection of multiple different poolsets in the data configuration, or you can migrate to a specified poolset as long as the capacity is large enough. In this way, the data can be guaranteed to be secure when it is irrecoverable due to external factors. The migration progress can be judged by the records in the migration table. If all the migration data in the record is deleted, all data has been successfully migrated.

[0068] Figure 3 Schematic diagram of the architecture of the manager executing the scheduling program provided in the embodiment of the present application, such as Figure 3 As shown, the manager executes the scheduler to scan, repair, and migrate the data in the corresponding poolset through the object storage gateway. The read-write table is a description of the overall object storage architecture. For example, data upload and download require metadata operations. The table to be repaired and the table to be migrated are both tables in the read-write table.

[0069] The embodiment of the present application proposes to repair and migrate data in different ways according to abnormal situations in different scenarios for different types of data under the fourth-generation object storage architecture. Specifically, for active data, the user's operation on the data will be used to determine whether the object's shard is lost. For inactive data, through the intelligent scheduling of the manager, the poolset is used as the granularity to scan and repair, the poolset is changed in state, and the user's business is prohibited from writing to ensure that the scanning process reduces the impact of the business and reduces the disk pressure. At the same time, under large amounts of data, the data in the storage pool is obtained in a concurrent manner with the pgid as the granularity, and the data is scanned and repaired. In the case where a large number of disks in this group of disks cannot be accessed due to external factors such as disk damage in a rack, and the disk cannot be quickly replaced, the data is migrated at the poolset granularity to ensure the security of user data. In this way, the trigger mechanism for data repair is added, the data is classified, and the processing speed is improved. Through the intelligent scheduling of the manager, the poolset is used as the granularity, and the state of the poolset is changed before scanning, which ensures the normal customer business. At the same time, the pgid is used at a finer granularity and concurrent operations are performed to speed up the scanning and repair speed of the poolset and ensure data security. It adds a protection for data security by ensuring data security through data migration caused by failures in special physical scenarios.

[0070] Figure 4 is a schematic diagram of the structure of the data processing device provided in the embodiment of the present application, such as Figure 4 As shown, the data processing device includes:

[0071] The storage unit 401 is used to store data in the first storage pool set in a first manner; the first manner is to split the data into N data slices, and the N data slices correspond to the first storage pool set; wherein N is a positive integer;

[0072] An acquisition unit 402 is used to acquire status information of N data slices;

[0073] The processing unit 403 is configured to process the N data slices according to the status information of the N data slices.

[0074] In some embodiments, when the data is of the first category, the acquisition unit 402 is used to acquire the user's first operation process, and determine the status information of each data slice in N data slices according to the first operation process; the first operation process is used to acquire N data slices.

[0075] In some implementations, when the data is of the second type, the acquisition unit 402 is used to scan N data slices according to the first time period to generate scanning results; the scanning results are used to determine the status information of each data slice in the N data slices.

[0076] In some implementations, the apparatus further includes: a freezing unit 404; the freezing unit 404 is configured to freeze the first storage pool set before scanning the N data shards according to the first time period.

[0077] In some embodiments, the processing unit 403 is used to determine whether to store M first information corresponding to M data slices out of N data slices into the first table based on the status information; the M first information includes information of the M data slices and object information corresponding to the M data slices respectively; wherein M is a positive integer and M is less than N; and the M data slices are repaired according to the first table.

[0078] In some embodiments, when M first information corresponding to M data slices are stored in a first table, the acquisition unit 402 is used to acquire second information of the M data slices stored in the first table; the second information includes storage configuration information of the M data slices; and the processing unit 403 is used to adjust the storage configuration of the M data slices according to the second information.

[0079] In some implementations, the processing unit 403 is configured to migrate N data shards from a first storage pool set to a second storage pool set.

[0080] Those skilled in the art should understand that Figure 4 The implementation functions of each unit in the data processing device shown can be understood by referring to the relevant description of the aforementioned method. Figure 4 The functions of each unit in the data processing device shown may be implemented by a program running on a processor, or may be implemented by a specific logic circuit.

[0081] Figure 5 It is a schematic structural diagram of a data processing device 500 provided in an embodiment of the present application. Figure 5 The data processing device 500 shown includes a processor 510, which can call and run a computer program from a memory to implement the method in the embodiment of the present application.

[0082] Alternatively, if Figure 5 As shown, the data processing device 500 may further include a memory 520. The processor 510 may call and run a computer program from the memory 520 to implement the method in the embodiment of the present application.

[0083] The memory 520 may be a separate device independent of the processor 510 , or may be integrated into the processor 510 .

[0084] Alternatively, if Figure 5 As shown, the data processing device 500 may further include a transceiver 530, and the processor 510 may control the transceiver 530 to communicate with other devices, specifically, to send information or data to other devices, or to receive information or data sent by other devices.

[0085] The transceiver 530 may include a transmitter and a receiver. The transceiver 530 may further include an antenna, and the number of the antennas may be one or more.

[0086] The data processing device 500 can implement the corresponding processes implemented by the data processing device in each method of the embodiments of the present application, and for the sake of brevity, they are not described here in detail.

[0087] Figure 6 It is a schematic structural diagram of the chip of an embodiment of the present application. Figure 6 The chip 600 shown includes a processor 610, which can call and run a computer program from a memory to implement the method in the embodiment of the present application.

[0088] Alternatively, if Figure 6 As shown, the chip 600 may further include a memory 620. The processor 610 may call and run a computer program from the memory 620 to implement the method in the embodiment of the present application.

[0089] The memory 620 may be a separate device independent of the processor 610 , or may be integrated into the processor 610 .

[0090] Optionally, the chip 600 may further include an input interface 630. The processor 610 may control the input interface 630 to communicate with other devices or chips, and specifically, may obtain information or data sent by other devices or chips.

[0091] Optionally, the chip 600 may further include an output interface 640. The processor 610 may control the output interface 640 to communicate with other devices or chips, and specifically, may output information or data to other devices or chips.

[0092] The chip can implement the corresponding processes implemented by the data processing device in each method of the embodiments of the present application. For the sake of brevity, they will not be described here.

[0093] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0094] It should be understood that the processor of the embodiment of the present application has the ability to process signals. In the implementation process, each step of the above method embodiment can be completed by the processor.

[0095] An embodiment of the present application also provides a computer program product, including a computer program.

[0096] When the computer program is executed by the processor, it implements the corresponding processes implemented by the data processing device in each method of the embodiments of the present application. For the sake of brevity, it will not be repeated here.

[0097] An embodiment of the present application also provides a computer-readable storage medium for storing a computer program.

[0098] The computer program enables the computer to execute the corresponding processes implemented by the data processing device in each method of the embodiments of the present application, which will not be described in detail here for the sake of brevity.

[0099] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0100] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0101] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0102] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0103] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0104] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0105] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A data processing method, characterized in that: The method comprises: Storing the data in a first storage pool set in a first manner; the first manner is splitting the data into N data slices, the N data slices corresponding to the first storage pool set; wherein N is a positive integer; Acquire status information of the N data shards; and process the N data shards according to the status information of the N data shards.

2. The method according to claim 1, characterized in that When the data is of the first type, the obtaining of the status information of the N data slices includes: A first operation process of the user is obtained, and status information of each of the N data shards is determined according to the first operation process; the first operation process is used to obtain the N data shards.

3. The method according to claim 1, characterized in that When the data is the second type of data; the obtaining the status information of the N data slices includes: The N data slices are scanned according to a first time period to generate a scanning result; the scanning result is used to determine status information of each data slice in the N data slices.

4. The method according to claim 3, characterized in that Before scanning the N data slices according to the first time period, the method further includes: Freeze the first storage pool set.

5. The method according to any one of claims 2 to 4, characterized in that: The processing of the N data slices according to the state information includes: Determine whether to store M first information corresponding to M data slices among the N data slices into the first table according to the state information; the M first information includes information of the M data slices and object information corresponding to the M data slices respectively; wherein M is a positive integer, and M is less than N; The M data slices are repaired according to the first table.

6. The method according to claim 5, characterized in that When the M first information corresponding to the M data slices are stored in the first table, the method further includes: Acquire second information of the M data slices stored in the first table; the second information includes storage configuration information of the M data slices; Adjust the storage configuration of the M data shards according to the second information.

7. The method according to claim 1, characterized in that The processing of the N data slices according to the state information includes: Migrate the N data shards from the first storage pool set to the second storage pool set.

8. A data processing device, characterized in that: The device comprises: A storage unit, configured to store data in a first storage pool set in a first manner; the first manner is to split the data into N data slices, the N data slices corresponding to the first storage pool set; wherein N is a positive integer; An acquisition unit, used to acquire status information of the N data slices; A processing unit is used to process the N data slices according to the status information of the N data slices.

9. A data processing device, characterized in that: include: A processor and a memory, the memory being used to store a computer program, the processor being used to call and run the computer program stored in the memory to execute the method according to any one of claims 1 to 7.

10. A computer program product, characterized in that include: A computer program which, when executed by a processor, implements the method according to any one of claims 1 to 7.

11. A computer-readable storage medium, characterized in that: Used to store a computer program, wherein the computer program causes a computer to execute the method according to any one of claims 1 to 7.

Images