Data processing method, device, terminal device and computer-readable storage medium

Abstract

The present application is applicable to the field of data processing technology and provides a data processing method, apparatus, terminal device, and computer-readable storage medium. The method comprises: obtaining information about data to be classified, the information including all access heat values ​​of the data to be classified within a preset time period and first time information corresponding to each access heat value; calculating a first attenuation value for each access heat value based on the first time information and a preset first time attenuation function; calculating a first heat value based on the first attenuation value corresponding to each access heat value; and determining a heat category for the data to be classified based on the first heat value. The present application can solve the problem that current data classification schemes have low accuracy and are prone to cache contamination.

Description

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on December 11, 2019, with application number 201911268590.5 and application name “Method for Storing Data and Method for Recovering Data”, the Chinese patent application filed with the State Intellectual Property Office on December 31, 2019, with application number 201911425935.3 and application name “A Data Classification Method, Device and Terminal Device”, and the Chinese patent application filed with the State Intellectual Property Office on July 29, 2020, with application number 202010741937.X and application name “Data Storage Method, Terminal Device, Device and System Based on Erasure Code”. The entire contents of the above-mentioned Chinese patent applications are incorporated into this application by reference. Technical Field

[0002] The present application belongs to the field of data processing technology, and in particular relates to a data processing method, apparatus, terminal device, and computer-readable storage medium. Background Art

[0003] With the development of technology, smart terminal devices such as mobile phones, tablet computers, and car-mounted devices have entered people's lives and work. These smart terminal devices can provide users with data storage and access services.

[0004] However, the explosive growth of data volume places enormous storage pressure on terminal devices and affects data access efficiency. To address this, researchers have proposed a solution that classifies data popularity (e.g., hot data and cold data) based on its usage and then implements corresponding management strategies based on the data popularity category. This approach can conserve terminal device storage space and improve data access efficiency.

[0005] Currently, the most common algorithms used to manage data are the Least Recently Used (LRU) and Least Frequently Used (LFU) algorithms. However, these methods simply determine the popularity of data based on the number of accesses or recent accesses, resulting in low accuracy and a high risk of cache contamination.

[0006] Therefore, how to improve the accuracy of data classification has become a technical problem that technical personnel in this field urgently need to solve. Summary of the Invention

[0007] The embodiments of the present application provide a data processing method, apparatus, terminal device, and computer-readable storage medium, which can solve the problem that current data classification schemes have low accuracy and are prone to cache pollution.

[0008] In a first aspect, an embodiment of the present application provides a data processing method, applied to a first device, comprising:

[0009] Acquire information of the data to be classified, the information including all access popularity values ​​of the data to be classified within a preset time period and first time information corresponding to each access popularity value;

[0010] Calculate the first decay value of each access heat value according to the first time information and the preset first time decay function;

[0011] Calculating a first heat value according to the first decay value corresponding to each access heat value;

[0012] Determine a heat category of the data to be classified according to the first heat value.

[0013] It should be noted that when data to be classified needs to be classified, information of the data to be classified can be obtained. The above information may include all access records of the data to be classified within a preset time period, and the access records include access popularity values ​​and first time information.

[0014] The preset time period is a time period pre-set by the user and can be set according to actual needs. For example, if the current time is October 5, 2019, and the user wants to calculate the first popularity value of each data in the last five days, the user can set the preset time period to October 1, 2019 to October 5, 2019.

[0015] The preset first time decay function can be selected according to actual conditions. For example, the preset first time decay function can be any one of an exponential function, a linear function, an inverse S-curve function, a hyperbolic function, a logarithmic function, and the like.

[0016] In the process of calculating the first heat value, the first heat value is calculated based on the access heat value, which is related to the user's historical access situation, and can avoid occasional access contamination of the cache. In addition, when calculating the first heat value, a first time decay function is introduced. The first decay value of each access heat value is calculated based on the first time information and the preset first time decay function. The first heat value is calculated based on the first decay value, which reduces the cache contamination effect caused by historical data, thereby improving the accuracy of the calculated first heat value and more accurately classifying the heat categories of the data to be classified.

[0017] In a possible implementation of the first aspect, calculating the first decay value of each access popularity value according to the first time information and a preset first time decay function includes:

[0018] Calculate the first time decay coefficient of each access heat value according to the first time information and the preset first time decay function;

[0019] Multiplying the first time attenuation coefficient of each access heat value by each access heat value to obtain a first attenuation value corresponding to each access heat value;

[0020] The above-mentioned calculation of the first heat value according to the first attenuation value corresponding to each access heat value includes:

[0021] The first attenuation values ​​corresponding to the various access heat values ​​are summed to obtain the first heat value.

[0022] It should be noted that the first time attenuation coefficient of each access heat value can be calculated according to the first time information and the preset first time attenuation function, and the first time attenuation coefficient represents the attenuation of the access heat value.

[0023] By multiplying the first time attenuation coefficient of each access heat value by each access heat value, a first attenuation value corresponding to each access heat value can be obtained. The first attenuation value represents the actual heat utility of the access heat value.

[0024] In a possible implementation of the first aspect, the information further includes a second heat value and second time information corresponding to the second heat value, where the second heat value is a preset initial heat value or a most recently calculated first heat value;

[0025] Accordingly, after obtaining the information of the data to be classified, the above method further includes:

[0026] Calculating a second attenuation value of the second heat value according to the second time information and a preset second time attenuation function, where the second time information is the start time of the preset time period;

[0027] Accordingly, the above-mentioned calculation of the first heat value according to the first attenuation value corresponding to each access heat value includes:

[0028] Calculating the first heat value of the data to be classified according to the second attenuation value of the second heat value and the first attenuation value corresponding to each access heat value;

[0029] Update the second heat value to the first heat value.

[0030] It should be noted that the second heat value is the initial heat value set by the user or the most recently calculated first heat value. The second time information is the start time of the preset time period. For example, if the preset time period is from October 1, 2019 to October 5, 2019, the second time information is the start time of the preset time period, that is, October 1, 2019.

[0031] After the first heat value is calculated according to the access heat value and the second heat value, the second heat value is updated to the first heat value so as to calculate the next first heat value.

[0032] The preset second time decay function may be the same as the preset first time decay function, or the preset second time decay function may be different from the preset first time decay function.

[0033] The preset second time decay function can be selected according to actual conditions. For example, the preset second time decay function can be any one of an exponential function, a linear function, an inverse S-curve function, a hyperbolic function, a logarithmic function, and the like.

[0034] In the process of calculating the first heat value, combining the access heat value and the second heat value for comprehensive calculation can make the calculated first heat value more reasonable, thereby more accurately classifying the heat type of the data to be classified.

[0035] In a possible implementation of the first aspect, calculating the second attenuation value of the second heat value according to the second time information and a preset second time attenuation function includes:

[0036] Calculating a second time decay coefficient according to the second time information and a preset second time decay function;

[0037] multiplying the second time attenuation coefficient and the second heat value to obtain a second attenuation value of the second heat value;

[0038] Accordingly, the above-mentioned calculation of the first heat value of the data to be classified according to the second attenuation value of the second heat value and the first attenuation value corresponding to each access heat value includes:

[0039] The second attenuation value of the second heat value and the first attenuation values ​​corresponding to each access heat value are summed to obtain the first heat value.

[0040] It should be noted that the second time attenuation coefficient can be calculated based on the second time information and the preset second time attenuation function, and the second time coefficient represents the attenuation of the second heat value.

[0041] By multiplying the second time attenuation coefficient by the second heat value, a second attenuation value of the second heat value can be obtained, where the second attenuation value represents the actual heat utility of the second heat value.

[0042] After the first attenuation value and the second attenuation value are calculated, the first heat value can be obtained by summing the first attenuation value and the second attenuation value.

[0043] In a possible implementation of the first aspect, the popularity category includes warm data and cold data, and the method further includes:

[0044] Selecting a first preset proportion of warm data from the data to be classified whose heat type is warm data as first backup data;

[0045] Performing erasure coding on the first backup data and the data to be classified whose popularity category is cold data to obtain first verification data;

[0046] The data to be classified whose popularity category is cold data is sent to the first designated terminal, the first verification data is sent to the second designated terminal, and the locally stored data to be classified whose popularity category is cold data is deleted.

[0047] It should be noted that after determining the warm data and cold data, a first preset ratio of warm data can be selected as the first backup data. The first preset ratio can be set based on actual conditions. For example, if the data to be classified contains 60% warm data, the first preset ratio can be set to 20%, and 20% of the warm data from the data to be classified can be selected as the first backup data (which can be understood as selecting 1 / 3 of the current 60% warm data).

[0048] After selecting the first backup data, erasure coding is performed on the first backup data and the cold data to generate the first verification data. The cold data is sent to the first designated terminal, the first verification data is sent to the second designated terminal, and the locally stored cold data is deleted. This mixed encoding of warm and cold data ensures the integrity of the cold data and improves the fault tolerance of both warm and cold data.

[0049] In a possible implementation of the first aspect, performing erasure coding on the first backup data and the data to be classified whose popularity category is cold data to obtain the first verification data includes:

[0050] The first backup data is warm data with the same data volume as the cold data, and erasure coding is performed on the first backup data and the data to be classified whose popularity category is cold data to obtain first verification data;

[0051] Alternatively, the first backup data is warm data with a different data volume from the cold data. If the data volume of the first backup data is less than that of the cold data, the first backup data is padded with zeros to have the same data volume as the cold data, and then the first backup data padded with zeros and the data to be classified whose heat category is cold data are encoded with error correction and erasure codes to obtain first verification data.

[0052] It should be noted that when selecting the first backup data, the first backup data may be warm data with the same data volume as the cold data. In this case, the first backup data and the cold data may be directly erasure coded to obtain the first verification data.

[0053] Alternatively, the first backup data may be warm data with a different data size from the cold data. In this case, if the first backup data has a smaller data size than the cold data, the first backup data may be padded with zeros to bring the first backup data to the same data size as the cold data. Erasure coding is then performed on the zero-padded first backup data and the data to be classified, which has a temperature category of cold data, to obtain first verification data.

[0054] In a possible implementation of the first aspect, the method further includes:

[0055] Splitting the warm data other than the first backup data in the data to be classified whose heat type is warm data into the second backup data and the third backup data;

[0056] Performing erasure coding on the second backup data and the third backup data to obtain second verification data;

[0057] The second verification data is sent to a third designated terminal, and the backup data of the second backup data is sent to a fourth designated terminal.

[0058] It should be noted that, in addition to erasure coding the first backup data and the cold data, the warm data other than the first backup data in the warm data can also be split into the second backup data and the third backup data. When splitting the warm data other than the first backup data, the warm data other than the first backup data can be evenly divided into the second backup data and the third backup data. In this case, the data volume of the second backup data is the same as the data volume of the third backup data. Alternatively, the warm data other than the first backup data can be unequally divided into the second backup data and the third backup data. In this case, if the data volume of the second backup data is less than the data volume of the third backup data, the second backup data can be padded with zeros to make it the same as the data volume of the third backup data; if the data volume of the third backup data is less than the data volume of the second backup data, the third backup data can be padded with zeros to make it the same as the data volume of the second backup data.

[0059] Erasure coding is performed on the second and third backup data to generate second verification data. This improves the fault tolerance of warm data other than the first backup data. The second verification data is sent to a third designated terminal, and the backup data of the second backup data is sent to a fourth designated terminal. The first, second, and third backup data can all be retained in local storage, thereby ensuring the integrity of the warm data.

[0060] In a possible implementation of the first aspect, the popularity category includes hot data, and the method further includes:

[0061] The data to be classified whose popularity category is hot data is stored in the local storage space, and the backup data of the hot data is sent to the data backup terminal for backup.

[0062] It should be noted that when managing hot data, since hot data is data that users use more frequently, the backup data of the hot data can be sent to the data backup terminal for backup. This not only improves the fault tolerance performance of the hot data, but also makes it easier for users to call hot data on the data backup terminal.

[0063] In a possible implementation of the first aspect, the heat category includes warm data, and the method further includes:

[0064] Selecting a second preset proportion of warm data from the data to be classified whose heat category is warm data as fourth backup data;

[0065] Performing erasure coding on the fourth backup data and warm data other than the fourth backup data to obtain third verification data;

[0066] The third verification data is sent to the fifth designated terminal, and the fourth backup data is sent to the sixth designated terminal.

[0067] It should be noted that when managing warm data, in addition to mixed encoding of warm data and cold data, warm data can also be encoded independently.

[0068] At this time, a second preset proportion of warm data can be selected from the warm data as the fourth backup data, and the fourth backup data and the warm data other than the fourth backup data can be encoded with error correction code to obtain third verification data, thereby improving the fault tolerance performance of the warm data by error correction code encoding.

[0069] The second preset ratio can be set according to actual conditions. For example, if the data to be classified contains 60% warm data, the second preset ratio can be set to 20%, and 20% of the data to be classified is selected as the fourth backup data (which can be understood as selecting 1 / 3 of the current 60% warm data).

[0070] In a possible implementation of the first aspect, the popularity category includes cold data, and the method further includes:

[0071] Splitting the data to be classified as cold data in the popularity category into fifth backup data and sixth backup data;

[0072] Performing erasure coding on the fifth backup data and the sixth backup data to obtain fourth verification data;

[0073] The fifth backup data is sent to the seventh designated terminal, the sixth backup data is sent to the eighth designated terminal, the fourth verification data is sent to the ninth designated terminal, and the locally stored data to be classified whose popularity category is cold data is deleted.

[0074] It should be noted that when managing cold data, in addition to mixed encoding of warm data and cold data, cold data can also be encoded independently.

[0075] At this time, the cold data can be split into fifth backup data and sixth backup data, and the fifth backup data and the sixth backup data are erasure coded to obtain fourth verification data, thereby improving the fault tolerance performance of the cold data through erasure coding.

[0076] In a possible implementation of the first aspect, the method further includes:

[0077] Determine the stripe length of the erasure code stripe, which is less than or equal to the number of online devices in the distributed storage system;

[0078] Obtaining data to be backed up for each device to be backed up, where the device to be backed up is an online device with data to be backed up, the device to be backed up includes the first device, and the data to be backed up includes data to be classified;

[0079] Perform erasure coding based on the stripe length and the data to be backed up to obtain the target check data;

[0080] The target verification data is sent to the verification node for storage.

[0081] It should be noted that when a terminal device performs data redundancy backup on both its own data and other terminal devices, the terminal device can serve as an encoding node, or other terminal devices in the distributed storage system can be selected as encoding nodes.

[0082] When the terminal device acts as an encoding node, the terminal device may first determine the stripe length of the erasure code stripe.

[0083] The stripe length of an erasure code stripe should be less than or equal to the number of online devices in the distributed storage system. Otherwise, the amount of data to be backed up will be less than the stripe length.

[0084] The terminal device can then obtain the data to be backed up from each device to be backed up. The device to be backed up is an online device that has data to be backed up. The data to be backed up is data that requires data redundancy backup.

[0085] The data to be backed up may include data to be classified into preset heat types, such as warm data and cold data. And / or, the data to be backed up may also include other data that needs to be backed up.

[0086] After determining the stripe length and the data to be backed up, the terminal device can perform erasure coding to obtain the target verification data and send the target verification data to the verification node for storage.

[0087] Verification nodes are used to store verification data generated by erasure coding. Verification nodes can be terminal devices in the distributed storage system, or other terminal devices outside the system, such as cloud storage.

[0088] In a possible implementation of the first aspect, determining the stripe length of the erasure coded stripe includes:

[0089] The number of online devices determines the stripe length of the erasure coded stripe.

[0090] It should be noted that since only online terminal devices can perform data redundancy backup, the terminal device can determine the number of online devices as the stripe length of the erasure code stripe, so that each online device can act as an independent storage node to store its own data, which is equivalent to each online device storing a part of the erasure code stripe.

[0091] In a possible implementation of the first aspect, determining the stripe length of the erasure coded stripe includes:

[0092] Determine the number of devices to be backed up among the online devices;

[0093] The number of devices to be backed up is determined as the stripe length of the erasure coded stripe.

[0094] It should be noted that among the online devices, only some terminal devices (ie, devices to be backed up) may have data to be backed up, while other terminal devices may have no data to be backed up or the data to be backed up is 0.

[0095] Therefore, the terminal device can also determine the number of devices to be backed up as the stripe length of the erasure code stripe, so that the data to be backed up on the devices to be backed up are organized into stripes, each device to be backed up stores a part of the erasure code stripe, and the data to be backed up stored in each device to be backed up is greater than 0.

[0096] In a possible implementation of the first aspect, determining the stripe length of the erasure coded stripe includes:

[0097] Obtain the data volume of each data to be backed up;

[0098] Merging the data to be backed up whose data volume is less than a first threshold according to a preset merging rule to obtain data to be backed up whose data volume is greater than or equal to the first threshold;

[0099] The amount of data to be backed up after the merge process is determined as the stripe length of the erasure code stripe.

[0100] It should be noted that when the amount of data to be backed up on the devices to be backed up is similar or similar to the erasure code stripe depth, each device to be backed up can be used as an independent storage node to store the data to be backed up, and the data to be backed up corresponding to each device to be backed up can be used as independent storage node data for erasure coding. When the amount of data to be backed up on the devices to be backed up is significantly different or different from the erasure code stripe depth, the data to be backed up on multiple devices to be backed up can be combined into a single storage node data for erasure coding, which can save computing resources on the encoding node.

[0101] At this time, a merging rule may be preset to merge the data to be backed up whose data volume is less than the first threshold value, to obtain the data to be backed up whose data volume is greater than or equal to the first threshold value.

[0102] The preset merge rule can be determined based on actual circumstances. For example, the preset merge rule may be to merge data to be backed up whose data volume is less than a first threshold; or the preset merge rule may be to merge data to be backed up whose data volume is less than the first threshold with data to be backed up whose data volume is greater than the first threshold; or the preset merge rule may be other rules, and the specific content of the preset merge rule is not limited herein.

[0103] In a possible implementation of the first aspect, the method further includes:

[0104] The data volume of the largest data block among the data blocks divided by each online device is determined as the stripe depth of the erasure code stripe;

[0105] Alternatively, set the stripe depth of erasure coded stripes to a preset value.

[0106] It should be noted that, in addition to determining the stripe length, the terminal device can also determine the stripe depth.

[0107] The stripe depth is the amount of data in a data block within an erasure coded stripe. To determine the stripe depth, the data block partitioning method used by each online device can be obtained. The data size of the largest data block among the data blocks partitioned by each online device can be used as the stripe depth. Alternatively, the terminal device can directly set the stripe depth to a preset value. For example, the stripe depth can be set to 1MB, 2MB, 5MB, and other data sizes.

[0108] In a possible implementation of the first aspect, performing erasure coding according to the stripe length and the data to be backed up to obtain target check data includes:

[0109] Generate an erasure code stripe based on the stripe length and the original data blocks corresponding to each data to be backed up;

[0110] Perform erasure coding on the erasure coded stripe to obtain target check data.

[0111] It should be noted that when performing erasure coding, the data to be backed up needs to be divided into original data blocks. This division of the data to be backed up can be performed by the encoding node or by the device to be backed up.

[0112] Then, the terminal device can select the original data blocks of each data to be backed up to form one or more erasure code stripes according to the stripe length, and perform erasure code encoding on the erasure code stripes to obtain target verification data.

[0113] In a possible implementation of the first aspect, before performing erasure coding on the erasure coded stripe to obtain target parity data, the method further includes:

[0114] If the number of original data blocks in an erasure coded stripe is less than the stripe length, the erasure coded stripe is padded with zeros so that the number of original data blocks in the erasure coded stripe is equal to the stripe depth.

[0115] It should be noted that if the number of original data blocks in an erasure code stripe is less than the stripe length, the erasure code stripe needs to be padded with zeros, and data blocks containing all 0s are used as the missing original data blocks to complete the erasure code stripe, so that the number of original data blocks in the erasure code stripe is equal to the stripe depth.

[0116] In a possible implementation of the first aspect, before performing erasure coding on the erasure coded stripe to obtain target parity data, the method further includes:

[0117] If there are original data blocks with data volume less than the stripe depth in the erasure code stripe, the original data blocks with data volume less than the stripe depth are padded with zeros so that the data volume of the original data blocks with data volume less than the stripe depth is equal to the stripe depth.

[0118] It should be noted that if there are original data blocks in the erasure code stripe whose data volume is less than the stripe depth, it is necessary to perform zero padding on the original data blocks whose data volume is less than the stripe depth, and fill the missing data in these original data blocks with 0 so that the data volume of these original data blocks is equal to the stripe depth.

[0119] In a possible implementation of the first aspect, the check node is a cloud storage device or a network attached storage (NAS) device.

[0120] It should be noted that when the check node is a terminal device other than a distributed storage system, the check node may be a cloud storage device or a network attached storage (NAS) device.

[0121] In a possible implementation of the first aspect, before sending the target verification data to the verification node for storage, the method further includes:

[0122] Obtaining device parameters of each online device, including one or more parameters of device reliability, device online time, and device storage space;

[0123] A check node is selected from each online device according to the device parameters of each online device and a preset check node selection rule.

[0124] It should be noted that when the check node is a terminal device in a distributed storage system, the encoding node can obtain device parameters of each online device. The device parameters may include one or more parameters such as device reliability, device online time, and device storage space.

[0125] Then, the encoding node may determine the check node according to the device parameters of each online device and a preset check node selection rule.

[0126] The preset check node selection rules can be set according to actual needs. For example, the preset check node selection rule can be to select online devices with high device reliability as check nodes; or the preset check node selection rule can be to select online devices with long device online time as check nodes; or the preset check node selection rule can be to select online devices with large device storage space as check nodes; or the preset check node selection rule can be other rules. This application does not limit the specific content of the preset check node selection rules.

[0127] In a possible implementation of the first aspect, obtaining the data to be backed up includes:

[0128] Determine the device to be backed up based on metadata corresponding to each online device, where the metadata is used to indicate whether the online device has data to be backed up and / or the amount of data to be backed up;

[0129] Read the data to be backed up from each device to be backed up.

[0130] It should be noted that when obtaining data to be backed up, the encoding node may first determine which online devices have data to be backed up based on metadata corresponding to each online device, and determine the online devices with data to be backed up as terminals to be backed up.

[0131] Then, the encoding node reads the data to be backed up from each terminal to be backed up.

[0132] The metadata is used to indicate whether there is data to be backed up on the online device and / or the amount of data to be backed up.

[0133] In a possible implementation of the first aspect, performing erasure coding according to the stripe length and the data to be backed up to obtain target check data includes:

[0134] When the stripe length satisfies the condition that p is a prime number greater than or equal to 3, the original data blocks corresponding to each data to be backed up are obtained to obtain N groups of original data blocks, where p is N+1 and N is the stripe length. One data to be backed up corresponds to a group of original data blocks, and the original data blocks in the same group have different sorting positions.

[0135] Selecting an original data block with the same sorting position from each of the N groups of original data blocks for erasure coding, to obtain first check data blocks corresponding to different sorting positions;

[0136] Selecting an original data block with a different sorting position from each of the N groups of original data blocks for erasure coding, to obtain a second check data block corresponding to each different sorting position;

[0137] Alternatively, N groups of data blocks are selected from N groups of original data blocks and a group of first verification data blocks, and a data block with a different sorting position is selected from each of the N groups of data blocks for erasure coding to obtain second verification data blocks corresponding to each different sorting position.

[0138] It should be noted that when the stripe length satisfies the condition that p is a prime number greater than or equal to 3, it means that there are N data to be backed up, and the original data blocks corresponding to each data to be backed up can be obtained to obtain N groups of original data blocks.

[0139] Each original data block in each group of original data blocks has a different sorting position. The sorting position can be identified by sequentially marking Arabic numerals or letters. In other embodiments, the sorting position can be identified without sequential marking. This application does not limit the specific implementation of the sorting position.

[0140] For example, the original data blocks of the same device to be backed up can be represented by identifiers such as 1, 2, 3, 4, etc. to indicate the sorting position; or, they can be represented by identifiers such as A, B, C, D, etc. to indicate the sorting order; or, they can be represented by identifiers combining data and letters to indicate the sorting order.

[0141] Then, the encoding node can select an original data block with the same sorting position from each of the N groups of original data blocks to form an erasure code stripe, and perform erasure code encoding on these erasure code stripes to obtain the first check data block corresponding to each sorting position.

[0142] Furthermore, the encoding node may select an original data block with a different sorting position from each of the N groups of original data blocks to form an erasure code stripe, and perform erasure code encoding on these erasure code stripes to obtain a second check data block corresponding to each sorting position.

[0143] Alternatively, N groups of data blocks are selected from N groups of original data blocks and 1 group of first verification data blocks. From the N groups of data blocks, one data block with a different sorting position is selected from each group to form an erasure code stripe. These erasure code stripes are erasure coded, and second verification data blocks corresponding to each sorting position can be obtained.

[0144] The first verification data block and the second verification data block are target verification data.

[0145] In a possible implementation of the first aspect, selecting an original data block having the same sorting position from each of the N groups of original data blocks for erasure coding to obtain first check data blocks corresponding to different sorting positions includes:

[0146] Arrange a group of original data blocks into a column according to the sorting position, with different groups of original data blocks corresponding to different columns, to form a first data block array;

[0147] Performing erasure coding on each row in the first data block array to obtain first check data blocks corresponding to different sorting positions;

[0148] Accordingly, the above-mentioned method of selecting an original data block with a different sorting position from each of the N groups of original data blocks for erasure coding to obtain a second check data block corresponding to each different sorting position includes:

[0149] Selecting an original data block with a different sorting position from each column of the first data block array to perform erasure coding, to obtain second check data blocks corresponding to the different sorting positions;

[0150] Accordingly, the above method selects N groups of data blocks from the N groups of original data blocks and the group of first check data blocks, selects a data block with a different sorting position from each of the N groups of data blocks for erasure coding, and obtains second check data blocks corresponding to the different sorting positions, including:

[0151] Arranging a group of first check data blocks into a column according to the sorting position, and expanding the first data block array according to the column corresponding to the first check data block to obtain a first expanded array;

[0152] N columns are selected from the N+1 columns of the first extended array, and a data block with a different sorting position is selected from each of the N columns for erasure coding to obtain second check data blocks corresponding to the different sorting positions.

[0153] It should be noted that when performing erasure coding, N groups of original data blocks can be arranged into a first data block array of N columns. In the first data block array, one column represents a group of original data blocks, and each original data block in a column is arranged according to the sorting position.

[0154] Then, the terminal device may perform erasure coding on each row in the first data block array to obtain first check data blocks corresponding to different sorting positions.

[0155] Also, the terminal device selects an original data block with a different sorting position from each column of the first data block array for erasure coding, and obtains second verification data blocks corresponding to each different sorting position; or, the terminal device can also arrange a group of first verification data blocks into a column according to the sorting position, and expand the first data block array according to the column corresponding to the first verification data block to obtain a first extended array; then, the terminal device selects N columns from the N+1 columns of the first extended array, and selects a data block with a different sorting position from each of the N columns for erasure coding, and obtains second verification data blocks corresponding to each different sorting position.

[0156] In a possible implementation of the first aspect, selecting an original data block with a different sorting position from each column of the first data block array for erasure coding to obtain second check data blocks corresponding to the different sorting positions includes:

[0157] Performing erasure coding on different diagonal lines on the first data block array to obtain second check data blocks corresponding to different sorting positions;

[0158] Selecting N columns from the N+1 columns of the first extended array, and selecting a data block with a different sorting position from each of the N columns for erasure coding to obtain second check data blocks corresponding to the different sorting positions, including:

[0159] Erasure coding of different diagonal lines is performed on the first extended array to obtain second check data blocks corresponding to different sorting positions.

[0160] It should be noted that when calculating the second check data block, the encoding node can directly select the data blocks on the diagonal of the first data block array or the first extended array for erasure coding, thereby obtaining the second check data corresponding to each different sorting position.

[0161] In a possible implementation of the first aspect, selecting an original data block having the same sorting position from each of the N groups of original data blocks for erasure coding to obtain first check data blocks corresponding to different sorting positions includes:

[0162] Arrange a group of original data blocks into a row according to the sorting position, with different groups of original data blocks corresponding to different rows, to form a second data block array;

[0163] Performing erasure coding on each column in the second data block array to obtain first check data blocks corresponding to different sorting positions;

[0164] Accordingly, the above-mentioned method of selecting an original data block with a different sorting position from each of the N groups of original data blocks for erasure coding to obtain a second check data block corresponding to each different sorting position includes:

[0165] Selecting an original data block with a different sorting position from each row of the second data block array to perform erasure coding, to obtain second check data blocks corresponding to the different sorting positions;

[0166] Accordingly, N groups of data blocks are selected from the N groups of original data blocks and the group of first check data blocks, and a data block with a different sorting position is selected from each of the N groups of data blocks for erasure coding to obtain second check data blocks corresponding to the different sorting positions, including:

[0167] Arranging a group of first check data blocks into a row according to the sorting position, and expanding the second data block array according to the row corresponding to the first check data blocks to obtain a second expanded array;

[0168] N rows are selected from the N+1 rows of the second extended array, and a data block with a different sorting position is selected from each of the N rows for erasure coding to obtain second check data blocks corresponding to the different sorting positions.

[0169] It should be noted that, in addition to treating a group of original data blocks as a column, a group of original data blocks can also be treated as a row to form a second data block array.

[0170] Then, the terminal device may perform erasure coding on each column in the second data block array to obtain first check data blocks corresponding to different sorting positions.

[0171] Furthermore, the terminal device may select an original data block with a different sorting position from each row of the second data block array for erasure coding to obtain second verification data blocks corresponding to each different sorting position; or, arrange a group of first verification data blocks in a row according to the sorting position, and expand the second data block array according to the rows corresponding to the first verification data blocks to obtain a second extended array, select N rows from the N+1 rows of the second extended array, and select a data block with a different sorting position from each of the N rows for erasure coding to obtain second verification data blocks corresponding to each different sorting position.

[0172] In a possible implementation of the first aspect, selecting an original data block with a different sorting position from each row of the second data block array for erasure coding to obtain second check data blocks corresponding to the different sorting positions includes:

[0173] Performing encoding on different diagonals on the second data block array to obtain second check data blocks corresponding to different sorting positions;

[0174] The above method selects N rows from the N+1 rows of the second extended array, and selects a data block with a different sorting position from each of the N rows for erasure coding to obtain second check data blocks corresponding to different sorting positions, including:

[0175] The second extended array is coded along different diagonals to obtain second check data blocks corresponding to different sorting positions.

[0176] It should be noted that when calculating the second check data block, the encoding node can directly select the data blocks on the diagonal of the second data block array or the second extended array for erasure coding, thereby obtaining the second check data corresponding to each different sorting position.

[0177] In a possible implementation manner of the first aspect, the erasure code is encoded as XOR encoding or XOR encoding.

[0178] It should be noted that, when performing erasure coding, the terminal device may use an XOR coding method, an XOR coding method, or other coding methods.

[0179] In a possible implementation of the first aspect, the method further includes:

[0180] When a damaged data block exists, the damaged data block is repaired according to an undamaged original data block and undamaged target verification data, where the target verification data includes a first verification data block and a second verification data block.

[0181] It should be noted that, when there is a damaged data block, the terminal device can repair the damaged data block based on the undamaged original data block and the undamaged target verification data.

[0182] In a possible implementation of the first aspect, when a damaged data block exists, repairing the damaged data block according to the undamaged original data block and the undamaged target verification data includes:

[0183] When there are damaged data blocks and the number of groups of damaged data blocks is less than or equal to 2, the damaged data blocks are repaired according to the undamaged original data blocks and undamaged target verification data, where the target verification data includes a first verification data block and a second verification data block.

[0184] It should be noted that, since the above erasure coding method calculates two groups of check data blocks, the fault tolerance of the erasure coding method is 2, and at most two groups of data blocks are allowed to be damaged.

[0185] Therefore, when there are damaged data blocks, if the number of groups of the damaged data blocks is less than or equal to 2, the terminal device repairs the damaged data blocks based on the undamaged original data blocks and the undamaged target verification data.

[0186] In a possible implementation of the first aspect, when a damaged data block exists, repairing the damaged data block according to the undamaged original data block and the undamaged target verification data includes:

[0187] When any group of data blocks is damaged, obtain the type of the damaged data blocks;

[0188] If the damaged data block is the first parity data block, repair the damaged first parity data block according to the N groups of original data blocks;

[0189] If the damaged data block is the second parity data block, repair the damaged second parity data block according to the N groups of original data blocks, or repair the damaged second parity data block according to the N groups of original data blocks and the first parity data block;

[0190] If the damaged data block is an original data block, the damaged original data block is repaired based on the undamaged original data block and the first verification data block, or the damaged data block is repaired based on a mixture of the undamaged original data block, the first verification data block and the second verification data block.

[0191] It should be noted that when only one set of data blocks is damaged, the type of the damaged data block can be obtained. If the first check data block is damaged, the above N sets of original data can be used to repair the damaged first check data block according to the above erasure coding method.

[0192] If the second verification data block is damaged, the terminal device may repair the damaged second verification data block based on the above N groups of original data or the above N+1 groups of data blocks.

[0193] If the damaged data block is an original data block, the terminal device can directly repair the damaged original data block based on the first check data block and the undamaged original data block. Alternatively, the terminal device can repair the damaged data block based on a mixture of the undamaged original data block, the first check data block, and the second check data block, thereby reducing the number of data blocks that need to be acquired.

[0194] In a possible implementation of the first aspect, when a damaged data block exists, repairing the damaged data block according to the undamaged original data block and the undamaged target verification data includes:

[0195] When any two groups of data blocks are damaged, obtain the type of the damaged data blocks;

[0196] If the damaged data block is a group of first check data blocks and a group of second check data blocks, repair the damaged first check data blocks and second check data blocks according to the N groups of original data blocks;

[0197] If the damaged data block is a group of original data blocks and a group of second verification data blocks, repairing the damaged original data blocks and the second verification data blocks according to the undamaged original data blocks and the first verification data blocks;

[0198] If the damaged data block does not include the second verification data block, the damaged data block is repaired based on a mixture of the undamaged original data block and the second verification data block.

[0199] It should be noted that, when any two groups of data blocks are damaged, the terminal device may first obtain the types of the damaged data blocks.

[0200] If the damaged data blocks are a group of first check data blocks and a group of second check data blocks, the terminal device can repair the first check data blocks and the second check data blocks according to the above N groups of original data blocks using the above erasure code encoding method.

[0201] If the damaged data block is a group of original data blocks and a group of second verification data blocks, the terminal device can repair the damaged original data block based on the first verification data block and the undamaged original data block, and then repair the second verification data based on N groups of original data blocks.

[0202] If the damaged data block does not include the second check data block, it is possible that two groups of original data blocks are lost, or one group of original data blocks and one group of first check data blocks are lost. In this case, the terminal device can repair the damaged data block based on a mixture of the undamaged original data blocks and the second check data blocks.

[0203] In a second aspect, an embodiment of the present application provides a data processing device, including:

[0204] An information acquisition module is used to acquire information about the data to be classified, the information including all access popularity values ​​of the data to be classified within a preset time period and first time information corresponding to the access popularity values;

[0205] A first attenuation module, configured to calculate a first attenuation value of each access popularity value according to the first time information and a preset first time attenuation function;

[0206] A heat calculation module, configured to calculate a first heat value according to a first attenuation value corresponding to each access heat value;

[0207] The heat category module is used to determine the heat category of the data to be classified according to the first heat value.

[0208] In a possible implementation of the second aspect, the first attenuation module includes:

[0209] A first coefficient submodule, configured to calculate a first time decay coefficient of each access heat value according to the first time information and a preset first time decay function;

[0210] A first attenuation value submodule is configured to multiply the first time attenuation coefficient of each access heat value by each access heat value to obtain a first attenuation value corresponding to each access heat value;

[0211] The heat calculation module is specifically used to sum the first attenuation values ​​corresponding to each access heat value to obtain the first heat value.

[0212] In a possible implementation of the second aspect, the information further includes a second heat value and second time information corresponding to the second heat value, where the second heat value is a preset initial heat value or a most recently calculated first heat value;

[0213] Accordingly, the above device further includes:

[0214] A second attenuation module, configured to calculate a second attenuation value of the second heat value according to second time information and a preset second time attenuation function, wherein the second time information is a start time of a preset time period;

[0215] Correspondingly, the heat calculation module includes:

[0216] A first heat submodule, configured to calculate a first heat value of the data to be classified according to a second attenuation value of the second heat value and first attenuation values ​​corresponding to each access heat value;

[0217] The data updating submodule is used to update the second heat value to the first heat value.

[0218] In a possible implementation of the second aspect, the second attenuation module includes:

[0219] A second coefficient submodule, configured to calculate a second time decay coefficient according to the second time information and a preset second time decay function;

[0220] A second attenuation value submodule, configured to multiply the second time attenuation coefficient and the second heat value to obtain a second attenuation value of the second heat value;

[0221] Correspondingly, the first heat submodule is specifically configured to sum the second attenuation value of the second heat value and the first attenuation values ​​corresponding to each access heat value to obtain the first heat value.

[0222] In a possible implementation of the second aspect, the heat category includes warm data and cold data, and the apparatus further includes:

[0223] A first data module is configured to select a first preset proportion of warm data from the data to be classified whose heat type is warm data as first backup data;

[0224] A first verification module is configured to perform erasure coding on the first backup data and the data to be classified whose popularity category is cold data to obtain first verification data;

[0225] The first sending module is configured to send the data to be classified whose popularity category is cold data to a first designated terminal, send the first verification data to a second designated terminal, and delete the data to be classified whose popularity category is cold data stored locally.

[0226] In a possible implementation of the second aspect, the first verification module is specifically configured to:

[0227] The first backup data is warm data with the same data volume as the cold data, and erasure coding is performed on the first backup data and the data to be classified whose popularity category is cold data to obtain first verification data;

[0228] Alternatively, the first backup data is warm data with a different data volume from the cold data. If the data volume of the first backup data is less than that of the cold data, the first backup data is padded with zeros to have the same data volume as the cold data, and then the first backup data padded with zeros and the data to be classified whose heat category is cold data are encoded with error correction and erasure codes to obtain first verification data.

[0229] In a possible implementation of the second aspect, the apparatus further includes:

[0230] The second data module is used to split the warm data other than the first backup data in the data to be classified whose heat type is warm data into the second backup data and the third backup data;

[0231] A second verification module is used to perform erasure coding on the second backup data and the third backup data to obtain second verification data;

[0232] The second sending module is configured to send the second verification data to a third designated terminal, and send the backup data of the second backup data to a fourth designated terminal.

[0233] In a possible implementation of the second aspect, the heat category includes heat data, and the apparatus further includes:

[0234] The third sending module is used to store the to-be-classified data whose popularity category is hot data in a local storage space, and send the backup data of the hot data to a data backup terminal for backup.

[0235] In a possible implementation of the second aspect, the heat category includes temperature data, and the apparatus further includes:

[0236] A third data module is configured to select a second preset proportion of warm data from the data to be classified whose heat category is warm data as fourth backup data;

[0237] a third verification module, configured to perform erasure coding on the fourth backup data and warm data other than the fourth backup data to obtain third verification data;

[0238] The fourth sending module is configured to send the third verification data to the fifth designated terminal and send the fourth backup data to the sixth designated terminal.

[0239] In a possible implementation of the second aspect, the popularity category includes cold data, and the apparatus further includes:

[0240] A fourth data module is used to split the data to be classified whose popularity category is cold data into fifth backup data and sixth backup data;

[0241] a fourth verification module, configured to perform erasure coding on the fifth backup data and the sixth backup data to obtain fourth verification data;

[0242] The fifth sending module is used to send the fifth backup data to the seventh designated terminal, send the sixth backup data to the eighth designated terminal, send the fourth verification data to the ninth designated terminal, and delete the locally stored data to be classified whose heat category is cold data.

[0243] In a possible implementation of the second aspect, the apparatus further includes:

[0244] A length determination module, configured to determine a stripe length of an erasure coded stripe, wherein the stripe length is less than or equal to the number of online devices in the distributed storage system;

[0245] A data reading module is used to obtain the data to be backed up from each device to be backed up. The device to be backed up is an online device that has data to be backed up. The data to be backed up includes data to be classified in a preset popularity category.

[0246] The checksum generation module is used to perform erasure coding based on the stripe length and the data to be backed up to obtain the target checksum data;

[0247] The verification storage module is used to send the target verification data to the verification node for storage.

[0248] In a possible implementation manner of the second aspect, the length determination module is specifically configured to determine the number of online devices as the stripe length of the erasure code stripe.

[0249] In a possible implementation of the second aspect, the length determination module includes:

[0250] The quantity submodule is used to determine the number of devices to be backed up among the online devices;

[0251] The determination submodule is configured to determine the number of devices to be backed up as the stripe length of the erasure code stripe.

[0252] In a possible implementation of the second aspect, the length determination module includes:

[0253] The size submodule is used to obtain the data volume of each data to be backed up;

[0254] a merging submodule, configured to merge the to-be-backed-up data whose data volume is less than a first threshold according to a preset merging rule to obtain the to-be-backed-up data whose data volume is greater than or equal to the first threshold;

[0255] The quantity submodule is configured to determine the quantity of the merged data to be backed up as the stripe length of the erasure code stripe.

[0256] In a possible implementation of the second aspect, the apparatus further includes:

[0257] The depth determination module is configured to determine the data volume of the largest data block among the data blocks divided by each online device as the stripe depth of the erasure code stripe; or, to set the stripe depth of the erasure code stripe to a preset value.

[0258] In a possible implementation of the second aspect, the verification generation module includes:

[0259] The original submodule is used to generate erasure code stripes based on the stripe length and the original data blocks corresponding to each data to be backed up;

[0260] The encoding submodule is used to perform erasure code encoding on the erasure code stripe to obtain target verification data.

[0261] In a possible implementation of the second aspect, the apparatus further includes:

[0262] The stripe processing module is used to pad the erasure code stripe with zeros if the number of original data blocks in the erasure code stripe is less than the stripe length, so that the number of original data blocks in the erasure code stripe is equal to the stripe depth.

[0263] In a possible implementation of the second aspect, the apparatus further includes:

[0264] The block processing module is used to fill the original data blocks with data smaller than the stripe depth with zeros if there are original data blocks with data smaller than the stripe depth in the erasure code stripe, so that the data amount of the original data blocks with data smaller than the stripe depth is equal to the stripe depth.

[0265] In a possible implementation of the second aspect, the check node is a cloud storage device or a network attached storage (NAS) device.

[0266] In a possible implementation of the second aspect, the apparatus further includes:

[0267] A parameter acquisition module is used to obtain device parameters of each online device, including device reliability, device online time, and one or more parameters of device storage space;

[0268] The node selection module is used to select a verification node from each online device according to the device parameters of each online device and a preset verification node selection rule.

[0269] In a possible implementation of the second aspect, the data reading module includes:

[0270] A metadata submodule is used to determine the device to be backed up based on the metadata corresponding to each online device. The metadata is used to indicate whether there is data to be backed up and / or the amount of data to be backed up on the online device;

[0271] The reading submodule is used to read the data to be backed up from each device to be backed up.

[0272] In a possible implementation of the second aspect, the verification generation module includes:

[0273] The original data submodule is used to obtain the original data blocks corresponding to each data to be backed up when the stripe length satisfies the condition that p is a prime number greater than or equal to 3, obtaining N groups of original data blocks, where p is N+1 and N is the stripe length. Each data to be backed up corresponds to a group of original data blocks, and original data blocks in the same group have different sorting positions.

[0274] A first check submodule is configured to select an original data block with the same sorting position from each of the N groups of original data blocks for erasure coding, to obtain first check data blocks corresponding to different sorting positions;

[0275] A second check submodule is configured to select an original data block with a different sorting position from each of the N groups of original data blocks, perform erasure coding, and obtain a second check data block corresponding to each different sorting position;

[0276] Alternatively, N groups of data blocks are selected from N groups of original data blocks and one group of first verification data blocks, and a data block with a different sorting position is selected from each of the N groups of data blocks for erasure coding to obtain second verification data blocks corresponding to each different sorting position.

[0277] In a possible implementation of the second aspect, the first syndrome module includes:

[0278] A first array submodule is used to arrange a group of original data blocks into a column according to the sorting position, with different groups of original data blocks corresponding to different columns, to form a first data block array;

[0279] a row check submodule, configured to perform erasure coding on each row in the first data block array to obtain first check data blocks corresponding to different sorting positions;

[0280] Correspondingly, the second check submodule is specifically configured to select an original data block with a different sorting position from each column of the first data block array to perform erasure coding, thereby obtaining a second check data block corresponding to each different sorting position;

[0281] Alternatively, a group of first check data blocks are arranged into a column according to the sorting position, and the first data block array is expanded according to the column corresponding to the first check data block to obtain a first extended array; N columns are selected from the N+1 columns of the first extended array, and a data block with a different sorting position is selected from each of the N columns for erasure coding to obtain second check data blocks corresponding to each different sorting position.

[0282] In a possible implementation of the second aspect, the second syndrome module is specifically configured to:

[0283] Performing erasure coding on different diagonal lines on the first data block array to obtain second check data blocks corresponding to different sorting positions;

[0284] Alternatively, erasure coding is performed on the first extended array at different diagonal lines to obtain second check data blocks corresponding to different sorting positions.

[0285] In a possible implementation of the second aspect, the first syndrome module includes:

[0286] A second array submodule is used to arrange a group of original data blocks into a row according to the sorting position, with different groups of original data blocks corresponding to different rows, to form a second data block array;

[0287] a column check submodule, configured to perform erasure coding on each column in the second data block array to obtain first check data blocks corresponding to different sorting positions;

[0288] Accordingly, the second syndrome module is specifically configured to:

[0289] Selecting an original data block with a different sorting position from each row of the second data block array to perform erasure coding, to obtain second check data blocks corresponding to the different sorting positions;

[0290] Alternatively, a group of first check data blocks are arranged in a row according to the sorting position, and the second data block array is expanded according to the row corresponding to the first check data block to obtain a second extended array; N rows are selected from the N+1 rows of the second extended array, and a data block with a different sorting position is selected from each of the N rows for erasure coding to obtain second check data blocks corresponding to each different sorting position.

[0291] In a possible implementation of the second aspect, the second syndrome module is specifically configured to:

[0292] Performing encoding on different diagonals on the second data block array to obtain second check data blocks corresponding to different sorting positions;

[0293] Alternatively, different diagonal encodings are performed on the second extended array to obtain second check data blocks corresponding to different sorting positions.

[0294] In a possible implementation manner of the second aspect, the erasure code is encoded as XOR encoding or XOR encoding.

[0295] In a possible implementation of the second aspect, the apparatus further includes:

[0296] The data repair module is used to repair the damaged data block according to the undamaged original data block and the undamaged target verification data when there is a damaged data block, and the target verification data includes a first verification data block and a second verification data block.

[0297] In a possible implementation of the second aspect, the data repair module is specifically used to repair the damaged data blocks based on the undamaged original data blocks and the undamaged target verification data when there are damaged data blocks and the number of groups of damaged data blocks is less than or equal to 2, where the target verification data includes a first verification data block and a second verification data block.

[0298] In a possible implementation of the second aspect, the data repair module includes:

[0299] The first type submodule is used to obtain the type of damaged data blocks when any group of data blocks is damaged;

[0300] a first repair submodule, configured to repair the damaged first check data block according to the N groups of original data blocks if the damaged data block is a first check data block;

[0301] a second repair submodule, configured to repair the damaged second check data block based on the N groups of original data blocks if the damaged data block is a second check data block, or to repair the damaged second check data block based on the N groups of original data blocks and the first check data block;

[0302] The third repair submodule is configured to repair the damaged data block based on the undamaged original data block and the first verification data block if the damaged data block is an original data block, or to repair the damaged data block based on a mixture of the undamaged original data block, the first verification data block, and the second verification data block.

[0303] In a possible implementation of the second aspect, the data repair module includes:

[0304] The second type submodule is used to obtain the type of damaged data blocks when any two groups of data blocks are damaged;

[0305] a fourth repair submodule, configured to repair the damaged first check data block and the damaged second check data block according to the N groups of original data blocks if the damaged data block is a group of first check data blocks and a group of second check data blocks;

[0306] a fifth repair submodule, configured to repair the damaged original data block and the second check data block based on the undamaged original data block and the first check data block if the damaged data block comprises a group of original data blocks and a group of second check data blocks;

[0307] The sixth repair submodule is configured to repair the damaged data block by mixing the undamaged original data block and the second check data block if the damaged data block does not include the second check data block.

[0308] In a third aspect, a terminal device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0309] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the terminal device implements the steps of the above method.

[0310] In a fifth aspect, a chip system is provided. The chip system can be a single chip or a chip module composed of multiple chips. The chip system includes a memory and a processor. The processor executes a computer program stored in the memory to implement the steps of the above method.

[0311] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0312] In the data processing method of the present application, a first heat value is calculated based on the access heat values ​​in each access record of the data to be classified. The first heat value is related to the user's historical access history and can prevent occasional access from polluting the cache. Furthermore, in the process of calculating the first heat value, a first time decay function is introduced. The first decay value of each access heat value is calculated based on the first time decay function, which reduces the cache contamination effect caused by historical data and improves the accuracy of the calculated first heat value. This solves the problem of low accuracy and cache contamination caused by current data classification schemes. BRIEF DESCRIPTION OF THE DRAWINGS

[0313] Figure 1 This is a flow chart of a data classification method provided in an embodiment of the present application;

[0314] Figure 2 1 is a schematic diagram of a time attenuation coefficient curve provided in an embodiment of the present application;

[0315] Figure 3 This is a schematic diagram of an application scenario provided by an embodiment of the present application;

[0316] Figure 4 is a schematic diagram of another application scenario provided by an embodiment of the present application;

[0317] Figure 5 is a schematic diagram of another application scenario provided by an embodiment of the present application;

[0318] Figure 6 is a schematic diagram of another application scenario provided by an embodiment of the present application;

[0319] Figure 7 is a schematic diagram of another application scenario provided by an embodiment of the present application;

[0320] Figure 8 is a schematic diagram of another application scenario provided by an embodiment of the present application;

[0321] Figure 9 is a schematic diagram of another application scenario provided by an embodiment of the present application;

[0322] Figure 10 is a schematic diagram of another application scenario provided by an embodiment of the present application;

[0323] Figure 11 is a schematic diagram of another application scenario provided by an embodiment of the present application;

[0324] Figure 12is a schematic diagram of another application scenario provided by an embodiment of the present application;

[0325] Figure 13 is a schematic diagram of another application scenario provided by an embodiment of the present application;

[0326] Figure 14 This is a schematic diagram of the structure of a distributed storage system provided by an embodiment of the present application;

[0327] Figure 15 This is a flow chart of a data backup method provided in an embodiment of the present application;

[0328] Figure 16 is a schematic block diagram of a terminal device in a distributed storage system provided in an embodiment of the present application;

[0329] Figure 17 is a schematic diagram of another application scenario provided by an embodiment of the present application;

[0330] Figure 18 is a schematic diagram of another application scenario provided by an embodiment of the present application;

[0331] Figure 19 This is a flowchart of an erasure code encoding method provided by an embodiment of the present application;

[0332] Figure 20 is a schematic diagram of another application scenario provided by an embodiment of the present application;

[0333] Figure 21 is a schematic diagram of another application scenario provided by an embodiment of the present application;

[0334] Figure 22 is a schematic diagram of another application scenario provided by an embodiment of the present application;

[0335] Figure 23 is a schematic diagram of another application scenario provided by an embodiment of the present application;

[0336] Figure 24 is a schematic diagram of another application scenario provided by an embodiment of the present application;

[0337] Figure 25 is a schematic diagram of another application scenario provided by an embodiment of the present application;

[0338] Figure 26 is a schematic diagram of another application scenario provided by an embodiment of the present application;

[0339] Figure 27 is a schematic diagram of another application scenario provided by an embodiment of the present application;

[0340] Figure 28is a schematic diagram of another application scenario provided by an embodiment of the present application;

[0341] Figure 29 is a schematic diagram of another application scenario provided by an embodiment of the present application;

[0342] Figure 30 is a schematic diagram of another application scenario provided by an embodiment of the present application;

[0343] Figure 31 is a schematic diagram of another application scenario provided by an embodiment of the present application;

[0344] Figure 32 This is a flow chart of a data recovery method provided in an embodiment of the present application;

[0345] Figure 33 is a schematic diagram of another application scenario provided by an embodiment of the present application;

[0346] Figure 34 is a schematic diagram of another application scenario provided by an embodiment of the present application;

[0347] Figure 35 is a schematic diagram of another application scenario provided by an embodiment of the present application;

[0348] Figure 36 is a schematic diagram of another application scenario provided by an embodiment of the present application;

[0349] Figure 37 is a schematic diagram of another application scenario provided by an embodiment of the present application;

[0350] Figure 38 is a schematic diagram of another application scenario provided by an embodiment of the present application;

[0351] Figure 39 is a schematic diagram of another application scenario provided by an embodiment of the present application;

[0352] Figure 40 is a schematic diagram of another application scenario provided by an embodiment of the present application;

[0353] Figure 41 is a schematic diagram of another application scenario provided by an embodiment of the present application;

[0354] Figure 42 is a schematic diagram of another application scenario provided by an embodiment of the present application;

[0355] Figure 43 is a schematic diagram of a terminal device provided in an embodiment of the present application;

[0356] Figure 44This is a functional structure diagram of a data heat calculation module provided in an embodiment of the present application;

[0357] Figure 45 This is a schematic diagram of another terminal device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0358] In the following description, specific details such as specific system structures and techniques are provided for purposes of illustration rather than limitation to facilitate a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obscuring the description of the present application with unnecessary detail.

[0359] In order to illustrate the technical solution described in this application, specific embodiments are provided below.

[0360] It will be understood that when used in this specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0361] It should also be understood that the terms used in this specification are for the purpose of describing specific embodiments only and are not intended to limit the present application. As used in this specification and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms unless the context clearly indicates otherwise.

[0362] In addition, in the description of the present application, the terms "first", "second", "third", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0363] References to "one embodiment" or "some embodiments" in this specification mean that a particular feature, structure, or characteristic described in conjunction with that embodiment is included in one or more embodiments of the present application. Thus, phrases such as "in one embodiment," "in some embodiments," "in other embodiments," and "in other embodiments" appearing in various places in this specification do not necessarily refer to the same embodiment, but rather mean "one or more but not all embodiments," unless otherwise specifically emphasized. The terms "including," "comprising," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized.

[0364] The data classification method provided in the embodiments of the present application can be applied to terminal devices such as mobile phones, tablet computers, wearable devices, vehicle-mounted devices, augmented reality (AR) / virtual reality (VR) devices, laptop computers, ultra-mobile personal computers (UMPCs), netbooks, and personal digital assistants (PDAs). The embodiments of the present application do not impose any restrictions on the specific types of terminal devices.

[0365] For example, the terminal device can be a station (STAION, ST) in a WLAN, a cellular phone, a cordless phone, a Session Initiation Protocol (SIP) phone, a Wireless Local Loop (WLL) station, a Personal Digital Assistant (PDA) device, a handheld device with wireless communication function, a computing device or other processing device connected to a wireless modem, a vehicle-mounted device, a vehicle networking terminal, a computer, a laptop computer, a handheld communication device, a handheld computing device, a satellite wireless device, a wireless modem card, a TV set-top box (STB), customer premise equipment (CPE) and / or other devices for communicating on a wireless system and a next-generation communication system, such as a mobile terminal in a 5G network or a mobile terminal in a future evolved Public Land Mobile Network (PLMN) network.

[0366] Among the existing data classification schemes, the LRU algorithm and the LFU algorithm are mainly used.

[0367] The LRU algorithm eliminates data based on how recently it was accessed. Its core concept is that if data has been accessed recently, it is more likely to be accessed in the future. Therefore, in the LRU algorithm, when new data is added or existing data is accessed, it is moved to the head of the queue. When data needs to be eliminated, it is removed from the tail of the queue.

[0368] When using the LRU algorithm, occasional or periodic batch operations can cause a sharp drop in the algorithm's accuracy and lead to severe cache pollution. For example, if some infrequently used data is occasionally called, it will immediately be placed ahead of other more frequently used data. When data needs to be eliminated, other frequently used data may be eliminated, while the infrequently used data may be retained.

[0369] The LFU algorithm eliminates data based on its historical access count. Its core concept is that if data has been accessed frequently in the past, it will likely be accessed more frequently in the future. Therefore, the LFU algorithm records the access count of each piece of data. When data needs to be eliminated, it starts with the data with the lowest access count.

[0370] Using the LFU algorithm can avoid cache pollution caused by occasional data calls. However, since the LFU algorithm simply counts the number of data accesses, ancient access records and recent access records have the same validity. Therefore, once the data access pattern changes, the LFU algorithm needs a long time to adapt to the new access pattern. For example, if a user frequently accesses data A within a certain period of time, and the number of accesses to data A is much greater than that to data B, the LFU algorithm will consider data A to be the most popular data. However, after a period of time, the user changes their needs and frequently accesses data B. At this time, since data A has accumulated a large number of accesses in the previous period, the LFU algorithm still considers data A to be the most popular data. Therefore, in the LFU algorithm, there is a cache pollution effect in which historical data affects future data.

[0371] In summary, current data classification schemes simply determine the popularity of data based on the number of historical accesses or recent accesses, which is inaccurate and prone to cache contamination. To address the above issues, the present application provides a data processing method that can include a data classification method, a data backup method, and an erasure code encoding method, as described below.

[0372] The following describes a data classification method provided by an embodiment of the present application from the perspective of a terminal device. Figure 1 , the data classification method in the embodiment of the present application includes:

[0373] S101. Acquire information of data to be classified, the information including all access popularity values ​​of the data to be classified within a preset time period and first time information corresponding to the access popularity values;

[0374] The preset time period is a time period pre-set by the user and can be set according to actual needs. For example, if the current time is October 5, 2019, and the user wants to calculate the first popularity value of each data in the last five days, the user can set the preset time period to October 1, 2019 to October 5, 2019.

[0375] When classifying data to be classified, information about the data to be classified may be obtained first, and the information may include access records of the data to be classified.

[0376] In an embodiment of the present application, a data list may be set to record data access records. The access records may include access popularity values ​​and first time information. The first time information is used to identify the time point or time period corresponding to the access record.

[0377] In some possible implementations, the first time information may be a real standard time. For example, when days are used as the time span, the first time information may be a specific date, such as November 6, 2019.

[0378] In some other possible implementations, the first time information may be the time interval between the recording time of the access record and the current time. For example, the first time information in the access information recorded 30 days ago may be recorded as 30.

[0379] In other possible implementations, the first time information can be recorded using a timer, with a specific day being the starting point, and the first time information being the time interval between the recording time of the access record and the starting point. For example, if the starting point is 30 days ago, the first time information of an access record recorded 25 days ago can be recorded as 5.

[0380] The access heat value is used to indicate whether the data to be classified is accessed at the time point or time period corresponding to the first time information.

[0381] In some possible implementations, if the data to be classified is accessed at the time point or time period corresponding to the first time information, the access heat value Q is accumulated each time the data to be classified is accessed. The access heat value can be expressed as Q*M, where M is the number of times the data to be classified is accessed at the time point or time period corresponding to the first time information. If the data to be classified is not accessed at the time point or time period corresponding to the first time information, the access heat value is recorded as P. Wherein, Q is a first preset value, and P is a second preset value, and the specific values ​​of Q and P can be set according to actual conditions. For example, in some embodiments, Q can be set to 1 and P can be set to 0; if the data to be classified is accessed 5 times at the time point or time period corresponding to the first time information, the access heat value of the access record is 1*5=5; if the data to be classified is not accessed at the time point or time period corresponding to the first time information, the access heat value of the access record is set to 0.

[0382] In some other possible implementations, if the data to be classified is accessed at the time point or time period corresponding to the first time information, the access heat value is recorded as K. Even if the data to be classified is accessed multiple times within the time point or time period corresponding to the first time information, it is only recorded once. If the data to be classified is not accessed at the time point or time period corresponding to the first time information, the access heat value is recorded as L. Here, K is the third preset value, L is the fourth preset value, and the specific values ​​of K and L can be set according to actual conditions. For example, in some embodiments, K can be set to 1 and L can be set to 0; if the data to be classified is accessed at the time point or time period corresponding to the first time information, the access heat value in the access record is set to 1; if the data to be classified is not accessed at the time point or time period corresponding to the first time information, the access heat value in the access record is set to 0.

[0383] In engineering applications, if the first implementation method is used to record access popularity values, the access popularity values ​​can more accurately represent the popularity of the data. However, using the first implementation method to record access popularity values ​​requires constant monitoring of data access, which consumes a large amount of system resources.

[0384] If the second implementation method is used to record access popularity values, even if the data is accessed multiple times at the time point or time period corresponding to the first time information, it will only be recorded once, which will reduce the accuracy of the access popularity value to a certain extent. However, when the second implementation method is used to record access popularity values, once the data is monitored to be accessed, the access popularity value is set to K, and monitoring of the data access situation stops until the time point or time period corresponding to the next access record, which can save a lot of system resources.

[0385] For example, assuming that a certain data to be classified is accessed only once a day; if the first implementation method mentioned above is used to record the access heat value, it is necessary to monitor the data access situation for 24 hours on that day corresponding to the data to be classified; if the second implementation method mentioned above is used to record the access heat value, assuming that the data is monitored to be accessed at 8:23, then for the access record of that day, the access heat value is directly set to K, and there is no need to continue monitoring the access situation of the data for the remaining 15 hours and 37 minutes, saving a lot of system resources.

[0386] For example, with a time span of days and a specific date as the first time information, the access heat value is recorded using the second implementation method described above, and K = 1, L = 0. A file is first written to a terminal device on October 1, 2019, and is accessed on October 2, 2019, and October 5, 2019. The corresponding access records can be shown in Table 1.

[0387] Table 1

[0388] First time information October 1 October 2 October 3 October 4 October 5 Visit popularity value 1 1 0 0 1

[0389] For the same data to be classified, there may be one or more access records. In some possible implementations, when access records for the data to be classified are needed, all access records for the data to be classified may be obtained. In other possible implementations, based on user configuration, only access records for the data to be classified within a preset time period may be obtained. The specific method for obtaining access records can be configured based on actual needs.

[0390] S102: Calculate a first decay value of each access popularity value according to the first time information and a preset first time decay function;

[0391] After obtaining the access records of the data to be classified, a first time attenuation coefficient may be calculated according to the first time information in the access records, and a first attenuation value of each access heat value may be calculated according to the first attenuation coefficient.

[0392] The calculation method of the first time decay coefficient can be set according to actual conditions. For example, in some embodiments, the first time decay function for calculating the first time decay coefficient can be:

[0393] f(x)=T 2 / (x 2 +T 2 ) (1)

[0394] Wherein, T is the fifth preset value, and x represents the time interval between the recording time of a certain access record of the data to be classified and the current time. When the terminal device periodically executes the data classification method of this embodiment, the above-mentioned current time is the time when the terminal device executes the data classification method of this embodiment; when the user wants to calculate the first heat value of the data to be classified in a certain preset time period, the above-mentioned current time is the end time of the preset time period. In the case where there are multiple access records for the same data to be classified, x can be used. i (i=1, 2, 3…) represents the time interval between the recording time of the i-th access record of the data to be classified and the current time (for example, assuming that the access records are as shown in Table 1, the preset time period is from October 1, 2019 to October 5, 2019, then the current time is October 5, 2019, for the first access record on October 1, x1=5-1=4; for the second access record on October 2, x2=5-2=3; for the third access record on October 3, x3=5-3=2; for the fourth access record on October 4, x4=5-4=1; for the fifth access record on October 5, x5=5-5=0), x can be obtained according to the first time information, f() represents the first time attenuation function for calculating the first attenuation coefficient, and f(x) represents the first time attenuation coefficient.

[0395] The specific value of the fifth preset value T in Formula 1 can be set according to actual conditions. For example, if the user wants the first time attenuation coefficient to decay to 0.1 on the 90th day, T can be set to 30.

[0396] It should be noted that the calculation formula for the first attenuation coefficient is only one example of the embodiment of the present application and should not be understood as limiting the calculation method of the first time attenuation coefficient. In other possible implementations, the first time attenuation coefficient can also be calculated by setting a corresponding calculation formula in the form of a linear function, an exponential function, or other function. For example:

[0397] When the first time decay function is an exponential function, the first time decay function can be expressed as:

[0398] f(x)=Be -bx (2)

[0399] Wherein, B and b are both preset constants greater than 0.

[0400] When the first time decay function is a linear function, the first time decay function can be expressed as:

[0401] f(x)=-Cx+c (3)

[0402] Wherein, C and c are both preset constants greater than 0.

[0403] When the first time decay function is an inverse S-shaped curve function, the first time decay function can be expressed as:

[0404]

[0405] Wherein, D, d, and H are all preset constants greater than 0, and h is a preset constant greater than 1.

[0406] When the first time decay function is a hyperbolic function, the first time decay function can be expressed as:

[0407]

[0408] Wherein, E and k are both preset constants greater than 0.

[0409] In addition to the above examples, the first time decay function can also be other forms of decay functions, such as a logarithmic decay function, which are not listed here. The function form and corresponding function parameters of the first time decay function should be set according to actual needs. In this embodiment, the first time decay function is not limited.

[0410] The first decay value of the access heat value can be obtained by multiplying the first time decay coefficient by the access heat value.

[0411] S103, calculating a first popularity value according to the first attenuation value corresponding to each access popularity value;

[0412] After calculating and obtaining the first attenuation value corresponding to each access record corresponding to the data to be classified, the first attenuation values ​​corresponding to each access record may be summed, and the first heat value may be calculated based on the summation result.

[0413] In some possible implementations, the first heat value may be calculated based only on the first decay coefficient and the access heat value.

[0414] For example, in some embodiments, the formula for calculating the first popularity value according to the first time decay coefficient and the access popularity value can be expressed as:

[0415]

[0416] Among them, V represents the first heat value, x i represents the time interval between the i-th access record of the data to be classified and the current time, m is the number of days from the time the data is written to the user device to the current time, and x i It can be calculated by the first time information of the i-th access record, f(x i ) represents the first time attenuation coefficient corresponding to the i-th access record, A irepresents the access heat value in the i-th access record, f(x i )*A i Indicates the first decay value of the access record in the i-th access record.

[0417] For example, taking the access records shown in Table 1 as an example, if the current time is October 5, 2019, then m=5. Then the first heat value of the data to be classified at the current time is:

[0418]

[0419] After simplification, we can get:

[0420] V=f(4)+f(3)+f(0) (8)

[0421] By using the calculation formula (6) of the first heat value, the access heat values ​​corresponding to each access record can be weighted and summed according to the first time attenuation coefficient corresponding to each access record, thereby obtaining the first heat value.

[0422] In some other possible implementations, the first heat value may be calculated based on the second heat value, the second time decay coefficient, and the first time decay coefficient and access heat value corresponding to each access record.

[0423] The second heat value can be a pre-set initial heat value, or the second heat value can also be the first heat value obtained last time, that is, the most recent first heat value. The second time attenuation coefficient is the time attenuation coefficient corresponding to the second heat value. The second time attenuation coefficient can be calculated based on the second time information corresponding to the second heat value, and the second time information is the starting time of the preset time period. For example, if the preset time period is from October 1, 2019 to October 5, 2019, then the second time information is the starting time of the preset time period, that is, October 1, 2019. The second time attenuation coefficient can be calculated using the same calculation formula as the first time attenuation coefficient, or other calculation formulas can be selected as the second time attenuation function for calculation. The specific calculation method of the second time attenuation coefficient can be set according to actual conditions. The second attenuation value can be obtained by multiplying the second time attenuation coefficient by the second heat value.

[0424] In some embodiments, a formula for calculating the first heat value based on the second heat value, the second time decay coefficient, and the first time decay coefficient and access heat value corresponding to each access record can be expressed as:

[0425]

[0426] Among them, l represents the time interval between the recording time of the second heat value and the current time, l can be calculated through the second time information, f(l) represents the second time attenuation coefficient, V2 represents the second heat value, and f(l)*V2 represents the second attenuation value.

[0427] At the same time, since the second heat value is the initial heat value or the first heat value obtained last time, in order to facilitate the calculation of the first heat value next time, after the first heat value is calculated, the second heat value can be updated to the first heat value.

[0428] In the calculation formula of the first heat value shown in formula (9), in addition to the access heat value, the second heat value is also considered. The second heat value is a pre-set initial heat value, or the second heat value can also be the first heat value obtained last time, that is, the most recent first heat value.

[0429] For example, referring to the access records shown in Table 1, if October 1, 2019 is the first entry time of the data, the data is pre-set with an initial heat value of 5 as the second heat value when entering, the current time is October 5, 2019, the preset time period is October 1, 2019 to October 5, 2019, and the second time information is October 1, 2019, then m=5, l=5-1=4, and the first heat value of the data to be classified on October 5, 2019 is:

[0430]

[0431] After simplification, we can get:

[0432] V=f(4)*6+f(3)+f(0) (11)

[0433] After calculating the first heat value, the second heat value can be updated to the first heat value to facilitate the next calculation. In this case, the second heat value is updated to f(4)*6+f(3)+f(0), rather than the initial heat value of 5.

[0434] Assuming that the file is subsequently accessed on October 6, 2019, October 7, 2019, and October 8, 2019, the corresponding access records can be shown in Table 2.

[0435] Table 2

[0436] First time information October 6 October 7 October 8 October 9 October 10 Visit popularity value 1 1 1 0 0

[0437] If the current time is October 10, 2019, the preset time period is October 6, 2019 to October 10, 2019, and the second time information is October 6, 2019, the second heat value is updated to f(4)*6+f(3)+f(0), then m=5, l=10-6=4, and the first heat value of the data to be classified at the current time is:

[0438]

[0439] After simplification, we can get:

[0440] V=f(4) 2 *6+f(3)*f(4)+f(0)*f(4)+f(4)+f(3)+f(2) (13)

[0441] In current data classification methods, such as the LFU algorithm, it is generally assumed that long-term data access records and recent data access records have the same heat utility. This is obviously unreasonable and can easily lead to cache pollution effects where historical data affects future data.

[0442] In the data classification method of this embodiment, the first heat value is calculated based on the access heat values ​​in each access record of the data to be classified. The first heat value is related to the user's historical access situation, which can avoid occasional access from polluting the cache.

[0443] At the same time, the method of this embodiment believes that recent data access records should have higher heat than long-ago data access records. To this end, the method of this embodiment introduces a time attenuation coefficient in the process of calculating the first heat value. The first time attenuation coefficient and the second time attenuation coefficient represent the utility attenuation of the access heat value and the second heat value. Multiplying the first time attenuation coefficient by the corresponding access heat value can obtain the first attenuation value, and multiplying the second time attenuation coefficient by the second heat value can obtain the second attenuation value. The first attenuation value represents the actual heat utility of the access heat value, and the second attenuation value represents the actual heat utility of the second heat value.

[0444] For example, if T in the above formula (1) is set to 30, the function curve corresponding to formula (1) is as follows: Figure 2As shown in the figure. When T is set to 30, the time decay coefficient remains high for the first 15 days, with a time decay coefficient greater than or equal to 0.8. The time decay coefficient drops sharply after 15 days, and after 90 days, it drops to 0.1 and gradually approaches 0. The smaller the time decay coefficient, the more severe the heat utility decay. Multiplying the access heat value and the second heat value by the corresponding time decay coefficient, the older the heat value, the more severe the decay, and the more recent the heat value, the less severe the decay. This greatly reduces the cache contamination effect of historical data on future data and improves the accuracy of the first heat value.

[0445] S104: Determine a heat category of the data to be classified according to the first heat value.

[0446] After the first heat value of the data to be classified is calculated, the heat category of the data to be classified can be determined according to the first heat value and the division method of the heat categories.

[0447] In some possible implementations, a preset heat threshold may be set to classify data with a first heat value greater than the preset heat threshold as hot data, and data with a first heat value less than or equal to the preset heat threshold as cold data.

[0448] In some other possible implementations, a first preset heat threshold and a second preset heat threshold may be set, where the first preset heat threshold is greater than the second preset heat threshold. Data to be classified whose first heat value is greater than the first preset heat threshold is classified as hot data; data to be classified whose first heat value is less than or equal to the first preset heat threshold and greater than the second preset heat threshold is classified as warm data; and data to be classified whose first heat value is less than or equal to the second preset heat threshold is classified as cold data.

[0449] In other possible implementations, more preset heat thresholds may be set, and multiple heat categories may be divided according to each preset heat threshold. The first heat value may be compared with each preset heat threshold to determine the heat category of the data to be classified.

[0450] In some possible implementations, the preset heat thresholds may be fixed values. For example, a user may set the first preset heat threshold to 20 and the second preset heat threshold to 10. In this case, data to be classified with a first heat value greater than 20 is considered hot data; data to be classified with a first heat value less than or equal to 20 and a first heat value greater than 10 is considered warm data; and data to be classified with a first heat value less than 10 is considered cold data.

[0451] In some other possible implementations, the above-mentioned preset heat threshold may not be a fixed value. After calculating the first heat value of each data to be classified, one or some first heat values ​​may be set as the preset heat threshold according to the preset rules. For example, the user may classify the first 20% of the data to be classified with the highest first heat value as hot data, the last 20% of the data to be classified with the lowest first heat value as cold data, and the data to be classified with the first heat value ranked between 20% and 80% as warm data. Then the first preset heat threshold is the first heat value ranked at 20%, and the second preset heat threshold is the first heat value ranked at 80%. The first preset heat threshold and the second preset heat threshold will change with the changes in the first heat values ​​ranked at 20% and 80%.

[0452] In other possible implementations, the heat category of the data to be classified can also be determined based on a proportion, without setting a preset heat threshold. For example, a user can classify the first 20% of the data to be classified with the highest first heat value as hot data, the last 20% of the data to be classified with the lowest first heat value as cold data, and the data to be classified with a first heat value between 20% and 80% as warm data. However, in this case, no preset heat threshold is set, and the hot data, warm data, and cold data are simply selected based on the proportion.

[0453] It should be noted that the terms hot data, warm data, and cold data described above are relative. Hot data represents the popularity category corresponding to the data users use most frequently; cold data represents the popularity category corresponding to the data users use least frequently; and warm data represents the popularity category between hot and cold data. The specific classification of popularity categories can be set based on actual circumstances. In addition to the two categories of hot data and cold data, and the three categories of hot data, warm data, and cold data, more popularity categories can be set as needed.

[0454] After determining the popularity category of the data to be classified, the terminal device may perform a data management operation corresponding to the popularity category on the data to be classified.

[0455] In some embodiments, the terminal device can perform data redundancy backup for the data of the device. In other embodiments, the terminal device can serve as a terminal device in a distributed storage system and perform data redundancy backup together with other terminal devices in the distributed storage system.

[0456] When the terminal device performs data redundancy backup for the data of the terminal device, the terminal device can set corresponding data management operations for data of different popularity categories. After the terminal device determines the popularity category of the data to be classified, the terminal device can perform data management operations corresponding to its popularity category on the data to be classified.

[0457] Among them, users can set one or more terminal devices as data backup terminals. After the terminal device determines the popularity category of the data to be classified, the terminal device can retain the data to be classified that is classified as hot data in the local storage space and send a backup copy of the hot data to the above-mentioned data backup terminal for backup.

[0458] Hot data is data that users use more frequently. Backing up hot data to a data backup terminal can not only improve the fault tolerance of hot data, but also make it easier for users to call hot data on the above data backup terminal.

[0459] by Figure 3 、 Figure 4 and Figure 5 Take the scenario shown in the figure as an example. Figure 3 As shown, the user has terminal device 301 and terminal device 302. Terminal device 301 is a terminal device to which the above data classification method is applied, and terminal device 302 is a data backup terminal designated by the user.

[0460] After the terminal device 301 determines the data set A as hot data using the above data classification method, the data set A may be sent to the terminal device 302 for backup.

[0461] After receiving data set A, terminal device 302 stores data set A in a local memory.

[0462] like Figure 4 As shown, when a user needs to use terminal device 302 for work due to reasons such as a business trip, if they need to access data in dataset A, terminal device 302 can directly access the data from local storage without having to interact with terminal device 301. Even if the data exchange channel between terminal device 301 and notebook 302 is disconnected, it will not affect the user's ability to work with the data in dataset A on terminal device 302.

[0463] like Figure 5 As shown, if the data in dataset A stored in terminal device 301 is lost due to unexpected factors, or the user wants to verify dataset A stored in terminal device 301, the user can ask terminal device 302 to send dataset A to terminal device 301. Terminal device 301 can recover or verify the locally stored dataset A based on the dataset A sent by terminal device 302.

[0464] In summary, sending unclassified data categorized as hot data to a data backup terminal for backup improves the fault tolerance of hot data and enables cross-verification between multiple backups. Furthermore, backing up data in the data backup terminal provides a pre-reading function for the data backup terminal. When a user needs to access this hot data on the data backup terminal, they can directly access the required data from their locally stored data without frequently interacting with the current terminal device.

[0465] When managing data to be classified as warm data and cold data, since the amount of warm data and cold data is large, if the warm data and cold data are backed up to other terminal devices, more storage space of other terminal devices will be occupied.

[0466] Therefore, the terminal device can perform erasure coding on the warm data and the cold data, and perform data redundancy backup on the warm data and the cold data through erasure coding.

[0467] In some embodiments, the terminal device may perform mixed encoding of warm data and cold data. The terminal device selects a first preset ratio of warm data as first backup data, performs erasure coding on the first backup data and the cold data to obtain first verification data. The terminal device may then send the cold data to a first designated terminal and the first verification data to a second designated terminal.

[0468] The remaining warm data can be split into second and third backup data. The terminal device performs erasure coding on the second and third backup data to obtain second verification data. The terminal device can then send the second verification data to the third designated terminal and send the backup data of the second backup data to the fourth designated terminal.

[0469] Afterwards, the terminal device may delete the locally stored first verification data, the second verification data, and the data to be classified whose popularity category is cold data.

[0470] The terminal device performs mixed encoding of warm data and cold data, which can ensure the integrity of the above-mentioned cold data and improve the fault tolerance performance of warm data and cold data.

[0471] The above-mentioned first preset ratio can be set according to actual conditions. For example, in some embodiments, it is assumed that the data to be classified includes 20% hot data, 60% warm data, and 20% cold data. In this case, the first preset ratio can be set to 20%, and 20% of the warm data can be selected from the data to be classified as the first backup data (which can be understood as selecting 1 / 3 of the current 60% warm data), so that the data volume of the first backup data is consistent with the data volume of the cold data, so that the first backup data and the cold data are exclusive OR (XOR) encoded to obtain the first verification data. The other 40% of the warm data is independently erasure coded to obtain the second verification data. Alternatively, the user can also set the first preset ratio to a value such as 10%, 40%, 50%, etc. according to actual conditions. In this case, the data volume of the first backup data is inconsistent with the data volume of the cold data. The terminal device can pad the first backup data with 0, or pad the cold data with 0, so that the data volume of the first backup data and the cold data is consistent, so that the first backup data and the cold data are exclusive OR encoded.

[0472] In some other embodiments, the warm data and the cold data may also be independently erasure coded. In this case, the terminal device may independently perform erasure coding on the warm data to obtain third check data.

[0473] For example, when using XOR encoding, the warm data can be divided into two equal sub-data, each containing 50% of the warm data. The two sub-data are then erasure-coded to obtain the third parity data. Alternatively, if the data amounts of the two sub-data are inconsistent, for example, the first sub-data contains 60% of the warm data and the second sub-data contains 40% of the warm data, the second sub-data should be padded with zeros to make the data amount of the second sub-data consistent with the data amount of the first sub-data. The two sub-data are then erasure-coded to obtain the third parity data. When using Reed-Solomon (RS) encoding, the warm data does not need to be split. The warm data can be directly operated on with a preset generator matrix to obtain the third parity data.

[0474] In this embodiment, the warm data can be stored on the current terminal device, and the third verification data can be sent to other terminal devices. This not only ensures the integrity and accessibility of the warm data on the current terminal device, making it easier for users to read data, but also improves the fault tolerance of the warm data. If the warm data stored on the current terminal device is partially damaged due to unexpected factors, the third verification data can be obtained from other terminal devices and restored based on the third verification data.

[0475] In some embodiments, only the third verification data can be sent to the fifth designated terminal. When the warm data stored on the current terminal device is partially damaged, the damaged warm data can be restored based on the undamaged warm data and the third verification data. However, if all the warm data stored on the current terminal device is lost, the warm data cannot be restored based on the third verification data alone. For example, referring to the previous example, when XOR coding is used, if the first sub-data is damaged, the terminal device can use the second sub-data and the third verification data to restore the first sub-data; if the second sub-data is damaged, the terminal device can use the first sub-data and the third verification data to restore the first sub-data; if both the first sub-data and the second sub-data are damaged, the terminal device cannot restore the first sub-data and the second sub-data based on the third verification data alone. When RS coding is used, the damaged warm data can be restored based on the undamaged warm data and the third verification data through a preset generation matrix.

[0476] In other embodiments, in addition to sending the third verification data to the fifth designated terminal, a second preset ratio of warm data may be selected as the fourth backup data. The second preset ratio may be set based on actual circumstances. For example, if the data to be classified contains 60% warm data, the second preset ratio may be set to 20%, and 20% of the data to be classified may be selected as the fourth backup data (which can be understood as selecting 1 / 3 of the current 60% warm data).

[0477] Afterwards, the terminal device performs erasure coding on the fourth backup data and the warm data other than the fourth backup data to obtain third verification data, and sends the third verification data and the fourth backup data to different designated terminals for storage.

[0478] For example, when using XOR encoding, the terminal device can select the first sub-data or the second sub-data in the above example as the fourth backup data; when using RS encoding, the terminal device can arbitrarily select a preset proportion of warm data as the fourth backup data.

[0479] The terminal device sends the third verification data to the fifth designated terminal and the fourth backup data to the sixth designated terminal. If all the warm data stored on the current terminal device is damaged, the terminal device can obtain the third verification data from the fifth designated terminal and the backup warm data (i.e., the fourth backup data) from the sixth designated terminal, and restore the damaged warm data based on the third verification data and the fourth backup data.

[0480] by Figure 6 、 Figure 7 、 Figure 8 and Figure 9Taking the scenario shown as an example, a user owns terminal device 601 and terminal device 602. Terminal device 601 is a terminal device to which the above data classification method is applied, and terminal device 602 is a fifth designated terminal designated by the user.

[0481] After determining that data set B is classified as warm data using the data classification method, terminal device 601 can perform erasure coding on data set B to obtain third verification data. Terminal device 601 sends the third verification data to terminal device 602.

[0482] When data set B stored on terminal device 601 is partially damaged, terminal device 602 can send the third verification data to terminal device 601. Terminal device 601 restores the damaged data in data set B based on the third verification data and the intact data in data set B, ensuring that the user can continue to access the data in data set B normally.

[0483] In addition, the user owns terminal device 801 and terminal device 802. Terminal device 801 is a terminal device to which the above-mentioned data classification method is applied, terminal device 802 is a fifth designated terminal designated by the user, and terminal device 803 is a sixth designated terminal designated by the user.

[0484] After terminal device 801 determines that dataset B is classified as warm data using the aforementioned data classification method, it can select a portion of data from dataset B as dataset C (fourth backup data) and perform erasure coding on dataset C and the warm data outside of dataset C to obtain third verification data. Terminal device 801 sends the third verification data to terminal device 802 and dataset C to terminal device 803.

[0485] If data set B stored on terminal device 801 is completely damaged, terminal device 802 can send the third verification data to terminal device 801, and terminal device 803 can send data set C to terminal device 801. Terminal device 801 restores data set B based on the third verification data and data set C, ensuring that the user can continue to access data in data set B normally.

[0486] Alternatively, the terminal device 802 may send the third verification data to the terminal device 803, and the terminal device 803 may restore the data set B according to the data set C and the third verification data, and send the data set B to the terminal device 801 after restoration, thereby ensuring that the user can continue to call the data in the data set B normally on the terminal device 801.

[0487] In summary, storing warm data on the current terminal device and sending the third verification data to other terminal devices ensures the integrity and accessibility of the warm data on the current terminal device, making it easier for users to read the data. Furthermore, if the warm data stored on the current terminal device is damaged due to unexpected factors, the third verification data can be retrieved from other terminal devices and restored based on the third verification data, thereby improving the fault tolerance of the warm data.

[0488] In addition, the terminal device may also independently perform erasure coding on the data to be classified whose heat category is cold data to obtain fourth verification data.

[0489] In some embodiments, the cold data can be split into fifth backup data and sixth backup data, and the fifth backup data and the sixth backup data can be erasure coded to obtain fourth verification data. Then, the terminal device can send the fifth backup data to the seventh designated terminal, send the sixth backup data to the eighth designated terminal, send the fourth verification data to the ninth designated terminal, and delete the locally stored data to be classified with a heat category of cold data and the fourth verification data. When cold data needs to be called, the backed-up cold data can be obtained from the seventh designated terminal or the eighth designated terminal. When the fifth backup data or the sixth backup data is damaged, the damaged cold data can be restored based on the fourth verification data and the undamaged fifth backup data or the sixth backup data.

[0490] In other embodiments, the terminal device may also send the cold data in its entirety to a tenth designated terminal, send the fourth verification data to an eleventh designated terminal, and delete the locally stored data to be classified as cold data and the fourth verification data. When cold data is needed, the cold data can be retrieved from the tenth designated terminal. If the cold data in the tenth designated terminal is partially damaged, the damaged cold data can be restored based on the fourth verification data and the undamaged cold data. However, if all the cold data stored in the tenth designated terminal is damaged, restoration cannot be performed based solely on the fourth verification data.

[0491] Since cold data is data that is less frequently used by users, the cold data and the fourth verification data can be sent to other terminal devices for storage, and the cold data and the fourth verification data in the current terminal device can be deleted to save storage space of the current terminal device.

[0492] by Figure 10 、 Figure 11 、 Figure 12 and Figure 13Taking the scenario shown as an example, a user owns terminal device 1001, terminal device 1002, terminal device 1003, and terminal device 1004. Terminal device 1001 is the terminal device to which the above data classification method is applied, terminal device 1002 is the seventh terminal device designated by the user, terminal device 1003 is the eighth terminal device designated by the user, and terminal device 1004 is the ninth terminal device designated by the user.

[0493] After terminal device 1001 determines that dataset D is classified as cold data using the aforementioned data classification method, it can split dataset D into dataset E and dataset F, and perform erasure coding on datasets E and F to obtain fourth verification data. Terminal device 1001 sends the fourth verification data to terminal device 1002, dataset E to terminal device 1003, and dataset F to terminal device 1004.

[0494] When dataset E is completely damaged, dataset E can be restored based on dataset F and the fourth verification data. When dataset F is completely damaged, dataset F can be restored based on dataset E and the fourth verification data.

[0495] The above data restoration process can occur on any terminal device among terminal device 1001, terminal device 1002, terminal device 1003 and terminal device 1004.

[0496] The user owns terminal device 1201 and terminal device 1202. Terminal device 1201 is the terminal device to which the above data classification method is applied, terminal device 1202 is the tenth designated terminal designated by the user, and terminal device 1203 is the eleventh designated terminal designated by the user.

[0497] After terminal device 1201 determines that dataset D is classified as cold data using the aforementioned data classification method, it can perform erasure coding on dataset D to obtain fourth verification data. Terminal device 1201 sends dataset D to terminal device 1202, sends the fourth verification data to terminal device 1203, and deletes dataset D and the fourth verification data stored on terminal device 1201, thereby conserving storage space on terminal device 1201.

[0498] When a user needs to access data from dataset D on terminal device 1201, terminal device 1201 can obtain the data from dataset D from terminal device 1202. If dataset D is damaged while stored on terminal device 1202, terminal device 1202 can restore dataset D based on the fourth verification data stored on terminal device 1203 and then send the data accessed by terminal device 1201 to terminal device 1201. Alternatively, terminal device 1202 can send dataset D to terminal device 1201, and terminal device 1203 can send the fourth verification data to terminal device 1201. Terminal device 1201 can then restore dataset D based on the fourth verification data and access the user's desired data from the restored dataset D.

[0499] In summary, storing the cold data and the fourth verification data on other terminal devices can save storage space on the current terminal device. Furthermore, if the cold data is damaged, it can be restored based on the fourth verification data, thereby improving the fault tolerance of the cold data.

[0500] At the same time, the data backup terminal and the first to eleventh designated terminals mentioned above can be set according to actual conditions. Users can apply some or all of the terminal devices according to actual needs. In the process of setting the above-mentioned terminal devices, users can set the data backup terminal and the first to eleventh designated terminals as different terminal devices. Alternatively, users can also set some designated terminal devices with similar functions to the same terminal device. For example, the functions of the second designated terminal, the fourth designated terminal, the fifth designated terminal, and the ninth designated terminal are similar. The user can set the second designated terminal, the fourth designated terminal, the fifth designated terminal, and the ninth designated terminal as two or more different terminal devices, or the user can also set a certain terminal device as the second designated terminal, the fourth designated terminal, the fifth designated terminal, and the ninth designated terminal at the same time.

[0501] When the above-mentioned terminal device performs data redundancy backup together with other terminal devices in the distributed storage system, each terminal device in the distributed storage system can serve as an encoding node or a storage node, and the encoding nodes in the distributed storage system can use the following data backup method to perform data redundancy backup.

[0502] First, a distributed storage system to which the data backup method provided in the embodiment of the present application is applicable is introduced as an example. Figure 14 As shown, if there are terminal devices 1401, 1402, 1403, 1404, and 1405, and these terminal devices can communicate and form a network, then these terminal devices can form a distributed storage system 1400.

[0503] In the distributed storage system 1400, terminal device 1401, terminal device 1402, terminal device 1403, terminal device 1404, and terminal device 1405 can serve as storage nodes to store their respective data, and then use erasure code fault tolerance technology to perform data redundancy backup on the data in these terminal devices, generate verification data, and store it.

[0504] When a terminal device in the distributed storage system 1400 fails, such as being lost or damaged, the distributed storage system 1400 can use the original data and verification data stored in other terminal devices to restore the data in the failed device, thereby ensuring that each terminal device in the distributed storage system 1400 can provide reliable data services to users.

[0505] In the distributed storage system 1400, each terminal device may be user equipment (UE), an access terminal, a terminal, a subscriber unit, a subscriber station, a mobile station, a mobile station, a remote station, a remote terminal, a mobile device, a user terminal, a wireless network device, a user agent, or a user apparatus. The terminal device may also be a cellular phone, a cordless phone, a Session Initiation Protocol (SIP) phone, a smart phone, a wireless local loop (WLL) station, a personal digital assistant (PDA), a handheld device with wireless communication capabilities, a computing device or other device connected to a wireless modem, an in-vehicle device, a wearable device, an unmanned aerial vehicle device, or a terminal in the Internet of Things (IoT), the Internet of Vehicles (IoV), or any other terminal in future networks, a relay user device, or a terminal device in a future-evolved public land mobile network (PLMN). The embodiments of the present application do not impose any restrictions on the type of terminal device.

[0506] Furthermore, in the distributed storage system 1400, the terminal devices may communicate with each other via wired communication and / or wireless communication. Wireless communication includes long-distance wireless communication and short-distance wireless communication, and may include microwave communication, satellite communication, wireless fidelity (Wi-Fi), Bluetooth, ZigBee, infrared, and near-field communication (NFC).

[0507] In addition, in the distributed storage system 1400, terminal devices can be divided into storage nodes, encoding nodes, and verification nodes according to the functions implemented by the terminal devices during the redundant backup process. The relevant descriptions of each type of node are as follows:

[0508] Storage nodes are used to store raw data, which can include both backed-up data and data to be backed up. For example, each terminal device can store its own photos, videos, audio, documents, and other files, as well as its own application data and device parameters. Within a storage node, data that has already been backed up is considered backed-up data, while data that requires redundant backup is considered to be backed-up data. Furthermore, storage nodes may also contain non-backed-up data that has not yet been redundantly backed up and does not require redundant backup.

[0509] It should be noted that each terminal device in the distributed storage system 1400 can serve as an independent storage node.

[0510] The encoding node has computing capabilities and can apply the data backup method provided in the embodiments of the present application to encode the data to be backed up on each terminal device to generate verification data. The encoding node can be one of the multiple terminal devices in the distributed storage system 1400. The encoding node can be specified by the user or defaulted to a device with stronger computing capabilities.

[0511] When performing data redundancy backup, the encoding node can read the data to be backed up from each storage node. The data to be backed up is data that needs to be backed up or requires fault tolerance. The encoding node can then use a preset erasure coding algorithm to calculate verification data for the data to be backed up and send the verification data to the verification node for backup.

[0512] In some embodiments, the coding node itself can also serve as a storage node to store data. When the coding node has data that needs to be backed up, the coding node can perform encoding calculations on its own data to be backed up and the data to be backed up on other devices.

[0513] In other embodiments, the encoding node may not be a storage node. In this case, the encoding node obtains the data to be backed up on other devices to perform encoding calculations.

[0514] The check nodes are used to store the check data generated by the encoding nodes. The check data is used to restore the original data on the failed device when a terminal device in the distributed storage system 1400 fails.

[0515] In some embodiments, the check node can be specified by the user. For example, the user can specify a terminal device that implements the above data classification method as an encoding node.

[0516] In other embodiments, the check node may be assumed by default by the terminal device with the highest priority. The priority of the terminal device may be determined based on one or more conditions including device reliability, online time, and storage space. The encoding node then determines the check node from the multiple terminal devices in the distributed storage system 1400 based on the aforementioned priorities.

[0517] The device reliability can be determined based on the ease of loss and / or damage of the terminal device. The easier the terminal device is to be lost or the easier it is to be damaged, the lower the device reliability of the terminal device.

[0518] For example, the distributed storage system 1400 may select terminal devices with high reliability (eg, not easily lost or damaged), long online time, and sufficient storage space as verification nodes.

[0519] In other embodiments, the check node may also be a terminal device in the distributed storage system 1400 specifically used to store check data. Figure 14 As shown, the distributed storage system 1400 may further include a cloud or storage device 1406, which is specifically used to store the verification data generated by the encoding node, thereby improving the security and reliability of the stored verification data.

[0520] Exemplary cloud or storage devices 1406 may include cloud servers, cloud storage devices, and network attached storage (NAS). NAS refers to a device connected to a network that stores data and is also known as "network storage." NAS is a dedicated data storage server that centrally manages data, freeing up bandwidth, improving performance, reducing total cost of ownership, and protecting investments.

[0521] In summary, a check node can be at least one of the multiple terminal devices included in the distributed storage system. A check node can be user-specified, or the check node can be assumed by default by the terminal device with the highest priority. Alternatively, a check node can be a terminal device specifically used to store check data.

[0522] For ease of understanding, the following is still based on Figure 14 Take the following example to illustrate. Figure 14 In the figure, solid lines represent the transmission of original data, and dashed lines represent the transmission of verification data. Distributed storage system 1400 includes multiple terminal devices, such as terminal device 1401, terminal device 1402, terminal device 1403, terminal device 1404, and terminal device 1405. Terminal devices 1402, 1403, 1404, and 1405 all have data to be backed up.

[0523] At this time, in the embodiment of the present application, terminal device 1401 can be selected as the encoding node. When performing redundant backup, terminal device 1401 obtains the data to be backed up from other terminal devices (i.e., terminal device 1402, terminal device 1403, terminal device 1404, and terminal device 1405), and uses a preset erasure coding algorithm to encode the backup data to generate verification data.

[0524] Afterwards, terminal device 1401 may store the verification data locally, or terminal device 1401 may send the verification data to other terminal devices in distributed storage system 1400 for storage. For example, the terminal device may send the verification data to terminal device 1402 and / or terminal device 1403 for storage.

[0525] Optionally, distributed storage system 1400 may further include a cloud or storage device 1406. Cloud or storage device 1406 is specifically used to store verification data. After generating verification data, terminal device 1401 may send the verification data to cloud or storage device 1406 for storage. In this case, during this data redundancy backup process, terminal device 1401 serves as an encoding node, terminal devices 1402, 1403, 1404, and 1405 serve as storage nodes, and cloud or storage device 1406 serves as a verification node.

[0526] It should be noted that the above description of the distributed storage system 1400 is only an illustrative example of the embodiment of the present application. In the actual application process, the distributed storage system 1400 may have more Figure 14 The five terminal devices shown in FIG. 14 may include more or fewer terminal devices. For example, the distributed storage system 1400 may have two terminal devices, three terminal devices, four terminal devices, six terminal devices, eight terminal devices, etc.

[0527] It should also be understood that the multiple terminal devices included in the distributed storage system 1400 can be terminal devices of the same type or terminal devices of different types. The embodiments of the present application do not impose any restrictions on the types of terminal devices included in the distributed storage system 1400.

[0528] In addition, multiple terminal devices in a distributed storage system can belong to the same user, or they can belong to different users, as long as the above-mentioned multiple terminal devices can be networked and communicate with each other through certain communication methods (such as Bluetooth connection, WI-FI connection, logging in to the same account, etc.).

[0529] also, Figure 14The terminal devices in the distributed storage system 1400 shown in FIG are all in an online state. In some other embodiments, the distributed storage system 1400 may also include terminal devices that are not online (i.e., offline devices). When the offline devices come online, these offline devices may also participate in the above-mentioned data redundancy backup process.

[0530] See also Figure 15 , Figure 15 A flow chart of a data backup method provided in an embodiment of the present application is shown. Figure 15 The data backup method shown can be applied to a distributed storage system. For example, the data backup method can be applied to Figure 14 The distributed storage system 1400 is shown.

[0531] The distributed storage system in the embodiment of the present application includes multiple terminal devices that can communicate with each other. The above-mentioned data backup method can be executed by any terminal device with computing capabilities among the multiple terminal devices. The terminal device that executes the above-mentioned data backup method can be called an encoding node.

[0532] It should be noted that the multiple terminal devices in a distributed storage system that can communicate with each other can be understood as having the ability to communicate with other terminal devices. However, whether terminal devices can communicate with each other depends on the device status of the terminal devices. This device status includes online and offline status.

[0533] When a terminal device in a distributed storage system is in an online state, the terminal device (ie, the online device) can communicate with other online devices in the distributed storage system.

[0534] When a terminal device in a distributed storage system is offline, the terminal device (ie, offline device) cannot communicate with other terminal devices in the distributed storage system.

[0535] The following will Figure 15 The data backup method shown is introduced in detail:

[0536] S1501: The coding node determines the stripe length.

[0537] In the embodiment of the present application, multiple terminal devices that can communicate with each other can be networked to form a distributed storage system. The above encoding node is a terminal device in the online device of the distributed storage system, and the encoding node is used to execute the data backup method provided in the embodiment of the present application.

[0538] The encoding node is a terminal device with computing power. As mentioned above, the encoding node can be specified by the user or can be assumed to be a device with stronger computing power.

[0539] In some possible implementations, the encoding node can be specified by the user. For example, when multiple terminal devices are networked to form a distributed storage system, the user's currently used terminal device can display a selection interface through an application. The selection interface is used to display multiple terminal devices in the distributed storage system, and the user can select the encoding node to execute the above-mentioned data backup method based on the multiple terminal devices displayed on the selection interface.

[0540] In other possible implementations, the encoding node may be assumed by default to be a device with greater computing power. For example, when multiple terminal devices are networked to form a distributed storage system, the terminal devices communicate with each other to negotiate and, based on preset rules, select the terminal device with the greatest computing resources as the encoding node. Alternatively, a terminal device with computing resources greater than a preset computing resource threshold may be selected as the encoding node, or a terminal device of a preset type (e.g., a smartphone) may be selected as the encoding node.

[0541] Furthermore, in some embodiments, the encoding node can be fixed. For example, if a terminal device is always online (or has the longest online time) and has strong computing capabilities, then this terminal device can be set as a fixed encoding node and selected as the encoding node each time data redundancy backup is performed.

[0542] In other embodiments, the encoding node may also be determined each time data redundancy backup is performed based on parameters such as computing resources, online time, and storage resources of each online device in the distributed storage system.

[0543] In an embodiment of the present application, the encoding node can combine striping technology and erasure code technology to perform data redundancy backup on the data to be backed up in the distributed storage system. The data redundancy backup process is the process of the encoding node using a preset erasure code algorithm to encode the backup data by erasure code.

[0544] Striping technology involves dividing a continuous block of data into many small parts and storing them on different physical nodes such as disks, hard drives, servers, or other geographical locations. This balances the input / output (I / O) load across multiple physical nodes, thus avoiding conflicts caused by multiple processes accessing different parts of the data at the same time.

[0545] Taking disk systems as an example, striping (also called data striping) is the process of dividing continuous data into blocks of equal size, writing each piece of data to different disks in the array. This allows multiple processes to access different parts of the data simultaneously without causing disk conflicts. A stripe spans all data disks, with each disk storing a portion of the stripe, and the data stored on all disks together forming a complete copy. The stripe length refers to the number of stripes that can be read or written concurrently, or the number of sectors or blocks contained in a stripe, and can be equal to the number of disks spanned by the stripe. For example, a striped array with five physical hard drives has a stripe length of 5. The stripe depth refers to the size of the stripe data block written on each disk.

[0546] When an encoding node performs erasure coding, a stripe refers to an erasure code stripe, and the erasure code stripe includes multiple original data blocks participating in the erasure coding.

[0547] The stripe length refers to the number of original data blocks in an erasure-coded stripe. The stripe depth refers to the amount of data in the original data blocks in an erasure-coded stripe.

[0548] When performing erasure coding, the encoding node may first determine the stripe length and the stripe depth of the erasure code stripe.

[0549] There are multiple ways for a coding node to determine the stripe length of an erasure code stripe. The coding node can select an appropriate method to determine the stripe length based on the needs of the actual scenario.

[0550] In some possible implementations, the encoding node may determine the stripe length of the erasure code stripe based on the number of online devices in the distributed storage system, and the stripe length of the erasure code stripe is equal to the number of online devices in the distributed storage system.

[0551] Because only online devices can participate in data redundancy in a distributed storage system, the encoding node can determine the number of online devices as the stripe length of the erasure code stripe. In this case, each online device in the distributed storage system can act as an independent storage node to store its own data, meaning each online device stores a portion of the erasure code stripe.

[0552] In some other possible implementations, the encoding node may determine the stripe length of the erasure code stripe according to the number of devices to be backed up in the distributed storage system, and the length of the erasure code stripe is equal to the number of devices to be backed up in the distributed storage system.

[0553] The aforementioned device to be backed up is an online device in the distributed storage system, and is a terminal device that contains data to be backed up.

[0554] Among online devices, only some may have data requiring backup (i.e., data to be backed up), while other devices may have no data to back up or zero data to back up. Therefore, the encoding node can use the number of devices to be encoded as the stripe length of the erasure code stripe. In this case, an erasure code stripe consists of the data to be backed up on the devices to be backed up. Each device to be backed up stores a portion of the erasure code stripe, and the amount of data to be backed up stored on each device to be backed up is greater than zero.

[0555] When determining the number of devices to be backed up, the encoding node may determine whether the online device has data to be backed up and / or the amount of data to be backed up based on metadata of each online device.

[0556] In an embodiment of the present application, each terminal device in the distributed storage system may include metadata of other terminal devices, where the metadata indicates whether other terminal devices have data to be backed up and / or the amount of data to be backed up.

[0557] Metadata, also known as intermediary data or relay data, is data about data and is descriptive information about data and information resources. Metadata is an electronic catalog that primarily describes data properties and supports functions such as indicating storage location, historical data, resource retrieval, and file logging.

[0558] Therefore, when the data information of a terminal device in the distributed storage system changes, the terminal device can notify other terminal devices in the distributed storage system of the content of the data information change. Other terminal devices can obtain the corresponding content through the metadata corresponding to the terminal device, such as changes in data storage location, file size changes, file creation, update, deletion, etc., which makes it easier for other terminal devices to determine whether there is data to be backed up on the terminal device and / or the amount of data to be backed up based on the metadata.

[0559] The encoding node determines whether the online device has data to be backed up and / or the amount of data to be backed up based on the metadata of each online device, and determines the terminal device with data to be backed up and the amount of data to be backed up is not 0 as data to be backed up, thereby determining the number of devices to be backed up.

[0560] In some other possible implementations, the encoding node may determine the stripe length of the erasure code stripe according to the data volume of each data to be backed up in the distributed storage system.

[0561] If the amount of data to be backed up of each device to be backed up is greater than the first threshold, the stripe length of the erasure code stripe is equal to the number of devices to be backed up.

[0562] If the amount of data to be backed up of one or more devices among the data to be backed up is less than or equal to the first threshold, the length of the erasure code stripe is less than the amount of data to be backed up.

[0563] At this time, the encoding node may merge the one or more to-be-backed-up data with a data volume less than or equal to the first threshold value to obtain one or more to-be-backed-up data with a data volume greater than the first threshold value.

[0564] For example, assume that the amount of data to be backed up of terminal device A, the data to be backed up of terminal device B, the data to be backed up of terminal device C, and the data to be backed up of terminal device D are all less than a first threshold. The encoding node may merge the data to be backed up of terminal device A and the data to be backed up of terminal device B into data to be backed up E, and merge the data to be backed up of terminal device C and the data to be backed up of terminal device D into data to be backed up F. The amount of data to be backed up E and data to be backed up F is greater than the first threshold.

[0565] Alternatively, the encoding node may also merge the one or more to-be-backed-up data with a data volume smaller than the first threshold with other to-be-backed-up data with a data volume larger than the first threshold to obtain one or more to-be-backed-up data with a data volume larger than the first threshold.

[0566] For example, assuming that the amount of data to be backed up of terminal device A is less than a first threshold, and the amount of data to be backed up of terminal device B is greater than the first threshold, the encoding node may merge the data to be backed up of terminal device A and the data to be backed up of terminal device B into data to be backed up C. The amount of data to be backed up C is greater than the first threshold.

[0567] The first threshold can be set according to actual conditions. For example, the first threshold can be set to one-half, one-third, three-fifths, etc. of the stripe depth of the erasure coded stripe, or can be set to two times, three times, etc. of the stripe depth of the erasure coded stripe.

[0568] For example, if the first threshold is equal to the stripe depth of the erasure coded stripe, and the amount of data to be backed up on each device to be backed up is not significantly different from one another, or is not significantly different from the stripe depth of the erasure coded stripe, the encoding node can treat each device to be backed up as an independent storage node to store the data to be backed up, and the data to be backed up corresponding to each device to be backed up can be independently encoded with the erasure code.

[0569] If the amount of data to be backed up on some devices to be backed up is significantly different from that of other data to be backed up, or the stripe depth of the erasure code stripe is significantly different, the encoding node can merge some of the data to be backed up into one or more data to be backed up to participate in erasure code encoding, thereby saving the computing resources of the encoding node.

[0570] Through the above method, the encoding node can merge at least two to-be-backed-up data into one to-be-backed-up data to participate in the data redundancy backup (ie, erasure coding) process.

[0571] In some other possible implementations, the encoding node may determine the stripe length of the erasure code stripe according to the type of a preset erasure code algorithm.

[0572] When encoding nodes use different erasure coding algorithms, the stripe length of the erasure coded stripe may need to meet different constraints. Therefore, when determining the stripe length of the erasure coded stripe, the encoding device can determine the corresponding constraints based on the selected erasure coding algorithm. The encoding device then determines the length of the erasure coded stripe based on these constraints and the number of online devices.

[0573] For example, when the preset erasure coding algorithm is the RS-encoded erasure coding algorithm, the stripe length must meet the constraints corresponding to the RS-encoded erasure coding algorithm. When the preset erasure coding algorithm is the XOR-encoded erasure coding algorithm, the stripe length must meet the constraints corresponding to the XOR-encoded erasure coding algorithm.

[0574] In addition to determining the stripe length of an erasure coded stripe, an encoding node can also determine the stripe depth of an erasure coded stripe.

[0575] In some possible implementations, some or all of the online devices in a distributed storage system may divide their stored data into blocks and store them in the form of data blocks. When different terminal devices divide the data blocks, the data size of the divided data blocks may be the same or different.

[0576] At this time, the encoding node may obtain the data volume of the data blocks in the data block partitioning manner of each storage node, and determine the data volume of the largest data block as the stripe depth of the erasure code stripe.

[0577] In some other possible implementations, the encoding node may also set the stripe depth of the erasure code stripe to a preset value.

[0578] For example, the encoding node can pre-set the stripe depth of the erasure code stripe to 1MB, 2MB, 5MB, 10MB, etc.

[0579] It should be noted that the smaller the stripe depth of the erasure code stripe, the smaller the data granularity of the erasure code stripe, the smaller the blocks, and the stronger the processing capability of the encoding node.

[0580] S1502: The encoding node obtains the data to be backed up.

[0581] As described above, the data to be backed up is stored on the corresponding device to be backed up. The encoding node can determine whether the data to be backed up exists on each online device and / or the amount of the data to be backed up based on the metadata of other devices stored locally.

[0582] Therefore, the encoding node can determine the devices to be backed up according to the metadata, and read the corresponding data to be backed up from each device to be backed up.

[0583] In some embodiments, each terminal device in the online device has data to be backed up. In this case, the encoding node can read the data to be backed up corresponding to each online device from each online device.

[0584] In other embodiments, some of the terminal devices (i.e., the devices to be backed up) in the online device have data to be backed up, while the other terminal devices do not have data to be backed up. In this case, the encoding node can read the data to be backed up corresponding to each device to be backed up from the device to be backed up.

[0585] After obtaining the data to be backed up, the encoding node may read the obtained data to be backed up into a cache so as to perform subsequent processing on the data to be backed up.

[0586] In the embodiment of the present application, there is no restriction on the content and type of the data to be backed up.

[0587] In some embodiments, the data to be backed up may be the data that needs to be backed up (eg, warm data and cold data) determined according to the heat category after the terminal device executes the above data classification method and determines the heat category of the data to be classified.

[0588] In other embodiments, the data to be backed up may include updated data, which is data obtained after the backed up data is modified by the user.

[0589] That is to say, after the user modifies the backed up data, the storage node can use the modified backed up data as new data to be backed up and perform data redundancy backup again.

[0590] In some other embodiments, the data to be backed up may include reconstructed data, which refers to other backed up data on the same erasure code stripe as the backed up data deleted by the user.

[0591] That is to say, when a user deletes the backed-up data of a storage node, other storage nodes can use the other backed-up data in the same erasure code stripe as the deleted backed-up data as the data to be backed up and perform redundancy backup again.

[0592] These reconstructed data can still be on the same erasure code stripe or on different erasure code stripes when performing data redundancy backup.

[0593] In other embodiments, if the user modifies the backed up data, the modified data (i.e., updated data) and other backed up data in the same erasure code stripe as the modified backed up data can be used as new data to be backed up and redundantly backed up again.

[0594] That is to say, if the user modifies the backed up data and does not store the modified data as a new copy, the verification node needs to delete the previously stored verification data related to the modified backed up data, and use the updated data and other backed up data in the same erasure code stripe as the modified backed up data as the new data to be backed up, and re-back up the redundant data.

[0595] In other embodiments, the data to be backed up may include re-encoded data. When one or more storage nodes are offline (i.e., offline devices), other terminal devices may use other backed-up data in the same erasure code stripe as the backed-up data stored on the offline device as the re-encoded data.

[0596] That is to say, when a terminal device exits the distributed storage system, the other backed-up data in the erasure code stripe corresponding to the backed-up data on the terminal device can be used as re-encoded data for redundancy backup.

[0597] S1503: The encoding node performs erasure coding on the backup data according to the stripe length of the erasure code stripe to obtain verification data.

[0598] After determining the stripe length of the erasure coded stripe and obtaining the data to be backed up, the encoding node can perform erasure coding on the data to be backed up based on the stripe length to obtain verification data. Furthermore, the encoding node can define the verification data obtained by performing redundant data backup on multiple terminal devices in the distributed storage system as target verification data.

[0599] Specifically, the encoding node can generate one or more erasure code stripes based on the stripe length. Each erasure code stripe contains the same number of original data blocks as the stripe length, each original data block comes from the backup data of a different backup device, and the data volume of each original data block is equal to the stripe depth.

[0600] Then, the encoding node performs erasure coding on each erasure code stripe using a preset erasure code algorithm to obtain verification data corresponding to each erasure code stripe.

[0601] In some possible implementations, some or all of the devices to be backed up may block their respective stored data to be backed up and store them in the form of data blocks. The encoding node obtains the data to be backed up as original data blocks that have been block-processed.

[0602] At this time, the encoding node can select original data blocks corresponding to each data to be backed up with the same number of stripe lengths to form an erasure code stripe. Each original data block in the erasure code stripe comes from the data to be backed up on different devices to be backed up.

[0603] Since different devices to be backed up may divide data blocks according to different data volumes, it is possible that the data volume of some or all original data blocks is smaller than the aforementioned stripe depth.

[0604] For example, some devices to be backed up may divide data blocks according to the data volume of 0.5MB, some devices to be backed up may divide data blocks according to the data volume of 0.7MB, and some devices to be backed up may divide data blocks according to the data volume of 1MB, resulting in the data volume of the original data blocks corresponding to some or all of the data to be backed up being less than the stripe depth.

[0605] When the data volume of the original data blocks in the erasure code stripe is less than the above stripe depth, the encoding node can align the original data blocks whose data volume is less than the stripe depth so that the data volume of these original data blocks is padded to the stripe depth.

[0606] Furthermore, the number of original data blocks corresponding to each piece of data to be backed up may not be the same. Therefore, some erasure coded stripes may have fewer original data blocks than the stripe length. If an erasure coded stripe has fewer original data blocks than the stripe length, the encoding node can pad the stripe with zeros, using data blocks containing all zeros as original data blocks, so that the number of original data blocks in the stripe equals the stripe length.

[0607] For example, assume that the number of original data blocks corresponding to data A to be backed up is 3, the number of original data blocks corresponding to data B to be backed up is 2, and the number of original data blocks corresponding to data C to be backed up is 3, and the stripe length is 3. The encoding node assembles three erasure coded stripes based on data A, B, and C to be backed up. The third erasure coded stripe contains only one original data block corresponding to data A to be backed up and one original data block corresponding to data C to be backed up. The number of original data blocks in the third erasure coded stripe is 2, which is less than the stripe length of 3. In this case, the encoding node can pad the third erasure coded stripe with zeros so that the number of original data blocks in the third erasure coded stripe equals the stripe length.

[0608] In some other possible implementations, the data to be backed up obtained by the encoding node is data that has not been segmented.

[0609] At this time, the encoding node may perform block processing on the data to be backed up according to the stripe depth to obtain original data blocks corresponding to each piece of data to be backed up.

[0610] Afterwards, the encoding node can select original data blocks corresponding to each to-be-backed-up data with the same number of stripe lengths to form an erasure code stripe. Each original data block in the erasure code stripe comes from the to-be-backed-up data of a different to-be-backed-up device.

[0611] Since the amount of data to be backed up is not necessarily an integer multiple of the stripe depth, the amount of data in some original data blocks may be smaller than the stripe depth.

[0612] For example, assuming that the data size of a certain data to be backed up is 2.3MB and the stripe depth is 1MB, the data to be backed up can be divided into 3 original data blocks, and the data size of the third original data block is 0.3MB, which is less than the stripe depth.

[0613] Therefore, if there is an erasure code stripe with a number of original data blocks less than the stripe length, the encoding node can align the original data blocks with a data volume less than the stripe depth so that the data volume of these original data blocks is padded to the stripe depth.

[0614] Furthermore, since the amount of data to be backed up may not be the same for each data block, the number of original data blocks obtained by dividing each data block to be backed up may not be the same. Therefore, the number of original data blocks in some erasure coded stripes may be less than the stripe length.

[0615] If there is an erasure code stripe in which the number of original data blocks is less than the stripe length, the encoding node may pad the erasure code stripe with zeros so that the number of original data blocks of the erasure code stripe is equal to the stripe length.

[0616] The above alignment processing methods can be selected according to actual conditions. In the embodiments of the present application, the following alignment processing methods will be exemplified:

[0617] In some embodiments, when the data size of the original data block is less than the slice depth, the encoding node may pad the original data block with zeros so that the data size of the original data block is equal to the slice depth.

[0618] In other embodiments, when there are one or more original data blocks whose data volume is less than the stripe depth and these original data blocks come from different data to be backed up, the encoding node can merge these original data blocks into one or more original data blocks whose data volume is equal to the stripe depth.

[0619] In other embodiments, when there are multiple data to be backed up from different devices to be backed up that are smaller than the above-mentioned first threshold, the encoding node can merge these data to be backed up to obtain one or more data to be backed up that are larger than the above-mentioned first threshold, and then stripe the merged data to be backed up and participate in erasure code encoding.

[0620] When encoding an erasure code stripe, the encoding node can select a suitable erasure code algorithm based on the actual situation.

[0621] For example, in some embodiments, encoding nodes may employ an RS-encoded erasure coding algorithm for encoding. This can tolerate data loss across multiple storage nodes, improving user data reliability. In other embodiments, encoding nodes may employ an XOR-encoded erasure coding algorithm for encoding. This can reduce the difficulty of encoding and decoding and improve decoding accuracy.

[0622] After encoding the erasure coded stripe, the encoding node can obtain the verification data. The encoding node can then send the verification data to the verification node for storage.

[0623] The verification node can be a cloud storage device, a network attached storage NAS device, or one or more terminal devices among the above online devices.

[0624] In some possible embodiments, a storage device specifically used to store verification data is provided in the distributed storage system. The storage device may be a cloud server, a cloud storage, a network attached storage (NAS) device, or the like.

[0625] At this time, after obtaining the verification data, the encoding node may send the verification data to the above-mentioned storage device specifically used for storing verification data for storage.

[0626] In some other possible embodiments, no storage device specifically used for storing verification data is provided in the distributed storage system.

[0627] At this time, the encoding node may select one or more terminal devices from the online devices as verification nodes, and send the verification data to the verification nodes for storage.

[0628] When selecting a check node, to improve data storage reliability, you can choose a terminal device that does not participate in data redundancy backup as a check node, or you can choose a device to be backed up as a check node. However, when the check node is the device to be backed up, the original data on the device to be backed up that participates in data redundancy backup should be transferred to other storage nodes to avoid the loss of both the check data and the original data in the event of a check node failure, resulting in data recovery failure.

[0629] The coding node can select a terminal device as a verification node and send the verification data to the verification node for storage. Alternatively, the coding node can select multiple terminal devices as verification nodes and store the verification data in multiple copies in multiple verification nodes.

[0630] When verification data is stored in multiple replicas on multiple verification nodes, the verification nodes can also be used as backup devices for data redundancy. In this case, damage to some verification nodes does not result in the loss of both verification data and the original data. If a verification node fails, verification data can be retrieved from other verification nodes to restore the original data on the failed device.

[0631] When selecting a check node, the encoding node may determine the check node based on conditions such as device reliability, device online time, and device storage space.

[0632] Device reliability can be determined by the terminal device's vulnerability to damage or loss. For example, a desktop terminal device is less likely to be lost than a portable terminal device, so the device reliability of a desktop terminal device is higher than that of a portable terminal device. The coding node can determine the reliability of the terminal device based on its device attributes (e.g., device type).

[0633] For example, the coding node may select a terminal device with high device reliability as a verification node. A terminal device with high device reliability is not easily damaged or lost, which can reduce the risk of loss of verification data.

[0634] Alternatively, the encoding node can select a terminal device with a long online time as a verification node. Selecting a terminal device with a long online time as a verification node can avoid the situation where the verification node is offline when verification data is needed for data recovery, resulting in the inability to obtain verification data for data recovery.

[0635] Alternatively, the encoding node can also select a terminal device with large storage space as a verification node. When the verification node has a large storage space, the verification node has enough space to store the verification data, avoiding the situation where the verification node cannot store the verification data due to insufficient storage space during data redundancy backup.

[0636] Alternatively, the encoding node may also comprehensively consider multiple conditions including device reliability, device online time, and device storage space to determine the verification node.

[0637] When there is a damaged terminal device in the distributed storage system, the terminal device in the distributed storage system can obtain the device identification of the faulty device, search for undamaged original data and verification data based on the device identification of the faulty device and the verification information list, and restore the damaged original data on the faulty device based on the undamaged original data and verification data.

[0638] The terminal device that performs the data recovery operation may be the faulty terminal, or may be another terminal device other than the faulty terminal in the distributed storage system.

[0639] Furthermore, during the data redundancy backup process, the encoding node can generate and record a verification information list. The verification information list is used to indicate the encoding information. Specifically, the verification information list may include the device identifier of the device to be backed up corresponding to the original data block participating in the data redundancy backup, metadata information corresponding to the original data block, the offset of the original data block, the amount of data in the original data block, the file attributes corresponding to the original data block (e.g., file name, file path), metadata corresponding to the generated verification data, and the device identifier of the verification node storing the verification data.

[0640] That is, the terminal device can find the information of the original data and the verification data involved in the erasure coding through the verification information list.

[0641] For example, when a terminal device fails (ie, a faulty device), the terminal device can obtain the device identification of the faulty device and query the information of the original data and the verification data involved in the erasure coding from the above verification information list.

[0642] The information of the data to be backed up is used to indicate the original data involved in the erasure coding, and the information of the verification data is used to indicate the verification data generated by the erasure coding.

[0643] After obtaining the data, the terminal device can search for the undamaged original data according to the information of the data to be backed up, and search for the verification data according to the information of the verification data, and restore the damaged data on the faulty device based on the above undamaged original data and verification data.

[0644] In summary, the embodiments of the present application provide a data backup method that can flexibly apply erasure coding technology to terminal device scenarios. Each terminal device in a distributed storage system can independently serve as a storage node.

[0645] During the data redundancy backup process, the encoding node can dynamically adjust the stripe length of the erasure code stripe according to the online status of each terminal device in the distributed storage system, avoiding the problem that some terminal devices cannot use erasure codes for data fault tolerance after going online or offline due to the fixed stripe length, thereby improving the reliability of user data.

[0646] By using the above data backup method, the coding node can generate and store verification data. When a terminal device in the distributed storage system fails, the terminal device can restore the original data stored on the failed device based on the undamaged original data and verification data on the remaining terminal devices.

[0647] The following will explain the above data backup method in detail in conjunction with specific data recovery scenarios.

[0648] Application scenario 1: A terminal device in a distributed storage system fails (for example, is damaged or lost), and all data on the failed device needs to be restored.

[0649] When a terminal device in a distributed storage system fails (for example, is damaged or lost), the terminal device needs to recover all data in the failed device.

[0650] At this time, the terminal device can search the verification information list according to the device identification of the faulty device, and query the verification information list for information about the original data and verification data corresponding to each data redundancy backup of the faulty device.

[0651] Then, the terminal device reads the original data and verification data from the corresponding storage node based on the information of the original data and the verification data, and restores the data that needs to be backed up by the faulty device during each data redundancy backup process through the decoding algorithm, original data and verification data.

[0652] If all data on the faulty device has been backed up for data redundancy, the terminal device can restore all data on the faulty device through the above data recovery operations.

[0653] The terminal device performing the data recovery operation may be an encoding node in each data redundancy backup process, or a terminal device specified by the user, or a terminal device that is defaulted to an online device. The embodiment of the present application does not impose any restrictions on the terminal device performing the data recovery operation.

[0654] Application scenario 2: Some files on a terminal device in a distributed storage system are damaged and need to be restored.

[0655] When a user or an application on a terminal device finds that a file is damaged, the damaged file needs to be restored.

[0656] At this time, the terminal device can search the verification information list according to the file identifier (such as the file number) of the damaged file, and query the verification information list for information about the original data and verification data involved in the data redundancy backup process in which the file participates.

[0657] Then, the terminal device reads the original data and the verification data from the corresponding storage node according to the information of the original data and the verification data, and restores the damaged file through a decoding algorithm, the original data and the verification data.

[0658] The terminal device performing the data recovery operation may be an encoding node in each data redundancy backup process, or a terminal device specified by the user, or a terminal device that is defaulted to an online device. The embodiment of the present application does not impose any restrictions on the terminal device performing the data recovery operation.

[0659] The following will be combined Figures 16 to 18 The content shown in the figure further describes some non-limiting examples of the above data backup method. The following examples will be described using the verification node as a cloud or storage device as an example, but as mentioned above, the data backup method provided in the embodiment of the application can also be applied to scenarios where the verification node is other types of terminal devices in a distributed storage system.

[0660] Figure 16 The schematic block diagram of the terminal device in the distributed storage system provided by the embodiment of the present application is shown. The distributed storage system provided by the embodiment of the present application may include multiple terminal devices that can communicate with each other. For the convenience of description, Figure 16 Only two terminal devices in the distributed storage system are schematically shown, namely terminal device 1610 and terminal device 1620.

[0661] like Figure 16 As shown, the terminal device 1610 includes a striping module 1611, a data encoding module 1612, a data recovery module 1613, a storage module 1614, and an inter-device data communication module 1615. In the distributed storage system, the terminal device 1610 acts as an encoding node to perform data redundancy backup operations, that is, the erasure coding process. Specifically, the terminal device 1610 can perform Figure 15 The data backup method shown.

[0662] The terminal device 1620 includes a striping module 1621, a data encoding module 1622, a data recovery module 1623, a storage module 1624, and an inter-device data communication module 1625. In a distributed storage system, the terminal device 1620 can serve as a storage node to provide erasure coded data to be backed up.

[0663] Before executing the data backup method provided in the embodiments of the present application, pre-processing may be performed to determine encoding nodes and check nodes. The encoding nodes are used to perform the erasure coding process and are performed by terminal devices in the distributed storage system. The check nodes are used to store the verification data generated after the erasure coding. The check nodes can be performed by terminal devices in the distributed storage system or by dedicated storage devices (e.g., cloud or storage device 1630).

[0664] In some scenarios, the coding node can be specified by the user. Specifically, the terminal device currently used by the user can display an interface for selecting a coding node to the user through an application. In this interface, the application can recommend coding nodes to the user.

[0665] For example, the encoding node can be a terminal device with computing resources greater than a certain threshold in the distributed storage system, and the application can recommend the terminal device with computing resources greater than the certain threshold to the user. Alternatively, the encoding node can be a terminal device with strong computing power in the distributed storage system, and the application can sort the terminal device types according to computing power (for example, when sorting by computing power, desktop computers>tablet computers>smartphones>smart bracelets, etc.), and then the application will recommend the terminal device corresponding to the terminal device type with the highest ranking to the user. Alternatively, the application can recommend any terminal device with computing power in the distributed storage system to the user. Alternatively, the application can recommend other terminal devices to the user in addition to the terminal device currently used by the user.

[0666] The above application programs may be installed on some or all of the terminal devices in the distributed storage system.

[0667] In other scenarios, the encoding node can also be a default. For example, the encoding node can be the terminal device with the most computing resources in the distributed storage system by default. Alternatively, the encoding node can be the terminal device of a preset type (such as a mobile phone) by default. Alternatively, the encoding node can be the terminal device with the largest storage space in the distributed storage system by default.

[0668] The selection principle of the coding node can be determined based on any one of the above conditions, or can be determined based on a combination of multiple conditions. The embodiment of the present application does not impose any restrictions on the selection principle of the coding node.

[0669] Similarly, in some scenarios, the check nodes can also be specified by the user. Specifically, the terminal device currently used by the user can display an interface for selecting a check node to the user through an application. In this interface, the application can recommend a check node to the user.

[0670] For example, a check node can be Figure 16Alternatively, the check node can be a terminal device with high reliability in the distributed storage system. Alternatively, the check node can be a terminal device with long online time in the distributed storage system. Alternatively, the check node can be a terminal device with large storage space in the distributed storage system. Alternatively, the check node can be any terminal device with storage space in the distributed storage system. Based on the above conditions, the application selects the corresponding terminal device and recommends it to the user.

[0671] In other scenarios, the check node can also be a default. For example, the check node can default to one or more of the following types of terminal devices: dedicated storage devices (such as cloud or NAS devices), terminal devices with high reliability (or reliability above a certain level), terminal devices with long online time (or online time longer than a certain value), terminal devices with large storage space (or storage space greater than a certain value), and terminal devices of a preset type (such as smartphones).

[0672] The selection principle of the check node can be determined based on one of the above conditions, or can be determined based on a combination of multiple conditions. The embodiment of the present application does not impose any restrictions on the selection principle of the check node.

[0673] If there are multiple check nodes, the check data may be stored in the form of multiple copies on the multiple check nodes.

[0674] During pre-processing, that is, when multiple terminal devices are networked to form a distributed storage system, each terminal device within the distributed storage system can establish a metadata management mechanism. Specifically, each terminal device can store metadata information in its own storage module. This metadata information is used to indicate whether new data to be backed up is generated on each terminal device and / or the amount of data to be backed up.

[0675] After networking, the coding node can immediately perform data redundancy backup upon detecting new data to be backed up; alternatively, the coding node can also perform data redundancy backup at a fixed time (for example, late at night when the terminal device is idle); alternatively, the coding node can perform backup after receiving a backup instruction from the user. The embodiments of the present application do not impose any restrictions on the timing of the coding node performing data redundancy backup.

[0676] The frequency of data redundancy backup performed by the encoding node may be fixed. For example, the encoding node may perform data redundancy backup once a day or once a week.

[0677] Alternatively, the frequency of data redundancy backup performed by the coding node may not be fixed. For example, the coding node may perform data redundancy backup when new data to be backed up is detected, or the coding node may perform data redundancy backup when it detects that the amount of data to be backed up is greater than a second threshold.

[0678] The embodiment of the present application does not impose any restrictions on the frequency of data redundancy backup of the coding node.

[0679] In a distributed storage system, terminal devices may frequently go online or offline. Only online devices can participate in the data redundancy backup process, i.e., the erasure coding process.

[0680] For example, assuming that during a data redundancy backup process, terminal device 1610, terminal device 1620, and other terminal devices not shown are online, when there is data to be backed up in the distributed storage system, terminal device 1610 may perform the following data redundancy backup operations:

[0681] Step 1: The terminal device 1610 determines the stripe length and stripe depth of the erasure code stripe. Specifically, this step may be performed by the stripe division module 1611.

[0682] There are many ways for the encoding node to determine the stripe length and stripe depth. Compared with the design scheme of setting the stripe length to a fixed value in large-scale storage systems, the embodiment of the present application will adopt a design scheme with variable stripe length.

[0683] Exemplarily, the coding node may determine the stripe length in the following ways.

[0684] Method 1: The encoding node (eg, terminal device 1610 ) divides the stripe length according to whether the terminal device in the distributed storage system has data to be backed up.

[0685] For example, the stripe length can be the number of devices to be backed up in the distributed storage system. A device to be backed up is an online device that has data to be backed up and is used to store the data to be backed up. Alternatively, a device to be backed up can be considered an online device that generates data to be backed up.

[0686] Method 2: The encoding node divides the stripe length according to the number of online devices in the distributed storage system.

[0687] For example, the stripe length may be equal to the number of online devices in the distributed storage system, wherein some or all of the terminal devices in the online devices have data to be backed up.

[0688] Method 3: The encoding node divides the stripe length according to whether the terminal device in the distributed storage system has data to be backed up and the amount of data to be backed up on each device to be backed up.

[0689] Because the amount of data to be backed up can vary significantly between different backup devices, to conserve storage space and computing resources on encoding nodes, the backup data on multiple devices can be combined into a single backup data item for erasure coding. In this case, the stripe length is smaller than the number of devices to be backed up.

[0690] For example, the encoding node may merge the backup data of multiple backup devices and determine how many new backup data can be merged into and how many backup data of the backup devices each new backup data consists of. Then, the encoding node may determine the stripe length.

[0691] In addition, the encoding node can also determine the stripe depth in a variety of ways. In some embodiments, the stripe depth can be a preset value. For example, the stripe depth can be fixedly set to 1MB. In other embodiments, the stripe depth can also be determined based on the size of the data block on the device to be backed up. For example, data can be stored in blocks on the device to be backed up, and the stripe depth can be equal to the data volume of the largest data block stored on each device to be backed up. In other embodiments, since the choice of stripe depth will affect the calculation time of a single erasure code encoding, the smaller the stripe depth, the stronger the processing power of the encoding node and the faster the data access speed, but the randomness of data block access and the addressing time will increase accordingly. Therefore, the encoding node can also determine the stripe depth based on its own computing power and dynamically adjust the stripe depth each time a redundant data backup is performed.

[0692] After determining the stripe length and stripe depth, the encoding node can obtain the original data blocks corresponding to each data to be backed up. The data to be backed up can be divided into multiple original data blocks on the device to be backed up, or the data to be backed up can also be divided into multiple original data blocks by the encoding node.

[0693] It's important to note that data block sizes may or may not be the same in different storage systems. For example, in redundant array of independent disks (RAID) storage systems, common data block sizes range from 4KB to 128KB. In distributed storage systems, data blocks are larger, typically up to 64MB. Therefore, the size of the original data blocks corresponding to the data to be backed up on different backup devices may or may not be equal.

[0694] If the original data blocks corresponding to the data to be backed up have different data sizes, the encoding node can pad the original data blocks with data sizes less than the stripe depth with zeros before erasure coding. The original data blocks participating in the erasure coding cannot be modified during the erasure coding process.

[0695] In addition, during the erasure coding process, the encoding node may encounter a situation where the stripe length is greater than the amount of data to be backed up. In this case, the encoding node can use zero padding to fill the missing original data blocks with zeros during the erasure coding process.

[0696] The information used by the encoding node to determine the stripe length and stripe depth can be obtained through the cross-device data communication module 1615. In the distributed storage system, each terminal device can be provided with a cross-device data communication module, so that the terminal devices can communicate with each other through the cross-device data communication module.

[0697] Furthermore, in a distributed storage system, the storage module of each terminal device can store metadata information about other terminal devices in the distributed storage system. Based on this metadata information, the terminal device can determine whether other terminal devices have generated new data to be backed up and the amount of data to be backed up. In this embodiment of the present application, terminal device 1610 can obtain the online status and metadata information of terminal device 1620 and other terminal devices (not shown) in the distributed storage system.

[0698] Of course, in some embodiments, if the encoding node is fixed, then only the encoding node may be able to manage metadata information for all terminal devices in the distributed storage system. In other embodiments, since the encoding node is a terminal device with computing capabilities, terminal devices with computing capabilities in the distributed storage system may manage metadata information, while terminal devices without computing capabilities may not manage metadata information.

[0699] Step 2: The terminal device 1610 obtains the data to be backed up and performs striping. Specifically, this step can be performed by the cross-device data communication module 1615 and the striping module 1611.

[0700] In this step, the terminal device 1610 needs to read the data to be backed up from other terminal devices and perform centralized encoding locally. Centralized encoding means that all encoding work for an erasure code stripe is completed by a single encoding node.

[0701] In the distributed storage system provided in the present application, all terminal devices can serve as storage nodes, and the encoding node can be a terminal device in the storage node.

[0702] That is to say, the terminal device 1610 can only act as a coding node to perform erasure coding on the data to be backed up on other terminal devices, or the terminal device 1610 can also act as both a coding node and a storage node to perform erasure coding on the local data to be backed up and the data to be backed up on other terminal devices.

[0703] When acquiring data to be backed up, the terminal device 1610 can determine the device to be backed up and the storage location of the data to be backed up based on the metadata information. Then, the terminal device 1610 communicates with the cross-device data communication module of the device to be backed up via the cross-device data communication module 1615 to acquire the data to be backed up on the device to be backed up.

[0704] The cross-device data communication module 1615 may store the acquired data to be backed up in the storage module 1614 (eg, cache) so that the data encoding module 1612 may perform erasure coding on the data to be backed up.

[0705] Before erasure coding, the striping module 1611 may stripe the read data to be backed up according to the method of step 1. For example, the striping module 1611 may separate the data to be backed up into blocks according to different storage devices (or storage nodes) to obtain original data blocks. Parts that are less than the stripe length and stripe depth may be padded with zeros to align them, so that the number of original data blocks on the erasure coded stripe is consistent with the stripe length, and the size of the original data blocks on the same erasure coded stripe is consistent.

[0706] Step 3: The terminal device 1610 performs erasure coding. Specifically, this step may be performed by the data coding module 1612.

[0707] In some embodiments, the data encoding module 1612 may perform XOR encoding on the original data blocks on the same stripe.

[0708] Exclusive OR (XOR) is a mathematical operation that returns 0 or an error (false) when two values ​​are identical, and 1 or a true value (true) when two values ​​are different. In other words, XOR can be used to determine whether two values ​​are different. Leveraging the properties of XOR, redundant information can be removed from duplicate data, achieving information increment and data compression.

[0709] Specifically, after the data encoding module 1612 performs XOR encoding on the two original data blocks, it performs XOR encoding on the obtained XOR encoding result and the next original data block until all the original data blocks on the same erasure code stripe are encoded, and the verification data and verification information list corresponding to the erasure code stripe are obtained.

[0710] The parity data is the final XOR result generated by XORing all the original data blocks in the same erasure-coded stripe. The parity information list records the information of the original data blocks and the parity data for that erasure coding. For example, this list includes the device ID of the terminal device corresponding to the original data block, the device ID of the parity node corresponding to the parity data, metadata, data block offset, stripe depth, and the file name and path of the original data involved in the erasure coding.

[0711] Encoding nodes use XOR encoding to achieve single-node fault tolerance, meaning they can tolerate data loss on a single terminal device. This encoding method achieves the same fault tolerance as single-copy backup, but reduces storage space by at least 50% and reduces encoding and decoding time, allowing for backup of user data without impacting the user experience.

[0712] In other embodiments, the data encoding module 1612 may perform Reed-Solomon (RS) encoding on the original data blocks on the same stripe.

[0713] Using RS encoding can tolerate data loss on multiple devices, improving the reliability of user data. When the encoding node uses RS encoding, the encoding node can determine the stripe length based on the constraints of the RS encoding and the number of online devices.

[0714] The encoding node may perform erasure coding immediately upon detecting new data to be backed up; alternatively, the encoding node may perform erasure coding at a fixed time (e.g., late at night when the terminal device is idle); alternatively, the encoding node may perform erasure coding on the data to be backed up periodically (e.g., daily, weekly, monthly); alternatively, the encoding node may perform backing up after receiving a backup instruction from the user. The embodiments of the present application do not impose any restrictions on the timing of the encoding node performing erasure coding.

[0715] Step 4: The terminal device 1610 sends the generated verification data and verification information list to the verification node for storage.

[0716] Verification nodes can be pre-set as shown above. For example, if a user purchases a cloud or storage device (such as Figure 3 If the user does not purchase a cloud or storage device, the terminal device currently used by the user can recommend a terminal device with high reliability, long online time and sufficient storage space as a verification node to the user during the pre-processing phase. The terminal device 1610 sends the verification result to the terminal device serving as the verification node ( Figure 3(not shown) for storage.

[0717] In addition, the terminal device 1610 may send the verification result to one verification node for storage, or the terminal device 1610 may send the verification result to multiple verification nodes for storage.

[0718] When the terminal device 1610 sends the verification result to a verification node for storage, in order to prevent the verification node from being damaged and causing the loss of both the original data and the verification data, the verification node may not participate in the erasure coding.

[0719] When terminal device 1610 sends verification results to multiple verification nodes for storage, the probability of simultaneous loss of multiple verification nodes is low. Therefore, terminal device 1610 can store the verification results in multiple copies across the multiple verification nodes, improving data security. If a verification node fails, the terminal device can obtain verification results from other verification nodes to restore the original data on the failed device.

[0720] In an embodiment of the present application, the encoding node adopts a design scheme with a variable stripe length. Compared with the traditional disk RAID storage system in which data is divided into stripes of fixed stripe length for storage, the encoding node of the embodiment of the present application determines the stripe length and stripe depth according to the online status of the terminal devices in the distributed storage system (such as the number of online devices and the number of devices to be backed up). During each data redundancy backup, the encoding node can dynamically adjust the stripe length according to the online and offline status of the terminal device. For example, the encoding node can dynamically adjust the stripe length according to the number of online devices, thereby realizing erasure code encoding of the data to be backed up in the terminal device and improving the reliability of user data. The embodiment of the present application realizes the application of erasure codes in terminal devices, so that files in the terminal device can be backed up regularly to prevent other terminal devices from being unable to use erasure codes for data fault tolerance after the terminal device goes online and offline.

[0721] In addition, the data to be backed up can be the initial original data after the terminal device is networked, the newly added data to be backed up, the modified backed up data, the backed up data with some data deleted, etc. Figure 16 Taking the terminal device 1610 as an encoding node as an example, several specific data redundancy backup situations are introduced in detail.

[0722] In one scenario, new data is added to the terminal device. For example, a user creates a new document, takes a photo, or downloads a video on the terminal device. Since the encoding node manages metadata information, the terminal device 1610 can obtain the new data on the terminal device through the metadata information.

[0723] At this point, terminal device 1610 can determine the stripe length and stripe depth based on the aforementioned data backup method, divide the newly added data into original data blocks, perform erasure coding on the newly added data, and generate verification data and a verification information list corresponding to this data redundancy backup. Terminal device 1610 then sends the generated verification results to the verification node for storage.

[0724] In another case, the backed-up data on the terminal device has been changed. For example, the user has modified the backed-up data on the terminal device. Therefore, the terminal device 1610 can obtain information about the modified backed-up data through the metadata information.

[0725] At this time, the terminal device can store the modified backed-up data as a new copy, and use the new copy as new data for data redundancy backup to generate new verification data, while the verification data corresponding to the original backed-up data can be retained.

[0726] Alternatively, the terminal device may directly use the modified backed-up data as newly added data for data redundancy backup to generate new verification data, and the verification data corresponding to the original backed-up data may be deleted.

[0727] Device 1610 may periodically count whether there is newly generated data in each terminal device, organize the newly generated data into new stripes, and encode and calculate verification results.

[0728] For example, Figure 17 A schematic diagram of an application scenario provided by an embodiment of the present application is shown. Figure 17 As shown, it is assumed that the distributed storage system includes device A, device B and device C.

[0729] Before the data is updated, the data to be backed up in device A includes files a and c, and the data to be backed up in device B includes files b and d.

[0730] When the encoding node (such as device C) performs data fault tolerance on the data to be backed up before the data is updated, striping is performed according to the stripe depth, and files a and c read from device A are divided into three original data blocks respectively, and files b and d read from device B are divided into three original data blocks respectively.

[0731] Device C encodes the original data blocks corresponding to file a and file b respectively (for example, XOR encoding) to generate verification data (a+b), and encodes the original data blocks corresponding to file c and file d respectively (for example, XOR encoding) to generate verification data (c+d).

[0732] Assume that the user later modifies file a on device A and file d on device B, and encodes the modified files as newly added data for fault tolerance. In other words, a new file a' is added to device A and a new file d' is added to device B.

[0733] At this point, the encoding node (such as device C) can perform fault-tolerant encoding on the newly added data a' and d' in devices A and B. Device C divides the read files a' from device A and d' from device B into three original data blocks, respectively. It then encodes the data blocks corresponding to files a' and d' (for example, using XOR encoding) to generate the checksum (a' + d'). When device C performs the encoding checksum, the original data blocks in the same erasure code stripe must belong to different devices to be backed up.

[0734] It should be understood that Figure 17 The number of terminal devices shown, the stripe length determined by the encoding node, and which original data blocks are XOR-encoded by the encoding node are merely exemplary and are only used to exemplify how to perform data fault tolerance on the modified data when the user modifies the backed-up data, and do not impose any limitations on the data backup method of the embodiments of the present application.

[0735] In another case, the user deletes the data stored in the terminal device, and the deleted data has been backed up. Since the encoding node manages metadata, the encoding node (such as terminal device 1610 or terminal device 1620, etc.) can obtain the deleted data information through the metadata information.

[0736] The encoding node can query other backed-up data in the same erasure code stripe as the deleted data from the verification information list based on the file number, file storage path and other information of the deleted data.

[0737] The encoding node can use other backed-up data in the same erasure code stripe as the deleted data as new data, reassemble the erasure code stripe for encoding, obtain a new verification result and send it to the verification node for storage.

[0738] For example, Figure 18 FIG. 1 is a schematic diagram showing another application scenario provided by an embodiment of the present application. Figure 18 As shown, it is assumed that the distributed storage system includes device A, device B and device C.

[0739] Before data deletion, the data to be backed up in device A includes files a and c, and the data to be backed up in device B includes files b and d.

[0740] When the encoding node (such as device C) performs data fault tolerance on the data to be backed up before data deletion, it performs striping according to the stripe depth, dividing the files a and c read from device A into three original data blocks respectively, and dividing the files b and d read from device B into three original data blocks respectively.

[0741] Device C encodes the original data blocks corresponding to file a and file b respectively (for example, XOR encoding) to generate verification data (a+b), and encodes the original data blocks corresponding to file c and file d respectively (for example, XOR encoding) to generate verification data (c+d).

[0742] Assume that the user later deletes file a in device A and encodes the other backed-up data in the same erasure code stripe as the deleted data as new data for fault tolerance.

[0743] At this time, device C can read file b in device B, encode file b and the newly added file e in device A (file e can also be a zero-padded data block) (for example, XOR encoding), and generate verification data (b+e).

[0744] After generating a new verification result (e.g., verification data (b+e)), the original verification result (e.g., verification data (a+b)) can be saved for a period of time, such as a week, before being deleted. This prevents users from accidentally deleting files on the terminal device and causing data loss. When device C performs erasure coding, the original data blocks in the same erasure-coded stripe should belong to different storage nodes.

[0745] It should be understood that Figure 18 The number of terminal devices shown, the stripe length determined by the encoding node, and which original data blocks are XOR-encoded by the encoding node are merely exemplary and are only used to exemplify how to perform data fault tolerance on the modified data when the user modifies the backed-up data, and do not impose any limitations on the data backup method of the embodiments of the present application.

[0746] In another case, when a new terminal device joins the distributed system, the newly added terminal device can serve as a new storage node, and the data to be backed up of the newly added terminal device can be combined with the data to be backed up of the existing terminal device to form an erasure code stripe for data redundancy backup.

[0747] For example, after the new terminal device joins the distributed storage system, other terminal devices can obtain information such as the online status of the new terminal device, whether new data to be backed up is generated, and so on.

[0748] It should be noted that, in addition to being able to serve as storage nodes, the above-mentioned new terminal devices can also serve as encoding nodes and verification nodes.

[0749] In another case, when a terminal device in the distributed storage system wants to exit or log out, other original data in the erasure coding in which the terminal device participates can be re-erasure coded and the new verification result can be stored.

[0750] For example, suppose a terminal device exits the distributed storage system. At this time, the encoding node can query the verification information list for other original data encoded with erasure codes in which the terminal device participates based on the device identification of the terminal device, and re-encode the original data with erasure codes.

[0751] In an embodiment of the present application, the encoding node can dynamically adjust the stripe length according to the number of online devices in the distributed storage system, and store the verification results on a storage device such as the cloud or NAS, which can ensure that the verification results on the terminal device are refreshed in real time without worrying about the erasure code being unavailable after the terminal device is disconnected.

[0752] In addition, when terminal devices join or exit the distributed storage system, the encoding node can flexibly adjust the stripe length to adapt to different numbers of online devices.

[0753] The above describes the process of data redundancy backup of terminal equipment. Figure 16 If a terminal device participating in data redundancy backup fails (for example, the device is lost or damaged, the file on the device is damaged or accidentally deleted, etc.), the terminal device can use the verification results stored in the verification node and the original data participating in the erasure code encoding stored on other terminal devices to restore the data on the failed device.

[0754] When a faulty device is damaged or lost, all data on the device needs to be restored. After the distributed storage system selects a terminal device to perform the recovery operation, the terminal device can obtain the verification results (including verification data and verification information list) of the erasure coding used by the faulty device from the verification node.

[0755] The terminal device can then use the device ID of the failed device to query the verification information list for the original data and verification data corresponding to the erasure coding used by the failed device. Based on the original data and verification data, the terminal device can retrieve the original data and verification data generated by other storage nodes that participated in the erasure coding during previous data redundancy backups. The terminal device can then restore the original data on the failed device based on the original data and verification data.

[0756] Alternatively, the terminal device may not obtain the verification result from the verification node, but directly query the original data information and verification data information corresponding to the erasure code encoding in which the faulty device participates at the verification node according to the faulty device identifier, and then perform data recovery.

[0757] When performing data recovery, the terminal device can recover data in sequence according to the data redundancy backup order in which the failed device participates.

[0758] Furthermore, after restoring the original data of the faulty device, the terminal device can store the restored original data in a preset storage device. The preset storage device can be a terminal device specified by the user, or a terminal device with a default storage space greater than a certain value.

[0759] When some files on a faulty device are damaged, lost, or accidentally deleted, data recovery is required. After the distributed storage system selects a terminal device to perform the recovery operation, the terminal device can use information that identifies the file, such as the file ID and storage path, to query the verification information list stored on the verification node for the original data and verification data corresponding to the erasure code used in the file. The terminal device then retrieves the corresponding original data and verification data based on the original data and verification data information, and uses a decoding algorithm to recover the data on the faulty device.

[0760] The terminal device performing the data recovery operation can be specified by the user, or can be determined according to a preset selection rule, or can be the faulty device itself that needs to recover some data. The embodiment of the present application does not impose any restrictions on the specific selection method of the terminal device performing the data recovery operation.

[0761] In addition, the terminal device may execute the above-mentioned data recovery process in response to a user's instruction, or the terminal device may automatically detect and execute the data recovery process according to a preset application.

[0762] refer to Figure 16, taking the data recovery process performed by terminal device 1620 as an example. Assuming that terminal device 1610 fails, the data recovery module 1623 in terminal device 1620 can obtain the verification results related to terminal device 1610 or the verification results related to some files from the cloud or storage device 1630, and then read the corresponding original data and verification data to perform data recovery. In addition, when querying the verification information list, the data recovery module 1623 in terminal device 1620 can directly query the original data information and verification data information from the verification node, without having to save the verification results to its own device and then query the original data information and verification data information from the stored verification information list.

[0763] It should be understood that Figure 16 The functions of the other modules in terminal device 1620 are similar to those of the corresponding modules in terminal device 1610. When terminal device 1620 serves as an encoding node, it can also execute the above-described data backup method to complete the data redundancy backup process. Alternatively, terminal device 1620 may not be used for encoding. In this case, terminal device 1620 may not include functional modules such as striping module 1621 and data encoding module 1622.

[0764] The above content describes the data backup method provided in an embodiment of the present application. When the coding node performs erasure coding, if the condition p=N+1, p is a prime number greater than or equal to 3, and N is the stripe length is satisfied, the coding node can use the following erasure coding method to calculate the verification data.

[0765] like Figure 19 As shown in FIG, a flowchart of an implementation method of an erasure code encoding method is provided for an embodiment of the present application. The method can be implemented by software and / or hardware of a terminal device. Figure 19 As shown, the method includes steps S1901 to S1904. The specific implementation principles of each step are as follows:

[0766] S1901, dividing the backup data of N different devices to be backed up into blocks to obtain multiple original data blocks of each device to be backed up, multiple original data blocks belonging to the same device to be backed up have different sorting positions, and N is a positive integer greater than or equal to 2.

[0767] Among them, when the stripe length is N, it means that there are N devices to be backed up participating in the erasure code encoding, and the backup data of N different devices to be backed up need to be backed up redundantly.

[0768] At this time, the encoding node may divide the data to be backed up of different devices to be backed up into stripes to obtain multiple original data blocks corresponding to each device to be backed up. The multiple original data blocks corresponding to each device to be backed up have different sorting positions.

[0769] The sorting position can be identified by sequentially marking Arabic numerals or by sequentially marking letters. In other embodiments, the sorting position can be identified without sequential marking. This application does not limit the specific implementation form of the sorting position.

[0770] For example, the original data blocks of the same device to be backed up can be represented by identifiers such as 1, 2, 3, 4, etc. to indicate the sorting position; or, they can be represented by identifiers such as A, B, C, D, etc. to indicate the sorting order; or, they can be represented by identifiers combining data and letters to indicate the sorting order.

[0771] When the number of original data blocks corresponding to different devices to be backed up is exactly the same, all original data blocks of the different devices to be backed up can be backed up together. Alternatively, multiple original data blocks can be selected or intercepted from the original data blocks of each device to be backed up for backup. In one encoding process, the number of original data blocks selected or intercepted from N different devices to be backed up is the same, G, where G is a positive integer greater than or equal to 2. The G selected or intercepted original data blocks have different sorting positions.

[0772] When the number of original data blocks corresponding to different devices to be backed up is different, the number of blocks can be adjusted to be the same. For example, the number of blocks of different devices to be backed up can be adjusted to be the same by filling data blocks.

[0773] The data to be backed up on each device to be backed up can be divided into blocks according to the stripe depth. If the data blocks after division are not equal to the stripe depth, they can be adjusted to the stripe depth by padding data. For example, a preset number of "0"s can be used to pad the data blocks to obtain the same size as the original data blocks.

[0774] In a specific embodiment of the present application, to simplify encoding calculations, the number of original data blocks obtained from different devices to be backed up can be the same as the number of devices to be backed up that participate in the encoding. In other words, the number of devices to be backed up is the same as the number of original data blocks to be encoded in each device to be backed up, with N equal to G. In this case, further, a data array can be formed using original data blocks from different devices to be backed up.

[0775] Optionally, the original data blocks of different devices to be backed up are arranged in sequence into a data array, with the original data blocks of each device to be backed up serving as a column or a row of the data array. Then, according to the erasure code encoding method provided in an embodiment of the present application, two columns or two rows of check data blocks are added, and the two columns or two rows of check data blocks and the data array constitute a codeword array.

[0776] At this time, the data array is of size (p-1)*(p-1), where p is a prime number greater than or equal to 3, and the value of p is the total number of nodes of the device to be backed up plus 1, that is, N+1 is a prime number greater than or equal to 3. The codeword array is of size (p-1)*(p+1). For example, assuming that the value of p is 5, the number of devices to be backed up is 4, the first 4 columns or the first 4 rows are original data, and the last 2 columns or the last 2 rows are check data. The first column or the first row of check data is generated by row or column check erasure coding, and the second column or the second row of check data is generated by diagonal erasure coding. The above-mentioned erasure coding method can be XOR coding or XOR coding.

[0777] As a non-limiting example, Figure 20 As shown in the figure, four different devices to be backed up, namely the first to fourth devices to be backed up, are used as the devices to be backed up. The data to be backed up of each device to be backed up, such as files 1 to 4, are divided into blocks to obtain G original data blocks corresponding to each device to be backed up, where G is equal to 4. The G original data blocks are marked as D in sequence. 0,i , D 1,i ,...,D g,i ,...,D G-1,i , g is a natural number, and 0≤g≤G-1, i is the identifier of the device to be backed up, and takes 0, 1, 2, 3; and then encoding is performed based on all the original data blocks of each device to be backed up.

[0778] As another non-limiting example, Figure 21 As shown in the figure, four different devices to be backed up, namely the first to fourth devices to be backed up, are used as the devices to be backed up. The data to be backed up of each device to be backed up, such as files 1 to 4, are divided into blocks to obtain multiple original data blocks corresponding to each device to be backed up. For example, the first device to be backed up includes R original data blocks, D 0,0 , D 1,0 ,...,D r,0 ,...,D R-1,0 The second device to be backed up includes S original data blocks, D 0,1 , D 1,1 ,...,D s,1 ,...,D S-1,1 The third device to be backed up includes T original data blocks, D 0,2 , D 1,2 ,...,D t,2 ,...,D T-1,2 The fourth device to be backed up includes U original data blocks, D 0,3 , D 1,3 ,...,D u,3 ,...,D U-1,3R, S, T and U are all positive integers greater than or equal to 4, and can be completely the same or not completely the same. In the case of not completely the same, each device to be backed up obtains the same number of original data blocks by filling in the data blocks. G original data blocks are selected from the multiple original data blocks of each device to be backed up for subsequent encoding steps, where G is less than R, S, T and U, and G is equal to 4. The G original data blocks are marked as D 0,i , D 1,i ,...,D g,i ,...,D G-1,i .

[0779] It should be understood that in the above example, after the encoding node completes encoding of the first four positions, it can also encode the subsequent four positions until all the original data blocks of the device to be backed up are encoded. In other words, during the encoding process, the original data blocks of the device to be backed up can be encoded in segments, and the encoding of different segments can be performed simultaneously, that is, in parallel, or sequentially. The selection and setting can be based on the performance of the encoding node (including computing power, storage space size, etc.), and this application does not limit this.

[0780] S1902 : Select original data blocks with the same sorting position from the original data blocks of each device to be backed up, and perform encoding to obtain first check data blocks corresponding to different sorting positions.

[0781] In step S1901, the data to be backed up of different devices to be backed up are divided into blocks to obtain a plurality of original data blocks arranged in order for the different devices to be backed up. In steps S1902 and S1903, the original data blocks are encoded with erasure codes.

[0782] In step S1902, original data blocks with the same sorting position are selected from the original data blocks of each device to be backed up for encoding to obtain first check data blocks corresponding to different sorting positions.

[0783] Optionally, the above encoding methods include XOR encoding, XOR encoding, etc. Among them, XOR encoding and XOR encoding are simple to calculate, have low computational complexity for encoding and decoding, and high data storage and recovery efficiency. However, due to the relatively high accuracy of XOR encoding, embodiments of the present application can set XOR encoding as the default encoding method.

[0784] As a non-limiting example of this application, Figure 21 In the example shown, four devices to be backed up are used as the devices to be backed up, and original data blocks at the same sorting position are selected from the original data blocks of the four devices to be backed up for encoding, to obtain four first check data blocks, which are P 0,0 , P 1,0 , P2,0 , P 3,0 , the result is as follows Figure 22 Therefore, in the embodiment of the present application, the code for generating the first check data block may be referred to as a row or column check code, and the first check data block may also be referred to as row or column check data (or row or column check information).

[0785] Optionally, the original data blocks of N devices to be backed up are arranged in sequence into a data array of size N*N, with the original data block of each device to be backed up as a column of the data array, and then two columns of check data blocks are calculated based on the original data blocks. The data array and the two columns of check data blocks constitute a codeword array.

[0786] The complete codeword array is represented as (p-1)*(p+1), where p is N+1. The first N columns are original data, and the last two columns are check data. The first column of check data is the first check data block, and the second column of check data is the second check data block. If each data block of the codeword array is represented by a c i,j The (0≤i≤p-2,0≤j≤p) codeword represents the first p-1 columns, i.e., 0≤j≤p-2, representing the original data block, and the last two columns, j=p-1 and j=p, representing the check data block. The check data block is obtained through encoding. In other words, the encoding process uses the data block (the original data block and / or the first check data block) to calculate the last two columns of the check data block.

[0787] The encoding process for obtaining the first column of check data is as follows: First, the check data block on column j = p - 1 (i.e., column p) is calculated using the original data blocks of the first p - 1 columns (0 ≤ j ≤ p - 2). The calculation formula is as follows: When p=5, the encoding process can be seen in Figure 22 shown.

[0788] It should be understood that in other application examples, the original data block of each device to be backed up can also be used as a row of the data array, and the check data block as the row data, together with the data array to form a codeword array. In addition, in other application examples, the check data block can also be calculated using a bitwise XOR encoding method.

[0789] As another non-limiting example of this application, Figure 21 In the example shown, four devices to be backed up are used as the devices to be backed up, and an original data block at the same sorting position is selected from each of the four devices to be backed up for encoding to obtain four first check data blocks, P 0,0 , P 1,0 , P 2,0 , P 3,0 .like Figure 23As shown, after completing the encoding of the first four positions, the encoding process of the subsequent four positions is performed to obtain another four first check data blocks, P 4,0 , P 5,0 , P 6,0 , P 7,0 .

[0790] In the example of the present application, the encoding node divides the original data block into multiple data segments, and encodes each data segment until all data segments complete the encoding process. The encoding process of each data segment can be carried out simultaneously or in sequence. The data processing is highly flexible. When multiple data segments are encoded at the same time, the encoding efficiency can be improved, thereby improving the efficiency of data redundancy backup.

[0791] S1903: Select original data blocks with different sorting positions from the original data blocks of each device to be backed up, and encode them to obtain second check data blocks corresponding to the different sorting positions; or select original data blocks and first check data blocks with different sorting positions from the original data blocks and first check data blocks of each device to be backed up, and encode them to obtain second check data blocks corresponding to the different sorting positions.

[0792] In step S1901, the data to be backed up of different devices to be backed up are divided into blocks to obtain a plurality of original data blocks of different devices to be backed up arranged in sequence. In steps S1902 and S1903, the original data blocks are encoded.

[0793] In some embodiments of the present application, in step S1903, the encoding node selects original data blocks with different sorting positions from the original data blocks of each of the N devices to be backed up, and encodes them to obtain second check data blocks corresponding to the different sorting positions.

[0794] It should be understood that in these embodiments, step S1902 may be executed first and then step S1903; step S1903 and step S1902 may also be executed simultaneously, i.e., parallel processing; or step S1903 may be executed first and then step S1902, and there is no specific limitation on the timing of the two steps.

[0795] As a non-limiting example of this application, Figure 22 In the example shown, four devices to be backed up are used as the devices to be backed up. One original data block at a different sorting position is selected from the original data blocks of each device to be backed up for encoding to obtain four second check data blocks, namely P 0,1 , P 1,1 , P 2,1 , P 3,1 .

[0796] See also Figure 24 and Figure 25 As shown, the four original data blocks marked by the circle are encoded to obtain the second verification data block P marked by the dotted circle. 0,1 ; Encode the four original data blocks marked by the rectangle to obtain the second check data block P marked by the dotted rectangle 1,1 The four original data blocks marked by the triangle are encoded to obtain the second check data block P marked by the dotted triangle 2,1 The four original data blocks marked with diamonds are encoded to obtain the second check data block P marked with a dotted diamond 3,1 .

[0797] Figure 24 The coding direction is regular oblique coding, that is, oblique coding in the diagonal direction, Figure 25 The coding direction is irregular oblique coding. Therefore, in the embodiment of the present application, the coding for generating the second check data block can be called oblique check coding, and the second check data block can also be called oblique check data (or oblique check information).

[0798] As another non-limiting example of this application, Figure 23 In the example shown, four devices to be backed up are used as the devices to be backed up, and an original data block at a different sorting position is selected from each device to be backed up for encoding to obtain four second check data blocks, P 0,1 , P 1,1 , P 2,1 , P 3,1 ;like Figure 26 As shown, after completing the encoding of the first four positions, the encoding process of the subsequent four positions is performed to obtain another four second check data blocks, P 4,1 , P 5,1 , P 6,1 , P 7,1 Repeat the encoding until all the original data blocks of the device to be backed up are encoded. Figure 26 The coding direction is the coding in the diagonal direction. It should be understood that in other examples, the coding node can also adopt an irregular diagonal coding method.

[0799] In other embodiments of the present application, in step S1903, N groups of original data blocks and a group of first check data blocks of N devices to be backed up constitute N+1 groups of data. At this point, the encoding node can select a data block with a different sorting position from each of the N groups of data in the N+1 groups of data for encoding, thereby obtaining a second check data block corresponding to each different sorting position.

[0800] Optionally, N groups of original data blocks of N devices to be backed up are formed into a data array, and then the data array and a group of first verification data blocks are formed into an extended array including N+1 groups of data. From the N groups of data in the N+1 groups of data in the extended array, a data block with a different sorting position is respectively selected for encoding, and N second verification data blocks corresponding to the different sorting positions can be obtained.

[0801] As a non-limiting example of this application, Figure 22 In the example shown, four devices to be backed up are used as the devices to be backed up. After obtaining the first verification data block, four groups of original data blocks of the four devices to be backed up and one group of first verification data blocks are used to form five groups of data blocks. A data block is selected from each of four groups of data blocks in these five groups of data blocks, and the four selected data blocks have different sorting positions. The four selected data blocks are encoded to obtain four second verification data blocks, namely P 0,1 , P 1,1 , P 2,1 , P 3,1 .

[0802] See also Figure 27 and Figure 28 As shown, the four data blocks marked with circles are encoded to obtain the second verification data block P marked with dotted circles. 0,1 Encode the four data blocks marked by the rectangle to obtain the second check data block P marked by the dotted rectangle 1,1 Encode the four data blocks marked by the triangle to obtain the second check data block P marked by the dotted triangle 2,1 Encode the four data blocks marked with diamonds to obtain the second check data block P marked with a dotted diamond 3,1 .

[0803] Figure 27 The coding direction shown is a regular oblique coding, that is, coding in the diagonal direction. Figure 27 There are four data blocks belonging to the same oblique direction that are not involved in the encoding. Figure 28 The coding direction shown is an irregular oblique coding direction, and the four data blocks not involved in the coding do not belong to the same oblique direction.

[0804] Optionally, the complete codeword array is expressed as (p-1)*(p+1), where p is N+1, the first N columns are original data information; the last two columns are check data columns, the first column of check data is the first check data block, and the second column of check data is the second check data block.

[0805] If each data block of the codeword array is represented by a c i,jThe (0≤i≤p-2,0≤j≤p) codeword representation is as follows: the first p-1 columns, i.e., 0≤j≤p-2, represent the original data block, and the last two columns, j=p-1 and j=p, represent the check data block.

[0806] The encoding node obtains the second column of parity data by encoding as follows: the original data blocks of the first p-1 columns (0≤j≤p-2) and the first parity data block on the j=p-1 column (i.e., the pth column) are used to encode the skew parity data on the j=p column, i.e., the second parity data block. The skew parity data is calculated as follows: in<i+j> =(i+j)mod p. For example, when p=5, the skew check encoding process can continue to refer to Figure 27 As shown, this is diagonal coding.

[0807] It should be understood that in other application examples, the encoding node may also treat the original data block of each device to be backed up as a row in a data array, and the check data block as row data, and together with the data array, form a codeword array. In other application examples, the encoding node may also use a bitwise XOR encoding method to perform erasure coding. Furthermore, in other application examples, the skew check encoding process may also use an irregular directional encoding method.

[0808] As another non-limiting example of this application, Figure 23 In the example shown, four devices to be backed up are used as the devices to be backed up. A data block is selected from each of three of the devices to be backed up, and a data block is selected from the first verification data block. The four selected data blocks have different sorting positions. The four selected data blocks are encoded to obtain four second verification data blocks, namely P 0,1 , P 1,1 , P 2,1 , P 3,1 ;like Figure 29 As shown, after the encoding of the first four positions is completed, the encoding process of the subsequent four positions is performed to obtain another four second check data blocks, namely P 4,1 , P 5,1 , P 6,1 , P 7,1 The encoding node repeats the oblique check encoding until all the original data blocks of the device to be backed up are encoded.

[0809] Figure 29 The encoding direction is diagonal encoding. It should be understood that in other examples, the encoding node may also use an irregular encoding method; or, for encoding different data segments in the same device to be backed up, the encoding direction may be different, for example, the first four sorting positions use a diagonal encoding method, while the subsequent four sorting positions use an irregular encoding method.

[0810] S1904: Store the first verification data block and the second verification data block.

[0811] The first check data block obtained in step S1902 and the second check data block obtained in step S1903 can be stored together or separately. In other words, the first check data block and the second check data block can be stored in the same or different check nodes.

[0812] In the embodiment of the present application, the check node can be used to store the first check data block and / or the second check data block. The check node can be the device to be backed up, or the check node can be other terminal devices other than the device to be backed up.

[0813] Optionally, when each device to be backed up does not store original data blocks, it is also necessary to store multiple original data blocks of the N devices to be backed up. For example, N groups of original data blocks are stored in the backup node. It should be understood that the backup node may or may not overlap with the device to be backed up.

[0814] In some embodiments of the present application, the original data blocks obtained after each device to be backed up is divided into blocks can be stored in the device to be backed up. Alternatively, N groups of original data blocks and a group of first verification data blocks and a group of second verification data blocks can be distributed and stored according to the network conditions and device status.

[0815] The network status includes bandwidth, network speed, etc.; the device status includes at least one of device online status and device performance. Device online status includes online duration or disconnection frequency. Device performance includes at least one of storage space size and processing speed.

[0816] Optionally, the original data blocks obtained after each device to be backed up can be stored on a backup node that has good communication with the device to be backed up, is less likely to be offline, or has relatively stable device performance. Storing data on different backup nodes based on network conditions and device status allows for flexible data storage.

[0817] After encoding the backup data, the encoding node can store the original data blocks and verification data blocks in different nodes according to different user needs and the network and device conditions. When a terminal device fails or goes offline, the terminal device performing the recovery operation can quickly retrieve the undamaged original data and verification data.

[0818] As a non-limiting example of the present application, data is stored in different placement methods. Taking the storage of 4 columns of original data blocks of 4 devices to be backed up as an example, 2 columns of verification data blocks (including the first verification data block and the second verification data block) are added, resulting in a total of 6 columns of data. These 6 columns of data can be stored in a distributed manner on up to 6 terminal devices.

[0819] For example, the six columns of data can be stored in a dispersed manner on six terminal devices, or can be stored in pairs on three terminal devices. When storing on three terminal devices, different combinations can be used for storage according to the terminal device conditions.

[0820] As another non-limiting example of this application, Figure 24 , Figure 25 , Figure 27 and Figure 28 In any of the examples shown, it is assumed that three of the four terminal devices to be backed up (i.e., the first terminal device (device 1), the second terminal device (device 2), the third terminal device (device 3) and the fourth terminal device (device 4)) are used, such as device 1, device 2 and device 3, to store the original data block, the first verification data block and the second verification data block.

[0821] In a specific implementation, device 1 and device 2 can store two columns of original data blocks respectively, and device 3 can store two columns of check data. Figure 30 As shown, device 1 stores the original data blocks of device 1 and device 2, device 2 stores the original data blocks of device 3 and device 4, and device 3 stores the first verification data block and the second verification data block.

[0822] In another specific embodiment, device 1 can store two columns of original data, and device 2 and device 3 can store one column of original data and one column of verification data respectively, for example, Figure 31 As shown, device 1 stores the original data blocks of device 1 and device 2, device 2 stores the original data blocks and the first verification data blocks of device 3, and device 3 stores the original data blocks and the second verification data blocks of device 4.

[0823] Optionally, if the communication and interaction capabilities of device 1 and device 2 are good, the following can be used: Figure 30 The storage method shown can obtain the required data by directly reading the original data on device 1 and device 2 without decoding and recovery. If the data transmission conditions of device 1, device 2 and device 3 are similar, the following can be used Figure 31 The storage method shown is used to achieve balanced load.

[0824] In the embodiment of the present application, the data is redundantly backed up by encoding through the above-mentioned erasure code encoding method, and only the first check data block and the second check data block are added, that is, two columns of check columns are added after encoding, which can tolerate the failure of two columns of data in the codeword array at the same time, and the fault tolerance performance is 2. The total storage overhead of the embodiment of the present application increases by 2 / (p-1). When p=5, N is 4, indicating that there are 4 columns of original data blocks. After encoding, 2 columns of check columns are added, and the distributed storage system needs to store 6 columns of data, that is, the total storage overhead is 6 / 4=1.5 times the original data storage space. Compared with the three-copy technology (i.e., the original data is copied twice, 1 original data and 2 backup data form 3 copies, and the total storage overhead is 3 times the original data storage space), under the same fault tolerance performance, half of the storage space can be reduced. It can be seen that compared with the backup fault tolerance method using copies, the embodiment of the present application greatly reduces the storage overhead. In addition, a first verification data block is obtained by encoding the original data blocks at the same sorting position; a second verification data block is obtained by obliquely encoding the original data blocks at different sorting positions, or by obliquely encoding the original data blocks at different sorting positions and the first verification data block. Compared with the existing erasure code encoding method, the encoding calculation complexity is low and the storage efficiency is high.

[0825] In addition, the data to be backed up in the above-mentioned erasure coding method may be data to be classified whose popularity category is identified as warm data and / or cold data by the above-mentioned data classification method.

[0826] Each device to be backed up can classify its data at a preset time, or at each preset time period, to determine the popularity of each data type. The device to be backed up can then back up hot data using data copies, and transfer warm and / or cold data as data to be backed up to an encoding node, which then performs data redundancy backup of the data using the erasure coding method described above.

[0827] Corresponding to the above-mentioned erasure coding method, a method for recovering data will be described in the following embodiment. Figure 32 , Figure 32 This is a flowchart of a method for recovering data provided in an embodiment of the present application. Figure 32 As shown, the method includes steps S3201 to S3202. The implementation principle of each step is as follows:

[0828] S3201: Obtain damage information for the device to be backed up and the check nodes. The faulty device is a device to be backed up that stores original data or a check node that stores check data. There are N devices to be backed up, each of which is a data node that provides original data blocks and participates in encoding. The check nodes are data nodes that store first check data blocks and / or second check data blocks. The damage information includes whether data on the device to be backed up and the check nodes has been damaged.

[0829] The data to be backed up of each device to be backed up may be divided into multiple original data blocks after being divided into blocks, and the multiple original data blocks belonging to the same device to be backed up have different sorting positions.

[0830] The first verification information block is generated by the encoding node selecting an original data block from each device to be backed up and encoding the original data blocks at the same sorting position.

[0831] The second verification information block is generated by the encoding node selecting an original data block from each device to be backed up, and selecting original data blocks at different sorting positions for encoding, or the second verification data block is generated by N+1 groups of data consisting of N groups of original data blocks and a group of first verification data blocks from N data nodes, and the encoding node selects an original data block with a different sorting position from each of the N groups of data in the N+1 groups of data for encoding.

[0832] As mentioned above, it is understandable that the check nodes and the devices to be backed up may or may not overlap.

[0833] S3202: If there is a faulty device among the devices to be backed up and the verification nodes, the damaged data blocks of the faulty device are restored based on the undamaged data blocks of other data nodes; the faulty device is the device to be backed up and / or verification node where data damage occurs; the other data nodes are the remaining terminal devices among the devices to be backed up and the verification nodes except the faulty device.

[0834] In some embodiments of the present application, step S3202 includes: if it is determined that the number of faulty devices with data corruption or failure in the devices to be backed up and the verification nodes meets the fault tolerance, the damaged data blocks of the faulty device are restored based on the undamaged data blocks of other data nodes.

[0835] Since two sets of verification information are added, namely, a set of first verification data blocks and a set of second verification data blocks, the fault tolerance for data loss in the embodiment of the present application is 2, that is, the maximum number of faulty devices that can tolerate data corruption or failure in the backup device and the verification node at the same time is 2.

[0836] In some embodiments of the present application, when it is determined that the number of faulty devices with data corruption among the devices to be backed up and the verification nodes is one, meaning that only one terminal device is a faulty device, the damaged data blocks of the faulty device are restored based on the undamaged data blocks of the remaining data nodes. In embodiments of the present application, the repair scenario with one faulty device is referred to as single-node fault repair. In other words, if it is determined that only one column of data blocks among the devices to be backed up and the verification nodes is damaged or faulty, data recovery for the single-node fault is performed.

[0837] Optionally, in a single-node fault repair example, when the faulty device is a verification node storing a second verification data block, that is, the damaged data block is the second verification data block, the terminal device restores the second verification data block in the verification node based on the undamaged data blocks of the remaining data nodes (such as the device to be backed up and the verification node storing the first verification data block).

[0838] For example, the terminal device can use the original data blocks of each device to be backed up to encode and restore the second parity data block. Alternatively, the terminal device can use the original data blocks of each device to be backed up and the first parity data block to encode and restore the second parity data block. Thus, if a data failure occurs in the second parity data block, the damaged data block can be restored by encoding the related undamaged data blocks that generated the second parity data block.

[0839] Optionally, in another single-node fault repair example, when the faulty device is a check node storing a first check data block, the terminal device encodes and restores the check node according to the original data blocks of each device to be backed up.

[0840] When the faulty device is one of the devices to be backed up, and a column of original data blocks is damaged or faulty, the damaged data block is the original data block, and the damaged terminal device is the device to be backed up. The terminal device uses the first check data block and the undamaged original data block to encode and restore the damaged data block.

[0841] Therefore, when a data failure occurs in the first verification data block or in the original data block, encoding can be performed based on the undamaged data blocks related to the generation of the first verification data block to restore the lost data block.

[0842] In the above two single-node fault repair examples, a single verification data set, i.e., one of the first verification data block set or the second verification data block set, is used to repair the damaged data. An enumeration method is used to repair the damaged data blocks, which requires reading a large number of remaining surviving data blocks. The amount of data read is large and the repair efficiency is low.

[0843] Therefore, for the case where data corruption occurs in a non-second check data block, the embodiment of the present application provides another example of single-node fault repair.

[0844] In this example, the corresponding Figure 27 、 Figure 28 and Figure 29 The encoding method shown corresponds to the case where the second verification data block is generated by encoding the original data block and the first verification data block. Since there is a non-empty data block intersection between the encoded data blocks when the first verification data block and the second verification data block are generated, two groups of verification data blocks can be used for mixed repair.

[0845] At this time, the terminal device performs a mixed repair on the damaged data block using the first verification data block and the second verification data block.

[0846] Specifically, the terminal device may repair half of the damaged data blocks using the first verification data block, and repair the other half of the damaged data blocks using the second verification data block.

[0847] The embodiment of the present application is called a method for hybrid repair of a single node failure. It should be noted that when a single node failure is determined to have occurred and the failed data node is not the check node storing the second check data block, the damaged data block can be hybrid repaired using two columns of check information.

[0848] As a non-limiting example of this application, Figure 33 As shown in the figure, four terminal devices are used as four backup devices, namely device 1, device 2, device 3 and device 4; two devices are used as verification nodes, namely device 5 and device 6. Device 5 stores a set of first verification data blocks, and device 6 stores a set of second verification data blocks. If a set of original data blocks of device 1 is damaged or lost ( Figure 33 If a single check data set, i.e., a set of the first check data block and the second check data block, is used for data repair, it is necessary to read 12 original data blocks from device 2 to device 4 and 4 first check data blocks from device 5, for a total of 16 blocks of data. Figure 33 As shown, using a single checksum enumeration repair requires reading data from 16 data blocks.

[0849] As another non-limiting example of this application, Figure 34 As shown, four terminal devices are used as four backup devices, namely device 1, device 2, device 3 and device 4; two devices are used as verification nodes, namely device 5 and device 6. Device 5 stores a set of first verification data blocks, and device 6 stores a set of second verification data blocks. It should be noted that if Figure 34 In the example shown, the second check data block uses Figure 27The encoding method shown is used to generate the second check data block. If a set of original data blocks of device 1 is damaged or lost ( Figure 34 The four blackened data blocks in the middle), when the hybrid repair algorithm is used, the two damaged data blocks ( Figure 34 The two blackened data blocks in the middle are repaired using the first check data block; the other two data blocks ( Figure 34 The two blackened data blocks in the middle and lower parts are repaired using the second check data block. There is no need to read the duplicate data blocks again. Only 12 data blocks need to be read to quickly repair a single node failure. Figure 33 The repair method of the single verification information set shown can save 25% of the data reading amount, thereby improving the data repair efficiency and shortening the time to recover data.

[0850] In some embodiments of the present application, when it is determined that the number of failed devices among the backup device and the check node is two, which means that there are two nodes other than the backup device and the check node that have lost data, two columns or two rows of data blocks are lost in the codeword array.

[0851] Optionally, if it is determined that the number of faulty devices among the devices to be backed up and the verification nodes is two, and the faulty devices are the verification nodes storing the first verification data block and the verification nodes storing the second verification data block, the original data blocks of the devices to be backed up are encoded to restore the damaged data blocks of the faulty devices.

[0852] If it is determined that the number of faulty devices among the devices to be backed up and the verification nodes is two, and the faulty device is one of the devices to be backed up and the verification node storing the second verification data block, then the damaged data blocks of the faulty device are restored based on the original data blocks and the first verification data blocks of the remaining devices to be backed up; the remaining devices to be backed up are terminal devices among the devices to be backed up in which no data damage has occurred.

[0853] If it is determined that the number of faulty devices among the devices to be backed up and the verification nodes is two, and the faulty device does not include the verification node storing the second verification data block, the damaged data blocks of the faulty device are restored by combining the original data blocks and the second verification data blocks of the remaining devices to be backed up; the remaining devices to be backed up are terminal devices among the devices to be backed up in which no data damage has occurred.

[0854] The following description is made by taking the case where two columns of data are lost, that is, the original data block of each device to be backed up is taken as one column. It should be understood that in other examples of the present application, the case where two rows of data are lost may also be made.

[0855] Losing two columns of data can be divided into the following situations:

[0856] 1) The two lost columns are check data columns. In this case, they can be repaired using the encoding algorithm described in the previous embodiment, that is, the first check data block is repaired using row check encoding, and the second check data block is repaired using skew check encoding.

[0857] 2) The two lost columns contain one column of original data information and one column of skew check information. In this case, the original data information is restored using the row check data column, and then the skew check data column is restored using the skew check code.

[0858] 3) The two lost columns of data blocks do not contain the skew check data column, that is, do not include the second check data block. In this case, the lost data blocks are repaired in sequence using the skew check set and row check set cross repair method (also known as hybrid repair).

[0859] Optionally, the cross-repair process includes:

[0860] 1. First, the lost data blocks are grouped to distinguish between those that can be repaired using the skew parity information and those that cannot be repaired. The two groups are then separated and the data blocks that can be repaired using the skew parity information are repaired using the skew parity data.

[0861] 2. Select the lost data block in the same row as the data block repaired in step 1 as the next member of the group to be repaired, and repair the member of the group;

[0862] 3. Identify the data blocks that can be repaired using the skew check data from the remaining lost data blocks, and repair the data blocks that can be repaired using the skew check data;

[0863] 4. Select the lost data block in the same row as the data block repaired in step 3 as the next member of the group to be repaired, and repair the member of the group;

[0864] 5. Alternately use the oblique checksum and row checksum repair sets to repair lost data blocks until all data blocks are repaired. By finding all data blocks that can be repaired using the oblique checksum data set, multiple lost data blocks can be repaired in parallel, improving data repair efficiency.

[0865] As a non-limiting example of this application. Figures 35 to 40 As shown, four terminal devices are used as four backup devices, namely device 1, device 2, device 3 and device 4; two devices are used as check nodes, namely device 5 and device 6. Device 5 stores row check data and device 6 stores oblique check data. It should be noted that the two columns of original data blocks of device 2 and device 4 are damaged or lost ( Figure 35 The two black columns of data blocks in the middle), the oblique check data uses Figure 27The encoding method shown is used to generate the diagonal encoding method to obtain the diagonal parity data. The process of recovering the two columns of original data blocks of device 2 and device 4 is as follows:

[0866] First, if Figure 35 As shown, the two lost columns of data blocks are distinguished and the data blocks that can be repaired with oblique check data are identified, that is, Figure 35 The two data blocks D marked by the dotted ellipse are shown in 1,3 and D 2,1 , first repair these two data blocks D 1,3 and D 2,1 .

[0867] Secondly, if Figure 36 As shown, select data block D 1,3 Lost data block D in the same row 1,1 , and data block D 2,1 Lost data block D in the same row 2,3 , as the repair object, for these two lost data blocks D 1,1 and D 2,3 Make repairs.

[0868] Afterwards, if Figure 37 As shown, the remaining lost data blocks are distinguished and the lost data blocks that can be repaired with oblique parity data are identified, that is, Figure 37 The lost data block D shown in 3,3 , repair this data block D 3,3 .

[0869] Then, if Figure 38 As shown, select data block D 3,3 Lost data block D in the same row 3,1 As the repair object, the lost data block D 3,1 Make repairs.

[0870] Then, if Figure 39 As shown, the remaining lost data blocks are distinguished and the lost data blocks that can be repaired with oblique parity data are identified, that is, Figure 39 The lost data block D shown in 0,3 , repair this data block D 0,3 .

[0871] Finally, if Figure 40 As shown, select data block D 0,3 Lost data block D in the same row 0,1 As the repair object, the lost data block D 0,1 Repair is performed. At this point, all lost data blocks are repaired.

[0872] Figure 41 A data backup system is applicable to the erasure code encoding method and data recovery method provided in the embodiment of the present application. Figure 41 The data backup system shown is used as an example to illustrate the applicable scenarios of the embodiments of the present application.

[0873] like Figure 41 As shown, the data backup system includes four terminal devices, namely a mobile phone 411, a laptop computer 412, a wristband 413 and a tablet computer 414.

[0874] Among them, the mobile phone 411, the laptop computer 412, the wristband 413 and the tablet computer 414 are connected to each other through a wired or wireless communication network. The mobile phone 411, the laptop computer 412, the wristband 413 and the tablet computer 414 each serve as a data node.

[0875] In order to avoid data loss due to loss or damage of the terminal device, or insufficient memory of the terminal device resulting in the inability to store more data, for example, to avoid loss of data of the wristband 413 due to damage or loss of the wristband 413, or limited storage space of the wristband 413, which may lead to insufficient memory and inability to store more data, the embodiment of the present application performs data redundancy backup through the above-mentioned erasure code encoding method, and restores data on the faulty device through the above-mentioned data recovery method.

[0876] The method of the embodiment of the present application can be performed on any of the four data nodes: mobile phone 411, laptop computer 412, wristband 413, and tablet computer 414. More generally, the method can also be performed on other data nodes that are communicatively connected to these four data nodes, such as a cloud server.

[0877] In one embodiment of the present application, a method for storing data includes: dividing the data to be backed up on four terminal devices into blocks to obtain a plurality of original data blocks arranged in order corresponding to each terminal device; selecting an original data block from each terminal device, and selecting original data blocks with the same sorting position, encoding them, and obtaining a first verification data block; selecting an original data block from each terminal device, and selecting original data blocks with different sorting positions, encoding them, and obtaining a second verification data block; or forming five groups of data blocks from four groups of original data blocks of four terminal devices and a group of first verification data blocks obtained by encoding, and selecting a data block with a different sorting position from each of four groups of data blocks in the five groups of data blocks, encoding them, and obtaining a second verification data block; and storing the first verification data block and the second verification data block.

[0878] Optionally, in some embodiments of the present application, the first verification data block and the second verification data block can be stored together in one of the four terminal devices, such as the laptop computer 412, or can be stored separately in two of the four terminal devices, such as the laptop computer 412 and the tablet computer 414. They can also be stored separately or together in other data nodes that are communicatively connected to the four data nodes, for example, the first verification data block and the second verification data block can be stored together in a cloud server, etc.

[0879] Optionally, in some embodiments of the present application, the original data blocks obtained after segmentation of each terminal device can be stored in the current data node, that is, the data to be backed up of the mobile phone 411 is stored in the mobile phone 411 after segmentation; the data to be backed up of the wristband 413 is stored in the wristband 413 after segmentation, and so on; or, the original data blocks obtained after segmentation of each terminal device can be stored in at least one of the other three data nodes that have better communication and interaction with the current data node, or are less offline, or have more stable device performance. For example, the data to be backed up of the wristband 413 is stored in the laptop computer 412 with stable device performance after segmentation.

[0880] When the right Figure 41 The data to be backed up of the four terminal devices shown in the figure are backed up, and the original data blocks obtained after each terminal device is divided into blocks are stored in the local data node, and the first verification data block and the second verification data block are stored together in the cloud server data node. Figure 41 The data of one or two of the four terminal devices shown is damaged and needs to be restored. For example, when the data in the bracelet 413 cannot be read due to loss or damage, the corresponding method of storing data, in one embodiment of the present application, the method of recovering data includes: obtaining information of the damaged data block to be recovered from the bracelet; the information includes information of the four terminal devices and the verification node (i.e., the cloud server); and recovering the damaged data block of the bracelet 413 based on the undamaged data blocks of the other three terminal devices participating in the encoding, i.e., the mobile phone 411, the laptop 412, and the tablet computer 414, and / or the cloud server.

[0881] Optionally, in some embodiments of the present application, the first check data block or the second check data block in the cloud server can be read to restore the damaged data block in the bracelet by enumeration. The amount of data read is large and the efficiency of repair is low.

[0882] Optionally, in some embodiments of the present application, the first and second check data blocks in the cloud server can be read to restore the damaged data blocks in the bracelet through a hybrid repair method. This reduces data reading overhead, improves repair efficiency, and shortens repair time.

[0883] The data storage method and data recovery method provided in the embodiments of the present application, on the one hand, redundantly back up the data by encoding, and add a group of first verification data blocks and a group of second verification data blocks by encoding, which greatly reduces the storage overhead compared with the backup fault tolerance method using copies; on the other hand, the first verification data block is obtained by encoding the original data blocks in the same sorting position, or encoding the data array in rows or columns; the second verification data block is obtained by obliquely encoding the original data blocks in different sorting positions, or by obliquely encoding the original data blocks and the first verification data blocks in different sorting positions. Compared with the existing erasure code encoding method, the encoding and decoding calculation complexity is low, the storage efficiency is high, and the data recovery speed is fast.

[0884] More generally, Figure 42 Another data backup system is applicable to the erasure code encoding method and data recovery method provided in the embodiment of the present application. Figure 42 The data backup system shown is used as an example to illustrate the applicable scenarios of the embodiments of the present application.

[0885] like Figure 42 As shown, the data backup system includes six terminal devices, namely terminal device 421, terminal device 422, terminal device 423, terminal device 424, terminal device 425 and terminal device 426. The six terminal devices are connected to each other via a wired or wireless communication network. Each terminal device serves as a data node.

[0886] In one embodiment of the present application, terminal devices 421 to 424 need to perform data backup as the source of the original data blocks. In other words, terminal devices 421 to 424 participate in encoding together as devices to be backed up. Terminal devices 421 to 424 each store their own original data blocks.

[0887] Terminal device 425 is used to store the first check data block, and terminal device 426 is used to store the second check data block; or, terminal device 425 is used to store the second check data block, and terminal device 426 is used to store the first check data block. In other words, terminal device 425 and terminal device 426 serve as check nodes.

[0888] In another embodiment of the present application, the terminal devices 421 to 424 need to perform data backup and serve as devices to be backed up. The terminal devices 421 to 424 each store their own original data blocks.

[0889] The terminal device 425 is configured to store the first verification data block and the second verification data block as a verification node.

[0890] In another embodiment of the present application, terminal device 421, terminal device 422, terminal device 423, and terminal device 425 need to perform data backup as devices to be backed up. Terminal device 421, terminal device 422, terminal device 423, and terminal device 425 each store their own original data blocks.

[0891] Terminal device 424 is configured to store the first and second check data blocks, with terminal device 424 acting as a check node. Alternatively, terminal device 424 is configured to store the first check data block, while terminal device 426 is configured to store the second check data block. In other words, terminal device 424 and terminal device 426 act as check nodes. Alternatively, terminal device 424 is configured to store the second check data block, while terminal device 426 is configured to store the first check data block. In other words, terminal device 424 and terminal device 426 act as check nodes.

[0892] In another embodiment of the present application, terminal devices 421 to 424 need to perform data backup and serve as devices to be backed up.

[0893] The original data blocks of terminal devices 421 and 422 are both stored on terminal device 421, and the original data blocks of terminal devices 423 and 424 are both stored on terminal device 422. The first and second check data blocks are stored on terminal device 423, which serves as a check node. In this application scenario, the device to be backed up and the check node overlap.

[0894] In another embodiment of the present application, terminal devices 421 to 424 need to perform data backup and serve as devices to be backed up.

[0895] The original data blocks of terminal devices 421 and 422 are both stored on terminal device 421. The original data blocks and first check data blocks of terminal device 423 are both stored on terminal device 422. The original data blocks and second check data blocks of terminal device 423 are both stored on terminal device 423. Terminal devices 422 and 423 serve as check nodes. In this application scenario, the devices to be backed up and the check nodes overlap.

[0896] It should be noted that, in addition to being applicable to the above-mentioned distributed storage system, the above-mentioned erasure coding method can also be applied to other systems, such as RAID storage systems. Furthermore, the data to be backed up in the above-mentioned erasure coding method can be warm data and / or cold data determined by the above-mentioned data classification method, or it can also be warm data and / or cold data determined by other data classification methods, or it can also be other types of data to be backed up. The embodiments of the present application do not impose any restrictions on the system to which the above-mentioned erasure coding method is applied and the data to be backed up that participates in the erasure coding.

[0897] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0898] See also Figure 43 and Figure 44 , this embodiment of the application provides a terminal device 4300. Figure 43 As shown, the terminal device may include a metadata management module 4301 , a data service module 4302 , a data heat calculation module 4303 , a hot and cold data separation module 4304 and a data classification management module 4305 .

[0899] The data service module 4302 is used to store the data to be classified. The metadata management module 4301 is used to store data attribute information, such as write time, data type, and other attribute information. The data heat calculation module 4303 is used to calculate the first heat value of the data to be classified. The hot and cold data separation module 4304 is used to classify the heat categories of the data to be classified based on the first heat value. The data classification management module 4305 is used to perform corresponding data management operations based on the heat category of the data to be classified.

[0900] Figure 44 4303 is a functional structure diagram of the data heat calculation module 4303 provided in this embodiment. Figure 44 As shown, the data heat calculation module 4303 can realize the functions of storing the second heat value, managing data access records, heat value attenuation calculation and first heat value calculation.

[0901] The function of storing the second heat value is used to store the initial heat value and the previous first heat value.

[0902] The data access log management function manages access records, retaining only access records within N days and deleting access records older than N days to prevent excessive storage space on the terminal device from being taken up by a large number of access records, where N is a preset positive integer. To implement this function, the terminal device can set up a data list in the data heat calculation module to record data access records. This data list manages data access records. Access records include access record values ​​and initial time information.

[0903] The popularity value decay calculation function is used to calculate the second decay value of the second popularity value based on a preset second time decay function and second time information, and to calculate the first decay value of the access popularity value in each access record based on a preset first time decay function and the first time information in each access record. The second time information is the start time of the preset time period.

[0904] The first heat value calculation function is used to calculate the sum of the first attenuation values ​​of each access record of the data, and calculate the current first heat value of the data based on the sum result.

[0905] In this embodiment, the information interaction, execution process, etc. between the various modules of the above-mentioned terminal device are based on the same concept as the method embodiment of this application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0906] See also Figure 45 The embodiment of the present application further provides a terminal device. The terminal device 45 includes: a processor 450, a memory 451, and a computer program 452 stored in the memory 451 and executable on the processor 450. When the processor 450 executes the computer program 452, the steps in the above-mentioned data classification method embodiment are implemented, for example Figure 1 Alternatively, when the processor 450 executes the computer program 452 , the functions of the modules / units in the above-mentioned device embodiments are realized.

[0907] Exemplarily, the computer program 452 may be divided into one or more modules / units, which are stored in the memory 451 and executed by the processor 450 to complete the present application. The one or more modules / units may be a series of computer program instruction segments capable of performing specific functions, which are used to describe the execution process of the computer program 452 in the terminal device 45. For example, the computer program 452 may be divided into a record acquisition module, a first attenuation module, a heat calculation module, and a classification management module, with the specific functions of each module being as follows:

[0908] A record acquisition module, configured to acquire access records of the data to be classified, wherein the access records include access popularity values ​​and first time information;

[0909] A first attenuation module, configured to calculate a first time attenuation coefficient according to the first time information;

[0910] a heat calculation module, configured to calculate a first heat value according to a first time decay coefficient and an access heat value corresponding to each access record, and determine a heat category of the data to be classified according to the first heat value;

[0911] The classification management module is used to perform data management operations corresponding to the popularity category on the data to be classified.

[0912] The terminal device 45 can be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server. The terminal device can include, but is not limited to, a processor 450 and a memory 451. Those skilled in the art will understand that Figure 45 It is merely an example of the terminal device 45 and does not constitute a limitation on the terminal device 45. The terminal device may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the terminal device may also include input and output devices, network access devices, buses, etc.

[0913] The processor 450 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0914] The memory 451 may be an internal storage unit of the terminal device 45, such as a hard disk or memory of the terminal device 45. The memory 451 may also be an external storage device of the terminal device 45, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the terminal device 45. Furthermore, the memory 451 may include both an internal storage unit of the terminal device 45 and an external storage device. The memory 451 is used to store the computer program and other programs and data required by the terminal device. The memory 451 may also be used to temporarily store data that has been output or is about to be output.

[0915] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0916] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0917] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0918] In the embodiments provided in this application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are merely illustrative. For example, the division of the modules or units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0919] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0920] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0921] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the process in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by the processor, it can implement the steps of the above-mentioned various method embodiments. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0922] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A data processing method, characterized in that: Applied to a first device, the method includes: Acquire information of the data to be classified, the information including all access popularity values ​​of the data to be classified within a preset time period and first time information corresponding to the access popularity values; Calculate a first decay value of each access popularity value according to the first time information and a preset first time decay function; Calculating a first popularity value according to the first attenuation values ​​corresponding to the respective access popularity values; Determining a popularity category of the data to be classified according to the first popularity value; Selecting a first preset proportion of warm data from the data to be classified whose heat type is warm data as first backup data; Performing erasure coding on the first backup data and the data to be classified whose popularity category is cold data to obtain first verification data; The data to be classified whose popularity category is cold data is sent to a first designated terminal, the first verification data is sent to a second designated terminal, and the locally stored data to be classified whose popularity category is cold data is deleted.

2. The method according to claim 1, characterized in that The calculating the first decay value of each access popularity value according to the first time information and a preset first time decay function includes: Calculating a first time decay coefficient of each access popularity value according to the first time information and the preset first time decay function; Multiplying the first time attenuation coefficient of each access heat value by each access heat value to obtain a first attenuation value corresponding to each access heat value; The calculating the first heat value according to the first attenuation value corresponding to each access heat value includes: The first attenuation values ​​corresponding to the respective access heat values ​​are summed to obtain the first heat value.

3. The method according to claim 1, characterized in that The information also includes a second heat value and second time information corresponding to the second heat value, wherein the second heat value is a preset initial heat value or a most recently calculated first heat value; Accordingly, after obtaining the information of the data to be classified, the method further includes: Calculating a second attenuation value of the second heat value according to the second time information and a preset second time attenuation function, wherein the second time information is a start time of the preset time period; Accordingly, the calculating of the first heat value according to the first attenuation value corresponding to each access heat value includes: Calculating the first heat value of the data to be classified according to the second attenuation value of the second heat value and the first attenuation values ​​corresponding to the respective access heat values; Update the second heat value to the first heat value.

4. The method according to claim 3, characterized in that The calculating the second attenuation value of the second heat value according to the second time information and a preset second time attenuation function includes: Calculating a second time decay coefficient according to the second time information and the preset second time decay function; multiplying the second time attenuation coefficient and the second heat value to obtain a second attenuation value of the second heat value; Accordingly, the calculating of the first heat value of the data to be classified according to the second attenuation value of the second heat value and the first attenuation values ​​corresponding to the respective access heat values ​​includes: The first heat value is obtained by summing the second attenuation value of the second heat value and the first attenuation values ​​corresponding to the respective access heat values.

5. The method according to any one of claims 1 to 4, characterized in that The performing erasure coding on the first backup data and the data to be classified as cold data to obtain first verification data includes: The first backup data is warm data having the same data volume as the cold data, and erasure coding is performed on the first backup data and the data to be classified whose popularity category is cold data to obtain the first verification data; Alternatively, the first backup data is warm data with a different data volume from the cold data. If the data volume of the first backup data is less than that of the cold data, the first backup data is padded with zeros to have the same data volume as the cold data, and erasure coding is performed on the zero-padded first backup data and the data to be classified whose heat category is cold data to obtain the first verification data.

6. The method according to any one of claims 1 to 4, characterized in that The method further comprises: Splitting the warm data other than the first backup data in the data to be classified whose popularity type is warm data into the second backup data and the third backup data; Performing erasure coding on the second backup data and the third backup data to obtain second verification data; The second verification data is sent to a third designated terminal, and the backup data of the second backup data is sent to a fourth designated terminal.

7. The method according to claim 1, characterized in that The heat category includes heat data, and the method further includes: The data to be classified whose popularity category is hot data is stored in a local storage space, and the backup data of the hot data is sent to a data backup terminal for backup.

8. The method according to claim 1, characterized in that Also includes: Determining a stripe length of an erasure coded stripe, wherein the stripe length is less than or equal to the number of online devices in the distributed storage system; Acquire data to be backed up for each device to be backed up, where the device to be backed up is an online device having data to be backed up, the device to be backed up includes the first device, and the data to be backed up includes the data to be classified; Performing erasure coding according to the stripe length and the data to be backed up to obtain target check data; The target verification data is sent to the verification node for storage.

9. The method according to claim 8, characterized in that Determining the stripe length of the erasure code stripe includes: The number of online devices is determined as a stripe length of an erasure code stripe.

10. The method according to claim 8, characterized in that Determining the stripe length of the erasure code stripe includes: Determining the number of devices to be backed up in the online devices; The number of the devices to be backed up is determined as the stripe length of the erasure code stripe.

11. The method according to claim 8, characterized in that Determining the stripe length of the erasure code stripe includes: Obtain the data volume of each data to be backed up; Merging the data to be backed up whose data volume is less than a first threshold according to a preset merging rule to obtain the data to be backed up whose data volume is greater than or equal to the first threshold; The amount of data to be backed up after the merge process is determined as the stripe length of the erasure code stripe.

12. The method according to any one of claims 8 to 11, characterized in that Also includes: Determining the data volume of the largest data block among the data blocks divided by each of the online devices as the stripe depth of the erasure code stripe; Alternatively, set the stripe depth of erasure coded stripes to a preset value.

13. The method according to claim 12, characterized in that The performing erasure coding according to the stripe length and the data to be backed up to obtain target verification data includes: Generate an erasure code stripe according to the stripe length and the original data blocks corresponding to each of the data to be backed up; Erasure coding is performed on the erasure code stripe to obtain target verification data.

14. The method according to claim 13, characterized in that Before performing erasure coding on the erasure coded stripe to obtain target check data, the method further includes: If the number of original data blocks in the erasure code stripe is less than the stripe length, zero padding is performed on the erasure code stripe so that the number of original data blocks in the erasure code stripe is equal to the stripe depth.

15. The method according to claim 13, characterized in that Before performing erasure coding on the erasure coded stripe to obtain target check data, the method further includes: If there is an original data block with a data volume less than the stripe depth in the erasure code stripe, the original data block with a data volume less than the stripe depth is padded with zeros so that the data volume of the original data block with a data volume less than the stripe depth is equal to the stripe depth.

16. The method according to claim 8, characterized in that The verification node is a cloud storage device or a network attached storage NAS device.

17. The method according to claim 8, characterized in that Before sending the target verification data to the verification node for storage, the method further includes: Acquire device parameters of each of the online devices, the device parameters including one or more parameters of device reliability, device online time, and device storage space; A check node is selected from each of the online devices according to the device parameters of each of the online devices and a preset check node selection rule.

18. The method according to claim 8, characterized in that The obtaining of the data to be backed up includes: Determining a device to be backed up based on metadata corresponding to each of the online devices, the metadata being used to indicate whether the online device has data to be backed up and / or the amount of data to be backed up; The data to be backed up is read from each of the devices to be backed up.

19. The method according to claim 8, characterized in that The performing erasure coding according to the stripe length and the data to be backed up to obtain target verification data includes: When the stripe length satisfies the condition that p is a prime number greater than or equal to 3, the original data blocks corresponding to each data to be backed up are obtained to obtain N groups of original data blocks, where p is N+1 and N is the stripe length. One data to be backed up corresponds to a group of original data blocks, and the original data blocks in the same group have different sorting positions. Selecting an original data block with the same sorting position from each of the N groups of original data blocks for erasure coding to obtain first check data blocks corresponding to different sorting positions; Selecting an original data block with a different sorting position from each of the N groups of original data blocks for erasure coding to obtain a second check data block corresponding to each different sorting position; Alternatively, N groups of data blocks are selected from the N groups of original data blocks and a group of first verification data blocks, and a data block with a different sorting position is selected from each of the N groups of data blocks for erasure coding to obtain second verification data blocks corresponding to each different sorting position.

20. The method according to claim 19, characterized in that The step of selecting an original data block having the same sorting position from each of the N groups of original data blocks for erasure coding to obtain first check data blocks corresponding to different sorting positions includes: Arrange a group of original data blocks into a column according to the sorting position, with different groups of original data blocks corresponding to different columns, to form a first data block array; Performing erasure coding on each row in the first data block array to obtain first check data blocks corresponding to different sorting positions; Accordingly, selecting an original data block with a different sorting position from each of the N groups of original data blocks for erasure coding to obtain a second check data block corresponding to each different sorting position includes: Selecting an original data block with a different sorting position from each column of the first data block array to perform erasure coding, to obtain second check data blocks corresponding to the different sorting positions; Accordingly, selecting N groups of data blocks from the N groups of original data blocks and a group of first check data blocks, selecting a data block with a different sorting position from each of the N groups of data blocks for erasure coding, and obtaining second check data blocks corresponding to each different sorting position, includes: Arranging a group of first check data blocks into a column according to the sorting position, and expanding the first data block array according to the column corresponding to the first check data block to obtain a first expanded array; N columns are selected from the N+1 columns of the first extended array, and a data block with a different sorting position is selected from each of the N columns for erasure coding to obtain second check data blocks corresponding to the different sorting positions.

21. The method according to claim 20, characterized in that The selecting an original data block with a different sorting position from each column of the first data block array to perform erasure coding to obtain second check data blocks corresponding to the different sorting positions includes: Performing erasure coding on different diagonal lines on the first data block array to obtain second check data blocks corresponding to different sorting positions; The step of selecting N columns from the N+1 columns of the first extended array, and selecting a data block with a different sorting position from each of the N columns for erasure coding to obtain second check data blocks corresponding to the different sorting positions, includes: Erasure coding of different diagonal lines is performed on the first extended array to obtain second check data blocks corresponding to different sorting positions.

22. The method according to claim 19, wherein The step of selecting an original data block having the same sorting position from each of the N groups of original data blocks for erasure coding to obtain first check data blocks corresponding to different sorting positions includes: Arrange a group of original data blocks into a row according to the sorting position, with different groups of original data blocks corresponding to different rows, to form a second data block array; Performing erasure coding on each column in the second data block array to obtain first check data blocks corresponding to different sorting positions; Accordingly, selecting an original data block with a different sorting position from each of the N groups of original data blocks for erasure coding to obtain a second check data block corresponding to each different sorting position includes: Selecting an original data block with a different sorting position from each row of the second data block array to perform erasure coding, to obtain second check data blocks corresponding to the different sorting positions; Accordingly, selecting N groups of data blocks from the N groups of original data blocks and a group of first check data blocks, selecting a data block with a different sorting position from each of the N groups of data blocks for erasure coding, and obtaining second check data blocks corresponding to each different sorting position, includes: Arranging a group of first check data blocks into a row according to the sorting position, and expanding the second data block array according to the row corresponding to the first check data blocks to obtain a second expanded array; N rows are selected from the N+1 rows of the second extended array, and a data block with a different sorting position is selected from each of the N rows for erasure coding to obtain second check data blocks corresponding to the different sorting positions.

23. The method according to claim 22, characterized in that The selecting an original data block with a different sorting position from each row of the second data block array to perform erasure coding to obtain second check data blocks corresponding to the different sorting positions includes: Performing encoding on different diagonals on the second data block array to obtain second check data blocks corresponding to different sorting positions; The step of selecting N rows from the N+1 rows of the second extended array, and selecting a data block with a different sorting position from each of the N rows for erasure coding to obtain second check data blocks corresponding to the different sorting positions, includes: The second extended array is coded along different diagonals to obtain second check data blocks corresponding to different sorting positions.

24. The method according to any one of claims 19 to 23, characterized in that The erasure code is encoded as XOR encoding or XOR encoding.

25. The method according to claim 19, wherein Also includes: When a damaged data block exists, the damaged data block is repaired according to an undamaged original data block and undamaged target verification data, where the target verification data includes the first verification data block and the second verification data block.

26. The method according to claim 25, characterized in that When a damaged data block exists, repairing the damaged data block according to the undamaged original data block and the undamaged target verification data includes: When there are damaged data blocks and the number of groups of the damaged data blocks is less than or equal to 2, the damaged data blocks are repaired according to the undamaged original data blocks and the undamaged target verification data, where the target verification data includes the first verification data block and the second verification data block.

27. The method according to claim 25, characterized in that When a damaged data block exists, repairing the damaged data block according to the undamaged original data block and the undamaged target verification data includes: When any group of data blocks is damaged, obtain the type of the damaged data blocks; If the damaged data block is a first parity data block, repairing the damaged first parity data block according to the N groups of original data blocks; If the damaged data block is a second parity data block, repairing the damaged second parity data block according to the N groups of original data blocks, or repairing the damaged second parity data block according to the N groups of original data blocks and the first parity data block; If the damaged data block is an original data block, the damaged original data block is repaired based on the undamaged original data block and the first verification data block, or the damaged data block is repaired based on a mixture of the undamaged original data block, the first verification data block and the second verification data block.

28. The method according to claim 25, characterized in that When a damaged data block exists, repairing the damaged data block according to the undamaged original data block and the undamaged target verification data includes: When any two groups of data blocks are damaged, obtain the type of the damaged data blocks; If the damaged data block is a group of first check data blocks and a group of second check data blocks, repairing the damaged first check data blocks and second check data blocks according to the N groups of original data blocks; If the damaged data block is a group of original data blocks and a group of second verification data blocks, repairing the damaged original data blocks and the second verification data blocks according to the undamaged original data blocks and the first verification data blocks; If the damaged data block does not include the second verification data block, the damaged data block is repaired based on a mixture of the undamaged original data block and the second verification data block.

29. A terminal device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 28 is implemented.

30. A computer-readable storage medium storing a computer program, wherein: When the computer program is executed by a processor, the method according to any one of claims 1 to 28 is implemented.

Images