A data processing method, device, computer device and cluster

Abstract

A data processing method and device, computer equipment and cluster are disclosed, and relate to the field of computers. Multiple sliding windows processing the to-be-stored data at a time can obtain multiple cut points, reducing the calculation time required by the computer equipment to obtain the cut points. If the number of sliding windows is too large, the cache storage space may not be sufficient, the cache occupancy rate is high, or multiple sliding windows determine multiple cut points in parallel, the calculation time of the cut points may also be longer. The computer equipment adjusts the parallelism according to at least one of the calculation time and the cache occupancy rate of the first cut point set, adjusts the number of sliding windows, so as to reduce the calculation time of the cut points, reduce the cache occupancy, obtain reasonable cut points of the to-be-stored data as soon as possible, adjust the length of the data block selected by the multiple sliding windows, improve the accuracy of the computer equipment to obtain the cut points, improve the acquisition rate of the data duplicated with the stored data in the to-be-stored data, and improve the data deduplication rate.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of storage, and particularly relates to a data processing method and device, computer equipment and cluster. BACKGROUND

[0002] Data deduplication is a data reduction technology, that is, a data stream is divided into multiple data blocks, and the same data block in the data stream as the stored data is deleted to reduce the storage capacity occupied by the redundant data in the storage device. Wherein, the data stream is divided according to a variable-length chunking algorithm. For example, a single sliding window is used to traverse the data stream to obtain multiple cut points, and multiple data blocks are obtained according to the multiple cut points. However, if the length of the sliding window is too long, the cut points cannot be accurately obtained, resulting in a low deduplication rate; if the length of the sliding window is too short, the calculation of the cut points is time-consuming. SUMMARY

[0003] The present application provides a data processing method and device, computer equipment and cluster, which solves the problems of high calculation time and low deduplication rate caused by the inappropriate length of the selected sliding window.

[0004] In a first aspect, the present application provides a data processing method. The method comprises: a computing device obtaining a first cut point set of a first data segment in to-be-stored data according to a first parallelism, and dividing the first data segment according to at least one cut point in the first cut point set to obtain a first data set. The first parallelism is used to indicate M sliding windows. The computing device adjusts the first parallelism according to at least one of the calculation time and the cache occupancy rate of the first cut point set to obtain a second parallelism. The first cache occupancy rate is used to indicate the cache occupancy rate of performing data deduplication on the first data segment according to the first cut point set. The second parallelism is used to indicate N sliding windows. M is not equal to N, and M and N are integers greater than or equal to 2. The computing device obtains a second cut point set of a second data segment in the to-be-stored data according to the second parallelism, and divides the second data segment according to at least one cut point in the second cut point set to obtain a second data set. The computing device performs data deduplication on the first data set and the second data set. The first data set comprises at least one data block, and the second data set comprises at least one data block.

[0005] Compared with the computing device determining the cut point of the to-be-stored data by using a single sliding window, in this application, the computing device processes the to-be-stored data by using multiple sliding windows in parallel to obtain a first cut point set including at least one cut point. The multiple sliding windows can obtain at least one cut point in a single processing of the to-be-stored data, thereby reducing the computing time required by the computing device to obtain the cut point. If the number of sliding windows is too large, the multiple sliding windows may slide in parallel to find the cut point of the to-be-stored data, the storage space of the cache may not be enough, the cache occupancy rate is relatively high, or the multiple sliding windows may slide in parallel to determine multiple cut points, and the computing time of the cut point may be relatively long. Therefore, the computing device adjusts the parallelism of processing the to-be-stored data according to at least one of the computing time and the cache occupancy rate of the first cut point set, adjusts the number of sliding windows, so as to reduce the computing time of the cut point and the cache occupancy as much as possible, to find the reasonable cut point of the to-be-stored data as soon as possible, adjusts the length of the data block selected by the multiple sliding windows, improves the accuracy of the computing device to obtain the cut point, improves the data deduplication rate of obtaining the data included in the to-be-stored data and the already-stored data, and improves the data deduplication rate.

[0006] In a possible implementation, M sliding windows slide on the to-be-stored data at least once, and the computing device obtains a first cut point set of the first segment of data.

[0007] In this way, the computing device adjusts the parallelism of processing the to-be-stored data after the M sliding windows slide on the to-be-stored data multiple times, thereby avoiding the consumption of computing and storage resources caused by frequent adjustment of the parallelism.

[0008] In another possible implementation, the computing device divides the first segment of data according to the M sliding windows to obtain a first initial data set, and obtains the first cut point set according to a cut point determination rule associated with the length and length distribution characteristics of the data block in the first initial data set.

[0009] In another possible implementation, the computing device obtains the first cut point set according to a cut point determination rule associated with a length interval determined according to the length, the maximum value, the minimum value and the expected value of the data block in the first initial data set.

[0010] In this way, the computing device uses different cut point determination rules to determine the cut points of data blocks of different lengths, thereby improving the accuracy of the computing device to obtain the cut point.

[0011] In another possible implementation, the M sliding windows divide the first segment of data at an interval of at least 1 byte, and the computing device obtains a first initial cut point set and a first initial data set.

[0012] In this way, the M sliding windows divide the first segment of data at an interval of at least 1 byte, so that each sliding window selects a different position of the first segment of data, thereby improving the efficiency of the sliding window in traversing the first segment of data.

[0013] In another possible implementation, the M sliding windows have the same length.

[0014] In this way, the M sliding windows have the same length, so that the lengths of the data blocks framed by each sliding window are the same, and the accuracy of the cut point calculated by the computing device is improved.

[0015] In another possible implementation, the time consumption for calculating the first cut point set includes at least one of a first time length, a second time length, and a sum of the first time length and the second time length. The first time length is used to indicate a time length required for dividing the first segment of data into the first initial data set by the M sliding windows. The second time length is used to indicate a time length required for obtaining the first cut point set by the computing device according to the cut point determination rule associated with the length and length distribution characteristics of the data blocks in the first initial data set.

[0016] In this way, the time consumption is obtained according to the time length required for obtaining the first initial data set and the time length required for obtaining the first cut point set. In a case where the time length required for obtaining the first initial data set is greater than the time length required for obtaining the first cut point set, the computing device reduces the time length required for obtaining the first initial data set. In a case where the time length required for obtaining the first initial data set is less than the time length required for obtaining the first cut point set, the computing device increases the time length required for obtaining the first initial data set. In this way, the time length required for obtaining the first initial data set and the time length required for obtaining the first cut point set are balanced.

[0017] In another possible implementation, in a case where at least one of the time consumption for calculating the first cut point set and the cache occupancy rate is greater than a corresponding upper limit value, the computing device reduces the M sliding windows to N sliding windows.

[0018] In this way, in a case where the time consumption for calculating the first cut point set is greater than the corresponding upper limit value and / or the cache occupancy rate is greater than the corresponding upper limit value, the resources that can be provided by the computing device cannot support the computing device to process the to-be-stored data by using the M sliding windows. The computing device reduces the number of sliding windows, so that the resources that can be provided by the computing device and the resources required for processing the to-be-stored data by using the sliding windows are matched.

[0019] In another possible implementation, in a case where at least one of the time consumption for calculating the first cut point set and the cache occupancy rate is less than a corresponding lower limit value, the computing device increases the M sliding windows to N sliding windows.

[0020] In this way, in a case where the time consumption for calculating the first cut point set is less than the corresponding upper limit value and / or the cache occupancy rate is less than the corresponding upper limit value, the computing device cannot effectively utilize the resources that can be provided by the computing device by using the M sliding windows to process the to-be-stored data. The computing device increases the number of sliding windows, so that the resources that can be provided by the computing device and the resources required for processing the to-be-stored data by using the sliding windows are matched.

[0021] In another possible implementation, the computing device deletes the data blocks in the first data set that match the stored data, and stores the data blocks in the first data set that do not match the stored data. In addition, the computing device deletes the data blocks in the second data set that match the stored data, and stores the data blocks in the second data set that do not match the stored data.

[0022] In this way, the computing device deletes the matched data blocks, and stores the unmatched data blocks. The storage capacity required by the computing device to store the to-be-stored data is reduced, and the deduplication rate is improved.

[0023] In another possible implementation, the stored data includes a second data block and a third data block. The at least one data block in the first data set matches the second data block. The computing device divides the third segment of the to-be-stored data by using the length of the third data block to obtain a fourth data block.

[0024] In this way, the computing device avoids selecting and dividing the to-be-stored data to obtain the data blocks by using the sliding window to slide byte by byte, reduces the time required to obtain the cut point, and improves the efficiency of data deduplication.

[0025] In another possible implementation, the first data set includes a fifth data block, a sixth data block, and a seventh data block. The lengths of the fifth data block and the sixth data block are the same as the maximum of the lengths. The first cut point divides the third segment of the to-be-stored data to obtain the fifth data block. The second cut point divides the fourth segment of the to-be-stored data to obtain the seventh data block. The computing device generates a cut point feature table of the fifth data block and the sixth data block. The cut point feature table includes: a cut point feature and a cut point distance. The cut point feature is used to indicate the fifth data block and the sixth data block. The cut point distance is used to indicate the distance between the second cut point and the first cut point.

[0026] In this way, the computing device generates a cut point feature table for a plurality of cut points, and saves the storage resources required by the computing device to store the cut point feature table.

[0027] Secondly, this application provides a data processing apparatus. The apparatus includes a cut-point determination module, an adjustment module, and a processing module. The cut-point determination module is configured to: obtain a first cut-point set of a first segment of data in the data to be stored based on a first degree of parallelism; divide the first segment of data according to at least one cut-point in the first cut-point set to obtain a first dataset. The first degree of parallelism is used to indicate M sliding windows. The adjustment module is configured to: adjust the first degree of parallelism based on at least one of the computation time and cache occupancy rate of the first cut-point set to obtain a second degree of parallelism. The first cache occupancy rate is used to indicate the cache occupancy rate for performing data deduplication on the first segment of data according to the first cut-point set. The second degree of parallelism is used to indicate N sliding windows, where M is not equal to N, and M and N are integers greater than or equal to 2. The cut-point determination module is further configured to: obtain a second cut-point set of a second segment of data in the data to be stored based on the second degree of parallelism; divide the second segment of data according to at least one cut-point in the second cut-point set to obtain a second dataset. The processing module is further configured to: perform data deduplication on the first dataset and the second dataset. The first dataset includes at least one data block, and the second dataset includes at least one data block.

[0028] In one possible scenario, the cut-point determination module is specifically used to: slide M sliding windows on the data to be stored at least once to obtain the first cut-point set of the first segment of data.

[0029] In another possible scenario, the cut-point determination module is specifically used to: divide the first data segment according to M sliding windows to obtain the first initial dataset. The cut-point determination module is also specifically used to: obtain the first cut-point set based on the cut-point determination rules relating the lengths and length distribution characteristics of the data blocks in the first initial dataset.

[0030] In another possible scenario, the cut-point determination module is specifically used to: obtain a first cut-point set based on the cut-point determination rules associated with the length intervals determined by the length of the data blocks in the first initial dataset and the maximum, minimum and expected lengths.

[0031] In another possible scenario, the cut-point determination module is specifically used to divide the first segment of data into M sliding window intervals of at least 1 byte to obtain the first initial dataset.

[0032] In another possible scenario, the lengths of the M sliding windows are the same.

[0033] In another possible scenario, the computation time for the first tangent set includes one of the following: first duration, second duration, first duration, and second duration.

[0034] The first duration indicates the time required to divide the data into the first segment using M sliding windows to obtain the first initial dataset. The second duration indicates the time required to obtain the first cut-point set based on the cut-point determination rules associated with the length and length distribution characteristics of the data blocks in the first initial dataset.

[0035] In another possible implementation, the adjusting module is specifically configured to: in a case where at least one of the calculation time consumption and the cache occupancy rate of the first set of cut points is greater than a respective upper limit value, reduce the M sliding windows to N sliding windows. M is greater than N.

[0036] In another possible implementation, the adjusting module is specifically configured to: in a case where at least one of the calculation time consumption and the cache occupancy rate of the first set of cut points is less than a respective lower limit value, increase the M sliding windows to N sliding windows. M is less than N.

[0037] In another possible implementation, the processing module is specifically configured to: delete at least one data block included in the first data set that matches the stored data, and store at least one data block included in the first data set that does not match the stored data. In addition, the processing module is also specifically configured to: delete at least one data block included in the second data set that matches the stored data, and store at least one data block included in the second data set that does not match the stored data.

[0038] In another possible implementation, the stored data includes a second data block and a third data block. At least one data block included in the first data set matches the second data block. The cut point dividing module is also configured to: divide a third segment of data of the to-be-stored data by using a length of the third data block to obtain a fourth data block.

[0039] In another possible implementation, the first data set includes a fifth data block, a sixth data block, and a seventh data block. The lengths of the fifth data block and the sixth data block are the same as the maximum value of the lengths. The first cut point divides the to-be-stored data to obtain the fifth data block, and the second cut point divides the to-be-stored data to obtain the seventh data block. The processing module is also configured to: generate a cut point feature table of the fifth data block and the sixth data block. The cut point feature table includes: a cut point feature and a cut point distance. The cut point feature is used to indicate the fifth data block and the sixth data block. The cut point distance is used to indicate the interval between the second cut point and the first cut point.

[0040] In a third aspect, a computer device is provided. The computer device includes a memory and a processor. The memory is configured to store a set of computer instructions. When the processor executes the set of computer instructions, the processor is configured to perform the operation steps of the data processing method in the first aspect or any possible design of the first aspect.

[0041] In a fourth aspect, a cluster is provided. The cluster includes a computing node and a storage node. The computing node performs the operation steps of the data processing method in the first aspect or any possible design of the first aspect, and the storage node is configured to store the data after the computing node performs data deduplication on the to-be-stored data.

[0042] In a fifth aspect, the present application provides a computer readable storage medium comprising computer software instructions, which when executed in a computing device, cause the computing device to perform the operations of the method according to the first aspect or any possible implementation of the first aspect.

[0043] In a sixth aspect, the present application provides a computer program product comprising instructions, which when executed in a cluster, cause the cluster to perform the operations of the method according to the first aspect or any possible implementation of the first aspect.

[0044] The advantages of the third aspect to the sixth aspect above can be referred to the description of any implementation of the first aspect or the second aspect, which will not be repeated here. The implementation of the present application in each aspect above can be further combined to provide more implementations. BRIEF DESCRIPTION OF DRAWINGS

[0045] Figure 1 A structural schematic diagram of a data processing system according to the present application is shown in the figure;

[0046] Figure 2 A schematic diagram of a fixed-length partitioning method according to the present application is shown in the figure;

[0047] Figure 3 A flowchart of obtaining data blocks according to the present application is shown in the figure;

[0048] Figure 4 A schematic diagram of obtaining data blocks by sliding window according to the present application is shown in the figure;

[0049] Figure 5 A schematic diagram of a variable-length partitioning method according to the present application is shown in the figure;

[0050] Figure 6 A schematic diagram of a combination of fixed-length and variable-length partitioning methods according to the present application is shown in the figure;

[0051] Figure 7 A flowchart of a data processing method according to the present application is shown in the figure;

[0052] Figure 8 A flowchart of obtaining a set of cut points according to the present application is shown in the figure;

[0053] Figure 9 A schematic diagram of data block length distribution according to the present application is shown in the figure;

[0054] Figure 10 A schematic diagram of the relationship between cut point determination probability and data block length according to the present application is shown in the figure;

[0055] Figure 11A corresponding diagram of a cut point determination rule and a length interval provided for the present application;

[0056] Figure 12 A diagram of obtaining a cut point provided for the present application;

[0057] Figure 13 A diagram of generating a cut point feature table provided for the present application;

[0058] Figure 14 A flow diagram of obtaining a cut point provided for the present application;

[0059] Figure 15 A diagram of adjusting parallelism provided for the present application;

[0060] Figure 16 A diagram of the relationship between cache hit rate and parallelism provided for the present application;

[0061] Figure 17 A data processing block diagram provided for the present application;

[0062] Figure 18 A structural diagram of a data processing apparatus provided for the present application;

[0063] Figure 19 A structural diagram of a computing device provided for the present application;

[0064] Figure 20 A diagram of a cluster provided for the present application;

[0065] Figure 21 A connection diagram of a computing device provided for the present application. DETAILED DESCRIPTION

[0066] For the sake of clear and concise description of the following embodiments, first give a brief introduction of related technologies.

[0067] Figure 1 A structural diagram of a data processing system provided for the present application, as shown in Figure 1 The data processing system 100 includes a client 110 and a computing device 120. In Figure 1 The application scenario shown, the user through the application program to access data. The computer running these application programs can be referred to as "client".

[0068] In a possible example, the client 110 can send data that needs to be stored to the computing device 120 through the network 130. For example, the network 130 can include a switch.

[0069] In another possible example, client 110 may also communicate with computing device 120 via a wired connection. For example, client 110 may communicate with computing device 120 via a universal serial bus (USB) or a Peripheral Component Interconnect Express (PCIe) bus.

[0070] Figure 1 The computing device 120 shown can be used to perform data deduplication operations on data to be stored, and to store the data after the deduplication operation. The computing device 120 can be a centralized data processing system or a distributed data processing system, which will be described below.

[0071] In the first possible scenario, the computing device 120 is a centralized data processing system.

[0072] In this case, the computing device 120 has a unified entry point through which all data from external devices must pass. This entry point can also be called engine 121.

[0073] like Figure 1 As shown, engine 121 may have one or more controllers. Figure 1 The following example illustrates the concept of engine 121 containing one controller. In one possible example, if engine 121 has multiple controllers, any two controllers can have a mirror channel, enabling any two controllers to serve as backups for each other, thereby preventing hardware failures from rendering the entire computing device 120 unusable. It should be understood that if engine 121 includes multiple controllers, then engine 121 can also be referred to as the array controller of computing device 120.

[0074] Engine 121 also includes a front-end interface 1211 and a back-end interface 1214. The front-end interface 1211 is used to communicate with client 110, thereby providing data access services to client 110. The back-end interface 1214 is used to communicate with hard drives to expand the capacity of computing device 120. Through the back-end interface 1214, engine 121 can connect to more hard drives, thus forming a storage resource pool with a very large storage capacity.

[0075] like Figure 1 As shown, each controller in engine 121 includes at least: processor 1212 and memory 1213.

[0076] The processor 1212 is a central processing unit (CPU) for processing data access requests from outside the computing device 120 (a server or other data processing system) as well as processing requests generated internally within the computing device 120. For example, when the processor 1212 receives a write data request from the client 110 via the front-end interface 1211, the processor 1212 temporarily stores the data in the write data request in the memory 1213. When the total amount of data in the memory 1213 reaches a certain threshold, the processor 1212 sends the data stored in the memory 1213 to at least one of the mechanical hard drive 1221, the mechanical hard drive 1222, the solid state drive (SSD) 1223, or the other hard drive 1224 via the back-end port for persistent storage.

[0077] The memory 1213 refers to an internal memory that directly exchanges data with the processor 1212, which can read and write data at any time and at a very fast speed, and is used as temporary data storage for an operating system or other programs that are running. The memory 1213 includes at least two types of memory, such as a random access memory or a read only memory (ROM). For example, the random access memory is a DRAM or an SCM. The DRAM is a type of semiconductor memory, and like most random access memories (RAMs), it is a type of volatile memory device. However, the DRAM and the SCM are only exemplary in this embodiment, and the memory can also include other random access memories, such as a static random access memory (SRAM), etc. For the read only memory, for example, it can be a programmable read only memory (PROM), an erasable programmable read only memory (EPROM), etc.

[0078] Additionally, memory 1213 can also be a dual in-line memory module (DIMM), i.e., a module composed of dynamic random access memory (DRAM), or an SSD. In practical applications, the controller can be configured with multiple memory modules 1213, and different types of memory modules 1213. This embodiment does not limit the number or type of memory modules 1213. Furthermore, memory modules 1213 can be configured to have a power-saving function. The power-saving function means that when the system experiences a power outage and then power is restored, the data stored in memory modules 1213 will not be lost. Memory with a power-saving function is called non-volatile memory.

[0079] The memory 1213 stores software programs, and the processor 1212 runs these programs to manage the hard drive. For example, the hard drive can be abstracted into a storage resource pool, and this pool can be provided to the server in the form of logical unit numbers (LUNs). Here, the LUN is essentially the hard drive as seen on the server. Of course, some centralized data processing systems are also file servers, providing shared file services to the server.

[0080] like Figure 1 As shown, in the data processing system 100, the engine 121 may not have a hard drive slot. In this case, the hard drive needs to be placed in the hard drive enclosure 122, and the back-end interface 1214 communicates with the hard drive enclosure 122. The back-end interface 1214 exists in the form of an adapter card in the engine 121. Two or more back-end interfaces 1214 can be used simultaneously on one engine 121 to connect multiple hard drive enclosures. Alternatively, the adapter card can also be integrated on the motherboard, in which case the adapter card can communicate with the processor 1212 via the PCIe bus.

[0081] It should be noted that, Figure 1 Only one engine 121 is shown in the figure. However, in actual applications, the data processing system may contain two or more engines 121, and redundancy or load balancing may be performed between multiple engines 121.

[0082] The hard drive enclosure 122 includes a control unit 1225 and several hard drives. The control unit 1225 can have various forms. In one case, the hard drive enclosure 122 is a smart enclosure, such as... Figure 1The control unit 1225 is shown to include a CPU and a memory. The CPU is used to perform address translation and read / write data operations. The memory is used to temporarily store data to be written to the hard drives or data read from the hard drives to be sent to the controller. Alternatively, the control unit 1225 is a programmable electronic component, such as a data processing unit (DPU). The DPU has the versatility and programmability of a CPU, but is more specialized and can efficiently operate on network packets, storage requests, or analytics requests. The DPU is distinguished from the CPU by a greater degree of parallelism (to handle a large number of requests). Alternatively, the DPU can be replaced by a graphics processing unit (GPU), a neural-network processing unit (NPU), or other processing chip. Typically, the number of control units 1225 can be one, two, or more. The functionality of the control unit 1225 can be offloaded to the network card 1226. In other words, in this embodiment, the hard drive chassis 122 does not have a control unit 1225, but instead the network card 1226 performs the data read / write, address translation, and other computational functions. In this case, the network card 1226 is a smart network card. It can include a CPU and a memory. The CPU is used to perform address translation and read / write data operations. The memory is used to temporarily store data to be written to the hard drives or data read from the hard drives to be sent to the controller. Alternatively, it can be a programmable electronic component, such as a DPU. The network card 1226 in the hard drive chassis 122 has no ownership relationship with the hard drives (e.g., the mechanical hard drives 1221, 1222, the solid state drive 1223, and other hard drives 1224) in the chassis 122, and the network card 1226 can access any hard drive in the chassis 122. Therefore, it is convenient to expand the hard drives when the storage space is insufficient. Figure 1 The mechanical hard drives 1221, 1222, the solid state drive 1223, and other hard drives 1224 are shown. The hard drive chassis 122 can be a SAS (serial attached small computer system interface) hard drive chassis, an NVMe (Non-Volatile Memory express) hard drive chassis, or other types of hard drive chassis, depending on the type of communication protocol between the engine 121 and the hard drive chassis 122. The SAS hard drive chassis uses the SAS3.0 protocol and supports 25 SAS hard drives per chassis. The engine 121 is connected to the hard drive chassis 122 through the on-board SAS interface or a SAS interface module. The NVMe hard drive chassis is more like a complete computer system, and the NVMe hard drive is plugged into the NVMe hard drive chassis. The NVMe hard drive chassis is connected to the engine 121 through the RDMA port.

[0083] The mechanical hard drives 1221, 1222, the solid state drive 1223, and other hard drives 1224 are shown. The hard drive chassis 122 can be a SAS (serial attached small computer system interface) hard drive chassis, an NVMe (Non-Volatile Memory express) hard drive chassis, or other types of hard drive chassis, depending on the type of communication protocol between the engine 121 and the hard drive chassis 122. The SAS hard drive chassis uses the SAS3.0 protocol and supports 25 SAS hard drives per chassis. The engine 121 is connected to the hard drive chassis 122 through the on-board SAS interface or a SAS interface module. The NVMe hard drive chassis is more like a complete computer system, and the NVMe hard drive is plugged into the NVMe hard drive chassis. The NVMe hard drive chassis is connected to the engine 121 through the RDMA port.

[0084] For example, the computing device 120 can refer to a storage array, such as an all-flash storage array in which all storage media are flash memory.

[0085] In an alternative implementation, the computing device 120 can be a disk- controlled centralized data processing system. The computing device 120 does not have the hard disk frame 122 described above, and the engine 121 is configured to manage a plurality of hard disks connected through hard disk slots. The functions of the hard disk slots can be implemented by the back-end interface 1214.

[0086] In a second possible scenario, the computing device 120 is a distributed data processing system.

[0087] In this scenario, the distributed data processing system includes a computing device cluster and a storage device cluster.

[0088] The computing device cluster includes one or more computing devices, which can communicate with each other. The computing device can be, but is not limited to, a server, a desktop computer, or a controller of a storage array, etc. In hardware, the computing device can include a processor, a memory, and a network card, etc. The processor is a CPU configured to process data access requests from outside the computing device or requests generated inside the computing device. For example, when the processor receives a write data request sent by a user, the processor temporarily stores the data in the write data request in the memory. When the total amount of data in the memory reaches a certain threshold, the processor sends the data stored in the memory to the storage device for persistent storage. In addition, the processor is also configured to perform data computation or processing, such as deduplication, data compression, garbage collection, data replication, virtualized storage space, and address translation, etc. In an example, any computing device in the computing device cluster can access any storage device in the storage device cluster through a network. The storage device cluster includes a plurality of storage devices. A storage device includes one or more controllers, a network card configured to communicate with the computing device, and a plurality of hard disks.

[0089] When the data processing system stores data, it can use a data reduction technique to delete data blocks in the data to be stored that are the same as the data already stored, so as to reduce the storage capacity occupied by redundant data in the storage device. The above process can be described in detail as follows. First, the computing device divides the data to be stored to obtain a plurality of data blocks. Second, the computing device generates a fingerprint feature corresponding to each data block. Finally, the computing device compares the fingerprint features of the data already stored with the fingerprint features of the data blocks, deletes the data blocks in the data to be stored that match the fingerprint features of the data already stored, and stores the data blocks in the data to be stored that do not match the fingerprint features of the data already stored.

[0090] In the above process, the computing device can divide the data to be stored into multiple data blocks in various ways, and store the data to be stored. For example, the computing device can divide the data to be stored into multiple data blocks using a combination of fixed-length block partitioning algorithm, variable-length block partitioning algorithm, fixed-length block partitioning algorithm, and variable-length block partitioning algorithm. The above three methods will be explained below.

[0091] Method 1: The computing device uses a fixed-length slicing algorithm to divide the data to be stored into multiple data blocks, and then stores the data to be stored.

[0092] The following example illustrates the process of storing two data blocks, Data1 and Data2, where the computing device divides the data into blocks of 4 bytes each.

[0093] Figure 2 A schematic diagram of a fixed-length partitioning method provided in this application is shown below. Figure 2 As shown, for the first storage of data 1: the computing device divides data 1 into data blocks 11 to 16, and stores data blocks 11 to 16. Each data block 11 to 16 contains 4 bytes. For data not stored for the first time: the computing device divides data 2 into data blocks 21 to 27. The computing device compares data blocks 11 to 16 with data blocks 21 to 27, determining that only data block 21 is identical to data block 11 within data blocks 21 to 27. The computing device then deletes data block 21 from data 2 and stores data blocks 22 to 27 from data 2. Thus, when some stored data changes, the fixed-length block algorithm suffers from a low deduplication rate when processing data to be stored.

[0094] Method 2: The computing device uses a variable-length slicing algorithm to divide the data to be stored into multiple data blocks, and then stores the data to be stored.

[0095] In this scenario, the computing device can use a sliding window to implement variable-length chunks of data to be stored. Let's take a sliding window of L bytes as an example.

[0096] Figure 3 A flowchart for obtaining a data block is provided in this application, such as Figure 3 As shown, the process includes steps ① through ③ below.

[0097] Step 1: The computing device uses a sliding window to select L bytes of data to be stored, generates cut-off point features based on the L bytes, and determines whether the cut-off point features meet the conditions.

[0098] For example, the computing device selects data corresponding to the first byte to the Lth byte of the to-be-stored data by using the sliding window. The computing device performs a hash algorithm on the data corresponding to the first byte to the Lth byte of the to-be-stored data to obtain a cutpoint feature f1 corresponding to the Lth byte of the to-be-stored data. The computing device performs f1 mod D to obtain an operation value, and determines whether the operation value is within a range indicated by r. D is a fixed value, and r is greater than 0 and less than or equal to D.

[0099] In step ②, if the cutpoint feature meets the condition, the computing device determines that the Lth character of the to-be-stored data is a cutpoint, and obtains data of the first byte to the Lth byte of the to-be-stored data as a data block. Then, the sliding window is stepped by L bytes, and the computing device continues to perform step ①.

[0100] For example, Figure 4 A schematic diagram of the sliding window obtaining a data block is provided in this application, as shown in Figure 4 If the operation value is within the range indicated by r, the cutpoint feature meets the condition, and the computing device obtains data corresponding to the first byte to the Lth byte of the to-be-stored data to obtain a data block containing L bytes. The sliding window 1 is stepped by L bytes, and the L+1th byte to the 2Lth byte of the to-be-stored data is taken as the content selected by the sliding window 1 next time, and the computing device continues to perform step ①.

[0101] In step ③, if the cutpoint feature does not meet the condition, and the sliding window 1 is stepped by 1 byte, the computing device continues to perform step ①.

[0102] For example, if the operation value is within the range indicated by r, the sliding window 1 is stepped by 1 byte, and the second byte to the L+1th byte of the to-be-stored data is taken as the content selected by the sliding window 1 next time, and the computing device continues to perform step ①. The computing device repeats steps ① to ③ above until the sliding window 1 traverses data 2 to obtain a plurality of data blocks.

[0103] Based on the above detailed description of the method of obtaining a data block by using a sliding window, the process of storing data by using a variable-length cut block algorithm is described by taking the computing device storing data 1 and data 2 as an example.

[0104] Figure 5 A schematic diagram of the fixed-length division method is provided in this application, as shown in Figure 5As shown, for the first storage of data 1: the computing device uses the sliding window data partitioning method described above to partition data 1 into data blocks 11 to 16, and stores data blocks 11 to 16. For data 2: the computing device uses the sliding window data partitioning method described above to partition data 2 into data blocks 21 to 26. The computing device compares data blocks 11 to 16 with data blocks 21 to 26, determining that data blocks 21 to 26 differ only in data block 22. The computing device stores data block 22 and deletes data blocks 21, 23 to 26. Although the block partitioning algorithm can achieve a high deduplication rate, during the process of the computing device using the sliding window to obtain data blocks, the computing device needs to calculate the segmentation feature once for each byte step of the sliding window. For example, for 1GB of data to be stored, the computing device needs to calculate approximately 10... 9 The single-point feature suffers from high computational cost and low efficiency.

[0105] Method 3: The computing device uses a combination of fixed-length slicing algorithm and variable-length slicing point algorithm to divide the data to be stored into multiple data blocks and store the data to be stored.

[0106] To address the inefficiency issue of method two, the computing device can first execute a fixed-length segmentation algorithm to divide the data to be stored into multiple data segments of equal length, such as... Figure 6 As shown, Figure 6 This application provides a schematic diagram of a combination of fixed-length and variable-length partitioning methods. It also illustrates the process of performing a segmentation algorithm on each data segment to divide the data into multiple data blocks. For a detailed description of the process of a computing device partitioning the data to be stored into multiple data segments, please refer to Method 1; for a detailed description of the process of a computing device partitioning data segments into multiple data blocks, please refer to Method 2. This will not be repeated here. While this method can improve efficiency, if some stored data is added or removed, all data segments after the changed data segment will change, resulting in a reduced number of data blocks that match the already stored data in the resulting data blocks, a low deduplication rate, and a large computational load due to the fact that each byte step in the sliding window still requires a segmentation feature calculation.

[0107] To solve the problems of high computing time and low data deduplication rate of a computing device in obtaining a cut point, the present application provides a data processing method. In the method, the computing device divides to-be-stored data by M sliding windows to obtain a first data set, adjusts the M sliding windows to N sliding windows according to at least one of the computing time and the cache occupancy rate of obtaining a first cut point set, and divides to-be-stored data by the N sliding windows to obtain a second data set. The computing device performs data deduplication on the first data set and the second data set. In this way, multiple sliding windows can obtain at least one cut point by processing to-be-stored data at one time, thereby reducing the computing time required by the computing device to obtain a cut point. The computing device adjusts the parallelism of processing to-be-stored data according to at least one of the computing time and the cache occupancy rate of obtaining a first cut point set, so as to adjust the length of to-be-stored data selected by multiple sliding windows. The accuracy of the computing device in obtaining a cut point is improved, the data deduplication rate is improved, and the data duplicated with stored data contained in to-be-stored data is obtained.

[0108] As a functional feature of a distributed storage system, data deduplication can cover the scenarios of main storage systems and backup storage systems (such as cloud storage, distributed storage, all-flash high-end storage, etc.) that currently support data deduplication functions, including storage systems managed based on open source distributed storage software (such as Ceph and HDFS).

[0109] Figure 7 A flowchart of a data processing method provided by the present application is shown in FIG. 1, which can be executed by the computing device described above, and the method includes the following S710-S740. Figure 7 Figure 1 The method includes the following S710-S740.

[0110] S710, the computing device obtains a first cut point set of a first segment of to-be-stored data according to a first parallelism, and divides the first segment of to-be-stored data according to at least one cut point in the first cut point set to obtain a first data set.

[0111] The process of the computing device obtaining the first data set can include the following first step and second step.

[0112] In the first step, the computing device obtains a first cut point set of a first segment of to-be-stored data by using M sliding windows.

[0113] The computing device can obtain a cut point in the first cut point set by sliding the M sliding windows on to-be-stored data, and also obtain a cut point in the first cut point set by selecting the cut point.

[0114] In the first possible case, the computing device obtains a cut point in the first cut point set by sliding the M sliding windows on to-be-stored data.

[0115] ​The computing device can slide M sliding windows on the data to be stored at least once to obtain a first set of cut points of a first data segment. Figure 8 A flowchart for obtaining a set of cut points is provided in the present application, as shown in Figure 8 The process includes the following S11 and S12.

[0116] S11, the computing device divides the first data segment by M sliding windows to obtain a first initial data set.

[0117] The first data segment is not pre-divided. The first data segment can refer to data covered by M sliding windows without sliding, or can refer to data covered by M sliding windows with sliding.

[0118] In one possible case, the computing device can divide the first data segment by M sliding windows with an interval of at least 1 byte to obtain a first initial data set. For example, with an interval of a bytes, a is a positive integer greater than or equal to 1.

[0119] The process of dividing the data to be stored according to whether M sliding windows are sliding on the data to be stored for the first time and dividing the data to be stored to obtain a first initial data set is different, which will be described below.

[0120] (1) M sliding windows are sliding on the data to be stored for the first time, and the data to be stored is divided.

[0121] Taking the data to be stored as data 1, M sliding windows as sliding window 1 to sliding window m, M sliding windows dividing data 1 with an interval of a bytes, and sliding window 1 dividing data 1 from the i-th byte of data 1 as an example.

[0122] Sliding window 1 frames the 1st byte to the Lth byte of data 1, sliding window 2 frames the a-th byte to the (L+a-1)th byte of data 1, and sliding window j frames the j*a-th byte to the (L+j*a-1)th byte of data 1. In this case, the first data segment can refer to the data segment corresponding to the 1st byte to the (L+j*a-1)th byte in data 1.

[0123] Sliding window 1 divides the 1st byte to the Lth byte of the first data segment to obtain a data block 11 with a length of L bytes. Sliding window 2 divides the a-th byte to the L+a-1th byte of the first data segment to obtain a data block 12 with a length of L bytes. Sliding window j divides the j*a-th byte to the (L+j*a-1)th byte of the first data segment to obtain a data block 1j with a length of L bytes. Wherein, i is a positive integer greater than or equal to 1, and j is a positive integer greater than or equal to 3 and less than or equal to m. The computing device obtains a first initial data set containing m data blocks according to the data block 11, the data block 12, and the data block 1j. In the following, taking the case of dividing the first data segment by M sliding windows with an interval of 1 byte as an example to illustrate the present application.

[0124] (2) M sliding windows are slid on the data to be stored for the second time, and the data to be stored is divided.

[0125] The computing device divides the data to be stored according to whether the position of the last byte of the data block in the first initial data set is a cut point.

[0126] I. For the case where the position of the last byte of the data block in the first initial data set is a cut point.

[0127] In this case, the data to be stored is divided to obtain a sliding window of the data block stepping L bytes, and the computing device re-divides the data to be stored to obtain a new data block, taking the position after stepping L bytes as the starting position of the sliding window.

[0128] For example, sliding window k is divided to obtain data block 1k, which contains data from the kth byte to the (L+k-1)th byte of data 1. The position of the last byte of data block 1k is a cut point. Sliding window k steps L bytes, and the sliding window k after stepping L bytes re-divides to obtain data block k1 of length L bytes, taking the (L+k)th byte of data 1 as the starting position. Data block k1 contains data from the (L+k)th byte to the (2L+k-1)th byte of data 1.

[0129] II. For the case where the position of the last byte of the data block in the first initial data set is not a cut point.

[0130] In this case, the data to be stored is divided to obtain a sliding window of the data block stepping 1 byte, and a data block of length (L+1) bytes is obtained by continuing to select the data to be stored.

[0131] For example, sliding window k is divided to obtain data block 1k, which contains data from the kth byte to the (L+k-1)th byte of data 1. The position of the last byte of data block 1k is not a cut point. Sliding window k steps 1 byte, and the sliding window k after stepping 1 byte re-divides to obtain data block k1 of length (L+1) bytes, taking the (k+1)th byte of data 1 as the starting position. Data block k1 contains data from the (k+1)th byte to the (L+k)th byte of data 1.

[0132] In this case, the first segment of data can refer to data covered by part of the M sliding windows stepping 1 byte and part of the M sliding windows stepping L bytes.

[0133] The above describes the process of dividing the to-be-stored data by the M sliding windows to obtain the first initial data set, taking the second sliding of the M sliding windows on the to-be-stored data and the division of the to-be-stored data as an example. After the M sliding windows slide twice, the computing device can obtain the first initial data set containing data blocks with a length of L bytes or (L+1) bytes. Similarly, the computing device can use the method described in (2) to slide and divide the to-be-stored data by the M sliding windows for the third time, the fourth time, and so on. For related content, please refer to the description in (2), which will not be repeated here.

[0134] The computing device can determine the length of the data block in the first initial data set according to the length of the sliding window and the number of times of sliding of the sliding window. After obtaining the first initial data set containing at least one data block, the computing device can use the method described in S12 to determine whether the length of the data block in the first initial data set is within the length interval determined by the maximum value and the minimum value of the length. If not, the computing device can continue to slide the sliding window to increase the length of the data block in the obtained first initial data set. If so, the computing device can use the cut point determination rule corresponding to the length to determine whether the position of the last byte of the data block is a cut point.

[0135] S12, the computing device obtains the first cut point set according to the length of the data block in the first initial data set and the cut point determination rule associated with the length distribution characteristics.

[0136] The distribution characteristics can refer to the distribution of various indicators of a group of data. The distribution characteristics can be described by the concentration tendency of the distribution, which can reflect the degree of convergence or aggregation of each data to the center value, which can also be referred to as the expectation in this application. The length distribution characteristics can refer to the maximum value, the minimum value, and the expectation value of the data block length.

[0137] In one possible case, the computing device obtains the first cut point set according to the length of the data block in the first initial data set and the cut point determination rule associated with the length interval determined by the maximum value, the minimum value, and the expectation value of the length. The length interval determined by the maximum value, the minimum value, and the expectation value of the length can be set by the user according to actual needs.

[0138] In the case where the content of the to-be-stored data is random, the cut point characteristics f generated by the data block selected by the sliding window are also random. In this case, from a statistical point of view, it can be considered that the probability of the position of the last byte of the data block selected by the sliding window each time being determined as a cut point is the same. Let this probability be p, then the probability of the last byte of the data block selected by the sliding window for the nth time being determined as a cut point is shown in formula (1).

[0139] P=p×(1-p) (n-1) Formula (1)

[0140] wherein p is the probability that the position of the last byte of the data block is determined as the cut point, (1-p) is the probability that the position of the last byte of the data block is not determined as the cut point, and n is the length of the data block. (n-1) is the probability that the position of the last byte of the (n-1) data blocks obtained by the (n-1) times of sliding of the sliding window is not determined as the cut point. According to formula (1), the probability that the position of the last byte of the data block obtained by each time of sliding of the sliding window is determined as the cut point is an exponential function. According to this property, a data block length distribution diagram as shown in FIG. 2 is established according to the given maximum, minimum and expected values of the length. Figure 9 Figure 9 is a data block length distribution diagram provided in the present application.

[0141] According to the different relationships between the length of the data block in the first initial data set and the maximum, minimum and expected values of the length, the process of obtaining the first cut point set by the computing device is also different, which is described below.

[0142] Case A, for the data block in the first initial data set whose length is less than the minimum value of the length.

[0143] In this case, the computing device does not determine whether the position of the last byte of the data block is the cut point. The computing device makes the sliding window of the divided data block step 1 byte. The sliding window after stepping 1 byte continues to divide the data to be stored to obtain a new data block by using the method described in S11.

[0144] For the new data block, the computing device obtains the length of the new data block, and determines the relationship between the length of the new data block and the maximum, minimum and expected values of the length again. If the length of the new data block is less than the minimum value of the length, the method described in case A is used to process the new data block. If the length of the new data block is in the length interval determined by the minimum and maximum values of the length, the method described in case B is used to process the new data block. If the length of the new data block is equal to the maximum value of the length, and the computing device determines that the last byte of the data block is not the cut point by using the method described in case B, the computing device uses the method described in case C to process the new data block to obtain one cut point in the first cut point set.

[0145] For example, the sliding window 1 divides the data 1 to obtain the data block 11, and the length of the data block 11 is l1. The minimum value of the length is Lmin. If l1

[0146] ​For the data block 11, the computing device acquires the length of the data block 11 (i.e. (l1+1) bytes), and determines the relationship between the length of the data block 11 and the maximum value (i.e. Lmax), the minimum value (i.e. Lmin) and the expected value of the length. If l1+1

[0147] Case B, for the data block whose length is within the length interval determined by the maximum value and the minimum value of the length among the data blocks in the first initial data set.

[0148] In one possible case, the computing device can divide the length interval determined by the maximum value and the minimum value of the length into multiple sub-length intervals, and determine different cut point determination rules for each sub-length interval. Figure 10 A diagram showing the relationship between the cut point determination probability and the length of the data block is provided for the present application, as shown in Figure 10 As shown, the probability of the position of the last 1 byte of the data block being determined as a cut point gradually increases in the process of the length of the data block changing from the minimum value to the expected value, and the probability of the position of the last 1 byte of the data block being determined as a cut point gradually increases in the process of the length of the data block changing from the expected value to the maximum value. In this way, the length of the data block obtained by the computing device using the sliding window division can be as long as possible under the condition that the length is not less than the expected value.

[0149] Exemplarily, Figure 11 A diagram showing the correspondence between the cut point determination rule and the length interval is provided for the present application, as shown in Figure 11As shown, the computing device divides the length interval into sub-length intervals 1 to sub-length interval i, sub-length interval (i+1) to sub-length interval n according to the maximum value of the length and the minimum value of the length. The sub-length intervals 1 to sub-length interval n correspond to the cut point determination rule 1 to the cut point determination rule n respectively. The sub-length interval i is the interval where the expected value is located. The cut point determination rule 1 to the cut point determination rule n gradually increase the probability of determining that the last byte of the data block is the cut point. The computing device can increase the probability of the cut point determination rule determining that the last byte of the data block is the cut point by reducing the number of judgment conditions shown in formula (2). As shown in Table 1, if the length of the data block is located in the length range of the sub-length interval 1 close to the minimum value of the length, the computing device determines that the last byte of the data block is the cut point only when the n judgment conditions shown in formula (2) are satisfied simultaneously. If the length of the data block is located in the length range of the sub-length interval i where the expected value is located, the computing device determines that the last byte of the data block is the cut point only when the i judgment conditions shown in formula (2) are satisfied simultaneously. If the length of the data block is located in the length range of the sub-length interval n close to the maximum value of the length, the computing device determines that the last byte of the data block is the cut point only when the 1 judgment condition shown in formula (2) is satisfied. In this way, the length of the data block obtained by dividing the cut point from the data to be stored is adjusted so that the length of the data block obtained by dividing the cut point is as long as possible without being less than the expected value.

[0150] f mod D = r formula (2)

[0151] Wherein, f is the cut point feature generated by the data block using the sliding window frame selection, D is a fixed value, r is greater than zero and less than or equal to D.

[0152] Table 1

[0153]

[0154]

[0155] The computing device can divide the length interval into multiple sub-length intervals in various ways, and the following gives several possible examples.

[0156] Example 11, the computing device divides the length interval into multiple sub-length intervals with different lengths. For example, the length of the length interval is 15 bytes, and the computing device divides the length interval into sub-length intervals 1 to sub-length interval 5. The lengths of the sub-length intervals 1 to sub-length interval 5 are 1 byte to 5 bytes respectively.

[0157] In example 12, the computing device divides the length interval into a plurality of sub-length intervals with equal sub-lengths and unequal sub-lengths. For example, the length interval has a length of 15 bytes, and the computing device divides the length interval into sub-length interval 1 to sub-length interval 10. The length of sub-length interval 1 to sub-length interval 5 is 1 byte, and the length of sub-length interval 6 to sub-length interval 10 is 2 bytes.

[0158] In example 13, the computing device divides the length interval into a plurality of sub-length intervals with equal sub-lengths, so that the length of each sub-length interval is equal. For example, the length interval has a length of 20 bytes, and the computing device divides the length interval into sub-length interval 1 to sub-length interval 10 uniformly. The length of sub-length interval 1 to sub-length interval 10 is 2 bytes.

[0159] After determining the cut point determination rule corresponding to the length interval, the computing device can determine whether the position of the last byte of the data block is a cut point using the following B11 to B13.

[0160] B11, the computing device obtains the data selected by the sliding window.

[0161] For example, the sliding window k divides the kth byte to the (k+x-1)th byte of data 1 to obtain a data block 1k with a length of x bytes. In this case, the computing device obtains the content of the ((k+x-1)-L)th byte to the (k+x-1)th byte of data 1 selected by the sliding window k.

[0162] B12, the computing device generates a cut point feature according to the data selected by the sliding window.

[0163] For example, the computing device performs a hash operation on the content of the ((k+x-1)-L)th byte to the (k+x-1)th byte of data 1 to obtain a cut point feature f k .

[0164] B13, the computing device determines whether the cut point feature meets the cut point determination rule associated with the length interval where the length of the data block is located.

[0165] The computing device determines the sub-length interval where the length of the data block is located according to the length of the data block, and determines whether the position of the last byte of the data block is a cut point according to the cut point determination rule corresponding to the sub-length interval. If yes, the computing device obtains a cut point in the first cut point set, and the computing device makes the sliding window k step L bytes and continues to divide the data to be stored to obtain a new data block using the method described in S11 above. If not, the computing device makes the sliding window k step 1 byte, and continues to divide the data to be stored to obtain a new data block using the method described in S11 above.

[0166] For example, the length of the data block 1k is in the sub-length interval n. The computing device determines whether the last byte of the data block 1k is the cut point according to the determination condition f of the row corresponding to the number n in Table 1 k mod D k = r k . If yes, the computing device obtains a cut point, and the computing device makes the sliding window k step L bytes, and continues to divide the data to be stored to obtain a new data block according to the method described in S11 above. If no, the computing device makes the sliding window k step 1 byte, and continues to divide the data to be stored to obtain a new data block according to the method described in S11 above.

[0167] Case C, for the data block in the first initial data set, the length of the data block is equal to the maximum value of the length, and the computing device determines that the last byte of the data block is not the cut point according to the method described in case B.

[0168] In this case, the computing device directly determines that the last byte of the data block in the data to be stored is the cut point, obtains a cut point in the first cut point set. And the computing device makes the sliding window step L bytes, and continues to divide the data to be stored according to the method described in S11 above to obtain a new data block.

[0169] For example, the sliding window 1 divides the data 1 to obtain the data block 11, and the length of the data block 11 is l1. The maximum value of the length is L max , l1 is equal to L max , and L max >L. The computing device determines that the last byte of the data block 1 is not the cut point according to the method described in case B. In this case, the computing device directly determines that the last byte of the data block 11 in the data 1 is the cut point, obtains a cut point in the first cut point set. And the computing device makes the sliding window 1 step L bytes, and continues to divide the data to be stored according to the method described in S11 above to obtain a new data block.

[0170] In the second possible case, the computing device obtains the cut point in the first cut point set in the way of selecting the cut point.

[0171] In this scenario, the computing device can also generate a cutoff feature table for the stored data and a cutoff feature for each cutoff in the first cutoff set. The computing device compares the cutoff features of each cutoff in the first cutoff set with the cutoff features in the cutoff feature table. If the cutoff feature of the first cutoff in the first cutoff set matches the cutoff feature in the cutoff feature table, the computing device directly shifts by a specific byte length from the position of the first cutoff to obtain another cutoff. The specific byte length can refer to the cutoff distance corresponding to the cutoff feature in the cutoff feature table. Thus, when the data to be stored has been partially modified relative to the stored data, the computing device does not need to use a sliding window to slide byte by byte across the data to be stored to obtain the cutoffs for the first data block. This saves the computing device the computational burden of calculating cutoffs and improves the efficiency of data deduplication.

[0172] For example, Figure 12 A schematic diagram of obtaining the tangent point provided in this application, such as Figure 12 As shown, the stored data includes data blocks 1 to n. The cut-point feature of data block 1 is cut-point feature 1, and the cut-point distance corresponding to cut-point feature 1 is L1. Cut-point 1 divides data 1 to obtain data block 11, which includes the first byte to the i-th byte of data 1. The position of cut-point 1 is the i-th byte of data block 11. The cut-point feature of data block 11 is cut-point feature 11. When cut-point feature 1 and cut-point feature 11 match, the computing device obtains the cut-point distance corresponding to cut-point feature 1 as L1. The computing device also shifts the position of cut-point 1 (i.e., the i-th byte of data 1) by L1 bytes to obtain cut-point 2 (i.e., the L1+i-th byte of data 1).

[0173] The cut-point feature table includes: cut-point features and cut-point distance. The cut-point features indicate the content selected by the sliding window of the divided data block. The cut-point distance indicates the distance between the first cut-point and the second cut-point. The first cut-point indicates the position of the last byte of the data block, and the second cut-point indicates the cut-point immediately following the first cut-point. The cut-point distance is the same as the length of the immediately following data block. Table 2 shows a cut-point feature table provided in this application.

[0174] Table 2

[0175] Data block Key Value Data block Key Value

[0176] For example, the stored data includes data block 1 and data block 2, and data block 2 is the subsequent adjacent data block of data block 1. Cutpoint 1 divides data 1 to obtain data block 1, and cutpoint 2 divides data 1 to obtain data block 2. The computing device can generate the cutpoint feature table of data block 1 by using the following process, which is specifically: ① The computing device generates the cutpoint feature 1 corresponding to data block 1. ② The computing device obtains the distance 1 between the interval between cutpoint 2 and cutpoint 1. ③ The computing device generates the cutpoint feature table of data block 1 according to the cutpoint feature 1 and the distance 1.

[0177] The computing device can generate the cutpoint feature table in various ways, two possible ways are given below.

[0178] Method a: The computing device generates the cutpoint feature table corresponding to each cutpoint.

[0179] This method is suitable for the cutpoint obtained by the computing device using the method described in the above case B. For example, the stored data includes data block 1 to data block K, and the computing device stores the cutpoint feature table corresponding to data block 1 to data 1K. The data to be stored is data 1, and cutpoint 1 divides data 1 to obtain data block 11. The process of generating the cutpoint feature table is described.

[0180] Step ①: The computing device obtains the content 11 selected by the sliding window frame of dividing data 1 to obtain data block 11, and generates the cutpoint feature 11 according to the content 11.

[0181] For example, the computing device can execute the hash algorithm on the content 11 to generate the cutpoint feature 11 of the content 11.

[0182] Step ②: The computing device compares the cutpoint feature 11 with the cutpoint features in the cutpoint feature table. If the cutpoint feature 11 matches any of the cutpoint features in the cutpoint feature table, the computing device performs step ③. If the cutpoint feature 11 does not match any of the cutpoint features in the cutpoint feature table, the computing device performs step ④.

[0183] For example, the computing device compares the cutpoint feature 11 with the cutpoint features corresponding to data block 1 to data block K, respectively. If the cutpoint feature 11 is the same as the cutpoint feature corresponding to data block 1, it is considered that the cutpoint feature 11 matches the cutpoint feature corresponding to data block 1. If the cutpoint feature 11 is not the same as the cutpoint features corresponding to data block 1 to data block K, it is considered that the cutpoint feature 11 does not match any of the cutpoint features in the cutpoint feature table.

[0184] In the third step, the computing device obtains the cutpoint distance corresponding to the cutpoint feature of the data block 1, and displaces the data block 1 from the position of the cutpoint 1 by the cutpoint distance corresponding to the cutpoint feature of the data block 1 to obtain the cutpoint 2. The computing device obtains the data between the cutpoint 1 and the cutpoint 2 in the data 1 to obtain the data block 21. The computing device continues to determine whether the cutpoint feature of the data block 21 is the same as the cutpoint feature in the stored cutpoint feature table by using the method described in the first step and the second step.

[0185] In the fourth step, the computing device stores the data block 11, generates the cutpoint feature table 11 of the data block 11, and adds the cutpoint feature table 11 to the stored cutpoint feature table to obtain a new cutpoint feature table containing the cutpoint feature table 11.

[0186] For example, the computing device generates the cutpoint feature table 11 according to the cutpoint feature 11 and the cutpoint distance 11. Specifically, the computing device generates the cutpoint feature 11 by using the method in the first step, the computing device obtains the cutpoint distance 2 corresponding to the cutpoint feature of the data block 2 of the stored data as the cutpoint distance 11, and updates the cutpoint distance 1 corresponding to the cutpoint feature of the data block 1 in the stored cutpoint feature table to the length of the data block 11. According to the actual application needs, the computing device can also determine the cutpoint distance 11 by using other methods, which are not limited in the present application.

[0187] In the method b, the computing device generates a cutpoint feature table corresponding to a plurality of cutpoints.

[0188] This case is suitable for the cutpoint obtained by the computing device by using the method described in case C above.

[0189] In a possible case, the first data set includes a fifth data block and a sixth data block adjacent to each other, and the fifth data block and the sixth data block are both data blocks obtained by dividing the data by using the method described in case C. The length of the fifth data block and the length of the sixth data block are the same as the maximum value of the length, and the computing device generates and stores a cutpoint feature table corresponding to the fifth data block and the sixth data block. The cutpoint feature in the cutpoint feature table is generated by the fifth data block and the sixth data block. The cutpoint distance in the cutpoint feature table is the distance between the third cutpoint and the fourth cutpoint. The third cutpoint is used to indicate the cutpoint of the third segment of data obtained by dividing the data to be stored to obtain the fifth data block, and the fourth cutpoint is used to indicate the cutpoint of the fourth segment of data obtained by dividing the data to be stored to obtain the sixth data block. Figure 13 A generation diagram of a cutpoint feature table provided in the present application is shown in FIG. 1. Figure 13As shown, in the case that there are multiple (e.g., A) cut points determined by the method described in Case C in the first data set, the computing device generates one cut point feature table for the multiple cut points, which saves the computing device's time for calculating the cut point features of (A-1) cut points, and saves the storage space for storing the cut point feature tables of the (A-1) cut points.

[0190] Figure 14 A flowchart for obtaining cut points is provided in the present application, as shown in Figure 14 As shown, the computing device determines the cut points by using differentiated cut point determination methods (i.e., different cut point determination rules are used for data blocks of different lengths). The computing device generates the cut point features corresponding to the cut points, and generates the cut point feature table. The computing device compares the cut point feature with the cut point features in the cut point feature table of the stored data, and in the case that the cut point feature is the same as a cut point feature in the cut point feature table, the computing device obtains the next cut point by shifting from the position of the cut point by a first distance, which can refer to the distance of the cut point corresponding to the cut point feature in the cut point feature table. The computing device determines whether the cut point feature of the next cut point is the same as the cut point feature in the cut point feature table of the stored data (i.e., satisfies the re-deletion locality). The re-deletion locality can refer to the existence of small part of data modification between two data, and the same of most of the contents.

[0191] Step ②, the computing device divides the first data set by using at least one cut point in the first cut point set.

[0192] For example, the data to be stored is data 1, and the first cut point set includes cut point 1 to cut point a. Cut point 1 indicates the xth byte of data 1, and cut point a indicates the yth byte of data 1. In this case, the first data set can refer to the 1st byte to the yth byte of data 1. Cut point 1 to cut point a divide the 1st byte to the yth byte of data 1, respectively, to obtain the first data set. The first data set includes data block 11 to data block 1x.

[0193] Optionally, for the case that the computing device adopts a length of a directly displaced specific byte of a current cutpoint to obtain another cutpoint in the first cutpoint set, the computing device can adopt the following process to divide the to-be-stored data to obtain the data blocks in the first data set. The stored data includes a second data block and a third data block. The first data set includes at least one data block matched with the second data block. The computing device divides a third segment of the to-be-stored data by a length of the third data block to obtain a fourth data block. The third data block can refer to a posterior adjacent data block of the second data block. In this way, in the case that there is only a small part of the to-be-stored data modified compared with the stored data, compared with the computing device adopting the sliding window to slide byte by byte on the to-be-stored data to obtain the initial data block, and determining the cutpoint according to the length of the initial data block and the cutpoint determination rule associated with the length distribution feature, and dividing the data to obtain the data block by using the cutpoint, the computing device adopts the sliding window to directly slide the length of the third data block to obtain the data block, which reduces the number of times of calculating the cutpoint feature, reduces the amount of calculation, and improves the efficiency of data re-deletion.

[0194] For example, the stored data includes a data block 1, a data block 2 to a data block K. The length of the data block 2 is x bytes. The first data set includes a data block 3. The data block 3 contains data of the i th byte to the y th byte of the data 1. If the data block 3 and the data block 1 match, the computing device starts from the (y+1) th byte of the data 1, and frames the (y+1) th byte to the (y+x) th byte of the data 1 to obtain a data block 4 with a length of x bytes.

[0195] S720, the computing device adjusts the first parallelism degree to obtain a second parallelism degree according to at least one of the calculation time consumption of the first cutpoint set and the cache occupancy rate.

[0196] The first cache occupancy rate is used to indicate the cache occupancy rate of performing data re-deletion on the first segment of data according to the first cutpoint set. The second parallelism degree is used to indicate N sliding windows. M is not equal to N, and M and N are integers greater than or equal to 2.

[0197] In a possible case, the computing device can obtain the calculation time consumption of the first cutpoint set in multiple ways, and three possible ways are given below.

[0198] A, the computing device adopts a first time length as the calculation time consumption of the first cutpoint set.

[0199] The first time length is used to indicate the time length required for dividing the first segment of data by the M sliding windows to obtain the first initial data set.

[0200] For example, the M sliding windows include sliding window 1 to sliding window m. The data to be stored is data 1. The first initial data set includes data block 11 to data block 1m. In this case, the first time length is used to indicate the time length required for dividing data 1 into data block 11 to data block 1m by sliding window 1 to sliding window m. For the process of dividing data into data blocks by sliding windows, please refer to the relevant description in S11 above, which will not be repeated here.

[0201] In mode B, the computing device uses the second time length as the calculation time consumption of the first cut point set.

[0202] The second time length is used to indicate the time length required for obtaining the first cut point set according to the length of the data block in the first initial data set and the cut point determination rule associated with the length distribution characteristics.

[0203] The second time length can include the time length required for the computing device to generate the cut point feature according to the content of the sliding window frame selected by dividing the data to be stored into data blocks, and the time length required for the computing device to determine whether the cut point feature conforms to the cut point determination rule. For the above process, please refer to the relevant description in S12 above, which will not be repeated here.

[0204] For example, the first initial data set includes data block 11 to data block 1m. Sliding window 1 to sliding window m respectively divides data 1 into data block 11 to data block 1m. The second time length includes the time length required for the computing device to generate cut point feature f1 to cut point feature f m required according to the content 1 to content m framed by sliding window 1 to sliding window m, and the time length required for determining whether the cut point feature f1 to cut point feature f m conforms to the cut point determination rule.

[0205] In mode C, the computing device uses the sum of the first time length and the second time length as the calculation time consumption of the first cut point set.

[0206] For the related content of the first time length and the second time length, please refer to the description of mode A and mode B above, which will not be repeated here.

[0207] According to the different relationships between the calculation time consumption and the cache occupancy rate and the respective upper limit values, the computing device can adjust the first parallel degree to the second parallel degree in different ways. The following will be described in different cases.

[0208] Figure 15 An example of adjusting the parallel degree provided in the present application is shown in FIG. 1, wherein the computing device adjusts the parallel degree according to the calculation time consumption and the cache occupancy rate, and the computing device can adjust the parallel degree by using formula (3). Figure 15

[0209]

[0210] wherein X is the parallel degree,​ is the length of each data block processed in parallel, is the time duration required for each parallel processing, i.e., the first time duration. k is the time duration required for the sliding window to slide by one byte. f(X) is the cache occupancy. g(X) is the time duration for the determination of the cut point, i.e., the second time duration. The computing device utilizes the computation time consumption and the cache occupancy (i.e., achieves performance monitoring, and optimizes the solution to obtain X that makes the formula take the minimum value, and adjusts the parallelism using the X.

[0211] There is a relationship between the cache occupancy and the parallelism as described in formula (4).

[0212]

[0213] wherein X m is the maximum parallelism, indicating that the cache occupancy is 100% when the parallelism is used. Figure 16 is a relationship diagram between the cache hit rate and the parallelism provided by the present application. As shown in Figure 16 , the cache occupancy increases with the increase of the parallelism, and when the cache occupancy reaches 100%, the cache hit rate gradually decreases.

[0214] In a possible case, when at least one of the computation time consumption and the cache occupancy of the first set of cut points is greater than the respective upper limit value, the computing device reduces the M sliding windows to N sliding windows. This case can include the following three possible examples.

[0215] Example 1, in the case that the computation time consumption is greater than the upper limit value of the computation time consumption, and the cache occupancy is greater than the upper limit value of the cache occupancy, the computing device reduces the M sliding windows to N sliding windows.

[0216] Example 2, in the case that the computation time consumption is greater than the upper limit value of the computation time consumption, and the cache occupancy is less than or equal to the upper limit value of the cache occupancy, the computing device reduces the M sliding windows to N sliding windows.

[0217] Example 3, in the case that the computation time consumption is less than or equal to the upper limit value of the computation time consumption, and the cache occupancy is greater than the upper limit value of the cache occupancy, the computing device reduces the M sliding windows to N sliding windows. Wherein M is greater than N.

[0218] In another possible case, when at least one of the computation time consumption and the cache occupancy of the first set of cut points is less than the respective lower limit value, the M sliding windows are increased to N sliding windows. Wherein M is less than N.

[0219] The computing device can adjust the number of sliding windows M and N in various ways, and the following gives several possible examples.

[0220] In example 1, the computing device adjusts the number of sliding windows according to a preset adjustment number. For example, the adjustment number is X. In a case where the computing time consumption is greater than an upper limit of the computing time consumption, and the cache occupancy rate is greater than an upper limit of the cache occupancy rate, the computing device reduces the M sliding windows by X to obtain N sliding windows.

[0221] In example 2, the computing device adjusts the number of sliding windows to a preset value according to the computing time consumption. For example, in a case where the computing time consumption reaches a first computing time consumption, the computing device adjusts the M sliding windows to a preset value 1. In a case where the computing time consumption reaches a second computing time consumption, the computing device adjusts the M sliding windows to a preset value 2, and so on.

[0222] In example 3, the computing device adjusts the number of sliding windows to a preset value according to the cache occupancy rate. For example, in a case where the cache occupancy rate reaches a first cache occupancy rate, the computing device adjusts the M sliding windows to a preset value 1. In a case where the cache occupancy rate reaches a second cache occupancy rate, the computing device adjusts the M sliding windows to a preset value 2, and so on.

[0223] The above describes three ways in which the computing device adjusts the number of sliding windows between M and N. Other ways of adjusting the number of sliding windows between M and N can also be used according to actual application requirements, such as adjusting the number of sliding windows one by one. The specific adjustment method of the sliding window is not limited in the present application.

[0224] In S730, the computing device obtains a second set of cut points of a second piece of data in the to-be-stored data according to the second parallelism. At least one cut point in the second set of cut points divides the second piece of data to obtain a second data set.

[0225] The second parallelism is used to indicate the N sliding windows.

[0226] In a possible case, the computing device can slide the N sliding windows on the to-be-stored data at least once to obtain a first set of cut points of the first piece of data.

[0227] In a possible case, the computing device divides the first piece of data according to the N sliding windows to obtain a first initial data set, and obtains the first set of cut points according to a cut point determination rule associated with a length interval of a data block in the first initial data set, the length interval being determined according to a maximum value, a minimum value, and an expected value of the length.

[0228] In a possible case, the computing device divides the first piece of data according to the N sliding windows to obtain a first initial data set, and obtains the first set of cut points according to a cut point determination rule associated with a length interval of a data block in the first initial data set, the length interval being determined according to a maximum value, a minimum value, and an expected value of the length.

[0229] In a possible case, the computing device obtains the first set of cut points according to a cut point determination rule associated with a length interval of a data block in the first initial data set, the length interval being determined according to a maximum value, a minimum value, and an expected value of the length.

[0230] The computing device obtains a second set of cut points of a second piece of data in the data to be stored by using N sliding windows, and at least one cut point in the second set of cut points divides the second piece of data to obtain a second data set. The process is the same as the process described in S710 above that the computing device obtains a first set of cut points of a first piece of data in the data to be stored by using M sliding windows, and at least one cut point in the first set of cut points divides the first piece of data to obtain a first data set. For details, please refer to the description of S710 above, which will not be repeated here.

[0231] In S740, the computing device performs data deduplication on the first data set and the second data set.

[0232] The first data set includes at least one data block, and the second data set includes at least one data block.

[0233] In a possible scenario, the computing device performing data deduplication on the first data set includes deleting, by the computing device, at least one data block included in the first data set that matches the stored data, and storing at least one data block included in the first data set that does not match the stored data. The computing device performing data deduplication on the second data set includes deleting, by the computing device, at least one data block included in the second data set that matches the stored data, and storing at least one data block included in the second data set that does not match the stored data.

[0234] The computing device can receive data to be stored, divide the data to be stored to obtain a data set (such as the first data set and the second data set) including at least one data block, and generate a content feature of each data block. After obtaining the content feature of each data block, the computing device queries whether the data block generating the content feature has performed data deduplication according to the content feature. The computing device includes an index subsystem, and the index subsystem includes a database (such as a teraDB database). In a general scenario, the teraDB database stores a mapping between a logical address and a physical address of data. In a data deduplication scenario, the teraDB database stores a mapping between a logical address of data and a content feature. The computing device determines whether the data has performed data deduplication by using the different mapping relationships in different scenarios. The computing device accesses the teraDB database, and if the content feature is stored in the data block, the data has performed data deduplication. Otherwise, the data has not performed data deduplication. The process can be performed in a global cache of the computing device.

[0235] Figure 17 A data processing block diagram provided in the present application is as follows: Figure 17As shown, the computing device receives the to-be-stored data sent by the client, and writes the to-be-stored data to the cache. The computing device can process the to-be-stored data by using a data chunking module to obtain at least one data chunk. The computing device calculates the content feature of each data chunk, queries the mapping in the database located in the index subsystem according to the content feature of the data chunk, determines whether the data chunk generating the content feature has performed data deduplication, and sets a deduplication flag for the data that has performed data deduplication. The computing device stores the data chunk that does not match the stored data to the distributed storage cluster. The data chunking module can include a multi-stream parallel computing module and a multi-strategy cut point judgment module. The multi-stream parallel computing module divides the to-be-stored data by using a plurality of parallel degrees (such as a first parallel degree) to obtain a first initial data set. The multi-strategy cut point judgment module uses different cut point judgment rules to determine whether the position of the last byte of the data chunk with different lengths in the first initial data set is a cut point to obtain a first cut point set. The multi-strategy cut point judgment module compares whether the cut point features of each cut point in the first cut point set match the cut point features in the stored cut point feature table. If they match, the computing device directly divides the to-be-stored data to obtain a data chunk with a length equal to the distance of the cut point in the stored cut point feature table, without the need to slide and divide the to-be-stored data byte by byte to obtain a new data chunk, thereby achieving deduplication calculation. The computing device adjusts the parallel degree according to the calculation time consumption and cache occupancy rate of the cut point calculation.

[0236] It can be understood that, in order to implement the functions in the above embodiments, the computing device includes corresponding hardware structures and / or software modules for performing various functions. Those skilled in the art should easily realize that, in combination with the units and method steps of the examples described in the embodiments disclosed in the present application, the present application can be realized in the form of hardware or a combination of hardware and computer software. Whether a certain function is implemented in hardware or computer software driven hardware depends on the specific application scenario and design constraints of the technical solution.

[0237] The data processing method provided by the present embodiment is described in detail above. Figures 1 to 17 The data processing device provided by the present embodiment is described below. Figure 18

[0238] Figure 18 A structural schematic diagram of a data processing device provided by the present application is shown in the figure. The data processing device can be used to implement the functions of the processor in the above method embodiments, and thus can also achieve the beneficial effects possessed by the above method embodiments. As shown in the figure, Figure 18 The data processing device 1800 includes a cut point judgment module 1810, an adjustment module 1820, a processing module 1830, and a storage module 1840. The data processing device 1800 is used to implement the functions of the processor in the above method embodiments, and thus can also achieve the beneficial effects possessed by the above method embodiments. Figure 7 ​The functions of the computing device in the method embodiments shown in the figures can be implemented by the cutting point determination module 1810. The cutting point determination module 1810 can be configured to implement the functions of S710 and S730 in the above method embodiments, the adjustment module 1820 can be configured to implement the function of S720 in the above method embodiments, and the storage module 1840 can be configured to store the data after performing data deduplication on the to-be-stored data.

[0239] The cutting point determination module 1810 is configured to: acquire a first cutting point set of a first piece of data in the to-be-stored data according to a first parallelism, and divide the first piece of data according to at least one cutting point in the first cutting point set to obtain a first data set. The first parallelism is used to indicate M sliding windows. The adjustment module 1820 is configured to: adjust the first parallelism according to at least one of a calculation time consumption and a cache occupancy rate of the first cutting point set to obtain a second parallelism. The first cache occupancy rate is used to indicate a cache occupancy rate of performing data deduplication on the first piece of data according to the first cutting point set. The second parallelism is used to indicate N sliding windows, and M is not equal to N, M and N are integers greater than or equal to 2. The cutting point determination module 1810 is further configured to: acquire a second cutting point set of a second piece of data in the to-be-stored data according to the second parallelism, and divide the second piece of data according to at least one cutting point in the second cutting point set to obtain a second data set. The processing module 1830 is configured to: perform data deduplication on the first data set and the second data set. The first data set includes at least one data block, and the second data set includes at least one data block.

[0240] In a possible case, the cutting point determination module 1810 is specifically configured to: acquire the first cutting point set of the first piece of data by sliding the M sliding windows on the to-be-stored data at least once.

[0241] In another possible case, the cutting point determination module 1810 is specifically configured to: divide the first piece of data according to the M sliding windows to obtain a first initial data set. In addition, the cutting point determination module 1810 is further specifically configured to: obtain the first cutting point set according to a cutting point determination rule associated with a length distribution feature of data blocks in the first initial data set.

[0242] In another possible case, the cutting point determination module 1810 is specifically configured to: obtain the first cutting point set according to a cutting point determination rule associated with a length interval determined according to a length of a data block in the first initial data set and a maximum value, a minimum value, and an expected value of the length.

[0243] In another possible case, the cutting point determination module 1810 is specifically configured to: divide the first piece of data according to the M sliding windows with an interval of at least one byte to obtain the first initial data set.

[0244] In another possible case, the lengths of the M sliding windows are consistent.

[0245] In another possible scenario, the time consumption of the calculation of the first set of cut points includes one of the first time length, the second time length, and one of the first time length and the second time length.

[0246] The first time length is used to indicate a time length required for dividing the first segment of data by the M sliding windows to obtain the first initial data set. The second time length is used to indicate a time length required for obtaining the first set of cut points according to the cut point determination rule associated with the length and the length distribution feature of the data block in the first initial data set.

[0247] In another possible scenario, the adjusting module 1820 is specifically configured to reduce the M sliding windows to N sliding windows in a case where at least one of the time consumption of the calculation of the first set of cut points and the cache occupancy rate is greater than a respective upper limit value. M is greater than N.

[0248] In another possible scenario, the adjusting module 1820 is specifically configured to increase the M sliding windows to N sliding windows in a case where at least one of the time consumption of the calculation of the first set of cut points and the cache occupancy rate is less than a respective lower limit value. M is less than N.

[0249] In another possible scenario, the processing module 1830 is specifically configured to delete at least one data block in the first data set that matches the stored data, and store at least one data block in the first data set that does not match the stored data. The processing module 1830 is also specifically configured to delete at least one data block in the second data set that matches the stored data, and store at least one data block in the second data set that does not match the stored data.

[0250] In another possible scenario, the storage module 1840 is configured to store the stored data. The stored data includes a second data block and a third data block. At least one data block in the first data set matches the second data block. The cut point division module is also configured to divide a third segment of data to be stored by using the length of the third data block to obtain a fourth data block.

[0251] In another possible scenario, the first data set includes a fifth data block, a sixth data block, and a seventh data block. The lengths of the fifth data block and the sixth data block are the same as the maximum value of the lengths. The first cut point divides the data to be stored to obtain the fifth data block, and the second cut point divides the data to be stored to obtain the seventh data block. The processing module 1830 is also configured to generate a cut point feature table of the fifth data block and the sixth data block. The cut point feature table includes a cut point feature and a cut point distance. The cut point feature is used to indicate the fifth data block and the sixth data block. The cut point distance is used to indicate the distance between the second cut point and the first cut point.

[0252] The cut point determination module 1810, the adjustment module 1820, the processing module 1830, and the storage module 1840 can be implemented by software or by hardware. For example, the implementation of the cut point determination module 1810 is described below. The implementation of the adjustment module 1820, the processing module 1830, and the storage module 1840 can be similar to the implementation of the cut point determination module 1810.

[0253] As an example of a software functional unit, the cut point determination module 1810 can include code running on a computing instance. The computing instance can include at least one of a physical host (computing device), a virtual machine, and a container. Further, the computing instance can be one or more. For example, the cut point determination module 1810 can include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers running the code can be distributed in the same region, or in different regions. Further, the multiple hosts / virtual machines / containers running the code can be distributed in the same availability zone (AZ), or in different AZs. Each AZ includes one data center or multiple data centers in close geographical proximity. Typically, one region includes multiple AZs.

[0254] Similarly, the multiple hosts / virtual machines / containers running the code can be distributed in the same virtual private cloud (VPC), or in multiple VPCs. Typically, one VPC is set up in one region, and communication between two VPCs in the same region or between VPCs in different regions requires a communication gateway in each VPC to achieve interconnection between VPCs.

[0255] As an example of a hardware functional unit, the cut-point determination module 1810 may include at least one computing device, such as a server. Alternatively, the cut-point determination module 1810 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0256] The multiple computing devices included in the cut-point determination module 1810 can be distributed in the same region or in different regions. Similarly, the multiple computing devices included in the cut-point determination module 1810 can be distributed in the same Availability Zone (AZ) or in different AZs. Likewise, the multiple computing devices included in the cut-point determination module 1810 can be distributed in the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0257] It should be noted that, in other embodiments, the cut-point determination module 1810 can be used to execute any step in the data processing method, the adjustment module 1820 can be used to execute any step in the data processing method, and the storage module 1840 can be used to execute any step in the data processing method. The steps implemented by the cut-point determination module 1810, the adjustment module 1820, the processing module 1830, and the storage module 1840 can be specified as needed. By implementing different steps in the data processing method through the cut-point determination module 1810, the adjustment module 1820, the processing module 1830, and the storage module 1840, the full functions of the data processing device can be realized.

[0258] Figure 19 A schematic diagram of the structure of a computing device provided in this application, such as... Figure 19 As shown, the computing device 1900 includes a processor 1910, a bus 1920, a memory 1930, and a communication interface 1940. The processor 1910, memory 1930, and communication interface 1940 communicate with each other via the bus 1920. The computing device 1900 can be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in the computing device 1900.

[0259] Bus 1920 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 19 Only one line is used in the figure to represent a bus, but this does not mean that there is only one bus or only one type of bus. Bus 104 can include paths for communicating information among the various components of computing device 1900 (e.g., memory 1930, processor 1910, communication interface 1940).

[0260] Processor 1910 can be configured to obtain a first data set and a second data set. Processor 1910 can include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP), etc.

[0261] Memory 1930 can include volatile memory (e.g., random access memory (RAM)), processor 1910 can also include non-volatile memory (e.g., read-only memory (ROM), flash memory, a hard disk drive (HDD), or a solid state drive (SSD)).

[0262] Memory 1930 can store executable program code, and processor 1910 executes the executable program code to respectively implement the functions of the aforementioned processing module, the cut point determination module, the adjustment module, and the storage module, thereby implementing the data processing method. That is, memory 1930 stores instructions for executing the data processing method. Memory 1930 can also store data after the data is re-deleted.

[0263] Alternatively, memory 1930 stores executable program code, and processor 1910 executes the executable program code to respectively implement the functions of the aforementioned data processing apparatus, thereby implementing the data processing method. That is, memory 1930 stores instructions for executing the data processing method.

[0264] The communication interface 1940 uses a transceiving module such as, but not limited to, a network interface card, a transceiver, to enable communication between the computing device 1900 and other devices or communication networks, such as can receive data to be stored sent by a client.

[0265] The present application also provides a cluster. The cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a notebook computer, or a smart phone.

[0266] Figure 20 A schematic diagram of a cluster provided by the present application is shown in FIG. 1. As shown in FIG. 1, the cluster includes at least one computing device 1900. The same instructions for performing a data processing method can be stored in the memory 1930 of one or more computing devices 1900 in the cluster. Figure 20

[0267] In some possible implementations, partial instructions for performing the data processing method can also be respectively stored in the memory 1930 of one or more computing devices 1900 in the cluster. In other words, the combination of one or more computing devices 1900 can collectively execute the instructions for performing the data processing method.

[0268] It should be noted that the memory 1930 in different computing devices 1900 in the cluster can store different instructions, respectively used to perform partial functions of the data processing apparatus. That is, the instructions stored in the memory 1930 in different computing devices 1900 can implement the functions of one or more of the processing module, the cut point determination module, the adjustment module, and the storage module.

[0269] In some possible implementations, one or more computing devices in the cluster can be connected through a network. The network can be a wide area network or a local area network, etc. Figure 21 A possible implementation is shown in FIG. 2. Figure 21 A connection diagram of a computing device provided by the present application is shown in FIG. 2. As shown in FIG. 2, two computing devices 1900A and 1900B are connected through a network. Specifically, the computing devices are connected to the network through the communication interface in each computing device. In this type of possible implementation, the memory 1930 in the computing device 1900A stores instructions for performing the functions of the processing module. Meanwhile, the memory 1930 in the computing device 1900B stores instructions for performing the functions of the cut point determination module, the adjustment module, and the storage module. Figure 21

[0270] Figure 21 ​​The connection mode between the clusters shown can be that a large amount of division of the data to be stored is required for the data processing method provided in the present application, and the cut point set is calculated. Therefore, the functions implemented by the adjustment module and the storage module are executed by the computing device 1900B.

[0271] It should be understood that Figure 21 The functions of the computing device 1900A shown in the middle can also be completed by multiple computing devices 1900. Similarly, the functions of the computing device 1900B can also be completed by multiple computing devices 1900.

[0272] The present application also provides another cluster. The connection relationship between the computing devices in the cluster can be similar to the connection mode of the cluster Figure 20 and Figure 21 of the cluster. The difference is that the same instructions for executing the data processing method can be stored in the memory 1930 of one or more computing devices 1900 in the cluster.

[0273] In some possible implementations, part of the instructions for executing the data processing method can also be stored in the memory 1930 of one or more computing devices 1900 in the cluster, respectively. In other words, the combination of one or more computing devices 1900 can collectively execute the instructions for executing the data processing method.

[0274] The present application also provides a computer program product containing instructions. The computer program product can be a software or program product containing instructions, which can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device is caused to execute the data processing method, or the data processing method.

[0275] The present application also provides a computer readable storage medium. The computer readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk), etc. The computer readable storage medium contains instructions, which instruct the computing device to execute the data processing method, or instruct the computing device to execute the data processing method.

[0276] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A data processing method, characterized by, The method comprises: obtaining a first cutpoint set of a first segment of data in the to-be-stored data according to a first parallelism, and dividing the first segment of data according to at least one cutpoint in the first cutpoint set to obtain a first data set, the first parallelism being used to indicate M sliding windows; adjusting the first parallelism according to at least one of a calculation time consumption of the first cutpoint set and a cache occupancy rate, the cache occupancy rate being used to indicate a cache occupancy rate of performing data deduplication on the first segment of data according to the first cutpoint set, the second parallelism being used to indicate N sliding windows, M not equal to N, M and N being integers greater than or equal to 2; obtaining a second cutpoint set of a second segment of data in the to-be-stored data according to the second parallelism, and dividing the second segment of data according to at least one cutpoint in the second cutpoint set to obtain a second data set; performing data deduplication on the first data set and the second data set, the first data set comprising at least one data block, and the second data set comprising at least one data block.

2. The method of claim 1, wherein the first cutpoint set of the first segment of data is obtained by sliding the M sliding windows on the to-be-stored data at least once.

3. The method of claim 1 or 2, wherein the first cutpoint set of the first segment of data according to the first parallelism comprises: dividing the first segment of data according to the M sliding windows to obtain a first initial data set; obtaining the first cutpoint set according to a cutpoint judgment rule associated with the length and length distribution characteristics of the data blocks in the first initial data set.

4. The method of claim 3, wherein the first cutpoint set is obtained according to the cutpoint judgment rule associated with the length and length distribution characteristics of the data blocks in the first initial data set, comprising: obtaining the first cutpoint set according to a cutpoint judgment rule associated with a length interval determined according to the maximum value, minimum value and expected value of the length of the data blocks in the first initial data set.

5. The method of claim 3, wherein the first initial data set is obtained by dividing the first segment of data according to the M sliding windows, comprising: the first initial data set is obtained by dividing the first segment of data according to the M sliding windows at an interval of at least 1 byte.

6. The method of any one of claims 1, 2, 4, or 5, wherein, The lengths of the M sliding windows are consistent.

7. The method of claim 3, wherein, The calculation time consumption of the first cutpoint set comprises: a first time length; or a second time length; or a sum of the first time length and the second time length; wherein the first time length is used to indicate a time length required for dividing the first segment of data by the M sliding windows to obtain the first initial data set, and the second time length is used to indicate a time length required for obtaining the first cutpoint set according to the cutpoint judgment rule associated with the length and length distribution characteristics of the data blocks in the first initial data set.

8. The method of any one of claims 1, 2, 4, 5 or 7, wherein the second parallelism is obtained by adjusting the first parallelism according to at least one of the calculation time consumption of the first cutpoint set and the cache occupancy rate, comprising: In a case that at least one of the calculation time consumption and the cache occupancy of the first cut point set is greater than a respective upper limit value, the M sliding windows are reduced to N sliding windows, where M is greater than N.

9. The method of any one of claims 1, 2, 4, 5, or 7, wherein, the adjusting the first parallel degree according to at least one of the calculation time consumption and the cache occupancy of the first cut point set comprises: In a case that at least one of the calculation time consumption and the cache occupancy of the first cut point set is less than a respective lower limit value, the M sliding windows are increased to N sliding windows, where M is less than N.

10. The method of any one of claims 1, 2, 4, 5, or 7, wherein, the performing data deduplication on the first data set and the second data set comprises: deleting at least one data block included in the first data set that matches stored data, and storing at least one data block included in the first data set that does not match the stored data; deleting at least one data block included in the second data set that matches the stored data, and storing at least one data block included in the second data set that does not match the stored data.

11. The method of claim 10, wherein, the stored data includes a second data block and a third data block, and at least one data block included in the first data set matches the second data block, the method further comprises: dividing a third segment of the to-be-stored data using a length of the third data block to obtain a fourth data block.

12. The method of any one of claims 1, 2, 4, 5, or 7, wherein, the first data set includes a fifth data block, a sixth data block, and a seventh data block, the fifth data block and the sixth data block have the same length or the same maximum length, the fifth data block is divided from the to-be-stored data by a first cut point, and the seventh data block is divided from the to-be-stored data by a second cut point, the method further comprises: generating a cut point feature table of the fifth data block and the sixth data block, the cut point feature table including a cut point feature and a cut point distance, the cut point feature being used to indicate the fifth data block and the sixth data block, and the cut point distance being used to indicate a distance between the second cut point and the first cut point.

13. A data processing apparatus, characterized by: the apparatus comprises: a cut point determination module configured to: obtain a first cut point set of a first segment of to-be-stored data according to a first parallel degree, and divide the first segment of data according to at least one cut point in the first cut point set to obtain a first data set, the first parallel degree being used to indicate M sliding windows; an adjustment module configured to: adjust the first parallel degree according to at least one of a calculation time consumption and a cache occupancy of the first cut point set to obtain a second parallel degree, the cache occupancy being used to indicate a cache occupancy of performing data deduplication on the first segment of data according to the first cut point set; and the second parallel degree being used to indicate N sliding windows, M not being equal to N, M and N being integers greater than or equal to 2; the cut point determination module is further configured to: obtain a second cut point set of a second segment of the to-be-stored data according to the second parallel degree, and divide the second segment of data according to at least one cut point in the second cut point set to obtain a second data set. The processing module is further configured to perform data deduplication on the first data set and the second data set, the first data set comprising at least one data block, and the second data set comprising at least one data block.

14. A computer device, comprising: A computer device comprising a memory and a processor, the memory being configured to store a set of computer instructions; when the processor executes the set of computer instructions, the processor performs the method of any one of claims 1-12.

15. A cluster, characterized by The cluster comprises a computing node and a storage node, the computing node being configured to perform the method of any one of claims 1-12, and the storage node being configured to store data after the computing node performs data deduplication on the data to be stored.

Images