Data processing method and device, storage medium and computer equipment

Abstract

The embodiment of the invention provides a data processing method and device, a storage medium and computer equipment, and the method comprises the steps: firstly obtaining a current change position and change data in a data table, and writing into a first data block; obtaining to-be-compared data blocks of a specific column of the change data record table and the last column with the data blocks, wherein each column of data blocks of the table corresponds to historical change information of the data table; comparing the to-be-compared data block with the first data block, and screening out a target data block and a target change position with the same change position; extracting target data and writing into a second data block; and writing the second data block corresponding to the target data block into the second row of the corresponding column, and writing the first data block into the next column of the last column. According to the method, the current data change condition can be recorded, the key point is that only the updated data of the recovery check point is read, so that excessive consumption of resources and time cost can be avoided, the data recovery efficiency is effectively improved, and the method has important significance in the field of data processing and can be applied to numerous data management scenes to optimize the data recovery process.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data processing method, device, storage medium and computer equipment. Background Art

[0002] Recommendation models are widely used in existing recommendation systems and usually consist of multiple embedding tables and multi-layer perceptrons (MLPs). The embedding table occupies the vast majority of the model, while the multi-layer perceptron only occupies a very small part. During the training of the recommendation model, the sparse features and dense features in the dataset are input into the embedding table and the MLP layer for processing respectively. Each embedding table consists of a large number of embedding vectors. The embedding table performs table lookup based on the sparse features to obtain the corresponding embedding vector, thereby mapping the sparse features to dense vectors for subsequent calculations. However, in each iteration, the embedding vector obtained by the table lookup only occupies a small part of all the embedding vectors, that is, only a small part of the embedding vectors participate in this iteration and are updated after this iteration. This partial update feature provides a unique opportunity for the recommendation model checkpoint mechanism, that is, only saving the updated data instead of saving all the data every time.

[0003] In the related art, an incremental checkpoint mechanism is used to save only updated data. Although a variety of intelligence data can be obtained from different channels, there is a large amount of duplicate data between multiple checkpoints. The duplicate data here refers to the data at the same location in the model that has undergone multiple updates and has been written into multiple incremental checkpoints. However, when restoring a checkpoint, only the most recently updated version of the data is needed. In other words, the incremental checkpoint reads a large amount of redundant data during recovery, resulting in greater resource consumption and time cost consumption (Input / Output Overhead, I / O), and reducing recovery efficiency. Therefore, the related art urgently needs to propose a data processing method to solve the above technical problems. Summary of the invention

[0004] The main purpose of the present application is to provide a data processing method, apparatus, storage medium and computer equipment, which can read only the data for the recovery checkpoint in the data at the location that has undergone multiple updates when restoring the checkpoint, without reading the data for each update, thereby avoiding resource consumption and time cost consumption and improving recovery efficiency.

[0005] In a first aspect, an embodiment of the present application provides a data processing method, including:

[0006] Obtaining a current change position where data change currently occurs in the data table, and change data of each current change position;

[0007] Writing each of the current change positions and the corresponding change data into the first data block;

[0008] Obtain the data block to be compared in the first row of each column of the change data record table, and the last column currently recording the data block, wherein all the data blocks recorded in each column of the change data record table correspond to the historical change position and the corresponding historical change data when the historical data in the data table is changed;

[0009] Comparing each of the data blocks to be compared with the first data block, selecting at least one target data block from each of the data blocks to be compared, and selecting a target change position that is the same as the current change position from the target data blocks;

[0010] Extracting the corresponding target change position and the corresponding target data from the target data block, and writing them into the second data block;

[0011] The second data block corresponding to the target data block is written into the second row of the corresponding column, and the first data block is written into the next column of the last column to record the current change position where the data change occurs and the change data of each current change position.

[0012] In a second aspect, an embodiment of the present application provides a data processing device, including:

[0013] A first acquisition unit, configured to acquire a current change position where data change currently occurs in a data table, and change data of each current change position;

[0014] A first writing unit, configured to write each of the current change positions and corresponding change data into a first data block;

[0015] A second acquisition unit is used to acquire the data block to be compared in the first row of each column of the change data record table, and the last column currently recording the data block, wherein all the data blocks recorded in each column of the change data record table correspond to the historical change position when the historical data in the data table is changed and the corresponding historical change data;

[0016] a comparing unit, configured to compare each of the to-be-compared data blocks with the first data block, select at least one target data block from each of the to-be-compared data blocks, and select a target change position that is the same as the current change position from the target data blocks;

[0017] A second writing unit, used for extracting a corresponding target change position and corresponding target data from the target data block, and writing them into a second data block;

[0018] The third writing unit is used to write the second data block corresponding to the target data block into the second row of the corresponding column, and write the first data block into the next column of the last column, so as to record the current change position where the data change occurs and the change data of each current change position.

[0019] In a third aspect, an embodiment of the present application provides a storage medium, wherein the computer-readable storage medium stores a plurality of instructions, which are suitable for loading by a processor to execute any of the above data processing methods.

[0020] In a fourth aspect, an embodiment of the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any of the above data processing methods when executing the computer program.

[0021] In an embodiment of the present application, by obtaining the current change position of the data change currently occurring in the data table, and the change data of each current change position; writing each current change position and the corresponding change data into the first data block; obtaining the data block to be compared in the first row of each column of the change data record table, and the last column where the data block is currently recorded, all the data blocks recorded in each column of the change data record table correspond to the historical change position and the corresponding historical change data when the historical data in the data table is changed; comparing each of the data blocks to be compared with the first data block, filtering out at least one target data block from each of the data blocks to be compared, and filtering out the target data blocks that are compared with the current data block. The target change position is the same as the previous change position; the corresponding target change position and the corresponding target data are extracted from the target data block, and written into the second data block; the second data block corresponding to the target data block is written into the second row of the corresponding column, and the first data block is written into the next column of the last column to record the current change position where the data change has occurred, and the change data of each current change position. Compared with the related art, when restoring the checkpoint, it is necessary to read the parameters of each update of the position that has undergone multiple updates, and a large amount of redundant data is read. The embodiment of the present application can only read the updated data for the recovery checkpoint, avoid resource consumption and time cost consumption, and improve recovery efficiency.

[0022] Other features and advantages of the present disclosure will be described in the following description, and partly become apparent from the description, or understood by practicing the present disclosure. The purpose and other advantages of the present disclosure can be realized and obtained by the structures particularly pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0024] Figure 1 A schematic diagram of a data processing system according to an embodiment of the present invention.

[0025] Figure 2 A flowchart of a data processing method provided in an embodiment of the present application.

[0026] Figure 3 A first schematic diagram of a change data record table provided in an embodiment of the present application.

[0027] Figure 4 A second schematic diagram of a change data record table provided in an embodiment of the present application.

[0028] Figure 5 A schematic diagram of the structure of a data processing method and device provided in an embodiment of the present application.

[0029] Figure 6 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0030] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0031] It should be noted that in some processes described in the specification, claims and the above-mentioned drawings, multiple steps appearing in a specific order are included, but it should be clearly understood that these steps may not be executed in the order in which they appear in this document or may be executed in parallel. The step numbers are only used to distinguish different steps, and the numbers themselves do not represent any execution order. In addition, descriptions such as "first", "second" or "target" in this document are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0032] Before further describing the embodiments of the present disclosure in detail, the nouns and terms involved in the embodiments of the present disclosure are described. The nouns and terms involved in the embodiments of the present disclosure are subject to the following interpretations:

[0033] Incremental checkpoint mechanism, 1. Saving process: First save a complete set of model parameters as a baseline checkpoint. For example, when training a large recommendation model, initially save the parameters of the entire model as a baseline. Assume that the size of this baseline checkpoint is 100GB (including all embedding tables and multi-layer perceptron parameters). In the subsequent training process, each checkpoint only saves the parameters updated since the last checkpoint. For example, after a period of training, the next time you save a checkpoint, assuming that only 0.1% (100MB) of the embedding table parameters are updated, then this incremental checkpoint only saves the 100MB updated parameters, rather than the parameters of the entire model.

[0034] 2. Recovery process: When you need to restore a checkpoint, you need to load the baseline checkpoint and all incremental checkpoints saved since the baseline checkpoint.

[0035] For example, if 5 incremental checkpoints are saved during the training process, when restoring to the 5th checkpoint, you need to load the 100GB baseline checkpoint first, and then load the 5 incremental checkpoints in sequence (assuming that the size of each incremental checkpoint is 100MB, 150MB, 80MB, 120MB, and 90MB respectively). Therefore, the total amount of data stored by the incremental checkpoint mechanism that needs to be loaded when the checkpoint is subsequently restored is 100GB+100MB+150MB+80MB+120MB+90MB=100GB+540MB.

[0036] Since there are a large number of duplicate parameters between multiple checkpoints (parameters at the same position in the model are updated multiple times and written into multiple incremental checkpoints, but only the most recently updated version is needed for recovery), a large amount of redundant data is read during recovery, resulting in greater I / O overhead and subsequent deserialization overhead, reducing recovery efficiency. For example, among the five incremental checkpoints mentioned above, there may be many parameters that have update records in multiple checkpoints, but only the last updated value is needed for actual recovery, but the old values ​​from many times ago are read.

[0037] Differential checkpoint mechanism,1. Saving process: Also save a complete set of model parameters as a baseline checkpoint, such as the 100GB baseline checkpoint in the above example. Each subsequent checkpoint saves the updated parameters since the baseline checkpoint.

[0038] For example, when saving a differential checkpoint for the first time, if 200MB of parameters have been updated since the baseline checkpoint, then this differential checkpoint will save these 200MB of parameters. When saving a differential checkpoint next time, if 180MB of new parameters have been updated (relative to the baseline checkpoint), then these 180MB of parameters will be saved instead of a cumulative total of 380MB (relative to the previous differential checkpoint).

[0039] 2. Recovery process: When restoring a checkpoint, you only need to load the base checkpoint and one differential checkpoint. For example, to restore to a specific checkpoint, you only need to load the 100GB base checkpoint first, and then load the corresponding differential checkpoint (assuming it is 200MB). The total amount of data that needs to be loaded is 100GB + 200MB.

[0040] For example, after multiple checkpoint saves, a differential checkpoint may reach 500MB or even 1GB. Compared with the relatively small and constant amount of incremental checkpoints saved each time (such as about 100MB in the above example), the storage pressure of differential checkpoints will gradually increase.

[0041] Compared with the incremental checkpoint mechanism, although differential checkpoints have advantages in recovery, they have the following disadvantages. Since differential checkpoints store the parameters updated since the baseline checkpoint, the amount of parameters saved each time increases rapidly as the number of checkpoints increases. The increase in the number of parameters leads to two problems. On the one hand, the training process will be blocked when saving checkpoints. The larger the number of parameters, the longer the blocking time. Long-term blocking makes it impossible for the training system to save checkpoints frequently, which will cause the training system to lose more training progress when a failure occurs. On the other hand, the large number of parameters causes more storage space overhead, resulting in expensive storage maintenance costs.

[0042] Incremental checkpoints avoid the above two problems. Since only the updated parameters since the last checkpoint are saved, the number of parameters saved by incremental checkpoints is small and constant, and will not increase with the increase of checkpoint times. This has huge advantages in terms of checkpoint saving time and storage space.

[0043] However, although the incremental checkpoint mechanism has been used in other fields, such as high-performance computing and stream processing, no work has yet applied the incremental checkpoint mechanism to recommendation model training. The main reason is that the efficiency of recovering from incremental checkpoints is too low. The root cause is that there are a large number of duplicate parameters between multiple checkpoints. Duplicate parameters here refer to the parameters at the same position in the model that have undergone multiple updates and are written to multiple incremental checkpoints. However, when restoring a checkpoint, only the most recently updated version of the parameter is required. In other words, incremental checkpoints read a lot of redundant data when restoring, resulting in greater I / O overhead and subsequent deserialization overhead, reducing recovery efficiency.

[0044] In order to solve the above problems, the embodiment of the present application obtains the current change position of the data change currently occurring in the data table, and the change data of each current change position; writes each current change position and the corresponding change data into the first data block; obtains the data block to be compared in the first row of each column of the change data record table, and the last column where the data block is currently recorded, and all the data blocks recorded in each column of the change data record table correspond to the historical change position and the corresponding historical change data when the historical data in the data table is changed; compares each of the data blocks to be compared with the first data block, filters out at least one target data block from each of the data blocks to be compared, and filters out the target data blocks that are compared with the target data blocks. The target change position is the same as the current change position; the corresponding target change position and the corresponding target data are extracted from the target data block, and written into the second data block; the second data block corresponding to the target data block is written into the second row of the corresponding column, and the first data block is written into the next column of the last column, so as to record the current change position where the data change occurs, and the change data of each current change position. Compared with the related art, when restoring the checkpoint, it is necessary to read the parameters of each update of the position that has undergone multiple updates, and a large amount of redundant data is read. The embodiment of the present application can only read the updated data for the recovery checkpoint, avoid resource consumption and time cost consumption, and improve recovery efficiency. Please continue to refer to the following specific embodiments for details.

[0045] See also Figure 1 , Figure 1 A schematic diagram of a data processing system provided in an embodiment of the present application, which includes a client 140, the Internet 130, a gateway 120, a computer device 110, etc.

[0046] The client 140 includes but is not limited to a pre-configured personal computer, or a tablet computer, a desktop computer, or other electronic device with data reporting capabilities. In addition, it can be a single device or a collection of multiple devices. The client 140 can communicate with the Internet 130 in a wired or wireless manner to exchange data.

[0047] The client 140 refers to a computer system that can report data (such as training data) to the computer device 110. Compared with ordinary terminals, the computer device 110 has higher requirements in terms of stability, security, performance, etc. The computer device 110 can be a high-performance computer in a network platform, a cluster of multiple high-performance computers, a part of a high-performance computer (such as a virtual machine), a combination of parts of multiple high-performance computers (such as virtual machines), etc.

[0048] The gateway 120 is also called an internetwork connector or a protocol converter. The gateway realizes network interconnection at the transport layer and is a computer system or device that acts as a converter. The gateway is a translator between two systems that use different communication protocols, data formats or languages, or even completely different architectures. At the same time, the gateway can also provide filtering and security functions. The message sent by the client 140 to the computer device 110 must be sent to the corresponding computer device 110 through the gateway 120. The message sent by the computer device 110 to the client 140 must also be sent to the corresponding client 140 through the gateway 120.

[0049] The data processing method of the embodiment of the present disclosure may be implemented on the computer device 110 .

[0050] It should be noted that Figure 1 The scenario diagram of the data processing system shown is merely an example. The data processing system and scenario described in the embodiment of the present application are intended to more clearly illustrate the technical solution of the embodiment of the present application, and do not constitute a limitation on the technical solution provided in the embodiment of the present application. A person of ordinary skill in the art can appreciate that with the evolution of data processing technology and the emergence of new business scenarios, the technical solution provided in the embodiment of the present application is equally applicable to similar technical problems.

[0051] In this embodiment, the description will be made from the perspective of a data processing device, which can be specifically integrated into a computer device having a storage unit and a microprocessor installed therein and having computing capabilities.

[0052] See also Figure 2 , Figure 2 A flow chart of a data processing method provided in an embodiment of the present application. The data processing method includes:

[0053] In step 201, the current change position where the data change currently occurs in the data table and the change data of each current change position are obtained.

[0054] Among them, the data table is a table structure for storing data. Taking the application scenario of training a recommendation model as an example, during the training process, the embedding vectors in the embedding table (Embedding Table) of the recommendation model and the perceptron data of the multi-layer perceptron (MLP) are mainly adjusted. During the training process, the embedding table occupies the vast majority of the model, while the multi-layer perceptron only occupies a very small part. The sparse features and dense features in the data set are input into the embedding table and the MLP layer for processing respectively. Each embedding table consists of a large number of embedding vectors. The embedding table performs a table query based on the sparse features to obtain the corresponding embedding vector, thereby mapping the sparse features to dense vectors for subsequent calculations. Based on this, the data table can be an embedding table, in which the data will change with the process of training iterations. It is the basic object of the entire data processing operation and contains multiple data items. Each data item has its specific position in the table. The current change position is the position identifier of the data item in the data table where the data change currently occurs, which is used to determine which data item has changed. For example, it can be a combination of row number and column number (if it is a data table with a two-dimensional table structure), or other methods that can uniquely identify the position of the data item in the table. The changed data is the new value of the data item after it is modified at the current change position, that is, the specific content of the data after the change.

[0055] Specifically, the embodiment of the present application provides a scenario in which the changed data is recorded when there is a change in the data. A recording period can also be set in advance. When the recording period is reached, the changed data in the current data table relative to the previous recording period is recorded (equivalent to periodically saving checkpoints). No limitation is made here.

[0056] In step 202, each current change position and the corresponding change data are written into the first data block.

[0057] The first data block is a data structure for temporarily storing the current change position and its corresponding change data. It can be a continuous storage space in the memory, or a data container in a specific format. Its purpose is to integrate the change information of the same batch together.

[0058] Specifically, the first data block may be an array structure or a user-defined structure array, wherein each element includes two parts, one for storing the change position information, and the other for storing the change data.

[0059] For example, there are currently two data items whose data have changed, and the changed positions and data are (2, 15) and (4, 20) respectively. 2 and 4 are the changed positions, and 15 and 20 are the changed data corresponding to the changed positions. Then the first data block can be represented as "((2, 15), (4, 20))".

[0060] In step 203, the data blocks to be compared in the first row of each column of the change data record table and the last column currently recording the data blocks are obtained. All data blocks recorded in each column of the change data record table correspond to the historical change positions and corresponding historical change data when the historical data in the data table is changed.

[0061] Among them, see Figure 3 , Figure 3 The first schematic diagram of the change data record table provided in the embodiment of the present application. The change data record table is a table structure specifically used to record the historical data changes of the data table. Each column records the relevant information when the data changes at different time points or different batches, including the historical change location and the historical change data. In this way, the change process of the data over time can be tracked. The data blocks to be compared are data blocks taken from the first row of each column of the change data record table. These data blocks contain the data change information of a previous time point or batch, and are used to compare with the current first data block to find duplicate or related change locations.

[0062] Specifically, Figure 3 As shown, Figure 3 The first column 10 includes two columns, the first column 10 is the historical change position (1, 2 and 3) and the corresponding historical change data (the historical change data corresponding to position 1, the historical change data corresponding to position 2 and the historical change data corresponding to position 3) recorded by the incremental checkpoint when the data change occurs for the first time since the baseline checkpoint is saved, the second column 20 is the historical change position (3, 4 and 5) and the corresponding historical change data (the historical change data corresponding to position 3, the historical change data corresponding to position 4 and the historical change data corresponding to position 5) recorded by the incremental checkpoint when the data change occurs again after the first column 10 is saved, and the last column currently recording the data block is the second column 20. The data blocks to be compared include the data blocks to be compared 11 in the first row of the first column 10 and the data blocks with comparison 21 in the first row of the second column 20.

[0063] In step 204, each data block to be compared is compared with the first data block, at least one target data block is selected from each data block to be compared, and a target change position that is the same as the current change position is selected from the target data blocks.

[0064] The target data block is a data block in the data block to be compared, which is determined to have the same change position as the current change after being compared with the first data block. After the target data block is screened out, the target change position that is the same as the current change position is screened out from the target data block.

[0065] In some implementations, comparing each of the to-be-compared data blocks with the first data block and selecting at least one target data block from each of the to-be-compared data blocks includes:

[0066] (1) obtaining a historical position range of a historical change position included in the data block to be compared, and a current position range of a current change position included in the first data block;

[0067] (2) comparing each of the historical position ranges with the current position range, and determining a to-be-compared data block corresponding to a historical position range that has an intersection with the current position range as a first candidate data block;

[0068] (3) Select at least one target data block from the first candidate data blocks.

[0069] The changed data record table maintains a position range for each data block to be compared as metadata information of the data block to be compared, and the position range is a range from the smallest changed position to the largest changed position in the data block to be compared.

[0070] Specifically, in order to improve the comparison speed and accelerate the screening rate of the target data block, the current position range of the current change position in the first data block can be compared with the historical position range of each data block to be compared. If the ranges do not have an intersection, it means that the current change position does not exist in the corresponding data block to be compared; if the ranges have an intersection, it means that the current change position falls within the historical position range and may exist in the corresponding data block to be compared. The data block to be compared corresponding to the historical position range that has an intersection with the current position range is determined as the first candidate data block, so as to screen out at least one target data block from the first candidate data block.

[0071] For example, Figure 3In the example, the data block 11 to be compared includes the change position 1 and the change position 2, so the historical position range of the data block 11 to be compared is [1,2]; the data block 21 to be compared includes the change position 3, the change position 4 and the change position 5, so the historical position range of the data block 21 to be compared is [3,5]. The current change positions are 2, 5 and 6, so the current position range of the first data block is [2,6]. Since the historical position range [1,2] of the data block 11 to be compared and the historical position range [3,5] of the data block 21 to be compared both intersect with the current position range [2,6], the data block 11 to be compared and the data block 21 to be compared are both first candidate data blocks.

[0072] In this way, by comparing the position range first, a large number of data blocks to be compared that are obviously irrelevant to the current change position can be quickly excluded. In actual data processing scenarios, if the data table is large and changes frequently, a large number of data blocks to be compared may accumulate in the change data record table. After using the position range comparison, only the data blocks to be compared whose position ranges intersect with the current change position range will enter the subsequent more detailed comparison, which greatly reduces unnecessary computing overhead and improves overall processing efficiency.

[0073] In some implementations, the step of selecting at least one target data block from the first candidate data blocks includes:

[0074] (1.1) obtaining a Bloom filter corresponding to each of the first candidate data blocks, wherein the Bloom filter includes a hash value tag of each hash value;

[0075] (1.2) calculating multiple hash values ​​corresponding to each of the current change positions in the first data block by using different preset hash functions;

[0076] (1.3) determining the number of positions where the hash value marks of the multiple hash values ​​corresponding to each of the current change positions in each of the Bloom filters are all first preset values, and obtaining the number of predicted repeated positions corresponding to each of the first candidate data blocks;

[0077] (1.4) calculating the ratio of the number of each predicted repeated position to the number of positions of the corresponding first candidate data block to obtain the predicted repetition ratio of each first candidate data block;

[0078] (1.5) selecting a target predicted repetition ratio that exceeds a first preset repetition ratio from each of the predicted repetition ratios, and determining a first candidate data block corresponding to the target predicted repetition ratio as a second candidate data block;

[0079] (1.6) Select at least one target data block from the second candidate data blocks.

[0080] Among them, the Bloom filter is a data structure based on a bit array, and its length is predetermined according to factors such as the amount of data and the expected error rate (for example, in processing large-scale data change scenarios, it may be set to 10,000 bits or longer). During initialization, all bits are set to a second preset value (for example, 0, indicating that the element has not been marked yet). For each first candidate data block, there is a corresponding Bloom filter, which is used to quickly determine whether a current change position may exist in the data block. For each current change position, multiple hash values ​​corresponding to the current change position are calculated by different preset hash functions. For example, if the current change position is 7, then hash1(7)=30, hash2(7)=400, hash3(7)=700.

[0081] Specifically, for each Bloom filter of the first candidate data block, the hash value marks of the hash values ​​calculated by traversing the current change position in the first data block are all 1. If they are all 1, it means that this change position may exist in the first candidate data block (there is a possibility of misjudgment). Count the number of all current change positions whose hash values ​​are marked as 1 in the Bloom filter. This number is the number of predicted repeated positions corresponding to the first candidate data block.

[0082] For example, for a first candidate data block, after checking, it is found that the hash values ​​of two currently changed positions are both marked as 1 in its Bloom filter, so the predicted number of repeated positions of the first candidate data block is 2.

[0083] A first preset repetition ratio is preset (for example, 0.2 or 20%, which is determined according to actual needs and data characteristics, and is used to determine whether the possible repetition degree in a data block is high enough to be worth further processing. If the predicted repetition ratio is less than the first preset repetition ratio, it means that a large amount of data needs to be read into the memory, but only a small part of the repeated data can be extracted, so there is no need to perform an extraction operation or read the actual data of the comparison block). The predicted repetition ratio of each first candidate data block obtained by traversal calculation is screened out if it is greater than the first preset repetition ratio.

[0084] For example, if there are 5 first candidate data blocks, and their predicted repetition ratios are 0.15, 0.25, 0.3, 0.18, and 0.22 respectively, then the predicted repetition ratios corresponding to the first candidate data blocks with predicted repetition ratios of 0.25, 0.3, and 0.22 exceed 0.2, and these first candidate data blocks are determined as second candidate data blocks.

[0085] Since the existence of hash collision may lead to misjudgment, after the second candidate data block is determined, at least one target data block needs to be screened out from the second candidate data blocks.

[0086] In this way, through the rapid judgment of the Bloom filter based on the hash value mark, it is possible to preliminarily estimate the possibility of duplication between each first candidate data block and the first data block without detailed comparison of each change position data, reducing the consumption of computing resources and processing time. The second candidate data block is further screened according to the predicted duplication ratio, avoiding reading a large number of data blocks with low duplication probability into the memory for detailed comparison. This screening can ensure that only the data blocks (second candidate data blocks) that are most likely to contain more duplicate data are read into the memory, improving the efficiency of memory use and ensuring stable operation of the system.

[0087] In some implementations, the step of selecting at least one target data block from the second candidate data blocks includes:

[0088] (1.1) comparing the change position in the second candidate data block with the current change position, and determining the actual number of repeated positions of the candidate change position in the second candidate data block that is the same as the current change position;

[0089] (1.2) calculating the ratio of the actual number of repeated positions of the second candidate data block to the number of corresponding positions to obtain an actual repetition ratio of the second candidate data block;

[0090] (1.3) The second candidate data block whose actual repetition ratio exceeds the second preset repetition ratio is determined as the target data block.

[0091] Among them, the Bloom filter has a certain misjudgment rate due to hash collision, so the repetition ratio obtained by the Bloom filter will be greater than the actual repetition ratio. For each second candidate data block, it is necessary to read the actual data in the block for key comparison. Therefore, for each second candidate data block, all the changed positions contained therein are traversed.

[0092] For example, a second candidate data block has change positions 3, 5, 7, 9, 11, and the current change positions are 5, 7, 12. Compare the change positions in the second candidate data block with the current change positions one by one, and find that change positions 5 and 7 exist in both sets, so these two positions are candidate change positions. Count the number of such candidate change positions. In this example, the actual number of repeated positions is 2.

[0093] Specifically, the second preset duplication ratio is set in advance according to the needs and characteristics of data processing. For example, in a data management system that has high requirements for data consistency and hopes to minimize the storage of duplicate data, the second preset duplication ratio may be set to 0.3 (or 30%), which means that if the actual repeated change position ratio in a second candidate data block exceeds 30%, the data block is considered worthy of further processing and determined as a target data block.

[0094] The specific method of determining the target data block is to compare the actual repetition ratio of each second candidate data block with the second preset repetition ratio. If the actual repetition ratio of a second candidate data block exceeds the preset second preset repetition ratio, then the second candidate data block is determined as the target data block. The target data block will contain the change position and its corresponding data that have a high degree of repetition with the current change position. These data blocks are of great significance in subsequent data processing (such as data deduplication, merging and updating, etc.) and can be processed in a targeted manner to improve data processing efficiency and optimize data storage.

[0095] Based on the second candidate data block obtained through the previous screening based on the Bloom filter and the predicted duplication ratio, a more accurate comparison is performed to determine the data block (target data block) that is truly highly relevant to the current change. By accurately calculating the actual duplication ratio and comparing it with the preset threshold, it is possible to further screen out data blocks with a high degree of duplication that have an important impact on the current data change processing, avoiding the incorrect processing of some data blocks due to possible misjudgment of the Bloom filter. At the same time, while ensuring accuracy, it also minimizes the number of data blocks that need to be processed in detail, improves the efficiency of the entire data processing process and the quality of the data processing results, and is especially suitable for large-scale data change processing scenarios where both data accuracy and processing efficiency are required.

[0096] Therefore, although the Bloom filter is highly efficient in the preliminary screening process, there is a certain misjudgment rate, and some data blocks that do not actually have repeated change positions may be misjudged as possibly having duplicates (i.e., the predicted repetition ratio is greater than the actual repetition ratio). By accurately comparing the actual change positions of the second candidate data blocks, the actual number of repeated positions can be accurately determined, thereby obtaining the true repetition ratio and correcting the misjudgment of the Bloom filter. Only the second candidate data block whose actual repetition ratio exceeds the second preset repetition ratio is determined as the target data block, which means that the target data block has a high correlation and repetition rate with the current change position. In subsequent data processing, operations are performed on these highly correlated data blocks to ensure the accuracy of the processing results and avoid unnecessary operations on irrelevant or low-correlation data blocks, thereby improving the data quality of the entire data processing flow.

[0097] In some implementations, obtaining a Bloom filter corresponding to each of the first candidate data blocks includes:

[0098] (1.1) constructing a preset Bloom filter for each of the first candidate data blocks, wherein the preset Bloom filter includes a plurality of hash values, and a hash value tag of each hash value is a second preset value;

[0099] (1.2) For each of the first candidate data blocks, a hash value of each changed position is calculated using a different preset hash function, and the hash value mark of the corresponding hash value in the corresponding preset Bloom filter is changed to a first preset value to obtain a Bloom filter corresponding to each of the first candidate data blocks.

[0100] Among them, the Bloom filter is essentially a bit array, and its length (that is, the number of hash values ​​contained) is predetermined based on factors such as the amount of data, the expected false positive rate, and the actual application scenario. For example, in scenarios where large-scale data changes are processed and the false positive rate requirements are strict, the Bloom filter length may be set to 10,000 bits or longer. During initialization, all bits in this bit array (that is, the hash value mark of each hash value) are set to a second preset value, usually 0, indicating that no element has been marked yet. This step prepares for the subsequent recording of the hash value of the changed position, ensuring that the Bloom filter does not contain any marked change position information in the initial state.

[0101] Select multiple different preset hash functions, which should have good randomness and distribution to minimize the probability of hash collision. For example, common hash functions such as MD5, SHA-1, etc., or customized hash functions based on specific data characteristics. For each changed position in each first candidate data block, use these selected hash functions to calculate its hash value respectively. Assume that there are changed positions 3, 5, and 7 in a first candidate data block, and three hash functions hash1, hash2, and hash3 are selected. Calculate hash1(3) = 100, hash2(3) = 500, hash3(3) = 800; hash1(5) = 200, hash2(5) = 600, hash3(5) = 900; hash1(7) = 300, hash2(7) = 700, hash3(7) = 1000 (here are just example hash values, the actual calculation results depend on the specific implementation of the hash function).

[0102] Specifically, after calculating the hash values ​​of the historical change positions, the hash value marks corresponding to these hash value index positions in the preset Bloom filter are changed from the second preset value (0) to the first preset value (usually 1). Taking the above example, the bits indexed as 100, 500, 800 (corresponding to the hash values ​​of change position 3), 200, 600, 900 (corresponding to the hash values ​​of change position 5) and 300, 700, 1000 (corresponding to the hash values ​​of change position 7) in the Bloom filter are all set to 1. In this way, the construction of the Bloom filter for the first candidate data block is completed. Through this Bloom filter, it is possible to quickly determine whether a certain change position may exist in the first candidate data block (there is a certain misjudgment rate because hash collisions may cause the hash values ​​of different change positions to be the same). This process is repeated for each first candidate data block to obtain the corresponding Bloom filter, which provides a basis for subsequent repeated position prediction and screening.

[0103] In this way, the construction of the Bloom filter provides a basis for quickly determining whether the change position may exist in the first candidate data block. In subsequent processing, when it is necessary to check the relationship between the current change position in the first data block and the first candidate data block, it is only necessary to calculate the hash value of the current change position and check the mark of the corresponding hash value position in the Bloom filter, without the need to perform a detailed comparison of each change position. A large number of candidate data blocks that obviously do not contain the change position can be quickly excluded, significantly improving the data screening speed.

[0104] In step 205, the corresponding target change position and the corresponding target data are extracted from the target data block and written into the second data block.

[0105] Among them, see Figure 3 and Figure 4 , Figure 4 A second schematic diagram of a change data record table provided in an embodiment of the present application. Figure 3 When the change data record table of two incremental checkpoints has been stored in the storage, the change data of the current incremental checkpoint (for example Figure 4The first data block including the change positions 2, 5, and 6 in the data block to be compared is finally determined to be the target data block after being compared with the data block to be compared 11 and the data block to be compared 21. The data block to be compared 11 is finally determined to be the target data block, and the target change position is position 2, which is the same as the first data block. The data block to be compared 21 is finally determined to be the target data block, and the target change position is position 5, which is the same as the first data block. Then, position 2 and the corresponding data in the data block to be compared 11 are extracted, and position 5 and the corresponding data in the data block to be compared 21 are extracted, and all the data extracted from each target data block are written into the second data block. For the data block to be compared 11, its corresponding second data block is the second data block 12 including position 2 and the corresponding data. For the data block to be compared 21, its corresponding second data block is the second data block 22 including position 5 and the corresponding data.

[0106] In step 206, the second data block corresponding to the target data block is written into the second row of the corresponding column, and the first data block is written into the next column of the last column to record the current change position where the data change occurs and the change data of each current change position.

[0107] The second data block including the target change position and the target data is written into the second row of the corresponding column in the change data record table. For example, if the target data block is from the first column of the change data record table, the second data block is written into the second row of the first column.

[0108] For example, from Figure 3 to Figure 4 In the example, for the first column, the second data block 12 is written into the second row of the first column, and for the second example, the second data block 22 is written into the second row of the second column.

[0109] Specifically, the first data block is written to the next column of the last column (i.e., a new column), thus completing the record of the current data changes, including the change location and the change data. You can subsequently query the change data record table to track the data change history, perform data analysis, restore data to a specific version, and other operations. At the same time, this recording process also provides a basis for possible data deduplication, storage optimization, and other operations. For example, if a large number of repeated change locations and data are found, you can consider merging or optimizing the storage structure to reduce storage space and improve data processing efficiency.

[0110] In some embodiments, the method further comprises:

[0111] (1) When a data restoration operation is detected, obtaining a first data change point that is desired to be restored;

[0112] (2) determining a target column corresponding to the first data change point from the change data record table;

[0113] (3) obtaining the last target row in the target column that records the data block;

[0114] (4) obtaining each record data block from the first row and the first column to the target row and the target column in the change data record table, as well as the initial data table;

[0115] (5) The data in the initial data table is updated according to the change position and the corresponding change data in each of the record data blocks to obtain a data table corresponding to the first data change point.

[0116] Among them, the data restoration operation can be triggered by a variety of situations, such as a user manually requesting to restore to the data state at a specific point in time, or the system automatically initiating a restoration operation after detecting a data error or anomaly. When the triggering condition is met, the system detects that a data restoration operation is required. The first data change point refers to the specific change moment or version (such as a specific incremental checkpoint) to which the data is expected to be restored. This can be specified in a variety of ways, such as in the form of a timestamp (for example, "2023-08-1510:30:00" means that the data is expected to be restored to the state at that point in time), or in the sequence number of the change point (if the system has numbered the data changes sequentially, such as the 50th data change point). After obtaining the first data change point that is expected to be restored, subsequent operations will revolve around how to restore the data to the state at that point.

[0117] Specifically, since the change data record table is a two-dimensional structure table for recording the history of data changes, each column corresponds to a data change event or a set of changes within a specific time period. The order of the columns records the changes from the earliest to the latest from left to right. According to the first data change point obtained, the corresponding column is searched in the change data record table. If the change data record table uses the timestamp as the column identifier, then the column matching the timestamp of the first data change point is searched; if the change sequence number is used as the identifier, the column with the sequence number equal to the sequence number of the first data change point is searched. The column found is the target column, which contains the relevant change position and change data information when the data change point occurs, and is one of the key bases for subsequent data restoration. The target column may contain multiple data blocks, each of which records the data changes of different parts or types under the data change point. Starting from the first row of the target column, check whether there is a data block recorded row by row until the last row containing the data block is found, and it is determined as the last target row. This step determines the scope of all changed data blocks that need to be considered at the data change point, ensuring that no relevant change information is missed. It also avoids processing unnecessary blank rows of data and improves data restoration efficiency.

[0118] Specifically, according to the previously determined range, all record data blocks starting from the first row and the first column to the target row and the target column are extracted from the change data record table. These data blocks contain all the change history information from the baseline checkpoint to the first data change point (a user-specified incremental checkpoint) and are the basic data source for gradually restoring data. These data blocks can be read and stored row by row and column by column in a loop traversal manner for subsequent unified processing.

[0119] The initial data table is the data at each location in the data table saved at the baseline checkpoint. It is the starting state of the data change and the basis for restoring the data. It can be a copy of the original data saved in the system in advance, or an initial state snapshot that has been maintained during the data change recording process. After obtaining the initial data table, it will be gradually restored to the state of the first data change point based on this initial state and the information in the change data record table.

[0120] For each acquired record data block, the corresponding position in the initial data table is found according to the change position information recorded therein, and then the data at the position is updated to the change data in the record data block.

[0121] For example, Figure 4 In the example, for the incremental checkpoint (i.e., data change point) restored to the second column, there are two rows in the second column, and each record data block in the first row and first column (i.e., the data block containing position 1) to the second row and second column (i.e., the data block containing position 5) is obtained. The record data block includes the data block containing position 1, the data block containing position 2, the data block containing positions 3 and 4, and the data block containing position 5. Then, the data at position 1, position 2, position 3, position 4, and position 5 in the initial data table are respectively updated to the data corresponding to each position in the data block, thereby restoring to the incremental checkpoint of the second column.

[0122] In this way, by changing the structural design of the data record table, each column corresponds to a specific data change event or time period, and the target column can be quickly located according to the first data change point. This search method based on column identifiers (timestamps or change sequence numbers) greatly improves the speed of data location compared to searching for related data in unstructured data. When acquiring record data blocks, by looping through the range from the first row and first column to the target row and target column, this orderly traversal method can efficiently extract the required data blocks, avoiding aimless searches of the entire data table and improving the efficiency of data acquisition.

[0123] In some implementations, after writing the second data block corresponding to the target data block into the second row of the corresponding column and writing the first data block into the next column of the last column, the method includes:

[0124] (1) splicing each data block in the first row to obtain a first spliced ​​file corresponding to the first row;

[0125] (2) splicing each data block in the second row to obtain a second spliced ​​file corresponding to the second row;

[0126] (3) Establishing a mapping relationship between the first splicing file and the second splicing file and the current data change point.

[0127] Among them, for the record of each incremental checkpoint, after writing the first data block into the next column of the last column, each data block in the first row will be spliced ​​to obtain the first splicing file corresponding to the first row, and each data block in the second row will be spliced ​​to obtain the second splicing file corresponding to the second row. Finally, a mapping relationship between the current checkpoint and the first splicing file and the second splicing file is established.

[0128] For example, in storage such as Figure 4 After the first data block is stored in the third column of , the data block including change position 1, the data block including change positions 3 and 4, and the first data block including change positions 2, 5 and 6 in the first row of the change data record table are spliced ​​to obtain a first spliced ​​file including change positions 1, 3, 4, 2, 5 and 6, and the data block including change position 2 and the data block including change position 5 in the second row are spliced ​​to obtain a second spliced ​​file including change positions 2 and 5, and a mapping relationship between the current data change point (the third incremental checkpoint) and the two files is established.

[0129] In some embodiments, the method further comprises:

[0130] (1) when a data restoration operation is detected, obtaining a second data change point that is desired to be restored;

[0131] (2) acquiring a target first splicing file and a target second splicing file that have a preset mapping relationship with the second data change point;

[0132] (3) According to the changed position and corresponding changed data of each data block in the target first splicing file and the target second splicing file, the data in the initial data table is updated to obtain a data table corresponding to the second data change point.

[0133] Among them, the second data change point refers to the specific change time or version to which the data is expected to be restored. In the previous data processing process, a preset mapping relationship between the data change point and the splicing file (the first splicing file and the second splicing file, etc.) has been established. According to the determined second data change point, the corresponding target first splicing file and the target second splicing file are searched in the mapping relationship. For example, if the second data change point is the timestamp "2023-09-2014:30:00", the record corresponding to the timestamp is searched in the mapping table. Assuming that the found record shows that the corresponding first splicing file is "file1.txt" and the second splicing file is "file2.txt", then these two files are the target files.

[0134] The first target splicing file and the second target splicing file contain multiple data blocks, each of which has a corresponding change position and change data. For example, a data block in the first target splicing file may record a change position of (3) (indicating the 3rd row and 5th column in the data table), and the change data is "new_value1"; a data block in the second target splicing file may record a change position of (2), and the change data is "new_value2".

[0135] Update the initial data table: Get the initial data table, which is the starting state of the data change. Update the initial data table based on the data block information obtained from the target splicing file. For the data block in the above example, find position 3 in the initial data table and update its value to "new_value1", find position 2 and update its value to "new_value2". Perform this update operation on all data blocks in the target first splicing file and the target second splicing file, and finally get the data table corresponding to the second data change point.

[0136] In this way, a mapping relationship between the splicing file and the current data change point is established, providing a clear and convenient way for data version management. Each data change point (such as each incremental checkpoint) can be accurately associated with the corresponding first splicing file and the second splicing file, just like affixing clear labels to different versions of data. Whether the system itself conducts data audits and version backtracking, or developers and operation and maintenance personnel need to understand the data status at a specific moment, they can quickly locate the corresponding files through this mapping relationship, which helps to accurately grasp the changes in data at different stages and achieve efficient version control. During the data restoration operation, only the second data change point to be restored can be used to quickly obtain the target first splicing file and the target second splicing file through the preset mapping relationship, without the need to filter and organize relevant data from a large amount of original data blocks or the entire change data record table. This rapid positioning mechanism based on mapping relationships greatly reduces the time cost of data search and screening, allowing the data restoration process to be started and executed more quickly, effectively shortening the time required for data restoration.

[0137] In some implementations, before obtaining the current change position where the data change currently occurs in the data table and the change data of each current change position, the method further includes:

[0138] (1) when it is detected that a data change currently occurs in the data table, a preset data table storage time for each data in the data table is obtained, and a preset reload time for reloading each saved data;

[0139] (2) calculating the sum of the preset data table storage time and the preset reload time to obtain a reset time for resetting the data table;

[0140] (3) obtaining a preset data storage time for storing the changed data, and gradually updating the data in the initial data table to the update time of the changed data where the data change currently occurs;

[0141] (4) calculating the sum of the preset data storage time, the preset reload time, and the update time to obtain a total restoration time;

[0142] (5) When the total restoration time is less than the reset time, the step of obtaining the current change position where the data change currently occurs in the data table and the change data of each current change position is executed.

[0143] Among them, the preset data table storage time C 基准It refers to the average time that is pre-set beforehand, which is the time it takes to store each data in the data table to the storage medium (such as hard disk, solid state drive and other storage devices). This time setting may be based on the statistical data of the past storage operations on the data table, or estimated based on the performance indicators of the storage device, the data structure and size of the data table and other factors.

[0144] For example, after multiple tests, it is found that it takes an average of 0.01 seconds to store a piece of data in a data table to a specific hard disk. This 0.01 second is the preset data table storage time for a single piece of data. In actual applications, these time data can be recorded in the configuration file or system metadata so that they can be easily obtained when needed.

[0145] Default reload time R 基准 It refers to the average time it takes to reload previously saved data from the storage medium back into the operating environment such as the memory so that the data can be used again by the system. Similarly, it is determined based on past experience or an evaluation of factors such as the storage device's read performance and data volume. For example, after testing, it takes an average of 0.02 seconds to read a piece of data from the data table from the hard disk and load it into the memory. This 0.02 seconds is the preset reload time. Similar to the storage time, these reload time data will also be recorded and saved in an appropriate manner for easy access when needed.

[0146] Preset data table storage time C 基准 With preset reload time R 基准 The sum of the values ​​is the reset time T of the reset data table. 基准 ,Reset time refers to resetting the baseline checkpoint, which means recreating a complete record of the model state, so it takes time to save it first, and then it takes time to load it back when it needs to be restored.

[0147] Preset data storage time C 增量 It is specifically for storing the changed data in the data table, and records the average time required to store the changed data in the storage medium. Its determination method is similar to the preset data table storage time, which is also preset after comprehensively considering factors such as the characteristics of the storage device, the scale and format of the changed data, etc. For example, after analysis, it is found that it takes an average of 0.005 seconds to store the changed data each time. This time value is the preset data storage time, which will also be properly recorded for subsequent retrieval and use.

[0148] Update time R iRepresents the time required to restore the parameters updated from the baseline checkpoint to the i-th checkpoint. When continuing to execute the incremental checkpoint strategy, in addition to restoring the baseline checkpoint, it is necessary to apply all incremental checkpoints from the baseline checkpoint to the i-th checkpoint in sequence to restore the model to the state of the i-th checkpoint. Each incremental checkpoint requires a certain amount of time to read and apply its updated data, which is the sum of these times.

[0149] For example, from the baseline checkpoint to the third incremental checkpoint, it takes 10 seconds to restore the first incremental checkpoint, 15 seconds to restore the second incremental checkpoint, and 20 seconds to restore the third incremental checkpoint, so R3=10+15+20=45 seconds.

[0150] Total recovery time T 增量 =C 增量 +R 基准 +R i , which means that when continuing to execute the incremental checkpoint strategy, the total time overhead includes saving the current incremental checkpoint (C 增量 ), restore the baseline checkpoint (R 基准 ) and restore all incremental checkpoint update parameters (R i ) time. It comprehensively considers the complete time cost of gradually recovering from the baseline to a specific checkpoint under the incremental checkpoint strategy. By comparing T 基准 and T 增量 , you can weigh which strategy is more favorable in terms of time in different situations (resetting the baseline or continuing to increment), and decide whether to reset the baseline checkpoint.

[0151] When the total restoration time is less than the reset time, the step of obtaining the current change position of the data in the data table where the data change currently occurs and the change data of each current change position is performed. This indicates that for the current change data, the time consumed for saving it to the change data record table and subsequent restoration is less than the time consumed for re-recording all the current data in the data table and subsequent restoration, so the step of obtaining the current change position of the data in the data table where the data change currently occurs and the change data of each current change position is performed.

[0152] In some embodiments, the method further comprises:

[0153] When the total restoration time is less than the reset time, the data of each current position in the data table is stored to obtain an updated initial data table.

[0154] Among them, when the total restoration time is less than the reset time, it means that for the current changed data, the time consumed to save it to the changed data record table and the subsequent recovery is greater than the time consumed to re-record all the current data in the data table and subsequently recover it. Therefore, the data at each current position in the data table is stored to obtain the updated initial data table.

[0155] In this way, by comparing the total restore time and reset time, the system can flexibly decide which data processing strategy to adopt based on the time cost, whether to reset the baseline checkpoint or continue to execute the incremental checkpoint strategy. For example, in a business scenario where data changes frequently, if the baseline checkpoint is reset blindly every time, a lot of unnecessary time may be consumed for complete data storage and reloading. By calculating and comparing these two times, when the total restore time is less than the reset time, the incremental checkpoint strategy is chosen to continue to execute, only processing the storage of changed data, baseline checkpoint recovery, and corresponding incremental checkpoint update parameter recovery, which can effectively reduce the overall time overhead of data processing and improve the efficiency of data operations.

[0156] As can be seen from the above, the embodiment of the present application obtains the current change position of the data change currently occurring in the data table, and the change data of each current change position; writes each current change position and the corresponding change data into the first data block; obtains the data block to be compared in the first row of each column of the change data record table, and the last column where the data block is currently recorded, and all the data blocks recorded in each column of the change data record table correspond to the historical change position and the corresponding historical change data when the historical data in the data table is changed; compares each of the data blocks to be compared with the first data block, filters out at least one target data block from each of the data blocks to be compared, and filters out the target data blocks that are compared with the target data blocks. The target change position is the same as the current change position; the corresponding target change position and the corresponding target data are extracted from the target data block, and written into the second data block; the second data block corresponding to the target data block is written into the second row of the corresponding column, and the first data block is written into the next column of the last column, so as to record the current change position where the data change occurs, and the change data of each current change position. Compared with the related art, when restoring the checkpoint, it is necessary to read the parameters of each update of the position that has undergone multiple updates, and a large amount of redundant data is read. The embodiment of the present application can only read the updated data for the recovery checkpoint, thereby avoiding resource consumption and time cost consumption, and improving recovery efficiency.

[0157] The specific implementation of the above steps can be found in the previous embodiments, which will not be described in detail here.

[0158] In order to better implement the data processing method provided in the embodiment of the present application, the embodiment of the present application also provides a device based on the above data processing method. The meanings of the terms are the same as those in the above data processing method, and the specific implementation details can refer to the description in the method embodiment.

[0159] See also Figure 5 , Figure 5 A structural schematic diagram of a data processing method device provided in an embodiment of the present application, wherein the data processing method device is applied to a computer device, wherein the data processing method device may include a first acquisition unit 601, a first writing unit 602, a second acquisition unit 603, a comparison unit 604, a second writing unit 605 and a third writing unit 606, etc.

[0160] The first acquisition unit 601 is used to acquire the current change position where the data change currently occurs in the data table, and the change data of each current change position;

[0161] A first writing unit 602, configured to write each of the current change positions and the corresponding change data into a first data block;

[0162] The second acquisition unit 603 is used to acquire the data block to be compared in the first row of each column of the change data record table and the last column currently recording the data block, wherein all the data blocks recorded in each column of the change data record table correspond to the historical change position when the historical data in the data table is changed and the corresponding historical change data;

[0163] The comparing unit 604 is used to compare each of the to-be-compared data blocks with the first data block, select at least one target data block from each of the to-be-compared data blocks, and select a target change position that is the same as the current change position from the target data blocks;

[0164] A second writing unit 605 is used to extract the corresponding target change position and the corresponding target data from the target data block, and write them into the second data block;

[0165] The third writing unit 606 is used to write the second data block corresponding to the target data block into the second row of the corresponding column, and write the first data block into the next column of the last column, so as to record the current change position where the data change occurs and the change data of each current change position.

[0166] In some embodiments, the comparison unit 604 includes:

[0167] An acquisition subunit, used for acquiring a historical position range of a historical change position included in the data block to be compared, and a current position range of a current change position included in the first data block;

[0168] A comparison subunit, configured to compare each of the historical position ranges with the current position range, and determine a to-be-compared data block corresponding to a historical position range that has an intersection with the current position range as a first candidate data block;

[0169] The screening subunit is used to screen out at least one target data block from the first candidate data blocks.

[0170] In some embodiments, the screening subunit is used to:

[0171] Obtaining a Bloom filter corresponding to each of the first candidate data blocks, wherein the Bloom filter includes a hash value tag of each hash value;

[0172] Calculating multiple hash values ​​corresponding to each of the current change positions in the first data block by using different preset hash functions;

[0173] Determine the number of positions where the hash value marks of the multiple hash values ​​corresponding to each of the current change positions in each of the Bloom filters are all first preset values, and obtain the number of predicted repetition positions corresponding to each of the first candidate data blocks;

[0174] Calculating the ratio of each of the predicted number of repetition positions to the number of positions of the corresponding first candidate data blocks to obtain a predicted repetition ratio of each of the first candidate data blocks;

[0175] Filter out a target predicted repetition ratio that exceeds a first preset repetition ratio from each of the predicted repetition ratios, and determine a first candidate data block corresponding to the target predicted repetition ratio as a second candidate data block;

[0176] At least one target data block is selected from the second candidate data blocks.

[0177] In some embodiments, the screening subunit is further specifically used for:

[0178] Comparing the change position in the second candidate data block with the current change position, determining the actual number of repeated positions of the candidate change position in the second candidate data block that is the same as the current change position;

[0179] Calculate the ratio of the actual number of repeated positions of the second candidate data block to the number of corresponding positions to obtain an actual repetition ratio of the second candidate data block;

[0180] The second candidate data block whose actual repetition ratio exceeds the second preset repetition ratio is determined as the target data block.

[0181] In some embodiments, the screening subunit is further specifically used for:

[0182] Constructing a preset Bloom filter for each of the first candidate data blocks, wherein the preset Bloom filter includes a plurality of hash values, and a hash value tag of each hash value is a second preset value;

[0183] For each of the first candidate data blocks, a hash value of each changed position is calculated using different preset hash functions, and the hash value mark of the corresponding hash value in the corresponding preset Bloom filter is changed to a first preset value to obtain a Bloom filter corresponding to each of the first candidate data blocks.

[0184] In some embodiments, the data processing device further includes:

[0185] A third acquisition unit, configured to acquire a first data change point desired to be restored when a data restoration operation is detected;

[0186] a determining unit, configured to determine a target column corresponding to the first data change point from a change data record table;

[0187] A fourth acquisition unit, used for acquiring the last target row in the target column where the data block is recorded;

[0188] A fifth acquisition unit, used to acquire each record data block from the first row and the first column to the target row and the target column in the change data record table, and an initial data table;

[0189] The first updating unit is used to update the data in the initial data table according to the change position and the corresponding change data in each of the record data blocks to obtain a data table corresponding to the first data change point.

[0190] In some embodiments, the data processing device further includes:

[0191] A first splicing unit, used for splicing each data block in the first row to obtain a first splicing file corresponding to the first row;

[0192] A second splicing unit, used for splicing each data block in the second row to obtain a second splicing file corresponding to the second row;

[0193] The mapping relationship establishing unit is used to establish a mapping relationship between the first splicing file and the second splicing file and the current data change point.

[0194] In some embodiments, the data processing device further includes:

[0195] A sixth acquisition unit, configured to acquire a second data change point expected to be restored when a data restoration operation is detected;

[0196] A seventh acquisition unit, configured to acquire a target first splicing file and a target second splicing file having a preset mapping relationship with the second data change point;

[0197] The second updating unit is used to update the data in the initial data table according to the changed position and corresponding changed data of each data block in the target first splicing file and the target second splicing file, so as to obtain the data table corresponding to the second data change point.

[0198] In some embodiments, the data processing device further includes:

[0199] an eighth acquisition unit, for acquiring a preset data table storage time for each data in the data table and a preset reload time for reloading each saved data when a data change is detected in the data table;

[0200] A first calculation unit, configured to calculate a sum of the preset data table storage time and the preset reload time, to obtain a reset time for resetting the data table;

[0201] A ninth acquisition unit, used to acquire a preset data storage time for storing the changed data, and to gradually update the data in the initial data table to the update time of the changed data where the data change currently occurs;

[0202] A second calculation unit, configured to calculate a sum of the preset data storage time, the preset reload time, and the update time to obtain a total restoration time;

[0203] An execution unit is used to execute the step of obtaining the current change position of the data change currently occurring in the data table and the change data of each current change position when the total restoration time is less than the reset time.

[0204] In some embodiments, the data processing device further includes:

[0205] The storage unit is used to store the data of each current position in the data table when the total restoration time is less than the reset time to obtain an updated initial data table.

[0206] The specific implementation of each of the above units can be found in the previous embodiments, which will not be described in detail here.

[0207] As can be seen from the above, the embodiment of the present application obtains the current change position of the data change currently occurring in the data table and the change data of each current change position through the first acquisition unit 601; the first writing unit 602 writes each current change position and the corresponding change data into the first data block; the second acquisition unit 603 obtains the data block to be compared in the first row of each column of the change data record table, and the last column where the data block is currently recorded. All the data blocks recorded in each column of the change data record table correspond to the historical change position and the corresponding historical change data when the historical data in the data table is changed; the comparison unit 604 compares each The data blocks to be compared and the first data blocks, at least one target data block is screened out from each of the data blocks to be compared, and a target change position that is the same as the current change position is screened out from the target data blocks; the second writing unit 605 extracts the corresponding target change position and the corresponding target data from the target data blocks, and writes them into the second data block; the third writing unit 606 writes the second data block corresponding to the target data block into the second row of the corresponding column, and writes the first data block into the next column of the last column, so as to record the current change position where the data change currently occurs, and the change data of each current change position. Compared with the related art, when restoring the checkpoint, it is necessary to read the parameters of each update of the position that has undergone multiple updates, and a large amount of redundant data is read. In this embodiment of the present application, only the updated data for restoring the checkpoint can be read, so as to avoid resource consumption and time cost consumption, and improve the recovery efficiency.

[0208] The specific implementation of each of the above units can be found in the previous embodiments, which will not be described in detail here.

[0209] Reference Figure 6 , Figure 6 The block diagram of the structure of part of the computer device 110 for implementing the embodiment of the present disclosure. The computer device 110 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPUs) 622 (for example, one or more processors) and memory 632, and one or more storage media 630 (for example, one or more mass storage devices) storing application programs 642 or data 644. Among them, the memory 632 and the storage medium 630 can be temporary storage or permanent storage. The program stored in the storage medium 630 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the server 600. Furthermore, the central processing unit 622 can be configured to communicate with the storage medium 630 and execute a series of instruction operations in the storage medium 630 on the server 600.

[0210] The computer device 110 may also include one or more power supplies 626, one or more wired or wireless network interfaces 650, one or more input and output interfaces 658, and / or one or more operating systems 641, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0211] The central processor 622 in the computer device 110 may be used to execute the data processing method of the embodiment of the present disclosure, for example:

[0212] Obtaining a current change position where data change currently occurs in the data table, and change data of each current change position;

[0213] Writing each of the current change positions and the corresponding change data into the first data block;

[0214] Obtain the data block to be compared in the first row of each column of the change data record table, and the last column currently recording the data block, wherein all the data blocks recorded in each column of the change data record table correspond to the historical change position and the corresponding historical change data when the historical data in the data table is changed;

[0215] Comparing each of the data blocks to be compared with the first data block, selecting at least one target data block from each of the data blocks to be compared, and selecting a target change position that is the same as the current change position from the target data blocks;

[0216] Extracting the corresponding target change position and the corresponding target data from the target data block, and writing them into the second data block;

[0217] The second data block corresponding to the target data block is written into the second row of the corresponding column, and the first data block is written into the next column of the last column to record the current change position where the data change occurs and the change data of each current change position.

[0218] The embodiments of the present disclosure further provide a computer-readable storage medium, which is used to store program codes, and the program codes are used to execute the data processing methods of the aforementioned embodiments.

[0219] The present disclosure also provides a computer program product, which includes a computer program. The processor of a computer device reads and executes the computer program, so that the computer device executes the above-mentioned data processing method. For example:

[0220] Obtaining a current change position where data change currently occurs in the data table, and change data of each current change position;

[0221] Writing each of the current change positions and the corresponding change data into the first data block;

[0222] Obtain the data block to be compared in the first row of each column of the change data record table, and the last column currently recording the data block, wherein all the data blocks recorded in each column of the change data record table correspond to the historical change position and the corresponding historical change data when the historical data in the data table is changed;

[0223] Comparing each of the data blocks to be compared with the first data block, selecting at least one target data block from each of the data blocks to be compared, and selecting a target change position that is the same as the current change position from the target data blocks;

[0224] Extracting the corresponding target change position and the corresponding target data from the target data block, and writing them into the second data block;

[0225] The second data block corresponding to the target data block is written into the second row of the corresponding column, and the first data block is written into the next column of the last column to record the current change position where the data change occurs and the change data of each current change position.

[0226] In addition, the terms "comprises" and "includes" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or apparatus that includes a series of steps or elements is not necessarily limited to those steps or elements explicitly listed, but may include other steps or elements not explicitly listed or inherent to such process, method, product or apparatus.

[0227] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0228] It should be understood that in the description of the embodiments of the present application, the meaning of multiple (or multiple items) is more than two, greater than, less than, exceed, etc. are understood to not include the number, and above, below, within, etc. are understood to include the number.

[0229] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0230] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0231] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0232] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store program codes.

[0233] It should also be understood that the various implementations provided in the embodiments of the present application can be combined arbitrarily to achieve different technical effects.

[0234] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0235] The above is a specific description of the implementation method of the present application, but the present application is not limited to the above-mentioned implementation method. Technical personnel familiar with the field can also make various equivalent modifications or substitutions without violating the spirit of the present application. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present application.

Claims

1. A data processing method, characterized in that: include: Obtaining a current change position where data change currently occurs in the data table, and change data of each current change position; Writing each of the current change positions and the corresponding change data into the first data block; Obtain the data block to be compared in the first row of each column of the change data record table, and the last column currently recording the data block, wherein all the data blocks recorded in each column of the change data record table correspond to the historical change position and the corresponding historical change data when the historical data in the data table is changed; Comparing each of the data blocks to be compared with the first data block, selecting at least one target data block from each of the data blocks to be compared, and selecting a target change position that is the same as the current change position from the target data blocks; Extracting the corresponding target change position and the corresponding target data from the target data block, and writing them into the second data block; The second data block corresponding to the target data block is written into the second row of the corresponding column, and the first data block is written into the next column of the last column to record the current change position where the data change occurs and the change data of each current change position.

2. The data processing method according to claim 1, characterized in that: The comparing each of the to-be-compared data blocks with the first data block, and selecting at least one target data block from each of the to-be-compared data blocks, comprises: Acquire a historical position range of a historical change position included in the data block to be compared, and a current position range of a current change position included in the first data block; Compare each of the historical position ranges with the current position range, and determine the to-be-compared data block corresponding to the historical position range that has an intersection with the current position range as the first candidate data block; At least one target data block is selected from the first candidate data blocks.

3. The data processing method according to claim 2, characterized in that: The step of selecting at least one target data block from the first candidate data blocks includes: Obtaining a Bloom filter corresponding to each of the first candidate data blocks, wherein the Bloom filter includes a hash value tag of each hash value; Calculating multiple hash values ​​corresponding to each of the current change positions in the first data block by using different preset hash functions; Determine the number of positions where the hash value marks of the multiple hash values ​​corresponding to each of the current change positions in each of the Bloom filters are all first preset values, and obtain the number of predicted repetition positions corresponding to each of the first candidate data blocks; Calculating the ratio of each of the predicted number of repetition positions to the number of positions of the corresponding first candidate data blocks to obtain a predicted repetition ratio of each of the first candidate data blocks; Filter out a target predicted repetition ratio that exceeds a first preset repetition ratio from each of the predicted repetition ratios, and determine a first candidate data block corresponding to the target predicted repetition ratio as a second candidate data block; At least one target data block is selected from the second candidate data blocks.

4. The data processing method according to claim 3, characterized in that: The step of selecting at least one target data block from the second candidate data blocks comprises: Comparing the change position in the second candidate data block with the current change position, determining the actual number of repeated positions of the candidate change position in the second candidate data block that is the same as the current change position; Calculate the ratio of the actual number of repeated positions of the second candidate data block to the number of corresponding positions to obtain an actual repetition ratio of the second candidate data block; The second candidate data block whose actual repetition ratio exceeds the second preset repetition ratio is determined as the target data block.

5. The data processing method according to claim 3, characterized in that: The obtaining of the Bloom filter corresponding to each of the first candidate data blocks includes: Constructing a preset Bloom filter for each of the first candidate data blocks, wherein the preset Bloom filter includes a plurality of hash values, and a hash value tag of each hash value is a second preset value; For each of the first candidate data blocks, a hash value of each changed position is calculated using different preset hash functions, and the hash value mark of the corresponding hash value in the corresponding preset Bloom filter is changed to a first preset value to obtain a Bloom filter corresponding to each of the first candidate data blocks.

6. The data processing method according to claim 1, characterized in that: The method further comprises: When a data restoration operation is detected, obtaining a first data change point that is expected to be restored; Determine a target column corresponding to the first data change point from the change data record table; Obtain the last target row in the target column that records the data block; Obtain each record data block from the first row and the first column to the target row and the target column in the change data record table, and the initial data table; The data in the initial data table is updated according to the change position and the corresponding change data in each of the record data blocks to obtain a data table corresponding to the first data change point.

7. The data processing method according to claim 1, characterized in that: After writing the second data block corresponding to the target data block into the second row of the corresponding column and writing the first data block into the next column of the last column, the method comprises: Splice each data block in the first row to obtain a first spliced ​​file corresponding to the first row; Splice each data block in the second row to obtain a second spliced ​​file corresponding to the second row; A mapping relationship between the first splicing file, the second splicing file and the current data change point is established.

8. The data processing method according to claim 7, characterized in that: The method further comprises: When a data restoration operation is detected, obtaining a second data change point that is desired to be restored; Acquire a target first splicing file and a target second splicing file that have a preset mapping relationship with the second data change point; According to the changed position and corresponding changed data of each data block in the target first splicing file and the target second splicing file, the data in the initial data table is updated to obtain the data table corresponding to the second data change point.

9. The data processing method according to any one of claims 1 to 8, characterized in that: Before obtaining the current change position where the data change currently occurs in the data table and the change data of each current change position, the method further includes: When a data change is detected in the data table, a preset data table storage time for each data in the data table is obtained, and a preset reload time for reloading each saved data is obtained; Calculating the sum of the preset data table storage time and the preset reload time to obtain a reset time for resetting the data table; Obtaining the preset data storage time for storing the changed data, and gradually updating the data in the initial data table to the update time of the changed data where the data change currently occurs; Calculate the sum of the preset data storage time, the preset reload time and the update time to obtain a total restoration time; When the total restoration time is less than the reset time, the step of obtaining the current change position where the data change currently occurs in the data table and the change data of each current change position is performed.

10. The data processing method according to claim 9, characterized in that: The method further comprises: When the total restoration time is less than the reset time, the data of each current position in the data table is stored to obtain an updated initial data table.

11. A data processing device, characterized in that: include: A first acquisition unit, configured to acquire a current change position where data change currently occurs in a data table, and change data of each current change position; A first writing unit, configured to write each of the current change positions and corresponding change data into a first data block; A second acquisition unit is used to acquire the data block to be compared in the first row of each column of the change data record table, and the last column currently recording the data block, wherein all the data blocks recorded in each column of the change data record table correspond to the historical change position when the historical data in the data table is changed and the corresponding historical change data; a comparing unit, configured to compare each of the to-be-compared data blocks with the first data block, select at least one target data block from each of the to-be-compared data blocks, and select a target change position that is the same as the current change position from the target data blocks; A second writing unit, used for extracting a corresponding target change position and corresponding target data from the target data block, and writing them into a second data block; The third writing unit is used to write the second data block corresponding to the target data block into the second row of the corresponding column, and write the first data block into the next column of the last column, so as to record the current change position where the data change occurs and the change data of each current change position.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor to execute the data processing method according to any one of claims 1 to 10.

13. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the data processing method according to any one of claims 1 to 10 is implemented.

Images