Data processing method, device, storage medium and computer equipment

Abstract

The embodiments of the present application provide a data processing method, apparatus, storage medium, and computer equipment. The method first obtains the current change position and change data in a data table and writes them into a first data block. Then, the method obtains the data block to be compared and the last column of the data block in a specific column of the change data record table. Each data block in the table corresponds to the historical change information of the data table. The method compares the data block to be compared with the first data block, filters out the target data block and the target change position with the same change position. The method extracts the target data and writes it into a second data block. The method writes the second data block corresponding to the target data block into the second row of the corresponding column and the first data block into the column next to the last column. This method can record the current data changes. The key is to only read the updated data of the recovery checkpoint. This can avoid excessive consumption of resources and time costs, effectively improve data recovery efficiency, and has important significance in the field of data processing. It can be applied to many data management scenarios to optimize the data recovery process.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data processing method, apparatus, storage medium, and computer equipment. Background Art

[0002] Recommendation models are widely used in existing recommendation systems and typically consist of multiple embedding tables and multi-layer perceptrons (MLPs). The embedding tables occupy the vast majority of the model, while the MLPs only account for a very small portion. During recommendation model training, sparse and dense features in the dataset are input into the embedding tables and MLP layers for processing, respectively. Each embedding table consists of a large number of embedding vectors. The embedding table performs table lookups based on the sparse features to obtain the corresponding embedding vectors, thereby mapping the sparse features into dense vectors for subsequent computation. However, during each iteration, the embedding vectors obtained from the table lookup only account for a small fraction of all embedding vectors. In other words, only a small portion of the embedding vectors participate in and are updated after that iteration. This partial update feature provides a unique opportunity for recommendation model checkpointing mechanisms, saving only the updated data rather than saving all the data every time.

[0003] In related technologies, incremental checkpoint mechanisms are used to save only updated data. However, while a variety of intelligence data can be obtained from different channels, a large amount of duplicate data exists between multiple checkpoints. Duplicate data here refers to data at the same location in the model that has been updated multiple times and written to multiple incremental checkpoints. However, when restoring a checkpoint, only the most recently updated version of the data is required. In other words, incremental checkpoints read a large amount of redundant data during recovery, resulting in greater resource consumption and time costs (Input / Output Overhead, I / O), reducing recovery efficiency. Therefore, related technologies urgently need to propose a data processing method to solve the above technical problems. Summary of the Invention

[0004] The main purpose of this application is to provide a data processing method, device, storage medium and computer equipment, which can only read the data for the recovery checkpoint in the data at the location that has undergone multiple updates when restoring the checkpoint, without reading the data for each update, thereby avoiding resource consumption and time cost consumption and improving recovery efficiency.

[0005] In a first aspect, an embodiment of the present application provides a data processing method, comprising:

[0006] Obtaining the current change position where data change has occurred in the data table, and the change data of each current change position;

[0007] Writing each of the current change positions and the corresponding change data into the first data block;

[0008] Obtain the data block to be compared in the first row of each column of the change data record table, and the last column currently recording the data block, wherein all data blocks recorded in each column of the change data record table correspond to the historical change position and corresponding historical change data when the historical data in the data table is changed;

[0009] Comparing each of the to-be-compared data blocks with the first data block, selecting at least one target data block from each of the to-be-compared data blocks, and selecting a target change position that is the same as the current change position from the target data blocks;

[0010] Extracting the corresponding target change position and the corresponding target data from the target data block, and writing them into the second data block;

[0011] The second data block corresponding to the target data block is written into the second row of the corresponding column, and the first data block is written into the next column of the last column to record the current change position where the data change occurs and the change data of each current change position.

[0012] In a second aspect, an embodiment of the present application provides a data processing device, including:

[0013] A first acquiring unit is configured to acquire a current change position where data change occurs in a data table, and change data at each current change position;

[0014] a first writing unit, configured to write each of the current change positions and corresponding change data into a first data block;

[0015] a second acquiring unit, configured to acquire the data block to be compared in the first row of each column of the change data record table and the last column currently recording the data block, wherein all the data blocks recorded in each column of the change data record table correspond to the historical change positions and corresponding historical change data when the historical data in the data table is changed;

[0016] a comparing unit, configured to compare each of the data blocks to be compared with the first data block, select at least one target data block from each of the data blocks to be compared, and select a target change position that is the same as the current change position from the target data blocks;

[0017] A second writing unit, configured to extract a corresponding target change position and corresponding target data from the target data block, and write the extracted data into a second data block;

[0018] The third writing unit is used to write the second data block corresponding to the target data block into the second row of the corresponding column, and write the first data block into the next column of the last column to record the current change position where the data change occurs and the change data of each current change position.

[0019] In a third aspect, an embodiment of the present application provides a storage medium, wherein the computer-readable storage medium stores a plurality of instructions, which are suitable for loading by a processor to execute any of the above data processing methods.

[0020] In a fourth aspect, an embodiment of the present application provides a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any of the above data processing methods when executing the computer program.

[0021] In an embodiment of the present application, the current change position of the data change currently occurring in the data table and the change data of each current change position are obtained; each current change position and the corresponding change data are written into the first data block; the data block to be compared in the first row of each column of the change data record table and the last column where the data block is currently recorded are obtained, and all the data blocks recorded in each column of the change data record table correspond to the historical change position and the corresponding historical change data when the historical data in the data table is changed; each of the data blocks to be compared is compared with the first data block, and at least one target data block is filtered out from each of the data blocks to be compared, and the target data blocks are filtered out from the target data blocks. The method further comprises the following steps: extracting the corresponding target change position and the corresponding target data from the target data block, and writing the corresponding target data into the second data block; writing the second data block corresponding to the target data block into the second row of the corresponding column, and writing the first data block into the next column of the last column to record the current change position where the data change occurs and the change data of each current change position. Compared with the related art, when restoring a checkpoint, it is necessary to read the parameters of each update of the position that has undergone multiple updates, and a large amount of redundant data is read. In this embodiment of the present application, only the updated data for the recovery checkpoint can be read, thereby avoiding resource consumption and time cost consumption and improving recovery efficiency.

[0022] Other features and advantages of the present disclosure will be described in the following description, and in part will become apparent from the description, or will be understood by practicing the present disclosure. The purposes and other advantages of the present disclosure can be realized and obtained by the structures particularly pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0024] Figure 1 A schematic diagram of a data processing system according to an embodiment of the present invention.

[0025] Figure 2 A flowchart of a data processing method provided in an embodiment of the present application.

[0026] Figure 3 This is a first schematic diagram of a change data record table provided in an embodiment of the present application.

[0027] Figure 4 This is a second schematic diagram of the change data record table provided in an embodiment of the present application.

[0028] Figure 5 A schematic diagram of the structure of the data processing method and device provided in an embodiment of the present application.

[0029] Figure 6 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0030] In order to enable those skilled in the art to better understand the solutions of this application, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without making any creative efforts shall fall within the scope of protection of this application.

[0031] It should be noted that some processes described in the specification, claims, and figures above include multiple steps that appear in a specific order. However, it should be understood that these steps may be executed in a different order than the order in which they appear herein or in parallel. The step numbers are used solely to distinguish between the different steps and do not themselves represent any order of execution. Furthermore, terms such as "first," "second," or "target" are used herein to distinguish similar objects and are not necessarily used to describe a specific order or precedence.

[0032] Before further explaining the embodiments of the present disclosure in detail, the nouns and terms involved in the embodiments of the present disclosure are explained. The nouns and terms involved in the embodiments of the present disclosure are subject to the following interpretations:

[0033] Incremental checkpoint mechanism, 1. Saving process: First, save a complete copy of the model parameters as a baseline checkpoint. For example, when training a large recommendation model, initially save the parameters of the entire model as a baseline. Assume that the size of this baseline checkpoint is 100GB (including all embedding tables and multi-layer perceptron parameters). In the subsequent training process, each checkpoint only saves the parameters updated since the previous checkpoint. For example, after a period of training, when saving the next checkpoint, assuming that only 0.1% (100MB) of the embedding table parameters are updated, then this incremental checkpoint only saves the updated parameters of the 100MB, rather than the parameters of the entire model.

[0034] 2. Recovery process: When a checkpoint needs to be restored, the baseline checkpoint and all incremental checkpoints saved since the baseline checkpoint need to be loaded.

[0035] For example, if five incremental checkpoints are saved during training, when restoring to the fifth checkpoint, you need to first load the 100GB baseline checkpoint, and then load the five incremental checkpoints in sequence (assuming that the size of each incremental checkpoint is 100MB, 150MB, 80MB, 120MB, and 90MB respectively). Therefore, the total amount of data stored by the incremental checkpoint mechanism that needs to be loaded when the checkpoint is subsequently restored is 100GB+100MB+150MB+80MB+120MB+90MB=100GB+540MB.

[0036] Because there are a large number of duplicate parameters across multiple checkpoints (parameters at the same location in the model undergo multiple updates and are written to multiple incremental checkpoints, but only the most recently updated version is required for recovery), recovery requires reading a large amount of redundant data, resulting in increased I / O overhead and subsequent deserialization overhead, reducing recovery efficiency. For example, among the five incremental checkpoints described above, many parameters may have update records in multiple checkpoints, but recovery only requires the most recently updated value, resulting in reading older values ​​from multiple times.

[0037] Differential Checkpoint Mechanism,1. Saving Process: Similarly, first save a complete set of model parameters as a baseline checkpoint, such as the 100GB baseline checkpoint in the above example. Each subsequent checkpoint saves the updated parameters since the baseline checkpoint.

[0038] For example, when saving a differential checkpoint for the first time, if 200MB of parameters have been updated since the baseline checkpoint, then this differential checkpoint will save these 200MB of parameters. When saving a differential checkpoint next time, if 180MB of new parameters have been updated (relative to the baseline checkpoint), then these 180MB of parameters will be saved instead of the cumulative 380MB (relative to the previous differential checkpoint).

[0039] 2. Recovery Process: When restoring a checkpoint, only the base checkpoint and one differential checkpoint need to be loaded. For example, to restore to a specific checkpoint, only the 100GB base checkpoint needs to be loaded first, followed by the corresponding differential checkpoint (assuming it is 200MB). The total amount of data to be loaded is 100GB + 200MB.

[0040] For example, after multiple checkpoint saves, a differential checkpoint may reach 500MB or even 1GB. Compared with the relatively small and constant amount of incremental checkpoints saved each time (such as about 100MB in the above example), the storage pressure of differential checkpoints will gradually increase.

[0041] While differential checkpoints offer advantages in recovery over incremental checkpoints, they also have the following disadvantages. Because differential checkpoints store updated parameters since the baseline checkpoint, the number of parameters saved each time increases rapidly as the number of checkpoints increases. This increase in parameters leads to two problems. First, saving a checkpoint blocks the training process, and the larger the number of parameters, the longer the blockage. This prolonged blockage prevents the training system from saving checkpoints frequently, potentially leading to further loss of training progress in the event of a failure. Second, the large number of parameters increases storage space overhead, leading to expensive storage maintenance costs.

[0042] Incremental checkpoints avoid both of these issues. Since they only save parameters updated since the last checkpoint, the number of parameters saved with each incremental checkpoint is small and relatively constant, and does not increase with the number of checkpoints. This offers significant advantages in both checkpoint storage time and storage space.

[0043] However, while incremental checkpointing has been used in other fields, such as high-performance computing and stream processing, no work has yet applied it to recommendation model training. The main reason is that restoring from incremental checkpoints is extremely inefficient. The root cause is the large number of duplicate parameters between multiple checkpoints. Duplicate parameters here refer to parameters at the same location in the model that have been updated multiple times and written to multiple incremental checkpoints. However, when restoring a checkpoint, only the most recently updated version of the parameter is required. In other words, incremental checkpoints read a large amount of redundant data during recovery, resulting in greater I / O overhead and subsequent deserialization overhead, reducing recovery efficiency.

[0044] In order to solve the above problems, the embodiment of the present application obtains the current change position of the data change currently occurring in the data table, and the change data of each current change position; writes each current change position and the corresponding change data into the first data block; obtains the data block to be compared in the first row of each column of the change data record table, and the last column where the data block is currently recorded, and all the data blocks recorded in each column of the change data record table correspond to the historical change position and the corresponding historical change data when the historical data in the data table is changed; compares each of the data blocks to be compared with the first data block, filters out at least one target data block from each of the data blocks to be compared, and filters out the target data blocks that are compared with the target data blocks. The target change position is the same as the current change position; the corresponding target change position and the corresponding target data are extracted from the target data block and written into the second data block; the second data block corresponding to the target data block is written into the second row of the corresponding column, and the first data block is written into the next column of the last column to record the current change position where the data change occurs and the change data of each current change position. Compared with the related art, when restoring the checkpoint, it is necessary to read the parameters of each update of the position that has undergone multiple updates, and a large amount of redundant data is read. The embodiment of the present application can only read the updated data for the recovery checkpoint, avoid resource consumption and time cost consumption, and improve recovery efficiency. Please refer to the following specific embodiments for details.

[0045] See also Figure 1 , Figure 1 This is a schematic diagram of a data processing system provided in an embodiment of the present application, which includes a client 140, the Internet 130, a gateway 120, a computer device 110, etc.

[0046] Client 140 includes, but is not limited to, pre-configured personal computers, tablet computers, desktop computers, and other electronic devices with data reporting capabilities. Furthermore, it can be a single device or a collection of multiple devices. Client 140 can communicate with Internet 130 via wired or wireless means to exchange data.

[0047] Client 140 refers to a computer system that can provide data (e.g., training data) to computer device 110. Compared to ordinary terminals, computer device 110 has higher requirements for stability, security, and performance. Computer device 110 can be a single high-performance computer in a network platform, a cluster of multiple high-performance computers, a portion of a single high-performance computer (e.g., a virtual machine), or a combination of portions of multiple high-performance computers (e.g., virtual machines).

[0048] Gateway 120, also known as a gateway or protocol converter, implements network interconnection at the transport layer and is a computer system or device that acts as a converter. It acts as a translator between two systems using different communication protocols, data formats, languages, or even completely different architectures. Gateways can also provide filtering and security functions. Messages sent by client 140 to computer device 110 are sent through gateway 120 to the corresponding computer device 110. Messages sent by computer device 110 to client 140 are also sent through gateway 120 to the corresponding client 140.

[0049] The data processing method according to the embodiment of the present disclosure may be implemented on the computer device 110 .

[0050] It should be noted that Figure 1 The scenario diagram of the data processing system shown is only an example. The data processing system and scenario described in the embodiment of the present application are intended to more clearly illustrate the technical solution of the embodiment of the present application, and do not constitute a limitation on the technical solution provided by the embodiment of the present application. Ordinary technicians in this field can know that with the evolution of data processing technology and the emergence of new business scenarios, the technical solution provided by the embodiment of the present application is also applicable to similar technical problems.

[0051] In this embodiment, the description will be made from the perspective of a data processing device, which can be specifically integrated into a computer device having a storage unit and a microprocessor installed therein and having computing capabilities.

[0052] See also Figure 2 , Figure 2 This is a flow chart of a data processing method provided in an embodiment of the present application. The data processing method includes:

[0053] In step 201 , the current change position where data change occurs in the data table and the change data of each current change position are obtained.

[0054] A data table is a tabular structure used to store data. For example, during training, the embedding vectors in the recommendation model's embedding table and the perceptron data of the Multi-Layer Perceptron (MLP) are primarily adjusted. The embedding table occupies the vast majority of the model, while the MLP layer only occupies a very small portion. The sparse and dense features in the dataset are input into the embedding table and MLP layer, respectively, for processing. Each embedding table consists of a large number of embedding vectors. The embedding table performs table lookups based on sparse features to obtain corresponding embedding vectors, thereby mapping the sparse features into dense vectors for subsequent computation. Therefore, a data table can be an embedding table. The data in the table changes over the course of training iterations. It serves as the fundamental object for all data processing operations and contains multiple data items, each with a specific position within the table. The current changed position is the position of the data item in the data table where the data change occurred. This position identifies the data item that has changed. For example, this can be a combination of row and column numbers (for a two-dimensional table), or other unique identifiers that uniquely identify the position of the data item within the table. The changed data is the new value of the data item after it is modified at the current change position, that is, the specific content of the data after the change.

[0055] Specifically, the embodiment of the present application provides a scenario in which the changed data is recorded when there is a change in the data. A recording period can also be set in advance. When the recording period is reached, the changed data in the current data table relative to the previous recording period is recorded (equivalent to regularly saving checkpoints). This is not limited here.

[0056] In step 202, each current change position and the corresponding change data are written into the first data block.

[0057] The first data block is a data structure used to temporarily store the current change position and its corresponding change data. It can be a continuous storage space in the memory or a data container in a specific format. Its purpose is to integrate the change information of the same batch together.

[0058] Specifically, the first data block may be an array structure or a user-defined structure array, wherein each element includes two parts, one for storing the change position information, and the other for storing the change data.

[0059] For example, if two data items have changed, and the changed positions and data are (2, 15) and (4, 20), respectively, 2 and 4 are the changed positions, and 15 and 20 are the changed data corresponding to the changed positions. The first data block can be represented as "((2, 15), (4, 20))".

[0060] In step 203, the data blocks to be compared in the first row of each column of the change data record table and the last column currently recording the data blocks are obtained. All the data blocks recorded in each column of the change data record table correspond to the historical change positions and corresponding historical change data when the historical data in the data table changes.

[0061] Among them, see Figure 3 , Figure 3 The first schematic diagram of the change data record table provided in the embodiment of the present application. The change data record table is a table structure specifically used to record the historical data changes in the data table. Each column records the relevant information when the data changes at different time points or different batches, including the historical change location and historical change data. In this way, the change process of data over time can be tracked. The data blocks to be compared are data blocks taken from the first row of each column of the change data record table. These data blocks contain data change information at a previous time point or batch, and are used to compare with the current first data block to find duplicate or related change locations.

[0062] Specifically, such as Figure 3 As shown, Figure 3 The table includes two columns. The first column 10 is the historical change position (1, 2, and 3) and the corresponding historical change data (the historical change data corresponding to position 1, the historical change data corresponding to position 2, and the historical change data corresponding to position 3) recorded by the incremental checkpoint when the data change occurs for the first time since the baseline checkpoint was saved. The second column 20 is the historical change position (3, 4, and 5) and the corresponding historical change data (the historical change data corresponding to position 3, the historical change data corresponding to position 4, and the historical change data corresponding to position 5) recorded by the incremental checkpoint when the data change occurs again after the first column 10 is saved. The last column currently recording the data block is the second column 20. The data blocks to be compared include the data block to be compared 11 in the first row of the first column 10 and the data block with comparison 21 in the first row of the second column 20.

[0063] In step 204 , each data block to be compared is compared with the first data block, at least one target data block is selected from each data block to be compared, and a target change position that is the same as the current change position is selected from the target data blocks.

[0064] The target data block is a data block in the data block to be compared that is determined to have the same change position as the current change after comparison with the first data block. After the target data block is screened out, the target change position that is the same as the current change position is screened out from the target data block.

[0065] In some embodiments, comparing each of the to-be-compared data blocks with the first data block and selecting at least one target data block from each of the to-be-compared data blocks includes:

[0066] (1) obtaining a historical position range of a historical change position included in the data block to be compared, and a current position range of a current change position included in the first data block;

[0067] (2) comparing each of the historical position ranges with the current position range, and determining the to-be-compared data block corresponding to the historical position range that intersects with the current position range as the first candidate data block;

[0068] (3) Select at least one target data block from the first candidate data blocks.

[0069] The change data record table maintains a position range for each data block to be compared as metadata information of the data block to be compared. The position range is the range from the minimum change position to the maximum change position in the data block to be compared.

[0070] Specifically, in order to improve the comparison speed and accelerate the screening rate of the target data block, the current position range of the current change position in the first data block can be compared with the historical position range of each data block to be compared. If there is no intersection in the ranges, it means that the current change position does not exist in the corresponding data block to be compared; if there is an intersection in the ranges, it means that the current change position falls within the historical position range and may exist in the corresponding data block to be compared. The data block to be compared corresponding to the historical position range that has an intersection with the current position range is determined as the first candidate data block to screen out at least one target data block from the first candidate data block.

[0071] For example, Figure 3In the example, data block 11 to be compared includes change positions 1 and 2, so the historical position range of data block 11 to be compared is [1, 2]. Data block 21 to be compared includes change positions 3, 4, and 5, so the historical position range of data block 21 to be compared is [3, 5]. The current change positions are 2, 5, and 6, so the current position range of the first data block is [2, 6]. Since the historical position ranges [1, 2] of data block 11 to be compared and the historical position range [3, 5] of data block 21 to be compared both intersect with the current position range [2, 6], both data blocks 11 and 21 to be compared are first candidate data blocks.

[0072] By first comparing the location range, we can quickly eliminate a large number of data blocks that are clearly unrelated to the current change location. In real-world data processing scenarios, if the data table is large and frequently updated, a large number of data blocks awaiting comparison may accumulate in the change data record table. However, using location range comparison only allows for subsequent, more detailed comparisons of data blocks whose location ranges overlap with the current change location range. This significantly reduces unnecessary computational overhead and improves overall processing efficiency.

[0073] In some implementations, selecting at least one target data block from the first candidate data blocks includes:

[0074] (1.1) obtaining a Bloom filter corresponding to each of the first candidate data blocks, wherein the Bloom filter includes a hash value tag for each hash value;

[0075] (1.2) calculating multiple hash values ​​corresponding to each of the currently changed positions in the first data block using different preset hash functions;

[0076] (1.3) determining the number of positions where the hash value marks of the multiple hash values ​​corresponding to each of the current change positions in each of the Bloom filters are all first preset values, and obtaining the predicted number of repeated positions corresponding to each of the first candidate data blocks;

[0077] (1.4) calculating the ratio of the number of each predicted repeated position to the number of positions of the corresponding first candidate data block to obtain the predicted repetition ratio of each first candidate data block;

[0078] (1.5) selecting a target predicted repetition ratio exceeding a first preset repetition ratio from each of the predicted repetition ratios, and determining the first candidate data block corresponding to the target predicted repetition ratio as the second candidate data block;

[0079] (1.6) Select at least one target data block from the second candidate data blocks.

[0080] Among them, the Bloom filter is a data structure based on a bit array, and its length is predetermined according to factors such as the amount of data and the expected error rate (for example, in processing large-scale data change scenarios, it may be set to 10,000 bits or longer). During initialization, all bits are set to a second preset value (for example, 0, indicating that the element has not yet been marked). For each first candidate data block, there is a corresponding Bloom filter, which is used to quickly determine whether a current change position may exist in the data block. For each current change position, multiple hash values ​​corresponding to the current change position are calculated using different preset hash functions. For example, if the current change position is 7, then hash1(7)=30, hash2(7)=400, hash3(7)=700.

[0081] Specifically, for each first candidate data block's Bloom filter, the hash values ​​calculated by traversing the current change position in the first data block are all marked as 1. If they are all 1, it means that this change position may exist in the first candidate data block (there is a possibility of misjudgment). The number of all current change positions whose hash values ​​are all marked as 1 in the Bloom filter is counted. This number is the predicted number of repeated positions corresponding to the first candidate data block.

[0082] For example, for a first candidate data block, after checking, it is found that the hash values ​​of two currently changed positions are both marked as 1 in its Bloom filter, so the predicted number of repeated positions of the first candidate data block is 2.

[0083] A first predetermined duplication ratio is pre-set (e.g., 0.2 or 20%. This value is determined based on actual needs and data characteristics and is used to determine whether the potential duplication level in a data block is high enough to warrant further processing. If the predicted duplication ratio is less than the first predetermined ratio, it indicates that a large amount of data needs to be read into memory, but only a small amount of duplicate data can be extracted. Therefore, no extraction operation is required, and no actual data reading of the comparison block is required.) The predicted duplication ratio of each first candidate data block obtained by traversal and calculation is filtered out, and those with a ratio greater than the first predetermined ratio are selected.

[0084] For example, if there are 5 first candidate data blocks, and their predicted repetition ratios are 0.15, 0.25, 0.3, 0.18, and 0.22 respectively, then the predicted repetition ratios corresponding to the first candidate data blocks with predicted repetition ratios of 0.25, 0.3, and 0.22 exceed 0.2, and these first candidate data blocks are determined as second candidate data blocks.

[0085] Since hash collision may cause misjudgment, after determining the second candidate data block, it is necessary to screen out at least one target data block from the second candidate data blocks.

[0086] In this way, the Bloom filter's rapid hash value-based judgment allows for a preliminary estimate of the likelihood of duplication between each first candidate data block and the first data block, without requiring a detailed comparison of each change location's data. This reduces computing resource consumption and processing time. Second candidate data blocks are further screened based on the predicted duplication ratio, avoiding the need to load a large number of data blocks with low duplication likelihood into memory for detailed comparison. This screening ensures that only the data blocks most likely to contain a high proportion of duplicate data (second candidate data blocks) are loaded into memory, improving memory usage efficiency and ensuring stable system operation.

[0087] In some implementations, selecting at least one target data block from the second candidate data blocks includes:

[0088] (1.1) comparing the change position in the second candidate data block with the current change position, and determining the actual number of repeated positions of the candidate change position in the second candidate data block that is identical to the current change position;

[0089] (1.2) calculating the ratio of the actual number of repeated positions of the second candidate data block to the number of corresponding positions to obtain an actual repetition ratio of the second candidate data block;

[0090] (1.3) The second candidate data block whose actual repetition ratio exceeds the second preset repetition ratio is determined as the target data block.

[0091] Bloom filters have a certain false positive rate due to hash collisions, so the duplication ratio obtained through Bloom filters may be greater than the actual duplication ratio. For each second candidate data block, the actual data in the block must be read for key comparison. Therefore, for each second candidate data block, all the changed locations contained in it are traversed.

[0092] For example, a second candidate data block has change positions 3, 5, 7, 9, and 11, while the current change position is 5, 7, and 12. Comparing the change positions in the second candidate data block with the current change position, we find that change positions 5 and 7 exist in both sets. These two positions are candidate change positions. Counting the number of such candidate change positions reveals that the actual number of duplicate positions in this example is 2.

[0093] Specifically, the second preset duplication ratio is set in advance based on the needs and characteristics of data processing. For example, in a data management system with high data consistency requirements and the desire to minimize the storage of duplicate data, the second preset duplication ratio may be set to 0.3 (or 30%). This means that if the actual percentage of repeated change positions in a second candidate data block exceeds 30%, the data block is considered worthy of further processing and is determined as a target data block.

[0094] The specific method for determining the target data block is to compare the actual duplication ratio of each second candidate data block with the second preset duplication ratio. If the actual duplication ratio of a second candidate data block exceeds the preset second preset duplication ratio, then this second candidate data block is determined as the target data block. The target data block will contain the change position and its corresponding data that have a high degree of duplication with the current change position. These data blocks are of great significance in subsequent data processing (such as data deduplication, merging and updating, etc.) and can be processed in a targeted manner to improve data processing efficiency and optimize data storage.

[0095] Based on the second candidate data block obtained through the previous screening based on the Bloom filter and the predicted duplication ratio, a more precise comparison is performed to determine the data block (target data block) that is truly highly relevant to the current change. By accurately calculating the actual duplication ratio and comparing it with the preset threshold, it is possible to further screen out data blocks with a high degree of duplication and a significant impact on the current data change processing. This avoids the incorrect processing of some data blocks due to possible misjudgments of the Bloom filter. At the same time, while ensuring accuracy, it also minimizes the number of data blocks that require detailed processing, improving the efficiency of the entire data processing process and the quality of the data processing results. This is particularly suitable for large-scale data change processing scenarios where both data accuracy and processing efficiency are highly demanding.

[0096] Therefore, although the Bloom filter is highly efficient in the preliminary screening process, there is a certain misjudgment rate, and some data blocks that do not actually have repeated change positions may be misjudged as possibly having duplicates (i.e., the predicted duplication ratio is greater than the actual duplication ratio). By accurately comparing the actual change positions of the second candidate data blocks, the actual number of duplicate positions can be accurately determined, thereby obtaining the true duplication ratio and correcting the misjudgment of the Bloom filter. Only the second candidate data block whose actual duplication ratio exceeds the second preset duplication ratio is determined as the target data block, which means that the target data block has a high correlation and duplication rate with the current change position. In subsequent data processing, operations are performed on these highly correlated data blocks to ensure the accuracy of the processing results, avoid unnecessary operations on irrelevant or low-correlation data blocks, and thus improve the data quality of the entire data processing flow.

[0097] In some implementations, obtaining a Bloom filter corresponding to each first candidate data block includes:

[0098] (1.1) constructing a preset Bloom filter for each of the first candidate data blocks, wherein the preset Bloom filter includes a plurality of hash values, and a hash value tag of each hash value is a second preset value;

[0099] (1.2) For each of the first candidate data blocks, a hash value of each changed position is calculated using a different preset hash function, and the hash value mark of the corresponding hash value in the corresponding preset Bloom filter is changed to a first preset value to obtain a Bloom filter corresponding to each of the first candidate data blocks.

[0100] Among them, the Bloom filter is essentially a bit array, and its length (that is, the number of hash values ​​contained) is predetermined based on factors such as the amount of data, the expected false positive rate, and the actual application scenario. For example, in scenarios where large-scale data changes are processed and the false positive rate requirements are strict, the Bloom filter length may be set to 10,000 bits or longer. During initialization, all bits in this bit array (that is, the hash value mark of each hash value) are set to a second preset value, usually 0, indicating that no element has been marked yet. This step prepares for the subsequent recording of the hash value of the changed position, ensuring that the Bloom filter does not contain any marked change position information in the initial state.

[0101] Select multiple different preset hash functions. These hash functions should have good randomness and distribution to minimize the probability of hash collisions. For example, common hash functions such as MD5, SHA-1, etc., or customized hash functions based on specific data characteristics. For each changed position in each first candidate data block, use these selected hash functions to calculate its hash value respectively. Assume that there are changed positions 3, 5, and 7 in a first candidate data block, and three hash functions hash1, hash2, and hash3 are selected. Calculate hash1(3) = 100, hash2(3) = 500, hash3(3) = 800; hash1(5) = 200, hash2(5) = 600, hash3(5) = 900; hash1(7) = 300, hash2(7) = 700, hash3(7) = 1000 (these are just example hash values. The actual calculation results depend on the specific implementation of the hash function).

[0102] Specifically, after calculating the hash values ​​of the historical change positions, the hash value marks corresponding to these hash value index positions in the preset Bloom filter are changed from the second preset value (0) to the first preset value (usually 1). Taking the above example, the bits indexed as 100, 500, 800 (corresponding to the hash values ​​of change position 3), 200, 600, 900 (corresponding to the hash values ​​of change position 5), and 300, 700, 1000 (corresponding to the hash values ​​of change position 7) in the Bloom filter are all set to 1. In this way, the Bloom filter construction of the first candidate data block is completed. Through this Bloom filter, it is possible to quickly determine whether a certain change position is likely to exist in the first candidate data block (there is a certain error rate because hash collisions may cause the hash values ​​of different change positions to be the same). This process is repeated for each first candidate data block to obtain the corresponding Bloom filter, which provides a basis for subsequent repeated position prediction and screening.

[0103] Thus, the construction of a Bloom filter provides a basis for quickly determining whether a change location is likely to exist in the first candidate data block. In subsequent processing, when the relationship between the current change location in the first data block and the first candidate data block needs to be checked, it is only necessary to calculate the hash value of the current change location and check the mark of the corresponding hash value location in the Bloom filter, without having to perform a detailed comparison of each change location. This can quickly eliminate a large number of candidate data blocks that clearly do not contain the change location, significantly improving data screening speed.

[0104] In step 205 , the corresponding target change position and the corresponding target data are extracted from the target data block and written into the second data block.

[0105] Among them, see Figure 3 and Figure 4 , Figure 4 This is a second schematic diagram of a change data record table provided in an embodiment of the present application. Figure 3 When the change data record table of two incremental checkpoints has been stored in the , the change data of the current incremental checkpoint is stored (for example Figure 4(a first data block including change positions 2, 5, and 6), after comparing with data blocks 11 to be compared and data blocks 21 to be compared, it is finally determined that data block 11 to be compared is the target data block, and it has the same target change position 2 as the first data block, and data block 21 to be compared is the target data block, and it has the same target change position 5 as the first data block. Then, position 2 and the corresponding data in data block 11 to be compared are extracted, and position 5 and the corresponding data in data block 21 to be compared are extracted, and all the data extracted from each target data block are written into the second data block. For data block 11 to be compared, its corresponding second data block is second data block 12 including position 2 and the corresponding data. For data block 21 to be compared, its corresponding second data block is second data block 22 including position 5 and the corresponding data.

[0106] In step 206, the second data block corresponding to the target data block is written into the second row of the corresponding column, and the first data block is written into the column next to the last column to record the current change position where the data change occurs and the change data of each current change position.

[0107] The second data block including the target change position and target data is written to the second row of the corresponding column in the change data record table. For example, if the target data block is from the first column of the change data record table, the second data block is written to the second row of the first column.

[0108] For example, from Figures 3 to 4 In the example, for the first column, the second data block 12 is written into the second row of the first column, and for the second example, the second data block 22 is written into the second row of the second column.

[0109] Specifically, the first data block is written to the column next to the last column (i.e., a new column). This completes the record of the current data changes, including the change location and the changed data. You can subsequently query the change data record table to track the data change history, perform data analysis, and restore data to a specific version. This recording process also provides a foundation for possible data deduplication and storage optimization. For example, if a large number of duplicate change locations and data are found, you can consider merging or optimizing the storage structure to reduce storage space and improve data processing efficiency.

[0110] In some embodiments, the method further comprises:

[0111] (1) When a data restoration operation is detected, obtaining a first data change point to be restored;

[0112] (2) determining the target column corresponding to the first data change point from the change data record table;

[0113] (3) obtaining the last target row in the target column that records the data block;

[0114] (4) obtaining each record data block from the first row and first column to the target row and target column in the change data record table, as well as the initial data table;

[0115] (5) The data in the initial data table is updated according to the change position and the corresponding change data in each of the record data blocks to obtain a data table corresponding to the first data change point.

[0116] Among them, the data restoration operation can be triggered by a variety of situations, such as a user manually requesting to restore to the data state at a specific point in time, or the system automatically initiating a restoration operation after detecting a data error or anomaly. When the triggering conditions are met, the system detects that a data restoration operation is required. The first data change point refers to the specific change moment or version (such as a specific incremental checkpoint) to which the data is expected to be restored. This can be specified in a variety of ways, such as in the form of a timestamp (for example, "2023-08-1510:30:00" indicates that the data is expected to be restored to the state at that point in time), or explicitly specified in the form of a change point number (if the system has sequentially numbered the data changes, such as the 50th data change point). After obtaining the first data change point that is expected to be restored, subsequent operations will revolve around how to restore the data to the state at that point.

[0117] Specifically, since the change data record table is a two-dimensional table used to record data change history, each column corresponds to a data change event or a set of changes within a specific time period. The columns are ordered from left to right, recording changes from earliest to latest. Based on the first data change point obtained, the corresponding column is searched in the change data record table. If the change data record table uses timestamps as column identifiers, the column matching the timestamp of the first data change point is searched; if the change sequence number is used as the identifier, the column with the sequence number equal to the first data change point is searched. This column is the target column, which contains the relevant change location and change data information at the time of the data change point and is a key basis for subsequent data restoration. The target column may contain multiple data blocks, each of which records a different part or type of data change at that data change point. Starting from the first row of the target column, each row is checked to see if it contains a data block until the last row containing a data block is found and identified as the last target row. This step determines the scope of all changed data blocks that need to be considered at the data change point, ensuring that no relevant change information is missed. It also avoids processing unnecessary blank rows of data and improves data restoration efficiency.

[0118] Specifically, based on the previously determined range, all record data blocks, starting from the first row and first column to the target row and target column, are extracted from the change data record table. These data blocks contain the entire change history from the baseline checkpoint to the first data change point (a user-specified incremental checkpoint) and serve as the foundation for incremental data restoration. A loop can be used to read and store these data blocks row by row and column by column for subsequent unified processing.

[0119] The initial data table contains the data at every location in the data table saved at the baseline checkpoint. It represents the starting state for data changes and serves as the baseline for restoring data. This can be a copy of the original data previously saved in the system, or an initial state snapshot maintained during the data change logging process. After obtaining the initial data table, this initial state is used to gradually restore the data to the state at the first data change point, based on the information in the change data logging table.

[0120] For each acquired record data block, the corresponding position in the initial data table is found according to the change position information recorded therein, and then the data at the position is updated with the change data in the record data block.

[0121] For example, Figure 4 In the example, for recovering to the incremental checkpoint (i.e., data change point) of the second column, there are two rows in the second column. Then, each record data block from the first row and first column (i.e., the data block containing position 1) to the second row and second column (i.e., the data block containing position 5) is obtained. The record data blocks include the data block containing position 1, the data block containing position 2, the data blocks containing positions 3 and 4, and the data block containing position 5. Then, the data at positions 1, 2, 3, 4, and 5 in the initial data table are updated with the data corresponding to each position in the data block, thereby recovering to the incremental checkpoint of the second column.

[0122] In this way, by changing the structure of the data record table, each column corresponds to a specific data change event or time period, and the target column can be quickly located based on the first data change point. This search method based on column identifiers (timestamps or change sequence numbers) greatly improves the speed of data location compared to searching for related data in unstructured data. When obtaining record data blocks, by looping through the range from the first row and first column to the target row and target column, this orderly traversal method can efficiently extract the required data blocks, avoiding aimless searching of the entire data table and improving data acquisition efficiency.

[0123] In some embodiments, after writing the second data block corresponding to the target data block into the second row of the corresponding column and writing the first data block into the column next to the last column, the method includes:

[0124] (1) Splicing each data block in the first row to obtain a first spliced ​​file corresponding to the first row;

[0125] (2) splicing each data block in the second row to obtain a second spliced ​​file corresponding to the second row;

[0126] (3) Establishing a mapping relationship between the first splicing file, the second splicing file and the current data change point.

[0127] Among them, for the record of each incremental checkpoint, after writing the first data block into the next column of the last column, each data block in the first row will be spliced ​​to obtain the first splicing file corresponding to the first row, and each data block in the second row will be spliced ​​to obtain the second splicing file corresponding to the second row. Finally, a mapping relationship between the current checkpoint and the first splicing file and the second splicing file is established.

[0128] For example, in storage such as Figure 4 After the first data block is stored in the third column of the change data record table, the data block including the change position 1, the data block including the change positions 3 and 4, and the first data block including the change positions 2, 5 and 6 in the first row are spliced ​​to obtain the first spliced ​​file including the change positions 1, 3, 4, 2, 5 and 6, and the data block including the change position 2 and the data block including the change position 5 in the second row are spliced ​​to obtain the second spliced ​​file including the change positions 2 and 5, and a mapping relationship between the current data change point (the third incremental checkpoint) and the two files is established.

[0129] In some embodiments, the method further comprises:

[0130] (1) When a data restoration operation is detected, obtaining a second data change point to be restored;

[0131] (2) obtaining a target first splicing file and a target second splicing file that have a preset mapping relationship with the second data change point;

[0132] (3) According to the changed position and corresponding changed data of each data block in the target first splicing file and the target second splicing file, the data in the initial data table is updated to obtain a data table corresponding to the second data change point.

[0133] Among them, the second data change point refers to the specific change time or version to which the data is expected to be restored. In the previous data processing process, a preset mapping relationship between the data change point and the splicing file (the first splicing file and the second splicing file, etc.) has been established. According to the determined second data change point, the corresponding target first splicing file and target second splicing file are searched in the mapping relationship. For example, if the second data change point is the timestamp "2023-09-2014:30:00", the record corresponding to the timestamp is searched in the mapping table. Assuming that the found record shows that the corresponding first splicing file is "file1.txt" and the second splicing file is "file2.txt", then these two files are the target files.

[0134] The first target splicing file and the second target splicing file contain multiple data blocks, each of which has a corresponding change position and change data. For example, a data block in the first target splicing file may record a change position of (3) (indicating row 3, column 5 in the data table) and a change data of "new_value1"; a data block in the second target splicing file may record a change position of (2) and a change data of "new_value2".

[0135] Update the initial data table: Obtain the initial data table, which represents the starting state for the data change. Update the initial data table based on the data block information obtained from the target spliced ​​file. For the data block in the example above, find position 3 in the initial data table and update its value to "new_value1." Find position 2 and update its value to "new_value2." Repeat this update operation for all data blocks in the first and second target spliced ​​files, ultimately obtaining the data table corresponding to the second data change point.

[0136] In this way, a mapping relationship between the splicing file and the current data change point is established, providing a clear and convenient way for data version management. Each data change point (such as each incremental checkpoint) can be accurately associated with the corresponding first splicing file and second splicing file, just like putting clear labels on different versions of data. Whether the system itself is conducting data audits and version backtracking, or developers and operation and maintenance personnel need to understand the data status at a specific moment, they can quickly locate the corresponding files through this mapping relationship, which helps to accurately grasp the changes in data at different stages and achieve efficient version control. During the data restoration operation, you only need to use the preset mapping relationship based on the second data change point you want to restore to quickly obtain the target first splicing file and the target second splicing file, without having to filter and organize the relevant data from the massive original data blocks or the entire change data record table. This rapid positioning mechanism based on mapping relationships greatly reduces the time cost of data search and screening, allowing the data restoration process to be started and executed more quickly, effectively shortening the time required for data restoration.

[0137] In some implementations, before obtaining the current change position where data change has occurred in the data table and the change data of each current change position, the method further includes:

[0138] (1) when a data change is detected in the data table, obtaining a preset data table storage time for each data in the data table and a preset reloading time for reloading each saved data;

[0139] (2) calculating the sum of the preset data table storage time and the preset reload time to obtain a reset time for resetting the data table;

[0140] (3) obtaining a preset data storage time for storing the changed data, and gradually updating the data in the initial data table to the update time of the changed data where the data change currently occurs;

[0141] (4) calculating the sum of the preset data storage time, the preset reload time, and the update time to obtain a total restoration time;

[0142] (5) When the total restoration time is less than the reset time, the step of obtaining the current change position where the data change currently occurs in the data table and the change data of each current change position is executed.

[0143] Among them, the preset data table storage time C 基准This is the pre-set average time it takes to store each piece of data in a table to a storage medium (such as a hard drive, solid-state drive, or other storage device). This time may be based on statistical data from past storage operations on the table, or estimated based on factors such as the storage device's performance, the table's data structure, and its size.

[0144] For example, after multiple tests, it was found that storing a row of data in a specific table on a specific hard drive takes an average of 0.01 seconds. This 0.01 second is the preset table storage time for a single row of data. In actual applications, this time data can be recorded in configuration files or system metadata for easy access when needed.

[0145] Default reload time R 基准 This refers to the average time it takes to reload previously saved data from a storage medium into an operating environment, such as memory, so that the data can be used again by the system. Similarly, it is determined based on past experience or an evaluation of factors such as the storage device's read performance and data volume. For example, after testing, it took an average of 0.02 seconds to read a row of data from the hard drive and load it into memory. This 0.02 seconds is the preset reload time. Similar to the storage time, this reload time data will also be recorded and saved in an appropriate manner for easy access when needed.

[0146] Preset data table storage time C 基准 With the preset reload time R 基准 The sum of the values ​​is the reset time T for resetting the data table. 基准 ,Reset time refers to resetting the baseline checkpoint, which means recreating a complete record of the model state, so it takes time to save it first, and then it takes time to load it back when it needs to be restored.

[0147] Preset data storage time C 增量 This is specifically for storing changed data within a data table, recording the average time required to store these changed data on the storage medium. This is determined similarly to the default data table storage time, taking into account factors such as the storage device characteristics, the size and format of the changed data, and other factors. For example, analysis revealed that the average storage time for each changed data change was 0.005 seconds. This time value is the default data storage time and is also recorded for future use.

[0148] Update time R iRepresents the time required to recover the updated parameters from the baseline checkpoint to the i-th checkpoint. When continuing with the incremental checkpoint strategy, in addition to restoring the baseline checkpoint, all incremental checkpoints from the baseline checkpoint to the i-th checkpoint must be applied sequentially to restore the model state to the i-th checkpoint. Each incremental checkpoint requires a certain amount of time to read and apply its updated data, which is the sum of these times.

[0149] For example, from the baseline checkpoint to the third incremental checkpoint, it takes 10 seconds to restore the first incremental checkpoint, 15 seconds to restore the second incremental checkpoint, and 20 seconds to restore the third incremental checkpoint, so R3=10+15+20=45 seconds.

[0150] Total restoration time T 增量 =C 增量 +R 基准 +R i , which means that when continuing to execute the incremental checkpoint strategy, the total time overhead includes saving the current incremental checkpoint (C 增量 ), Restoring the Baseline Checkpoint (R 基准 ) and recover all incremental checkpoint update parameters (R i ) time. It comprehensively considers the complete time cost of gradually recovering from the baseline to a specific checkpoint under the incremental checkpoint strategy. By comparing T 基准 and T 增量 , you can weigh which strategy is more favorable in terms of time under different situations (resetting the baseline or continuing the increment), and thus decide whether to reset the baseline checkpoint.

[0151] When the total restoration time is less than the reset time, the step of obtaining the current change position in the data table where the data change has occurred, as well as the change data for each of the current change positions, is performed. This indicates that the time required to save the current change data to the change data record table and subsequently restore it is less than the time required to re-record all current data in the data table and subsequently restore it. Therefore, the step of obtaining the current change position in the data table where the data change has occurred, as well as the change data for each of the current change positions, is performed.

[0152] In some embodiments, the method further comprises:

[0153] When the total restoration time is less than the reset time, the data of each current position in the data table is stored to obtain an updated initial data table.

[0154] Among them, when the total restoration time is less than the reset time, it means that for the current changed data, the time consumed to save it to the change data record table and the subsequent recovery is greater than the time consumed to re-record all the current data in the data table and subsequently recover it. Therefore, the data at each current position in the data table is stored to obtain the updated initial data table.

[0155] By comparing the total restore time and reset time, the system can flexibly decide which data processing strategy to adopt based on the time cost: resetting the baseline checkpoint or continuing with the incremental checkpoint strategy. For example, in a business scenario where data changes frequently, blindly resetting the baseline checkpoint each time may consume a significant amount of unnecessary time for complete data storage and reloading. By calculating and comparing these two times, when the total restore time is less than the reset time, the incremental checkpoint strategy is chosen to continue, processing only the storage of changed data, baseline checkpoint recovery, and corresponding incremental checkpoint update parameter recovery. This can effectively reduce the overall time overhead of data processing and improve the efficiency of data operations.

[0156] As can be seen from the above, the embodiment of the present application obtains the current change position of the data change currently occurring in the data table, and the change data of each current change position; writes each current change position and the corresponding change data into the first data block; obtains the data block to be compared in the first row of each column of the change data record table, and the last column where the data block is currently recorded, and all the data blocks recorded in each column of the change data record table correspond to the historical change position and the corresponding historical change data when the historical data in the data table is changed; compares each of the data blocks to be compared with the first data block, filters out at least one target data block from each of the data blocks to be compared, and filters out the target data blocks that match the target data blocks. The target change position is the same as the current change position; the corresponding target change position and the corresponding target data are extracted from the target data block and written into the second data block; the second data block corresponding to the target data block is written into the second row of the corresponding column, and the first data block is written into the next column of the last column to record the current change position where the data change occurs and the change data of each current change position. Compared with the related art, when restoring the checkpoint, it is necessary to read the parameters of each update of the position that has undergone multiple updates, and a large amount of redundant data is read. The embodiment of the present application can only read the updated data for the recovery checkpoint, avoid resource consumption and time cost consumption, and improve recovery efficiency.

[0157] The specific implementation of the above steps can be found in the previous embodiments and will not be repeated here.

[0158] To facilitate better implementation of the data processing method provided in the embodiment of the present application, the embodiment of the present application also provides a device based on the above data processing method. The meanings of the terms are the same as those in the above data processing method, and the specific implementation details can be referred to the description in the method embodiment.

[0159] See also Figure 5 , Figure 5 This is a structural schematic diagram of a data processing method device provided in an embodiment of the present application, which is applied to a computer device, wherein the data processing method device may include a first acquisition unit 601, a first writing unit 602, a second acquisition unit 603, a comparison unit 604, a second writing unit 605 and a third writing unit 606, etc.

[0160] The first acquiring unit 601 is configured to acquire a current change position where data change occurs in a data table, and change data at each current change position;

[0161] A first writing unit 602 is configured to write each of the current change positions and the corresponding change data into a first data block;

[0162] The second acquisition unit 603 is configured to acquire the data block to be compared in the first row of each column of the change data record table, and the last column currently recording the data block, wherein all the data blocks recorded in each column of the change data record table correspond to the historical change positions and corresponding historical change data when the historical data in the data table is changed;

[0163] a comparing unit 604, configured to compare each of the to-be-compared data blocks with the first data block, select at least one target data block from each of the to-be-compared data blocks, and select a target change position that is the same as the current change position from the target data blocks;

[0164] A second writing unit 605 is configured to extract a corresponding target change position and corresponding target data from the target data block and write the extracted data into a second data block;

[0165] The third writing unit 606 is used to write the second data block corresponding to the target data block into the second row of the corresponding column, and write the first data block into the next column of the last column to record the current change position where the data change occurs and the change data of each current change position.

[0166] In some embodiments, the comparison unit 604 includes:

[0167] an acquiring subunit, configured to acquire a historical position range of a historical change position included in the data block to be compared, and a current position range of a current change position included in the first data block;

[0168] a comparing subunit, configured to compare each of the historical position ranges with the current position range, and determine a to-be-compared data block corresponding to a historical position range that intersects with the current position range as a first candidate data block;

[0169] The screening subunit is configured to screen out at least one target data block from the first candidate data blocks.

[0170] In some embodiments, the screening subunit is configured to:

[0171] Obtaining a Bloom filter corresponding to each of the first candidate data blocks, where the Bloom filter includes a hash value tag for each hash value;

[0172] Calculating multiple hash values ​​corresponding to each of the current changed positions in the first data block by using different preset hash functions;

[0173] Determine the number of positions where the hash value marks of the multiple hash values ​​corresponding to each current change position in each Bloom filter are all first preset values, and obtain the predicted number of repeated positions corresponding to each first candidate data block;

[0174] Calculating the ratio of each of the predicted number of repeated positions to the number of positions of the corresponding first candidate data block to obtain a predicted repetition ratio of each of the first candidate data blocks;

[0175] Filtering out a target predicted repetition ratio exceeding a first preset repetition ratio from each of the predicted repetition ratios, and determining a first candidate data block corresponding to the target predicted repetition ratio as a second candidate data block;

[0176] At least one target data block is selected from the second candidate data blocks.

[0177] In some embodiments, the screening subunit is further specifically configured to:

[0178] comparing the changed position in the second candidate data block with the current changed position to determine the actual number of repeated positions of the candidate changed position in the second candidate data block that is identical to the current changed position;

[0179] Calculating a ratio of the actual number of repeated positions of the second candidate data block to the number of corresponding positions to obtain an actual repetition ratio of the second candidate data block;

[0180] The second candidate data block whose actual repetition ratio exceeds the second preset repetition ratio is determined as the target data block.

[0181] In some embodiments, the screening subunit is further specifically configured to:

[0182] Constructing a preset Bloom filter for each of the first candidate data blocks, wherein the preset Bloom filter includes a plurality of hash values, and a hash value tag of each hash value is a second preset value;

[0183] For each of the first candidate data blocks, a hash value of each changed position is calculated using a different preset hash function, and the hash value mark of the corresponding hash value in the corresponding preset Bloom filter is changed to a first preset value to obtain a Bloom filter corresponding to each of the first candidate data blocks.

[0184] In some embodiments, the data processing apparatus further includes:

[0185] a third acquiring unit, configured to acquire a first data change point desired to be restored when a data restoration operation is detected;

[0186] a determining unit, configured to determine a target column corresponding to the first data change point from a change data record table;

[0187] a fourth acquiring unit, configured to acquire the last target row in the target column where the data block is recorded;

[0188] A fifth acquiring unit, configured to acquire each record data block from the first row and first column to the target row and target column in the change data record table, and an initial data table;

[0189] The first updating unit is configured to update the data in the initial data table according to the change position and corresponding change data in each record data block to obtain a data table corresponding to the first data change point.

[0190] In some embodiments, the data processing apparatus further includes:

[0191] A first splicing unit is used to splice each data block in the first row to obtain a first splicing file corresponding to the first row;

[0192] A second splicing unit is used to splice each data block in the second row to obtain a second splicing file corresponding to the second row;

[0193] The mapping relationship establishing unit is configured to establish a mapping relationship between the first splicing file, the second splicing file, and the current data change point.

[0194] In some embodiments, the data processing apparatus further includes:

[0195] A sixth acquiring unit, configured to acquire a second data change point desired to be restored when a data restoration operation is detected;

[0196] a seventh acquiring unit, configured to acquire a target first splicing file and a target second splicing file having a preset mapping relationship established with the second data change point;

[0197] The second updating unit is configured to update the data in the initial data table according to the changed position and corresponding changed data of each data block in the target first splicing file and the target second splicing file, to obtain a data table corresponding to the second data change point.

[0198] In some embodiments, the data processing apparatus further includes:

[0199] an eighth acquiring unit, configured to acquire, when detecting that a data change currently occurs in the data table, a preset data table storage time for each data in the data table and a preset reloading time for reloading each stored data;

[0200] a first calculation unit, configured to calculate a sum of the preset data table storage time and the preset reload time to obtain a reset time for resetting the data table;

[0201] A ninth acquiring unit is configured to acquire a preset data storage time for storing the changed data, and to gradually update the data in the initial data table to the update time of the changed data at which the data change currently occurs;

[0202] a second calculation unit, configured to calculate a sum of the preset data storage time, the preset reload time, and the update time to obtain a total restoration time;

[0203] The execution unit is configured to execute the step of obtaining the current change position where the data change currently occurs in the data table and the change data of each current change position when the total restoration time is less than the reset time.

[0204] In some embodiments, the data processing apparatus further includes:

[0205] The storage unit is used to store the data of each current position in the data table when the total restoration time is less than the reset time to obtain an updated initial data table.

[0206] The specific implementation of each of the above units can be found in the previous embodiments and will not be described again here.

[0207] As can be seen from the above, the embodiment of the present application obtains the current change position of the data change currently occurring in the data table and the change data of each current change position through the first acquisition unit 601; the first writing unit 602 writes each current change position and the corresponding change data into the first data block; the second acquisition unit 603 obtains the data block to be compared in the first row of each column of the change data record table, and the last column of the current data block, all the data blocks recorded in each column of the change data record table correspond to the historical change position and the corresponding historical change data when the historical data in the data table is changed; the comparison unit 604 compares each of the The data blocks to be compared and the first data blocks, at least one target data block is screened out from each of the data blocks to be compared, and a target change position that is the same as the current change position is screened out from the target data blocks; the second writing unit 605 extracts the corresponding target change position and the corresponding target data from the target data blocks, and writes them into the second data block; the third writing unit 606 writes the second data block corresponding to the target data block into the second row of the corresponding column, and writes the first data block into the next column of the last column, so as to record the current change position where the data change occurs, and the change data of each current change position. Compared with the related art, when restoring a checkpoint, it is necessary to read the parameters of each update of the position that has undergone multiple updates, and read a large amount of redundant data. The embodiment of the present application can only read the updated data for restoring the checkpoint, avoiding resource consumption and time cost consumption, and improving recovery efficiency.

[0208] The specific implementation of each of the above units can be found in the previous embodiments and will not be described again here.

[0209] Reference Figure 6 , Figure 6 This is a block diagram of the structure of a portion of a computer device 110 for implementing an embodiment of the present disclosure. The computer device 110 may vary greatly due to different configurations or performance, and may include one or more central processing units (CPUs) 622 (for example, one or more processors) and memories 632, and one or more storage media 630 (for example, one or more mass storage devices) for storing application programs 642 or data 644. The memories 632 and storage media 630 may be temporary storage or permanent storage. The program stored in the storage medium 630 may include one or more modules (not shown in the figure), each module may include a series of instruction operations on the server 600. Furthermore, the central processing unit 622 may be configured to communicate with the storage medium 630 to execute a series of instruction operations in the storage medium 630 on the server 600.

[0210] The computer device 110 may also include one or more power supplies 626, one or more wired or wireless network interfaces 650, one or more input and output interfaces 658, and / or one or more operating systems 641, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0211] The central processing unit 622 in the computer device 110 may be used to execute the data processing method of the embodiment of the present disclosure, for example:

[0212] Obtaining the current change position where data change has occurred in the data table, and the change data of each current change position;

[0213] Writing each of the current change positions and the corresponding change data into the first data block;

[0214] Obtain the data block to be compared in the first row of each column of the change data record table, and the last column currently recording the data block, wherein all data blocks recorded in each column of the change data record table correspond to the historical change position and corresponding historical change data when the historical data in the data table is changed;

[0215] Comparing each of the to-be-compared data blocks with the first data block, selecting at least one target data block from each of the to-be-compared data blocks, and selecting a target change position that is the same as the current change position from the target data blocks;

[0216] Extracting the corresponding target change position and the corresponding target data from the target data block, and writing them into the second data block;

[0217] The second data block corresponding to the target data block is written into the second row of the corresponding column, and the first data block is written into the next column of the last column to record the current change position where the data change occurs and the change data of each current change position.

[0218] The embodiments of the present disclosure further provide a computer-readable storage medium, which is used to store program codes, and the program codes are used to execute the data processing methods of the aforementioned embodiments.

[0219] The present disclosure also provides a computer program product, which includes a computer program. A processor of a computer device reads and executes the computer program, so that the computer device executes the above-mentioned data processing method. For example:

[0220] Obtaining the current change position where data change has occurred in the data table, and the change data of each current change position;

[0221] Writing each of the current change positions and the corresponding change data into the first data block;

[0222] Obtain the data block to be compared in the first row of each column of the change data record table, and the last column currently recording the data block, wherein all data blocks recorded in each column of the change data record table correspond to the historical change position and corresponding historical change data when the historical data in the data table is changed;

[0223] Comparing each of the to-be-compared data blocks with the first data block, selecting at least one target data block from each of the to-be-compared data blocks, and selecting a target change position that is the same as the current change position from the target data blocks;

[0224] Extracting the corresponding target change position and the corresponding target data from the target data block, and writing them into the second data block;

[0225] The second data block corresponding to the target data block is written into the second row of the corresponding column, and the first data block is written into the next column of the last column to record the current change position where the data change occurs and the change data of each current change position.

[0226] In addition, the terms "comprises" and "comprising" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or apparatus that comprises a series of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, product or apparatus.

[0227] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0228] It should be understood that in the description of the embodiments of the present application, multiple (or multiple items) means more than two, greater than, less than, exceed, etc. are understood to exclude the number itself, and above, below, within, etc. are understood to include the number itself.

[0229] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0230] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0231] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0232] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0233] It should also be understood that the various implementation methods provided in the embodiments of the present application can be combined arbitrarily to achieve different technical effects.

[0234] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0235] The above is a specific description of the implementation methods of the present application, but the present application is not limited to the above implementation methods. Technical personnel familiar with the art can also make various equivalent modifications or substitutions without violating the spirit of the present application. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present application.

Claims

1. A data processing method, characterized in that: include: Obtaining the current change position where data change has occurred in the data table, and the change data of each current change position; Writing each of the current change positions and the corresponding change data into the first data block; Obtain the data block to be compared in the first row of each column of the change data record table, and the last column currently recording the data block, wherein all data blocks recorded in each column of the change data record table correspond to the historical change position and corresponding historical change data when the historical data in the data table is changed; Comparing each of the to-be-compared data blocks with the first data block, selecting at least one target data block from each of the to-be-compared data blocks, and selecting a target change position that is the same as the current change position from the target data blocks; Extracting the corresponding target change position and the corresponding target data from the target data block, and writing them into the second data block; Writing the second data block corresponding to the target data block into the second row of the corresponding column, and writing the first data block into the column next to the last column, so as to record the current change position where the data change occurs and the change data of each current change position; When a data restoration operation is detected, obtaining a first data change point desired to be restored; Determine the target column corresponding to the first data change point from the change data record table; Obtain the last target row in the target column that records the data block; Obtain each record data block from the first row and first column to the target row and target column in the change data record table, as well as the initial data table; The data in the initial data table is updated according to the changed position and corresponding changed data in each of the record data blocks to obtain a data table corresponding to the first data change point.

2. The data processing method according to claim 1, wherein: The comparing each of the to-be-compared data blocks with the first data block and selecting at least one target data block from each of the to-be-compared data blocks includes: Acquire a historical position range of a historical change position included in the data block to be compared, and a current position range of a current change position included in the first data block; Comparing each of the historical position ranges with the current position range, and determining a to-be-compared data block corresponding to a historical position range that intersects with the current position range as a first candidate data block; At least one target data block is selected from the first candidate data blocks.

3. The data processing method according to claim 2, characterized in that: The step of selecting at least one target data block from the first candidate data blocks includes: Obtaining a Bloom filter corresponding to each of the first candidate data blocks, where the Bloom filter includes a hash value tag for each hash value; Calculating multiple hash values ​​corresponding to each of the current changed positions in the first data block by using different preset hash functions; Determine the number of positions where the hash value marks of the multiple hash values ​​corresponding to each current change position in each Bloom filter are all first preset values, and obtain the predicted number of repeated positions corresponding to each first candidate data block; Calculating the ratio of each of the predicted number of repeated positions to the number of positions of the corresponding first candidate data block to obtain a predicted repetition ratio of each of the first candidate data blocks; Filtering out a target predicted repetition ratio exceeding a first preset repetition ratio from each of the predicted repetition ratios, and determining a first candidate data block corresponding to the target predicted repetition ratio as a second candidate data block; At least one target data block is selected from the second candidate data blocks.

4. The data processing method according to claim 3, characterized in that: The selecting at least one target data block from the second candidate data blocks includes: comparing the changed position in the second candidate data block with the current changed position to determine the actual number of repeated positions of the candidate changed position in the second candidate data block that is identical to the current changed position; Calculating a ratio of the actual number of repeated positions of the second candidate data block to the number of corresponding positions to obtain an actual repetition ratio of the second candidate data block; The second candidate data block whose actual repetition ratio exceeds the second preset repetition ratio is determined as the target data block.

5. The data processing method according to claim 3, wherein: The obtaining of the Bloom filter corresponding to each of the first candidate data blocks includes: Constructing a preset Bloom filter for each of the first candidate data blocks, wherein the preset Bloom filter includes a plurality of hash values, and a hash value tag of each hash value is a second preset value; For each of the first candidate data blocks, a hash value of each changed position is calculated using a different preset hash function, and the hash value mark of the corresponding hash value in the corresponding preset Bloom filter is changed to a first preset value to obtain a Bloom filter corresponding to each of the first candidate data blocks.

6. The data processing method according to claim 1, wherein: After writing the second data block corresponding to the target data block into the second row of the corresponding column and writing the first data block into the column next to the last column, the method includes: Splice each data block in the first row to obtain a first spliced ​​file corresponding to the first row; Splice each data block in the second row to obtain a second spliced ​​file corresponding to the second row; A mapping relationship between the first splicing file, the second splicing file and the current data change point is established.

7. The data processing method according to claim 6, characterized in that: The method further comprises: When a data restoration operation is detected, obtaining a second data change point desired to be restored; Acquire a target first splicing file and a target second splicing file that have a preset mapping relationship with the second data change point; According to the changed position and corresponding changed data of each data block in the target first splicing file and the target second splicing file, the data in the initial data table is updated to obtain a data table corresponding to the second data change point.

8. The data processing method according to any one of claims 1 to 7, characterized in that: Before obtaining the current change position where the data change has occurred in the data table and the change data of each current change position, the method further includes: When a data change is detected in the data table, a preset data table storage time for each data in the data table is obtained, and a preset reload time for reloading each saved data; Calculating the sum of the preset data table storage time and the preset reload time to obtain a reset time for resetting the data table; Obtain the preset data storage time for storing the changed data, and gradually update the data in the initial data table to the update time of the changed data where the data change currently occurs; Calculate the sum of the preset data storage time, the preset reload time, and the update time to obtain a total restoration time; When the total restoration time is less than the reset time, the step of obtaining the current change position where the data change currently occurs in the data table and the change data of each current change position is performed.

9. The data processing method according to claim 8, characterized in that: The method further comprises: When the total restoration time is less than the reset time, the data of each current position in the data table is stored to obtain an updated initial data table.

10. A data processing device, characterized in that: include: A first acquiring unit is configured to acquire a current change position where data change occurs in a data table, and change data at each current change position; a first writing unit, configured to write each of the current change positions and corresponding change data into a first data block; a second acquiring unit, configured to acquire the data block to be compared in the first row of each column of the change data record table and the last column currently recording the data block, wherein all the data blocks recorded in each column of the change data record table correspond to the historical change positions and corresponding historical change data when the historical data in the data table is changed; a comparing unit, configured to compare each of the data blocks to be compared with the first data block, select at least one target data block from each of the data blocks to be compared, and select a target change position that is the same as the current change position from the target data blocks; A second writing unit, configured to extract a corresponding target change position and corresponding target data from the target data block, and write the extracted data into a second data block; a third writing unit, configured to write the second data block corresponding to the target data block into the second row of the corresponding column, and write the first data block into the column next to the last column, so as to record a current change position where data change occurs and change data of each current change position; a third acquiring unit, configured to acquire a first data change point desired to be restored when a data restoration operation is detected; a determining unit, configured to determine a target column corresponding to the first data change point from a change data record table; a fourth acquiring unit, configured to acquire the last target row in the target column where the data block is recorded; A fifth acquiring unit, configured to acquire each record data block from the first row and first column to the target row and target column in the change data record table, and an initial data table; The first updating unit is configured to update the data in the initial data table according to the change position and corresponding change data in each record data block to obtain a data table corresponding to the first data change point.

11. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor to execute the data processing method according to any one of claims 1 to 9.

12. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the data processing method according to any one of claims 1 to 9 is implemented.

Images