Data platform file migration method, computer program product, and data platform

Abstract

The present application relates to a data platform file migration method, a computer program product, and a data platform, wherein the data platform file migration method includes: scanning the source data file to be migrated to obtain the type of the source data file; if the type of the source data file is in the state set of the target reinforcement learning model, using the type of the source data file as the current state of the target reinforcement learning model, and obtaining the current action of the target reinforcement learning model in response to the current state; using the current action of the target reinforcement learning model to block the source data file; constructing a Merkle tree of the source data file based on the block result of the source data file; locating the difference data block by comparing the Merkle tree of the source data file and the target data file by hashing layer by layer; and migrating the difference data block to the target end. This method solves the problem that the unreasonable data block strategy in the current data platform file migration method makes the difference location easy to fail.

Description

Technical Field

[0001] The present application relates to the field of data migration, and in particular to a data platform file migration method, a computer program product, and a data platform. Background Art

[0002] Complex fault-handling assistance technology based on cross-business scenarios requires integrating fault data from different business areas, including fault type, fault occurrence time, impact scope, and handling measures, in order to establish a fault resolution measures library. Association analysis algorithms are used to mine association rules between different faults and identify potential relationships between faults. Fault data from different business areas is pre-stored in different data platforms. During the data integration process, it is usually necessary to migrate fault data between different data platforms in order to integrate the fault data into one data platform. When file data in the source data platform changes, it needs to be updated synchronously on the target data platform.

[0003] Traditional data migration uses full data transfer. When there are local data changes, the entire data needs to be re-migrated, resulting in extremely low migration efficiency. To improve data migration efficiency, file migration methods have gradually emerged that partition data and construct data trees. When some data changes, the changed data blocks can be located in the data tree (the Diff algorithm can be used to locate differences), allowing the changed data blocks to be re-migrated in a targeted manner. However, these file migration methods typically use a fixed partitioning strategy, that is, all types of files use a unified partitioning strategy, without in-depth research on the characteristics of different file types. This leads to unreasonable partitioning strategies for at least some types of files, which in turn affects subsequent difference location. For example, semi-structured files such as Parquet and HFile in data warehouses and data lakes are often merged and stored. Fixed partitioning easily crosses logical record boundaries and cannot recognize the connections between different files, causing the Diff algorithm to fail and the migration to fall back to full transfer. Essentially, the partitioning logic of a single partitioning strategy doesn't match the storage logic of all data files. When the partitioning logic and storage logic differ, it's easy for the changed local data to be assigned to different data partitions, resulting in more differential data blocks. This reduces the efficiency of differential data block location and transmission. Furthermore, when there are too many differential data blocks, differential location becomes ineffective, forcing the differential data block transfer during data migration to be relegated to a full transfer, significantly reducing transmission efficiency.

[0004] There is currently no effective solution to the problem of low data migration efficiency caused by unreasonable data segmentation strategies in current data platform file migration methods. Summary of the Invention

[0005] The present invention provides a data platform file migration method, a computer program product, and a data platform to solve the problem of low data migration efficiency caused by unreasonable data block strategy in the current data platform file migration method.

[0006] In a first aspect, the present invention provides a data platform file migration method for migrating data files to a target end, comprising:

[0007] Scan the source data file to be migrated to obtain the type of the source data file;

[0008] If the type of the source data file is in the state set of the target reinforcement learning model, use the type of the source data file as the current state of the target reinforcement learning model, and obtain a current action of the target reinforcement learning model in response to the current state; wherein the state set of the target reinforcement learning model includes multiple file types, the action set of the target reinforcement learning model includes multiple file segmentation strategies; and the reward of the target reinforcement learning model includes the same ratio between data segments generated by the current file segmentation strategy before and after the local change of the data file of the current file type;

[0009] Using the current action of the target reinforcement learning model to divide the source data file into blocks;

[0010] Constructing a Merkle tree of the source data file based on the block result of the source data file;

[0011] Locate the difference data blocks by comparing the Merkle tree of the source data file and the target data file layer by layer;

[0012] Migrate the difference data block to the target end.

[0013] In a second aspect, the present invention provides a data platform file acquisition method for acquiring file data from a source end, comprising:

[0014] Scan the target end data file to be updated to obtain the type of the target end data file;

[0015] If the type of the target-side data file is in the state set of the target reinforcement learning model, use the type of the target-side data file as the current state of the target reinforcement learning model, and obtain a current action of the target reinforcement learning model in response to the current state; wherein the state set of the target reinforcement learning model includes multiple file types, the action set of the target reinforcement learning model includes multiple file segmentation strategies; and the reward of the target reinforcement learning model includes the same ratio between data segments generated by the current file segmentation strategy for a data file of the current file type before and after the local data is changed;

[0016] Using the current action of the target reinforcement learning model to divide the target end data file into blocks;

[0017] Constructing a Merkle tree of the target-end data file based on the block result of the target-end data file;

[0018] Locating the difference data blocks by comparing the Merkle tree of the target data file and the source data file layer by layer;

[0019] The difference data block is obtained from the source end.

[0020] In the third aspect, the present invention provides a data platform for storing data files. When the data platform is a source end, it adopts the data platform file migration method described in the first aspect to migrate the data files to the target end; when the data platform is a target end, it adopts the data platform file acquisition method described in the second aspect to obtain data files from the target end.

[0021] In a fourth aspect, the present invention provides a computer program product, which includes a computer program. When the computer program is executed, it implements the data platform file migration method described in the first aspect or the data platform file acquisition method described in the second aspect.

[0022] Compared with the related art, the data platform file migration method provided by the present invention only migrates data blocks with differences, thereby improving the data migration efficiency. It is necessary to block the source data file and the target data file to be migrated and construct a Merkle tree, and then compare the Merkle trees of the source and target ends to determine the difference data blocks. Compared with the same type of file migration method in the prior art, when the data file is blocked, a dynamic blocking strategy is adopted, that is, a more appropriate blocking strategy is adopted for different types of files. When the blocking logic matches the file storage logic, when the data file generates local data changes, fewer difference data blocks can be generated, which improves the comparison efficiency of the subsequent Merkle tree, can locate the difference data blocks more quickly, and can transmit fewer difference data blocks during data migration, thereby improving data transmission efficiency, and solving the problem of low data migration efficiency caused by unreasonable data blocking strategy in the current data platform file migration method.

[0023] The details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 is a flow chart of the data platform file migration method provided in this embodiment;

[0025] Figure 2 is a flow chart of the data platform file acquisition method provided in this embodiment;

[0026] Figure 3 is a flow chart of the data migration process provided in this embodiment;

[0027] Figure 4 This is an architectural diagram of the data migration system provided in this embodiment. DETAILED DESCRIPTION

[0028] In order to more clearly understand the purpose, technical solutions and advantages of the present application, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments.

[0029] Unless otherwise defined, technical or scientific terms used in this application shall have the ordinary meanings as understood by persons of ordinary skill in the art to which this application belongs. The terms "a," "an," "the," "these," and similar expressions in this application do not denote limitations on quantity and may be singular or plural. The terms "comprise," "include," "have," and any variations thereof, as used in this application, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or device comprising a series of steps or modules (units) is not limited to the listed steps or modules (units) but may include unlisted steps or modules (units) or other steps or modules (units) inherent to the process, method, product, or device. The terms "connected," "connected," "coupled," and similar expressions used in this application are not limited to physical or mechanical connections but may include electrical connections, whether direct or indirect. As used in this application, "plurality" means two or more. "And / or" describes an association between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone; A and B exist simultaneously; or B exists alone. Generally, the character " / " indicates that the objects in the preceding and following relationship are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0030] Migrating critical files to or from a data platform is often necessary for scenarios like enterprise cloud migration, data disaster recovery, and multi-data center collaboration. Generally speaking, the destination where data files are migrated is called the target, and the destination where data files are migrated is called the source. The following provides a data platform migration method, executed from the source.

[0031] In this embodiment, a data platform file migration method is provided. Figure 1 This is a flow chart of the data platform file migration method provided in this embodiment. Figure 1As shown, the process includes step S110, step S120, step S130, step S140, step S150 and step S160.

[0032] Step S110 : Scan the source data files to be migrated to obtain the types of the source data files.

[0033] The source side stores data files, which are defined as source data files. When migrating source data files, the source data files to be migrated are first scanned to determine their type. Specifically, the system first determines whether the source data file is a row-based data file. If so, the system then directly blocks the data by row, with n rows of data being considered a data block, where n is an integer greater than or equal to 1. If not, the system then determines whether the source data file is a database file, data lake file, or data warehouse file. If yes, read the configuration file of the source data file and read the file header to obtain the file format (type) of the source data file. The specific types of data files include B-tree index files, heterogeneous database files, Oracle database files, PostgreSQL database files, MySQL database files, data lake files and data warehouse files. These specific types are in the state set of the target reinforcement learning model. If no (that is, the type of the source data file is not in the state set of the target reinforcement learning model), continue to determine whether the source data file has a custom block strategy. If yes, then use the custom block strategy to block the source data file. The custom block strategy refers to a specific block strategy set in advance for some special file types. If not, use a block strategy in which the block size is inversely proportional to the modification frequency to block the source data file. The block strategy in which the block size is inversely proportional to the modification frequency is:

[0034]

[0035] MB is megabytes, S is the block size of data blocks, and N mod The number of historical modifications to the source data file.

[0036] In step S120, if the type of the source data file is in the state set of the target reinforcement learning model, the type of the source data file is used as the current state of the target reinforcement learning model, and the current action of the target reinforcement learning model in response to the current state is obtained. The state set of the target reinforcement learning model includes multiple file types, and the action set of the target reinforcement learning model includes multiple file segmentation strategies. The reward of the target reinforcement learning model includes the same ratio (defined as the block cache hit rate) between the data segments generated by the current file segmentation strategy before and after a local change for a data file of the current file type. By setting this reward, the target reinforcement learning model can match the optimal action (file segmentation strategy) for each state (file type). The principle is that when the segmentation of a data file of a certain type is reasonable, the number of data segments that result in corresponding data changes after a local data change should be as small as possible. For example, for data files using a logical page storage structure, related data is stored in the same logical page, and a single local data change typically occurs in the same logical page. When using a logical page partitioning strategy, a single local data change will only cause a change to a single data block. In this case, only that single data block can be migrated in subsequent data migrations, minimizing the number of migrations. When using other partitioning strategies, the same logical page may be divided into two or more data blocks. A single local data change may cause changes to multiple data blocks. In this case, multiple data blocks need to be migrated in subsequent data migrations, increasing the number of migrations. Therefore, a higher block cache hit rate indicates that the partitioning logic of the current file partitioning strategy tends to be exactly the same as the storage logic of the current file type, improving the accuracy of subsequent difference location and minimizing the number of subsequent data block migrations.

[0037] The aforementioned reward includes the similarity ratio between the data blocks generated by the current file segmentation strategy before and after the local data change for a data file of the current file type. The higher the similarity ratio between the two sets of data blocks, the lower the difference ratio. In this case, the target reinforcement learning model can obtain a higher reward. By continuously traversing different file types as the current file type and continuously traversing different file segmentation strategies as the current file segmentation strategy, the target reinforcement learning model can be driven to match the most reasonable file segmentation strategy for each file type. For example, if a data file of file type A is segmented using file segmentation strategy B before and after the local data change, resulting in 100 data blocks, assuming there are 97 identical data blocks and 3 different data blocks, the similarity ratio between the data blocks (block cache hit rate) is 97%. Alternatively, if a data file of file type A is segmented using file segmentation strategy C before and after the local data change, resulting in 50 data blocks, assuming there are 49 identical data blocks and 1 different data block, the similarity ratio between the data blocks (block cache hit rate) is 98%. For data files of file type A, when the C file chunking strategy is adopted, the target reinforcement learning model can obtain higher rewards, so the target reinforcement learning model tends to match the C file chunking strategy for data files of file type A.

[0038] By setting the above rewards, the target reinforcement learning model can match the best action for each state.

[0039] For example, the various file types include B-tree index files, heterogeneous database files, Oracle database files, PostgreSQL database files, MySQL database files, data lake files, and data warehouse files. The various file segmentation strategies include timestamp segmentation, logical page boundary segmentation, a mixed segmentation strategy of logical page boundaries and nodes, and a fixed segmentation strategy. The target reinforcement learning model is a DQN agent consisting of a three-layer convolutional neural network.

[0040] Specifically, based on the DQN method in reinforcement learning, a three-layer convolutional neural network agent is trained. Various file types are used as states, various block strategies are used as action sets, and the data synchronization block cache hit rate after data changes using the block strategy is used as a reward to drive the agent to make decisions. The agent is implemented by constructing a three-layer shallow neural network based on the convolutional neural network. The states here can be divided into: B-tree index files, heterogeneous database files, Oracle database files, PostgreSQL database files, MySQL database files, data lake and data warehouse files. The corresponding block strategies can be classified as follows:

[0041] Timestamp block strategy: For example, the appended content every 10 minutes is divided into a logical block, and the block boundary is set at the end of the log line to avoid cross-line splitting.

[0042] Logical page boundary segmentation: According to the preset file format parsing algorithm, the file header is read, and the file header is a separate block. Then the content data is read and the content is segmented according to the preset parsing algorithm for different formats, and the segment hash value and related metadata are recorded.

[0043] For data files such as Oracle, PostgreSQL, and MySQL database files, logical page alignment and block division are implemented. For some data files that use a logical page storage structure, related data is stored in the same logical page, and there are numerous "data holes" (i.e., empty pages not occupied by valid data) caused by deletion operations. By strictly aligning blocks to logical page boundaries (for example, according to Oracle's DB_BLOCK_SIZE or MySQL's 16KB page size), data integrity within a single page is ensured. This approach offers the following advantages: improved cache hit rates, strong data correlation within the same logical page, and easier triggering of locality during post-migration access, reducing cache invalidations; improved compression efficiency: the data structure within the logical page is regular, and empty areas can be efficiently identified and skipped for transmission. Combined with compression algorithms (such as Zstandard), this significantly reduces the amount of data transmitted; and improved difference detection accuracy: single-page modifications only affect the corresponding blocks, avoiding the difference detection failures caused by traditional fixed blocks crossing logical boundaries.

[0044] Optimizing logical page chunking for columnar storage in data lakes and warehouses. This solution addresses the frequent multi-file merge scenarios in data lakes (such as Parquet files) and data warehouses. The merged file data remains unchanged, but the metadata changes. Traditional chunking algorithms, limited to single-file comparisons, fail to identify consistent column chunks across files, leading to duplicate transfers. This solution improves file schema parsing to extract common column chunks across files. This decouples the chunking strategy from the merge logic, ensuring that when merging newly added files, only the newly added column chunks, rather than the entire file, need to be compared.

[0045] Mixed blocking of logical page boundaries and nodes: Data files are blocked according to the page size of tablespace files, and index files are blocked using logical page boundaries. Combined with the hierarchical characteristics of the B-tree structure, blocking is done in nodes and data blocks.

[0046] Index files, such as MySQL and PostgreSQL index files, are partitioned into B-Tree nodes. Index files use a hierarchical B-Tree structure, with each node containing a key value range and child node pointers. The partitioning strategy uses B-Tree nodes as the smallest unit. B-Tree updates typically affect only a limited number of nodes. After partitioning, only the changed nodes need to be transmitted, not the entire index file.

[0047] Fixed block: Split the file into fixed blocks.

[0048] First, the DQN agent is trained using pre-training data, and then the block strategy is selected based on the trained agent for actual operation.

[0049] For example, the block strategy It can be expressed as follows:

[0050]

[0051] in, 、 and Represents database files, data lake files, and data warehouse files respectively, and f represents the data file to be partitioned. 、 、 、 and They represent the timestamp blocking strategy, the logical page boundary blocking strategy, the logical page boundary and node mixed blocking strategy, the fixed blocking strategy, and the blocking strategy where the block size is inversely proportional to the modification frequency.

[0052] The above-mentioned block strategy is further described as follows.

[0053] Data files are logically page-aligned and partitioned. For some data files that use a logical page storage structure, related data is stored in the same logical page, and there are numerous "data holes" (i.e., empty pages not occupied by valid data) caused by deletion operations. By strictly aligning partitions to logical page boundaries (e.g., according to Oracle's DB_BLOCK_SIZE or MySQL's 16KB page size), data integrity within a single page is ensured. This approach offers the following advantages: improved cache hit rates, strong data association within the same logical page, and easier triggering of the principle of locality during post-migration access, reducing cache failures; optimized compression efficiency: the data structure within the logical page is regular, and empty areas can be efficiently identified and skipped for transmission. Combined with compression algorithms (such as Zstandard), this significantly reduces the amount of data transmitted; and improved difference detection accuracy: single-page modifications only affect the corresponding blocks, avoiding the difference detection failures caused by traditional fixed blocks crossing logical boundaries.

[0054] Index files are divided into B-Tree nodes. The index file uses a hierarchical B-Tree structure, with each node containing a key range and child node pointers. The block strategy uses the B-Tree node as the smallest unit. B-Tree updates typically only affect a local node. After block division, only the changed nodes need to be transmitted, not the entire index file.

[0055] Data lake / warehouse columnar storage block optimization addresses the frequent multi-file merge scenarios in data lakes (such as Parquet files) and data warehouses. The merged file data remains unchanged, but the metadata changes. Traditional block partitioning algorithms, limited to single-file comparisons, cannot identify consistent column blocks across files, leading to duplicate transfers. This solution improves file schema parsing to extract common column blocks across files. This decouples the block partitioning strategy from the merge logic, ensuring that when merging newly added files, only the newly added column blocks need to be compared, not the entire file.

[0056] Step S130: Use the current action of the target reinforcement learning model to segment the source data file. The current action of the target reinforcement learning model is the segmentation strategy selected by the target reinforcement learning model for the source data file, and the source data file is segmented according to the segmentation strategy.

[0057] Step S140: constructing a Merkle tree of the source data file based on the block division result of the source data file.

[0058] The block result of the source data file includes the data blocks generated after the source data file is block-wise. The specific construction process of the Merkle tree is as follows:

[0059] Calculate the hash value of each data block in the block result of the source data file and store metadata, which includes the hash value, block size and starting offset of each data block and the type identifier of the source data file; use the hash value of each data block as the bottom node in the Merkle tree, and build the upper node in sequence until the root node; in the Merkle tree, an upper node is the hash value of at least two lower nodes. For example, assuming there are 10 data blocks, there are 10 nodes in the bottom layer of the Merkle tree, which are the hash values ​​of the 10 data blocks respectively. Then, the hash value of each two bottom nodes can be hashed to obtain 5 nodes in the second-to-last layer, and so on until the root node is generated.

[0060] Among them, the starting offset of the data block is:

[0061]

[0062] Where S is the block size of the data block, H is the file header size of the source data file, and filesize is the file size of the source data file.

[0063] Step S150 , locating the difference data blocks by comparing the Merkle trees of the source data file and the target data file layer by layer through hashing.

[0064] Correspondingly, on the target side, the data stored there uses the same Merkle tree construction method to obtain the corresponding Merkle tree, making it easier for both ends to compare. The layer-by-layer hash comparison process is as follows: first compare the root node. If the root node is the same, it means there are no different data blocks. If the root node is different, the different nodes are determined in the lower nodes. If there are different lower nodes, different lower nodes are determined. Until the bottom node, it can be determined that there are different data blocks.

[0065] Merkle tree comparison function for:

[0066]

[0067] Where k is the branching factor of the tree, h is the height of the tree, the number of comparisons is c, and the recursive process satisfies:

[0068]

[0069] The comparison complexity of the above Merkle tree is significantly better than the complexity of traditional full comparison.

[0070] Step S160: Migrate the difference data blocks to the target end.

[0071] During the data migration process, only the data blocks with differences are migrated, which greatly improves the data migration efficiency compared to full data migration.

[0072] In summary, the data platform file migration method provided by this embodiment only migrates data blocks with differences, thereby improving the efficiency of data migration. It is necessary to divide the source data file and the target data file to be migrated into blocks and construct a Merkle tree, and then compare the Merkle trees of the source and target ends to determine the difference data blocks. Compared with the same type of file migration method in the prior art, when dividing the data file into blocks, a dynamic block strategy is adopted, that is, a more appropriate block strategy is adopted for different types of files. When the block logic matches the file storage logic, when the data file generates local data changes, fewer difference data blocks can be generated, which improves the comparison efficiency of the subsequent Merkle tree, can locate the difference data blocks more quickly, and can transmit fewer difference data blocks during data migration, thereby improving data transmission efficiency, and solving the problem of low data migration efficiency caused by unreasonable data block strategy in the current data platform file migration method.

[0073] If the source data file contains multiple types, multiple Merkle trees can be created for each type (using the same construction strategy for the target data file). This reduces the depth of the Merkle tree and improves construction speed. Differences are quickly located by comparing the root hashes layer by layer. If the root hashes differ, the hashes of the child nodes are compared layer by layer, recursively locating the difference subtree. Finally, a list of difference blocks corresponding to all leaf nodes in the difference subtree is extracted. Only the hash values, metadata, and data content of these blocks are transmitted, and the Merkle tree on the target is updated. During transmission, each block is independently encrypted, and the key is generated using the TLS 1.3 key exchange protocol. A cache is used on the target to store data blocks, avoiding the need to transfer already cached blocks. A resumable transmission mechanism is implemented, recording the hash list of transferred blocks in a local log file. If interrupted, the log is read to compare the incomplete blocks and resume the transmission.

[0074] Furthermore, if there are multiple versions of data files on the source side, multiple versions of the Merkle tree can be constructed for each version of the data file, allowing users to selectively select different versions of the data files on the source side to update the target side. Each version generates a unique identifier (such as v1.0) and stores its corresponding Merkle tree root hash and version metadata (creation time, migration task ID). An example of version metadata is as follows:

[0075]

[0076] The rollback process loads the root hash of the corresponding Merkle tree based on the version number. The target's current tree is compared with the source's historical tree to locate the difference blocks that need to be restored. The difference blocks are then retransmitted from the cache or source, overwriting the target's data files.

[0077] At the same time, expired data cleaning and lifecycle management are carried out. Between different versions of the Merkle tree, a log file will be recorded to compare which data blocks were deleted in the previous version. When the current version of the Merkle tree expires or is deleted, the corresponding cached data blocks will be found and deleted.

[0078] Therefore, this embodiment uses a dynamic block strategy to adaptively select block modes (such as timestamp block, page alignment block, logical page block) for line-based files, heterogeneous databases, and data lake warehouse file types to avoid the problem of difference detection failure caused by cutting across logical boundaries. Combined with the Merkle tree layer-by-layer hash comparison mechanism, only the difference block data needs to be transmitted, which significantly reduces network bandwidth consumption. The Merkle tree root hash is deeply bound to the migration version to achieve the atomicity of migration operations and version management, support rapid positioning of difference blocks after interruption and resume transmission without full retransmission, and significantly improve recovery efficiency. By recording the deletion log of the difference blocks between versions and combining it with the cache mechanism, the automatic cleaning of expired data is achieved, reducing storage resource usage. At the same time, it supports the coexistence management of multiple versions to meet data auditing and historical tracing needs. In summary, the present invention has achieved improvements in version control, dynamic adaptability, and resource utilization.

[0079] It should be noted that the data migration method in this embodiment requires constructing a Merkle tree in the same way in the source end and the target end. Furthermore, this embodiment also provides a data platform file acquisition method, which is executed by the target end in data migration and is used to obtain file data from the source end, which includes step S210, step S220, step S230, step S240, step S250 and step S260.

[0080] Step S210 , scanning the target-end data file to be updated to obtain the type of the target-end data file.

[0081] In step S220, if the type of the target-side data file is in the state set of the target reinforcement learning model, the type of the target-side data file is used as the current state of the target reinforcement learning model, and the current action of the target reinforcement learning model in response to the current state is obtained; wherein the state set of the target reinforcement learning model includes B-tree index files, heterogeneous database files, Oracle database files, PostgreSQL database files, MySQL database files, data lake files, and data warehouse files; the action set of the target reinforcement learning model includes a timestamp blocking strategy, a logical page boundary blocking strategy, a logical page boundary and node mixed blocking strategy, and a fixed blocking strategy.

[0082] Step S230: divide the target end data file into blocks using the current action of the target reinforcement learning model.

[0083] Step S240: constructing a Merkle tree of the target-end data file based on the block division result of the target-end data file.

[0084] Step S250 , locating the difference data blocks by comparing the Merkle trees of the target data file and the source data file layer by layer through hashing.

[0085] Step S260: Obtain the difference data block from the source end.

[0086] The Merkle tree construction method in this data platform file acquisition method is exactly the same as the data platform file migration method. The only difference is the processing object: the source data file to be migrated in the data platform file migration method, while the target data file to be updated in the data platform file acquisition method. Therefore, the Merkle tree construction method will not be detailed here.

[0087] As mentioned above, the data migration process has been described from the perspective of the source or target end. Figure 3 , the overall process of data migration in this embodiment is:

[0088] 1. Scan metadata. On the source side, the metadata is the metadata of the source data file. On the target side, the metadata is the metadata of the target data file.

[0089] 2. Determine whether the data file is a row-based data file. If so, proceed to step 3; otherwise, proceed to step 4.

[0090] 3. Divide the data file into blocks according to the row block strategy.

[0091] 4. Determine whether the data file is a database file. If so, proceed to step 6; otherwise, proceed to step 5.

[0092] 5. Determine whether the data file is a data lake or data warehouse file. If so, proceed to step 6; otherwise, proceed to step 7.

[0093] 6. Read the file header of the configuration file of the data file, identify the file format of the data file and provide it to the DQN agent, and the DQN agent will provide a block strategy, which includes timestamp block strategy, logical page boundary block strategy, logical page boundary and node mixed block strategy, and fixed block strategy.

[0094] 7. Determine whether the data file has a custom block interface. If so, execute step 8; otherwise, execute step 9.

[0095] 8. Use custom interface block strategy to block data files.

[0096] 9. Define the data file as a normal file and divide the data file into blocks according to the update frequency block strategy (block size is inversely proportional to the modification frequency).

[0097] Reference Figure 4In this embodiment, a data system is also provided, including a source file system and a target file system, both of which include a metadata scanning module, a dynamic block module, and a comparison module. The metadata scanning module is used to scan metadata to obtain the type of the data file (perform step 1 above). The dynamic block module is used to dynamically block the data file to construct a corresponding Merkle tree (perform steps 2-9 above). The comparison module is used to compare the Merkle trees on the source and target ends to determine different data blocks. The cache module is used to manage the cache of transmitted data blocks, avoid duplicate transmission, and support the cleanup of expired data.

[0098] This embodiment also provides a data platform for storing data files. When the data platform is a source, it uses the data platform file migration method provided in this embodiment to migrate data files to a target. When the data platform is a target, it uses the data platform file acquisition method provided in this embodiment to acquire data files from the target.

[0099] In this embodiment, a computer program product is also provided. The computer program product includes a computer program. When the computer program is executed, the data platform file migration method or the data platform file acquisition method provided in this embodiment is implemented.

[0100] It should be understood that the specific embodiments described herein are only used to explain this application and are not used to limit it. Based on the embodiments provided in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0101] Obviously, the accompanying drawings are merely examples or embodiments of the present application. A person skilled in the art can also apply the present application to other similar situations based on these drawings without inventive effort. Furthermore, it is understandable that, although the work involved in this development process may be complex and lengthy, certain design, manufacturing, or production changes based on the technical content disclosed in this application are merely routine technical means for a person skilled in the art and should not be considered to constitute a deficiency in the disclosure of the present application.

Claims

1. A data platform file migration method for migrating data files to a target end, characterized in that: include: Scan the source data file to be migrated to obtain the type of the source data file; If the type is in the state set of the target reinforcement learning model, the type is used as the current state of the target reinforcement learning model, and a current action of the target reinforcement learning model in response to the current state is obtained; wherein the state set of the target reinforcement learning model includes multiple file types, and the action set of the target reinforcement learning model includes multiple file segmentation strategies; the reward of the target reinforcement learning model includes the same ratio between data segments generated by the current file segmentation strategy before and after the local data change of the data file of the current file type: the higher the same ratio between the two groups of data segments, the lower the difference ratio, and the higher the reward obtained by the target reinforcement learning model; Using the current action of the target reinforcement learning model to divide the source data file into blocks; Constructing a Merkle tree of the source data file based on the block result of the source data file; Locate the difference data blocks by comparing the Merkle tree of the source data file and the target data file layer by layer; Migrate the difference data block to the target end.

2. The data platform file migration method according to claim 1, characterized in that: Also includes: Before scanning the source data file to be migrated, determining whether the source data file is a row-based data file; If yes, then the source data file is divided into blocks by row; if no, then the source data file is scanned; If the type of the source data file is not in the state set of the target reinforcement learning model, determine whether the source data file has a custom block strategy; if so, use the custom block strategy to block the source data file; if not, use a block strategy in which the block size is inversely proportional to the modification frequency to block the source data file.

3. The data platform file migration method according to claim 1, characterized in that: The target reinforcement learning model is a DQN agent comprising a three-layer convolutional neural network.

4. The data platform file migration method according to claim 1, characterized in that: The multiple file types include B-tree index files, heterogeneous database files, Oracle database files, PostgreSQL database files, MySQL database files, data lake files and data warehouse files, and the multiple file blocking strategies include timestamp blocking strategy, logical page boundary blocking strategy, logical page boundary and node mixed blocking strategy and fixed blocking strategy.

5. The data platform file migration method according to claim 1, characterized in that: Constructing the Merkle tree of the source data file based on the block result of the source data file includes: Calculating a hash value for each data block in the block result of the source data file, and storing metadata, the metadata including the hash value, block size and starting offset of each data block, and a type identifier of the source data file; The hash value of each data block is used as the bottom node in the Merkle tree, and the upper nodes are constructed in sequence until the root node; in the Merkle tree, an upper node is the hash value of at least two lower nodes.

6. The data platform file migration method according to claim 5, characterized in that: The starting offset of the data block for: Where S is the block size of the data block, H is the file header size of the source data file, and filesize is the file size of the source data file.

7. The data platform file migration method according to claim 2, characterized in that: The chunking strategy where chunk size is inversely proportional to modification frequency is: MB is megabytes, S is the block size of data blocks, and N mod The number of historical modifications to the source data file.

8. A data platform file acquisition method for acquiring file data from a source end, characterized in that: include: Scan the target end data file to be updated to obtain the type of the target end data file; If the type of the target-side data file is in the state set of the target reinforcement learning model, the type of the target-side data file is used as the current state of the target reinforcement learning model, and a current action of the target reinforcement learning model in response to the current state is obtained; wherein the state set of the target reinforcement learning model includes multiple file types, and the action set of the target reinforcement learning model includes multiple file segmentation strategies; the reward of the target reinforcement learning model includes the same ratio between data segments generated by the current file segmentation strategy before and after the local data change of the data file of the current file type: the higher the same ratio between the two groups of data segments, the lower the difference ratio, and the target reinforcement learning model obtains a higher reward; Using the current action of the target reinforcement learning model to divide the target end data file into blocks; Constructing a Merkle tree of the target-end data file based on the block result of the target-end data file; Locating the difference data blocks by comparing the Merkle tree of the target data file and the source data file layer by layer; The difference data block is obtained from the source end.

9. A data platform for storing data files, characterized by: When the data platform is a source end, it adopts the data platform file migration method described in any one of claims 1 to 7 to migrate the data file to the target end; When the data platform is the target end, it adopts the data platform file acquisition method described in claim 8 to acquire the data file from the target end.

10. A computer program product, comprising a computer program, characterized in that: When the computer program is executed, the data platform file migration method described in any one of claims 1 to 7 or the data platform file acquisition method described in claim 8 is implemented.

Images