Data extraction method and device, electronic equipment, storage medium and program product

By splitting and updating entry points during the data extraction process, the inefficiency problem caused by uneven partitioning of source disk data is solved, and the overall efficiency of data extraction is improved.

CN119046358APending Publication Date: 2024-11-29PICC INFORMATION TECH CO LTD
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN202411141285.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-19
Publication Date
2024-11-29

AI Technical Summary

Technical Problem

During the data extraction process, due to the uneven distribution of attributes of the source disk data, the amount of partitioned data is uneven, the thread concurrency is low, and the overall efficiency is reduced.

Method used

By determining whether the content of the extraction task exists in the target disk, if it does not exist, partition the source disk data in an equal way, determine the entry point, and adopt multi-threaded parallel extraction based on the entry point; if it exists, update the entry point to optimize the partition. Ensure that the partition data volume is basically consistent.

Benefits of technology

The concurrency of multi-threaded parallel extraction is improved and the overall efficiency of data extraction is improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119046358A_ABST
    Figure CN119046358A_ABST
Patent Text Reader

Abstract

The embodiment of the invention provides a data extraction method and device, electronic equipment, a storage medium and a program product. The method comprises the following steps: in response to an extraction task for a source disk, partitioning in an equal division manner when extraction content corresponding to the extraction task does not exist in a target disk so as to obtain an entry point of each partition, and updating the entry point of each partition through the extraction content when the extraction content corresponding to the extraction task exists in the target disk, and extracting the data in the source disk by adopting a multi-thread parallel extraction mode based on the entry point of each partition, and storing the extracted content to the target disk. Based on the method provided by the invention, the concurrency of multi-thread parallel extraction can be improved, so that the overall efficiency of data extraction of the source disk is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of big data, and in particular to a data extraction method, device, electronic device, storage medium and program product. Background Art

[0002] When extracting data from the source disk to the target disk, the server where the source disk is located is generally still running, and new data will be continuously generated on the source disk. Therefore, in the process of extracting data from the source disk, the amount of data already in the source disk before the extraction and the incremental data generated during the extraction process are often extracted to the target disk together.

[0003] Currently, the common solution is to partition the data in the source disk according to a certain attribute, such as partitioning according to generation time, and then use multi-threading to concurrently extract data from multiple partitions.

[0004] However, if the data in the source disk is unevenly distributed according to attributes, the amount of data in each partition will be uneven, and there will even be partitions with no data. The thread concurrency is low, resulting in low overall efficiency in extracting data from the source disk. Summary of the invention

[0005] The embodiments of the present application provide a data extraction method, device, electronic device, storage medium and program product to improve the efficiency of data extraction.

[0006] In a first aspect, an embodiment of the present application provides a data extraction method, the method comprising:

[0007] In response to an extraction task for a source disk, determining whether extraction content corresponding to the extraction task exists in a target disk;

[0008] If the extraction content corresponding to the extraction task does not exist in the target disk, the existing data in the source disk is partitioned in an equal manner to obtain the entry point of each partition;

[0009] Based on the entry point of each partition, the data in the source disk is extracted by multi-threaded parallel extraction, and the extracted content is stored in the target disk. The extracted content includes the data already on the source disk when the extraction task starts to execute and the incremental data generated during the execution of the extraction task;

[0010] If the extraction content corresponding to the extraction task exists in the target disk, the entry point of each partition is updated through the extraction content;

[0011] Based on the updated entry point of each partition, the data in the source disk is re-extracted using a multi-threaded parallel extraction method, and the extracted content is updated to the target disk.

[0012] In a possible implementation, determining whether there is extraction content corresponding to the extraction task in the target disk includes:

[0013] Probing the target disk to determine whether there is a historical extraction record of the source disk in the target disk, the historical extraction record including the entry point of each partition corresponding to the extraction task and the completion time of the extraction task;

[0014] If it exists, it is determined that the extraction content corresponding to the extraction task exists in the target disk;

[0015] If it does not exist, it is determined that the extraction content corresponding to the extraction task does not exist in the target disk.

[0016] In a possible implementation, before partitioning the existing data in the source disk in an equal manner, the method further includes:

[0017] Probe the source disk to count the amount of data already in the source disk.

[0018] In a possible implementation, partitioning the existing data in the source disk in an equal manner includes:

[0019] The existing data volume in the source disk is equally divided according to the preset partition quantity to obtain corresponding multiple partitions;

[0020] Determine the size relationship between the amount of data in the partition and the preset value. If the amount of data corresponding to the partition exceeds the preset value, divide the amount of data in each partition equally again according to the preset number of partitions to obtain new partitions, and determine the size relationship between the amount of data in the new partition and the preset value, and so on, until the amount of data corresponding to the last partition is equal to the preset value;

[0021] If the data volume of a partition is less than the preset value, the partitions are merged until the data volume corresponding to the final partition is equal to the preset value.

[0022] In a possible implementation, partitioning the existing data in the source disk in an equal manner includes:

[0023] The existing data in the source disk is equally divided according to the preset data volume to obtain corresponding multiple partitions.

[0024] In a possible implementation, data in the source disk is extracted by multi-threaded parallel extraction based on the entry point of each partition, including:

[0025] Mark the starting point of each partition obtained as the entry point of the corresponding partition;

[0026] Obtain the number of entry points in the source disk, and determine the number of threads based on the number of entry points;

[0027] Each thread is assigned to a corresponding entry point, so that each thread starts extracting data from the entry point and starts extracting data from the corresponding partition in sequence until it reaches the next entry point and stops.

[0028] In a possible implementation, updating the entry point of each partition by extracting content includes:

[0029] Get the entry point of each partition corresponding to the last extraction task from the historical extraction records;

[0030] Based on the incremental data in the extracted content corresponding to the last extraction task, the ntile function is used to adaptively adjust the entry point corresponding to the last extraction task to update the entry point of each partition.

[0031] In a second aspect, an embodiment of the present application provides a data extraction device, the device comprising:

[0032] A response module, used to respond to the extraction task of the source disk and determine whether there is extraction content corresponding to the extraction task in the target disk;

[0033] A processing module, for partitioning the existing data in the source disk in an equal manner to obtain an entry point for each partition if the extraction content corresponding to the extraction task does not exist in the target disk, and extracting the data in the source disk by multi-threaded parallel extraction based on the entry point of each partition, and storing the extracted content in the target disk, wherein the extracted content includes the existing data in the source disk when the extraction task starts to be executed and the incremental data generated during the execution of the extraction task;

[0034] The processing module is also used to update the entry point of each partition through the extracted content if there is extracted content corresponding to the extraction task in the target disk, and re-extract the data in the source disk by multi-threaded parallel extraction based on the updated entry point of each partition, and update the extracted content to the target disk.

[0035] In a third aspect, an embodiment of the present application provides a computer device, including: a memory, a processor;

[0036] Memory stores computer-executable instructions;

[0037] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the above first aspect and / or various possible implementations of the first aspect.

[0038] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the first aspect above and / or various possible implementations of the first aspect.

[0039] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the above first aspect and / or various possible implementation methods of the first aspect.

[0040] The data extraction method, device, electronic device, storage medium and program product provided by the embodiments of the present application partition the existing data in the source disk by dividing it equally, and obtain the entry point corresponding to each partition of the current data extraction, and obtain the entry point corresponding to each partition of the current data extraction by updating the entry point of each partition last time, when the source disk has been extracted. In this way, the amount of data in the obtained partitions is basically the same, which can improve the concurrency of multi-threaded parallel extraction, thereby improving the overall efficiency of data extraction from the source disk. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0042] Figure 1 A schematic diagram of a scenario for this application example;

[0043] Figure 2 Schematic diagram of the data extraction method provided for this application Figure 1 ;

[0044] Figure 3 A flowchart of partitioning a source disk is shown as an example;

[0045] Figure 4 Execute the flow chart for the extraction task of the example;

[0046] Figure 5 Schematic diagram of the data extraction method provided for this application Figure 2 ;

[0047] Figure 6 Schematic diagram of the data extraction method provided for this application Figure 3 ;

[0048] Figure 7 Another example of a flow chart for partitioning a source disk;

[0049] Figure 8 Schematic diagram of the data extraction method provided for this application Figure 4 ;

[0050] Fig. 9 The following is a schematic diagram of the partitions of the source disk used as an example;

[0051] Fig.10 Schematic diagram of the data extraction method provided for this application Figure 5 ;

[0052] Fig.11 Update the schematic diagram with the entry point of the source disk for the example;

[0053] Fig.12 Another example of a flow chart for partitioning a source disk;

[0054] Fig.13 A schematic diagram of the structure of the data extraction device provided by this application;

[0055] Fig.14 A schematic diagram of the structure of the computer device provided for this application.

[0056] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0057] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0058] Figure 1 A schematic diagram of a scenario for this application example, such as Figure 1 As shown, the specific application scenario of the present application mainly includes a source disk, a target disk and multiple threads, wherein multiple threads are used to extract data from the source disk to the target disk. In the process of extracting data from the source disk to the target disk, the server where the source disk is located is generally still in operation, so multiple threads often extract the amount of data in the source disk before the extraction together with the incremental data generated during the extraction process to the target disk. Since there are multiple threads to extract data from the source disk, the data in the source disk can be partitioned first, and the partition corresponding to each thread can be determined, and then the data in each partition can be extracted by multi-threaded concurrent extraction.

[0059] In combination with the above scenario, it can be seen that in the prior art, the data in the source disk can be partitioned according to a certain attribute, such as partitioning according to the generation time. However, if the data in the source disk is unevenly distributed according to the attribute, it will lead to uneven data volume in each partition, or even partitions without data, low thread concurrency, and thus low overall efficiency of extracting data from the source disk.

[0060] The data extraction method provided by the present application first determines whether there is extraction content corresponding to the extraction task in the target disk in response to the extraction task of the source disk; when there is no extraction content corresponding to the extraction task in the target disk, the existing data in the source disk is partitioned by equal division to obtain the entry point of each partition; based on the entry point of each partition, the data in the source disk is extracted by multi-threaded parallel extraction, and the extracted content is stored in the target disk; when there is extraction content corresponding to the extraction task in the target disk, the entry point of each partition is updated by the extraction content; based on the updated entry point of each partition, the data in the source disk is re-extracted by multi-threaded parallel extraction, and the extracted content is updated to the target disk. The present application divides the amount of data in the source data into multiple partitions by equal division, which can ensure that the amount of data in the obtained partitions is basically consistent, and can improve the concurrency of multi-threaded parallel extraction, thereby improving the overall efficiency of data extraction from the source disk.

[0061] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0062] Figure 2 Schematic diagram of the data extraction method provided for this application Figure 1 ,like Figure 2 As shown, the method includes:

[0063] S201. In response to an extraction task for a source disk, determine whether there is extraction content corresponding to the extraction task in a target disk.

[0064] The executor of this embodiment is a data extraction device, and the data extraction processing device can be implemented through a computer program, such as application software, etc.; or, it can also be implemented as a medium storing relevant computer programs, such as a U disk, a cloud disk, etc.; or, it can also be implemented through a physical device integrated or installed with relevant computer programs, such as a chip, etc.

[0065] Combined with the scene example, Figure 3A flowchart for partitioning a source disk is shown as an example. Figure 3 As shown, the extraction task of the source disk represents extracting all the data in the source disk and storing the extracted data in the target disk. Before extracting the data in the source disk, the exploration task of the target disk can be performed to determine whether there is relevant extraction content corresponding to the extraction task in the target disk, such as exploring the Hive of the target disk to determine whether there is relevant extraction content corresponding to the extraction task. Hive is a data warehouse framework built on Hadoop, which uses the distributed file system (Hadoop Distributed FileSystem, referred to as HDFS) to store data. The relevant extraction content corresponding to the extraction task represents the data in the source disk.

[0066] S202: If the extraction content corresponding to the extraction task does not exist in the target disk, the existing data in the source disk is partitioned in an equal manner to obtain an entry point for each partition.

[0067] Combined with scenario examples, such as Figure 3 As shown, if there is no extraction content corresponding to the extraction task in the target disk, it means that the data in the source disk corresponding to the extraction task has not been extracted before, that is, the extraction of the data from the source disk this time is the first extraction. When the source disk is extracted for the first time, the data in the source disk is not partitioned, then the data in the source disk is partitioned based on the amount of data in the source disk, and the entry point corresponding to each partition is determined. For example, the data in the source disk can be partitioned in an equal-division manner to obtain the equal-depth bucket information of the source disk, and the equal-depth bucket information of the source disk can be stored in the metadata of the extraction task, so that the equal-depth bucket information of the source disk can be obtained from the metadata later when the extraction task is executed. Among them, the equal-depth bucket represents the division of data into intervals containing approximately the same number of data points, the equal-depth bucket information can represent the endpoints of each interval, and the entry point of each partition can be obtained based on the equal-depth bucket information of the source disk.

[0068] S203, based on the entry point of each partition, the data in the source disk is extracted by multi-threaded parallel extraction, and the extracted content is stored in the target disk. The extracted content includes the data already existing on the source disk when the extraction task starts to execute and the incremental data generated during the execution of the extraction task.

[0069] Combined with the scene example, Figure 4 The extraction task execution flow chart for the example is as follows: Figure 4As shown, after the extraction task of the source disk starts to execute, the equal-depth bucket information of the source disk is obtained in the metadata of the extraction task to obtain the entry point of each partition. If the number of partitions is equal to the number of threads, and each thread is responsible for extracting the corresponding data in a partition, each thread can start from the entry point of the corresponding partition, and extract the data in the partition concurrently according to the extraction direction, and store the extracted data in the target disk. It is worth mentioning that if the server where the source disk is located is still in working state during the process of extracting data from the source disk, new data will be formed in the source disk in real time. After the thread completes the extraction of data in the corresponding partition, the extraction of the new data formed in the source disk is executed until the extraction of all data in the source disk is completed, and the extraction task for the source disk is ended.

[0070] S204: If the extraction content corresponding to the extraction task exists in the target disk, the entry point of each partition is updated through the extraction content.

[0071] Combined with scenario examples, such as Figure 3 As shown, if there is extraction content corresponding to the extraction task in the target disk, it means that the extraction of data in the source disk corresponding to the extraction task has been completed before, so the metadata contains the equal-depth bucket information corresponding to the last data extraction of the source disk. The server where the source disk is located is still in working state, so when the data in the source disk is extracted this time, the last equal-depth bucket information is inaccurate, but in order to improve efficiency, the method of re-exploring and re-partitioning the source disk is not adopted. The incremental data generated during the execution of the extraction task last time can be determined based on the extraction content in the target disk and the amount of data on the source disk detected when the extraction task was executed last time. The equal-depth bucket information of the last extraction task is adaptively updated through the generated incremental data to obtain the equal-depth bucket information of the source disk of this extraction task, and the equal-depth bucket information of the source disk is updated to the metadata of the extraction task. The entry point of each zone is adjusted through the equal-depth bucket information of the source disk this time to complete the update of the entry point.

[0072] S205 , re-extracting the data in the source disk by using a multi-threaded parallel extraction method based on the updated entry point of each partition, and updating the extracted content to the target disk.

[0073] Combined with scenario examples, such as Figure 4As shown, similarly, after the extraction task of the source disk starts to execute, the equal-depth bucket information of this source disk is obtained in the metadata of the extraction task to obtain a new entry point for each partition. If the number of partitions is equal to the number of threads, and each thread is responsible for extracting the corresponding data in a partition, then each thread can start from the entry point of the corresponding partition, extract the data in the partition concurrently according to the extraction direction, and store the extracted data in the target disk until all the data in the source disk is extracted, and then the extraction task for the source disk is terminated.

[0074] In this example, if the source disk has not been extracted, the existing data in the source disk is partitioned by equal division to obtain the entry point corresponding to each partition of this data extraction. If the source disk has been extracted, the entry point corresponding to each partition of this data extraction is obtained by updating the entry point of each partition last time. In this way, the amount of data in the partitions obtained is basically the same, which can improve the concurrency of multi-threaded parallel extraction, thereby improving the overall efficiency of data extraction from the source disk.

[0075] Optional, Figure 5 Schematic diagram of the data extraction method provided for this application Figure 2 ,like Figure 5 As shown, in S201, it is determined whether there is extraction content corresponding to the extraction task in the target disk, including:

[0076] S501. Detect the target disk to determine whether there is a historical extraction record of the source disk in the target disk. The historical extraction record includes the entry point of each partition corresponding to the extraction task and the completion time of the extraction task.

[0077] In combination with the scenario example, if the extraction task of the source disk has been completed before, a historical extraction record corresponding to the extraction task can be generated and stored in the target disk. When the target disk is detected, it can be detected whether there is a historical extraction record of the source disk in the target disk. The historical extraction record may include the entry point of each partition corresponding to the extraction and the time when the extraction is completed.

[0078] S502: If yes, determine whether the extraction content corresponding to the extraction task exists in the target disk.

[0079] If there are historical extraction records of the source disk in the target disk, it can be determined that the extraction task for the source disk has been completed at least once, and it can be determined that relevant data in the source disk exists in the target disk.

[0080] S503: If not, determine that the extraction content corresponding to the extraction task does not exist in the target disk.

[0081] If there is no historical extraction record of the source disk in the target disk, it can be determined that the extraction task for the source disk has been executed, and it can be determined that the relevant data in the source disk does not exist in the target disk. This example detects whether there is a historical extraction record in the target disk to achieve the purpose of determining whether there is extraction content corresponding to the extraction task in the target disk.

[0082] Optionally, before S202, the method further includes:

[0083] Probe the source disk to count the amount of data already in the source disk.

[0084] Combined with the scenario example, the source disk is explored to determine the amount of data currently in the source disk, and the data in the source disk can be partitioned in an equal manner based on the amount of data. This example obtains the amount of data on the target disk by exploring the target disk, which can improve the accuracy of the partition.

[0085] Optional, Figure 6 Schematic diagram of the data extraction method provided for this application Figure 3 ,like Figure 6 As shown, in S202, the existing data in the source disk is partitioned in an equal manner, including:

[0086] S601: Divide the existing data in the source disk equally according to the preset number of partitions to obtain corresponding multiple partitions.

[0087] Combined with the scenario example, after the data volume in the source disk is detected, the preset number of partitions can be determined according to the actual situation. For example, if the data volume in the source disk is 10 million and the preset number of partitions is 10, the 10 million data volume can be divided into 10 parts in a recursive N-equal division method to obtain 10 fine-grained partitions, each of which has a data volume of 1 million.

[0088] S602. Determine the size relationship between the data volume in the partition and the preset value. If the data volume corresponding to the partition exceeds the preset value, divide the data volume in each partition equally again according to the preset number of partitions to obtain new partitions, and determine the size relationship between the data volume in the new partition and the preset value, and so on, until the data volume corresponding to the final partition is equal to the preset value.

[0089] Combined with the scene example, Figure 7 Another example of a source disk partition flow chart is as follows: Figure 7As shown, the preset value of the amount of data in each partition corresponding to the source disk can be determined according to the actual situation. For example, the preset value of the amount of data in each partition is determined to be 100,000. After the amount of data in the source disk is divided equally into 10 partitions in a recursive N-equal manner, the amount of data in each partition exceeds the preset value of 100,000, so each partition needs to be partitioned further to obtain a smaller granularity. Each 100 partition is divided equally into 10 parts in this way to obtain a smaller granularity partition. The preset value of each partition is 100,000. At this time, the amount of data in each partition is equal to the preset value of 100,000, then the partitioning is stopped, and the 100 partitions of 100,000 finally obtained are used as the final partitions of the source disk.

[0090] S603: If the data volume of the partition is less than the preset value, the partitions are merged until the data volume corresponding to the final partition is equal to the preset value.

[0091] Combined with the scene example, combined with Figure 7 If the preset value of the data volume in each partition is set to 200,000, the data volume in the fine-grained partition is 100,000. In order to meet the requirement that the preset value of each partition is 200,000, every two fine-grained partitions can be merged to obtain 50 partitions of 200,000, and the 50 partitions of 200,000 are used as the final partitions of the source disk. Based on the method provided in this example, the data volume in the source disk can be accurately and evenly partitioned to ensure that the data volume in each partition is equal to the preset value.

[0092] Optionally, in S202, partitioning the existing data in the source disk in an equal manner includes:

[0093] The existing data in the source disk is equally divided according to the preset data volume to obtain corresponding multiple partitions.

[0094] Combined with the scenario example, the data volume in the source disk can be equally divided by first determining the preset value of the data volume corresponding to each partition according to the actual situation. For example, if the data volume in the source disk is 10 million and the preset value is 100,000, the data volume in the source disk can be divided into 100,000 data volumes per partition, and finally 100 partitions are obtained. Based on the method provided in this example, the data volume in the source disk can be accurately and evenly partitioned to ensure that the data volume in each partition is equal to the preset value.

[0095] Optional, Figure 8 Schematic diagram of the data extraction method provided for this application Figure 4 ,like Figure 8 As shown, in S203, data in the source disk is extracted by multi-threaded parallel extraction based on the entry point of each partition, including:

[0096] S801. Mark the starting point of each partition obtained as the entry point of the corresponding partition.

[0097] Combined with the scene example, Fig. 9 The following is a schematic diagram of the partition of the source disk as an example: Fig. 9 As shown, if the data in the source disk is divided into n zones, and the starting point of each zone is marked as P1, P2, P3 to Pn, the starting point of each marked zone can be used as the entry point of the zone.

[0098] S802: Obtain the number of entry points in the source disk, and determine the number of threads based on the number of entry points.

[0099] Combined with the scenario example, according to Fig. 9 It can be seen that if the number of entry points in the source disk is n, then it can be determined that the source disk has n partitions, and the number of threads can be determined according to the number of partitions that each thread is responsible for extracting. For example, if the number of partitions that each thread is responsible for extracting is 1, that is, the number of threads is the same as the number of partitions, then n threads can be set and the corresponding relationship between threads and partitions can be determined. For another example, if the number of partitions that each thread is responsible for extracting is m, then the number of threads can be set to n / m.

[0100] S803. Allocate each thread to a corresponding entry point, so that each thread starts extracting data from the corresponding partition from the entry point, and stops at the next entry point.

[0101] Combined with the scenario example, each thread is assigned to the entry point position of the corresponding partition. If the extraction direction is from left to right, each thread will extract the data in the partition from the corresponding entry point position in order from left to right until the entry point position of the next partition stops extracting and stores it in the target disk. For example, the thread corresponding to partition 1 is assigned to position P1, and the thread corresponding to partition 1 starts extracting from position P1 until it stops extracting at position P2. Similarly, the thread corresponding to partition 2 is assigned to position P2, and the thread corresponding to partition 2 starts extracting from position P2 until it stops extracting at position P3. Based on the method provided in this example, the corresponding start position and end position of each thread when extracting data are determined, which can ensure that multiple threads can extract data from the source data at the same time, thereby improving the efficiency of data extraction from the source data.

[0102] Optional, Fig.10 Schematic diagram of the data extraction method provided for this application Figure 5 ,like Fig.10 As shown, in S204, the entry point of each partition is updated by extracting content, including:

[0103] S1001. Obtain the entry point of each partition corresponding to the last extraction task from the historical extraction records.

[0104] Combined with the scene example, Fig.11 Update the diagram with the entry point for the example source disk, Fig.11 (a) is an example of the entry points of each partition corresponding to the last data extraction from the source disk, which are P1, P2, P3 to Pn.

[0105] S1002. Based on the incremental data in the extracted content corresponding to the last extraction task, the ntile function is used to adaptively adjust the entry point corresponding to the last extraction task to update the entry point of each partition.

[0106] Combined with the scenario example, it can be seen from the above that in the process of extracting data from the source disk to the source disk, in addition to the data in the source disk detected before the extraction, there is also incremental data generated during the extraction process. The extraction content corresponding to the extraction task of the source disk can be obtained from the target disk, thereby obtaining the incremental data generated during the extraction process. Fig.11 (b) Example In addition to the incremental data generated in the source disk, before extracting data from the source disk, in order to make the data extraction more comprehensive and balance the amount of data extracted by each thread, the incremental data generated in the source disk can be equally divided into each partition. Fig.12 Another example of a source disk partition flow chart is shown in FIG. Fig.12 As shown in the figure, the entry point can be adjusted by executing the Structured Query Language (SQL), using the ntile function to obtain the equal-depth bucket information of the source disk, and adaptively adjusting the entry point of each partition, thereby obtaining Fig.11 (c). Specifically, in the process of adjusting the entry point of each partition, the corresponding starting point is shifted backwards by the incremental data assigned to each partition. If the displacement corresponding to the incremental data assigned to each partition is taken as the unit displacement, the starting point of partition 1 remains unchanged, partition 2 is shifted backwards by one unit displacement, partition 3 is shifted backwards by two unit displacements, and so on, until partition n is shifted backwards by n-1 unit displacements, the following is obtained: Fig.11 (c) shows the updated entry point of each partition. Based on the method provided in this example, the entry point of each partition in the source disk can be updated to balance the amount of data extracted by each thread, improve the concurrency of the threads, and thus improve the overall efficiency of extracting data from the source disk.

[0107] In this embodiment, when the source disk has not been extracted, the existing data in the source disk is partitioned by equally dividing it to obtain the entry point corresponding to each partition of the current data extraction. In the case that the source disk has been extracted, the entry point corresponding to each partition of the current data extraction is obtained by updating the entry point of each partition last time. In this way, the amount of data in the obtained partitions is basically the same, which can improve the concurrency of multi-threaded parallel extraction, thereby improving the overall efficiency of data extraction from the source disk.

[0108] Fig.13 A schematic diagram of the structure of the data extraction device provided in this application, such as Fig.13 As shown, the data extraction device comprises:

[0109] A response module 131 is used to respond to the extraction task of the source disk and determine whether there is an extraction content corresponding to the extraction task in the target disk;

[0110] A partitioning module 132 is used to partition the existing data in the source disk by dividing it equally if the extraction content corresponding to the extraction task does not exist in the target disk, so as to obtain the entry point of each partition;

[0111] Processing module 133, used to extract data from the source disk by multi-threaded parallel extraction based on the entry point of each partition, and store the extracted content to the target disk, the extracted content includes the data already existing on the source disk when the extraction task starts to be executed and the incremental data generated during the execution of the extraction task;

[0112] An updating module 134, configured to update the entry point of each partition by extracting the content corresponding to the extraction task if the content corresponding to the extraction task exists in the target disk;

[0113] The processing module 133 is further used to re-extract the data in the source disk by using a multi-threaded parallel extraction method based on the updated entry point of each partition, and update the extracted content to the target disk.

[0114] Optionally, the response module 131 is specifically used to detect the target disk to determine whether there is a historical extraction record of the source disk in the target disk, and the historical extraction record includes the entry point of each partition corresponding to the extraction task and the completion time of the extraction task when executing the extraction task;

[0115] The response module 131 is further specifically used to determine whether the extraction content corresponding to the extraction task exists in the target disk if it exists;

[0116] The response module 131 is further specifically used to determine that the extraction content corresponding to the extraction task does not exist in the target disk if it does not exist.

[0117] Optionally, the processing module 133 is further configured to detect the source disk to count the amount of data already in the source disk.

[0118] Optionally, the partition module 132 is further configured to equally divide the existing data volume in the source disk according to a preset number of partitions to obtain corresponding multiple partitions;

[0119] The partition module 132 is further used to determine the size relationship between the data volume in the partition and the preset value. If the data volume corresponding to the partition exceeds the preset value, the data volume in each partition is further divided into equal parts according to the preset number of partitions to obtain new partitions, and the size relationship between the data volume in the new partition and the preset value is determined, and so on, until the data volume corresponding to the partition obtained finally equals the preset value;

[0120] The partitioning module 132 is further configured to merge partitions if the data volume of the partition is less than a preset value, until the data volume corresponding to the final partition is equal to the preset value.

[0121] Optionally, the partition module 132 is further configured to equally divide the existing data in the source disk according to a preset data volume to obtain a corresponding plurality of partitions.

[0122] Optionally, the processing module 133 is specifically configured to mark the starting point of each partition obtained as the entry point of the corresponding partition;

[0123] The processing module 133 is further specifically used to obtain the number of entry points in the source disk, and determine the number of threads based on the number of entry points;

[0124] The processing module 133 is further specifically used to assign each thread to a corresponding entry point, so that each thread starts to extract data from the corresponding partition from the entry point, and stops at the next entry point.

[0125] Optionally, the updating module 134 is specifically used to obtain the entry point of each partition corresponding to the last extraction task from the historical extraction records;

[0126] The updating module 134 is further specifically used to adaptively adjust the entry point corresponding to the last extraction task using the ntile function based on the incremental data in the extracted content corresponding to the last extraction task, so as to update the entry point of each partition.

[0127] The data extraction device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and this embodiment will not be described in detail here.

[0128] Fig.14 This is a schematic diagram of the structure of the computer device provided in this application. Fig.14As shown, the computer device 50 provided in this embodiment includes: at least one processor 501 and a memory 502. Optionally, the device 50 also includes a communication component 503. The processor 501, the memory 502 and the communication component 503 are connected via a bus 504.

[0129] In a specific implementation process, at least one processor 501 executes the computer-executable instructions stored in the memory 502, so that at least one processor 501 executes the above method.

[0130] The specific implementation process of the processor 501 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0131] In the above embodiments, it should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the invention may be directly implemented as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0132] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (Non-volatile Memory, NVM), such as at least one disk memory.

[0133] The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application is not limited to only one bus or one type of bus.

[0134] The present application also provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0135] The present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the above method is implemented.

[0136] The above-mentioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0137] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (Application Specific Integrated Circuits, referred to as: ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.

[0138] The division of units is only a logical function division, and there may be other divisions in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0139] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0140] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0141] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0142] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0143] Finally, it should be noted that those skilled in the art will readily conceive of other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses or adaptations of the present invention, which follow the general principles of the present invention and include common knowledge or customary technical means in the art not disclosed by the present invention, are not limited to the precise structure described above and shown in the drawings, and may be modified and changed in various ways without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. A data extraction method, characterized in that: The method comprises: In response to an extraction task for a source disk, determining whether extraction content corresponding to the extraction task exists in a target disk; If the extraction content corresponding to the extraction task does not exist in the target disk, partition the existing data in the source disk in an equal manner to obtain an entry point for each partition; Extracting data from the source disk using a multi-threaded parallel extraction method based on the entry point of each partition, and storing the extracted content to the target disk, wherein the extracted content includes the data already present on the source disk when the extraction task starts to be executed and the incremental data generated during the execution of the extraction task; If the extraction content corresponding to the extraction task exists in the target disk, updating the entry point of each partition by using the extraction content; Based on the updated entry point of each partition, the data in the source disk is re-extracted in a multi-threaded parallel extraction manner, and the extracted content is updated to the target disk.

2. The method according to claim 1, characterized in that The determining whether the extraction content corresponding to the extraction task exists in the target disk includes: Probing the target disk to determine whether there is a historical extraction record of the source disk in the target disk, the historical extraction record including the entry point of each partition corresponding to the extraction task and the completion time of the extraction task when executing the extraction task; If so, determining whether the extraction content corresponding to the extraction task exists in the target disk; If not, it is determined that the extraction content corresponding to the extraction task does not exist in the target disk.

3. The method according to claim 1, characterized in that Before partitioning the existing data in the source disk in an equal manner, the method further includes: The source disk is probed to count the amount of data already in the source disk.

4. The method according to claim 3, characterized in that Partitioning the existing data in the source disk in an equal manner includes: The existing data volume in the source disk is equally divided according to the preset partition quantity to obtain corresponding multiple partitions; Determine the size relationship between the amount of data in the partition and the preset value. If the amount of data corresponding to the partition exceeds the preset value, divide the amount of data in each partition equally again according to the preset number of partitions to obtain new partitions, and determine the size relationship between the amount of data in the new partition and the preset value, and so on, until the amount of data corresponding to the last partition is equal to the preset value; If the data volume of the partition is less than the preset value, the partitions are merged until the data volume corresponding to the final partition is equal to the preset value.

5. The method according to claim 3, characterized in that: Partitioning the existing data in the source disk in an equal manner includes: The existing data in the source disk is equally divided according to the preset data volume to obtain corresponding multiple partitions.

6. The method according to claim 4 or 5, characterized in that: The method of extracting data from the source disk by using a multi-threaded parallel extraction method based on the entry point of each partition includes: Mark the starting point of each partition obtained as the entry point of the corresponding partition; Obtain the number of entry points in the source disk, and determine the number of threads based on the number of entry points; Each thread is assigned to a corresponding entry point, so that each thread starts extracting data from the entry point and starts extracting data from the corresponding partition in sequence until it reaches the next entry point and stops.

7. The method according to claim 2, characterized in that The step of updating the entry point of each partition by extracting the content includes: Obtaining the entry point of each partition corresponding to the last extraction task from the historical extraction record; Based on the incremental data in the extracted content corresponding to the last extraction task, the ntile function is used to adaptively adjust the entry point corresponding to the last extraction task to update the entry point of each partition.

8. A data extraction device, characterized in that: The device comprises: A response module, used for responding to an extraction task for a source disk and determining whether there is an extraction content corresponding to the extraction task in a target disk; A partitioning module, used for partitioning the existing data in the source disk in an equal manner to obtain an entry point for each partition if the extraction content corresponding to the extraction task does not exist in the target disk; A processing module, used to extract data from the source disk by multi-threaded parallel extraction based on the entry point of each partition, and store the extracted content to the target disk, wherein the extracted content includes the data already present on the source disk when the extraction task starts to be executed and the incremental data generated during the execution of the extraction task; An updating module, configured to update the entry point of each partition by using the extracted content if the extracted content corresponding to the extraction task exists in the target disk; The processing module is also used to re-extract the data in the source disk by multi-threaded parallel extraction based on the updated entry point of each partition, and update the extracted content to the target disk.

9. A computer device, characterized in that: include: Memory, processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor performs the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 7 when executed by a processor.

11. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Citation Information

Patent Citations

  • Data paralleling abstracting method and apparatus and database system

    CN101329676A

  • Incremental data extraction method, device and server

    CN109213817A

  • Data partitioning method and device applied to data warehouse

    CN111753019A

  • Data transfer program, data transfer method, and data transfer unit

    JP2007122167A

  • Parallel decimation circuits

    US20030110194A1