Data scanning method, database system and related equipment

Abstract

The present application discloses a data scanning method, a database system, and related equipment, and relates to the field of computer technology. The method includes: obtaining the total unoccupied computing resources of the database system and the estimated amount of resources required for a scanning task to process data shards, where a data shard is a data set obtained by segmenting the data to be scanned; obtaining data statistical information of the data to be scanned, where the data statistical information is information used to describe the total amount of data to be scanned and the total number of data shards; calculating a target number of scanning tasks required to process the data to be scanned based on the total computing resources, the estimated amount of resources, and the data statistical information; and assigning the target number of scanning tasks to the data shards to achieve data scanning, and assigning multiple data shards to at least one scanning task. The present application solves the problem of large-scale scanning tasks affecting the stability of the entire database instance during data scanning.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data scanning method, a database system, and related equipment. Background Art

[0002] This section is intended to provide a background or context to the embodiments of the invention recited in the claims. No admission is made that the description herein is prior art by its inclusion in this section.

[0003] Data scanning is a crucial aspect of databases and big data products. It involves scanning data from underlying storage. In distributed scenarios, a single piece of data is split into multiple shards, which are then scanned concurrently by multiple scanning tasks to accelerate scanning. A shard is a portion of the data to be scanned (equivalent to a subset of the data). A scanning task contains specific computational logic and is responsible for processing the data.

[0004] In existing scanning solutions, such as Figure 1 As shown in the figure, one data split is processed by one scanning task, that is, the number of scanning tasks corresponds to the number of data splits. This scanning solution has the following disadvantages:

[0005] 1. When the number of data splits is particularly large, the corresponding number of scanning tasks will also be particularly large. When the number of scanning tasks is large, it will bring a large management cost to the entire scanning task, including the maintenance and management of scanning task management, monitoring statistics, heartbeat maintenance and other operation processes.

[0006] For example, if a customer's table contains approximately 1TB of data and each data split is approximately 64MB in size, this table will have approximately 16,000 data splits. This will generate 16,000 scan tasks, meaning the scan task manager will be responsible for scheduling and managing all 16,000 scan tasks simultaneously. The monitoring device needs to collect information about these 16,000 scan tasks and display it on the monitoring page. This large scale of scan tasks presents various challenges.

[0007] 2. The scanning task and the data shard Split are bound together and cannot be adjusted dynamically and flexibly according to the actual situation. As a result, the scanning task calculation will have a long tail. During the data scanning process, there may be two common long tail situations: 1) Data skew: The data shard Split itself is unevenly distributed. Some scanning tasks with large computational loads are calculated slowly. In this case, other scanning tasks need to wait until the large computational load scanning task is completed before they can be calculated. This leads to a natural long tail when calculating each task. 2) Inconsistent node performance: Due to different hardware or abnormalities, some nodes are slower than other nodes when processing the same amount of data, resulting in a long tail.

[0008] It can be seen that with the rapid growth of the amount of data in the user table, under the original scanning model, one data split was assigned to one scanning task for processing, resulting in a corresponding rapid increase in the number of scanning tasks responsible for data scanning and the scale of scanning tasks. The huge scanning task scale caused many problems in the operation processes such as scanning task scheduling, scanning task management, heartbeat maintenance, monitoring and statistics, affecting the stability of the entire database instance. Summary of the Invention

[0009] The embodiments of the present application provide a data scanning method, a database system, and related equipment to at least solve the problem in the prior art that the scale of scanning tasks in data scanning is huge, affecting the stability of the entire database instance.

[0010] According to one aspect of the present application, a data scanning method is also provided, which is applied to a database system and includes:

[0011] Obtaining the total unoccupied computing resources of the database system and an estimated amount of resources required for the scanning task to process data slices, where the data slices are datasets obtained by segmenting the data to be scanned;

[0012] Obtaining data statistical information of the data to be scanned, wherein the data statistical information is information used to describe the total amount of data of the data to be scanned and the total number of the data fragments;

[0013] Calculating a target number of scanning tasks required to process the data to be scanned based on the total computing resources, the estimated resource amount, and the data statistical information;

[0014] When allocating the data slices to be processed to the target number of scanning tasks to implement data scanning, a plurality of the data slices are allocated to at least one of the scanning tasks.

[0015] In some embodiments, the step of calculating a target number of scanning tasks required to process the data to be scanned based on the total computing resources, the estimated resource amount, and the data statistical information includes:

[0016] Calculating a first task number of the scanning task according to the total computing resources and the estimated resource amount;

[0017] Determine whether the numerical value of the first task number is smaller than the numerical value of the total number; if so, use the numerical value of the first task number as the value of the target number.

[0018] In some embodiments, the method further comprises:

[0019] If the numerical value of the first task number is greater than the numerical value of the total number, the numerical value of the total number is used as the value of the target number.

[0020] In some embodiments, before determining whether the numerical value of the first number of tasks is less than the numerical value of the total number of tasks, the method further includes:

[0021] Obtaining a pre-configured data processing volume of the scanning task;

[0022] Calculating the second task number of the scanning task according to the total amount of data and the data processing amount;

[0023] Based on the second number of tasks and the first number of tasks, determine whether the total computing resources are sufficient when processing the data to be scanned. If so, determine whether the numerical value of the first number of tasks is less than the numerical value of the total number.

[0024] In some embodiments, before determining whether the numerical value of the first number of tasks is less than the numerical value of the total number of tasks, the method further includes:

[0025] Obtaining a pre-configured processing number for the scanning task, where the processing number is the number of data shards that the scanning task can process;

[0026] Calculating a second task number of the scanning task according to the total number and the processed number;

[0027] Based on the second number of tasks and the first number of tasks, determine whether the total computing resources are sufficient when processing the data to be scanned. If so, determine whether the numerical value of the first number of tasks is less than the numerical value of the total number.

[0028] In some embodiments, the step of determining whether the total computing resources are sufficient for processing the data to be scanned based on the second number of tasks and the first number of tasks further includes:

[0029] Determine whether the numerical value of the second number of tasks is greater than the numerical value of the first number of tasks. If so, the total computing resources are insufficient. At this time, the numerical value of the first number of tasks is used as the value of the target number.

[0030] In some embodiments, in the process of allocating the target number of scan tasks to be processed, the method further includes:

[0031] It is determined whether the data volume of the data slices received by the scanning task is greater than a preset threshold. If so, the data slices with a data volume less than the threshold are sent to the scanning task.

[0032] In some embodiments, the method further comprises:

[0033] Obtain a preset upper limit value for the number of scanning tasks, determine whether the value of the target number is greater than the upper limit value, and if so, use the upper limit value as the value of the target number.

[0034] According to another aspect of the present application, a database system is also provided, wherein the database system is used to execute the above-mentioned data scanning method.

[0035] According to another aspect of the present application, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above method steps when executing the computer program.

[0036] According to another aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and the computer program implements the above method steps when executed by a processor.

[0037] According to another aspect of the present application, a computer program product is provided. The computer program product includes a computer program. When the computer program is executed by a processor, the above method steps are implemented.

[0038] In an embodiment of the present application, after calculating the target number of scanning tasks required to scan the data to be scanned, multiple data shards are assigned to at least one scanning task so that one scanning task can process multiple data shards concurrently, thereby reducing the task scale of the scanning task and ensuring the stability of the entire database instance. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The accompanying drawings, which constitute part of this application, are intended to provide a further understanding of this application. The exemplary embodiments and descriptions of this application are intended to explain this application and do not constitute an improper limitation on this application. In the accompanying drawings:

[0040] Figure 1 A schematic flow chart of a data scanning method provided in one embodiment of the present invention;

[0041] Figure 2 A schematic diagram of a process for obtaining a target value according to an embodiment of the present invention;

[0042] Figure 3 A flowchart of determining whether total computing resources are sufficient when processing data to be scanned, provided by an embodiment of the present invention;

[0043] Figure 4 A flowchart of determining whether total computing resources are sufficient when processing data to be scanned, provided by another embodiment of the present invention;

[0044] Figure 5 A schematic structural diagram of an adaptive data scanning device provided by an embodiment of the present invention;

[0045] Figure 6 A schematic diagram of a structure in which data slices are allocated to scanning tasks according to an embodiment of the present invention;

[0046] Figure 7 A schematic diagram of the structure of dynamically allocating data slices to scanning tasks provided by an embodiment of the present invention.

[0047] In the picture:

[0048] 11. First acquisition module; 12. Second acquisition module; 13. Calculation module; 14. Allocation module. DETAILED DESCRIPTION

[0049] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0050] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0051] With the rapid growth of the amount of data scanned by users, the number and scale of scanning tasks responsible for data scanning also increase rapidly under the original scanning model. The huge task scale poses a huge challenge to the stability of the entire database instance. In this regard, the first embodiment of the present invention provides a data scanning method, which is applied to a database system, such as Figure 1 As shown, the method includes the following steps:

[0052] Step S11: Obtain the total unused computing resources of the database system and the estimated amount of resources required to process the data shards of the scan task. A data shard is a dataset obtained by partitioning the data to be scanned. This estimated amount of resources can be calculated by the database optimizer based on cost estimates or determined using preconfigured resource parameters.

[0053] Step S12: Obtain data statistics information of the data to be scanned, which is information used to describe the total amount of data to be scanned and the total number of data slices. The total amount of data is the sum of the data amounts of each data slice.

[0054] Step S13: Calculate the target number of scanning tasks required to process the data to be scanned based on the total computing resources, the estimated amount of resources, and the data statistical information.

[0055] Step S14: for the target number of scanning tasks, allocating data slices to be processed to implement data scanning, and allocating multiple data slices to at least one scanning task.

[0056] After calculating the target number of scan tasks required to scan the data to be scanned, the embodiments of the present invention assign multiple data shards to at least one scan task, enabling a single scan task to concurrently process multiple data shards. This reduces the task size of the scan task and ensures the stability of the entire database instance. Compared to the prior art approach in which a single scan task only processes a single data shard, in the embodiments of the present invention, a single scan task processes multiple data shards. This not only reduces the number of scan tasks and the task size, but also, because a single scan task can concurrently execute multiple data shards, it accelerates the scanning of the entire data to be scanned and improves data scanning efficiency.

[0057] like Figure 2 As shown, in step S13, the step of calculating the target number of scanning tasks required for processing the data to be scanned based on the total computing resources, the estimated amount of resources, and the data statistical information includes:

[0058] Step S21: Calculate the first number of tasks for the scan task based on the total computing resources and the estimated resource amount. In this embodiment of the present invention, the value G of the first number of tasks is equal to the ratio of the total computing resources to the estimated resource amount. If the total computing resources are C, the estimated resource amount is D, and the value of the first number of tasks is G, then G = C / D.

[0059] Step S22: Determine whether the value of the first task number is less than the value of the total number. If so, use the value of the first task number as the target number. Therefore, since the number of scanning tasks is less than the number of data shards, after allocating at least one data shard to each scanning task, multiple data shards can be allocated to at least one scanning task. This allows the scanning task allocated multiple data shards to execute the multiple data shards concurrently, reducing the task size while improving data scanning efficiency.

[0060] In step S22, if the value of the first task number is greater than the value of the total task number, the total task number is used as the target task number. Specifically, when the total number of data shards is sufficiently small (less than the calculated first task number), this embodiment of the present invention uses the total number of data shards as the concurrency granularity to issue a corresponding number of scanning tasks. This avoids excessive scanning tasks, reduces the task scale, and ensures the stability of the entire database system and the reliability of data scanning.

[0061] like Figure 3 As shown, before step S22 determines whether the numerical value of the number of the first tasks is less than the numerical value of the total number, the method provided by the embodiment of the present invention further includes the following steps:

[0062] Step S31: obtaining the data processing volume of the pre-configured scanning task, that is, the total amount of data of the data slices that each pre-set scanning task needs to process.

[0063] Step S32: Calculate the second number of tasks for the scanning task based on the total amount of data and the data processing amount. In this embodiment of the present invention, the value of the second number of tasks is equal to the ratio of the total amount of data to the data processing amount. If the total amount of data is B, the data processing amount is E, and the value of the second number of tasks is H, then H = B / E.

[0064] Step S33: Based on the second task number and the first task number, determine whether the total computing resources are sufficient for processing the data to be scanned. If so, determine whether the value of the first task number is smaller than the value of the total number.

[0065] It can be seen that in the embodiment of the present invention, when the total computing resources available to the database system are sufficient, while ensuring that each scanning task processes at least one data shard, the smaller value between the number of first tasks and the total number of data shards is used as the target number of scanning tasks to be issued. Therefore, when the total number of data shards is small enough, the total number of data shards will be used as the concurrency granularity, and the number of scanning tasks corresponding to the total number of data shards will be issued. While ensuring that each data shard can be processed, the number of scanning tasks will not be too large. When the total number of data shards is large, a fixed value (i.e., the calculated number of first tasks) will be used as the concurrency granularity. At this time, at least one or more scanning tasks will process multiple data shards, reducing the task scale.

[0066] like Figure 4 As shown, before step S22 determines whether the numerical value of the number of the first tasks is less than the numerical value of the total number, the method provided by the embodiment of the present invention further includes the following steps:

[0067] Step S41: Obtain the pre-configured processing number of the scanning task, where the processing number is the number of data slices that the scanning task can process.

[0068] Step S42: Calculate the second task number of the scanning task according to the total number and the processing number;

[0069] Step S43: judging whether the total computing resources are sufficient for processing the data to be scanned based on the second task number and the first task number; if so, judging whether the value of the first task number is smaller than the value of the total number.

[0070] In addition to calculating the second number of scan tasks required to scan the data to be scanned based on the total amount of data to be scanned, embodiments of the present invention can also calculate this second number of tasks based on the total number of data shards and the number of data shards that each scan task can process. Thus, after determining that the total computing resources available to the database system are sufficient based on the second number of tasks and the first number of tasks, and ensuring that each scan task processes at least one data shard, the smaller value between the first number of tasks and the total number of data shards is used as the target number of scan tasks to be issued.

[0071] In step S33 or step S43, the step of determining whether the total computing resources are sufficient for processing the data to be scanned based on the number of the second tasks and the number of the first tasks includes:

[0072] It is determined whether the numerical value of the second task number is greater than the numerical value of the first task number. If not, the total computing resources are sufficient.

[0073] In step S33 or step S43, the step of determining whether the total computing resources are sufficient for processing the data to be scanned based on the number of the second tasks and the number of the first tasks further includes:

[0074] A determination is made as to whether the value of the second number of tasks is greater than the value of the first number of tasks. If so, total computing resources are insufficient, and the value of the first number of tasks is used as the target value. Thus, when the total computing resources of the database system are insufficient, this embodiment of the present invention uses the smaller value of the first number of tasks calculated as the target value to avoid issuing too many scanning tasks and to allocate multiple data shards to be processed to at least one or more scanning tasks, thereby reducing the task size and improving data scanning efficiency.

[0075] To solve the long-tail problem that occurs during data scanning, in step S14, in the process of allocating the target number of scanning tasks to be processed into data slices, the method provided by the embodiment of the present invention further includes:

[0076] Determine whether the amount of data slices received by the scanning task is greater than a preset threshold. If so, send the data slices with a data amount less than the threshold to the scanning task. Therefore, the embodiment of the present invention can dynamically allocate data slices according to the actual operation status of each scanning task during the execution of the scanning task, ensuring that the resources of each scanning task can be fully utilized and reducing the risk of long tail. Figure 7 As shown, a data shard with a large amount of data has been sent in the scanning task Task1 ( Figure 7 The Split shown in the figure represents a data shard. To illustrate the dynamic allocation of data shards, the data volume of a data shard is quantified by the length of the box containing the Split. Subsequently, a data shard with a smaller data volume is sent whenever possible (this is equivalent to sending a data shard with a data volume less than the threshold to scanning task Task 1) to ensure overall balance and reduce the risk of computational skew.

[0077] When the number of data shards is particularly large or the calculated number of first tasks is relatively large, the method provided by an embodiment of the present invention further includes: obtaining a pre-set upper limit value for the number of scanning tasks, determining whether the target value is greater than the upper limit value, and if so, using the upper limit value as the target value. Thus, the embodiment of the present invention uses the set upper limit value as the concurrency granularity to avoid the costs of heartbeat maintenance, task management, monitoring statistics, and other costs caused by too many scanning tasks.

[0078] The second embodiment of the present invention also provides an adaptive data scanning device, which is applied to a database system, such as Figure 5As shown, the device includes a first acquisition module 11, a second acquisition module 12, a calculation module 13 and an allocation module 14. The first acquisition module 11 is used to obtain the total unoccupied computing resources of the database system and the estimated amount of resources required for the scanning task to process the data slices, where the data slices are data sets obtained after segmenting the data to be scanned. The second acquisition module 12 is used to obtain data statistical information of the data to be scanned, which is information used to describe the total amount of data to be scanned and the total number of data slices. The calculation module 13 is used to calculate the target number of scanning tasks required to process the data to be scanned based on the total computing resources, the estimated amount of resources and the data statistical information. The allocation module 14 is used to allocate multiple data slices to at least one scanning task when allocating the target number of scanning tasks to realize data scanning.

[0079] After calculating the target number of scan tasks required to scan the data to be scanned, the device provided by an embodiment of the present invention allocates multiple data shards to at least one scan task. This allows a single scan task to concurrently process multiple data shards, reducing the size of the scan task and ensuring the stability of the entire database instance. Furthermore, because a single scan task can concurrently execute multiple data shards, the scanning speed of the entire data to be scanned is accelerated, improving data scanning efficiency.

[0080] like Figure 2 As shown, the calculation module 13 calculates the target number of scanning tasks required for processing the data to be scanned based on the total computing resources, the estimated amount of resources, and the data statistical information, including the following steps:

[0081] Step S21: Calculate the first number of tasks for the scan task based on the total computing resources and the estimated resource amount. In this embodiment of the present invention, the value G of the first number of tasks is equal to the ratio of the total computing resources to the estimated resource amount. If the total computing resources are C, the estimated resource amount is D, and the value of the first number of tasks is G, then G = C / D.

[0082] Step S22: Determine whether the value of the first task number is less than the value of the total number. If so, use the value of the first task number as the target number. Therefore, since the number of scanning tasks is less than the number of data shards, after allocating at least one data shard to each scanning task, multiple data shards can be allocated to at least one scanning task. This allows the scanning task allocated multiple data shards to execute the multiple data shards concurrently, reducing the task size while improving data scanning efficiency.

[0083] In step S22, if the value of the first task number is greater than the value of the total task number, the total task number is used as the target task number. Specifically, when the total number of data shards is sufficiently small (less than the calculated first task number), this embodiment of the present invention uses the total number of data shards as the concurrency granularity to issue a corresponding number of scanning tasks. This avoids excessive scanning tasks, reduces the task scale, and ensures the stability of the entire database system and the reliability of data scanning.

[0084] like Figure 3 As shown, before step S22 determines whether the numerical value of the number of the first tasks is less than the numerical value of the total number, the calculation module 13 provided by the embodiment of the present invention further performs the following steps:

[0085] Step S31: obtaining the data processing volume of the pre-configured scanning task, that is, the total amount of data of the data slices that each pre-set scanning task needs to process.

[0086] Step S32: Calculate the second number of tasks for the scanning task based on the total amount of data and the data processing amount. In this embodiment of the present invention, the value of the second number of tasks is equal to the ratio of the total amount of data to the data processing amount. If the total amount of data is B, the data processing amount is E, and the value of the second number of tasks is H, then H = B / E.

[0087] Step S33: Based on the second task number and the first task number, determine whether the total computing resources are sufficient for processing the data to be scanned. If so, determine whether the value of the first task number is smaller than the value of the total number.

[0088] It can be seen that when the total computing resources available to the database system are sufficient, the device provided by the embodiment of the present invention uses the smaller value between the number of first tasks and the total number of data shards as the target number of scanning tasks to be issued, while ensuring that each scanning task processes at least one data shard. Thus, when the total number of data shards is small enough, the total number of data shards will be used as the concurrency granularity, and the number of scanning tasks corresponding to the total number of data shards will be issued. While ensuring that each data shard can be processed, the number of scanning tasks will not be too large. When the total number of data shards is large, a fixed value (i.e., the calculated number of first tasks) will be used as the concurrency granularity. At this time, at least one or more scanning tasks will process multiple data shards, reducing the task scale.

[0089] like Figure 4 As shown, before step S22 determines whether the numerical value of the number of the first tasks is less than the numerical value of the total number, the calculation module 13 provided by the embodiment of the present invention further performs the following steps:

[0090] Step S41: Obtain the pre-configured processing number of the scanning task, where the processing number is the number of data slices that the scanning task can process.

[0091] Step S42: Calculate the second task number of the scanning task according to the total number and the processing number;

[0092] Step S43: judging whether the total computing resources are sufficient for processing the data to be scanned based on the second task number and the first task number; if so, judging whether the value of the first task number is smaller than the value of the total number.

[0093] In addition to calculating the second number of scan tasks required to scan the data to be scanned based on the total amount of data to be scanned, the calculation module 13 in this embodiment of the present invention can also calculate the second number of tasks based on the total number of data shards and the number of data shards that each scan task can process. Thus, after determining that the total computing resources available to the database system are sufficient based on the second number of tasks and the first number of tasks, and ensuring that each scan task processes at least one data shard, the smaller value between the first number of tasks and the total number of data shards is used as the target number of scan tasks to be issued.

[0094] In step S33 or step S43, the calculation module 13 determines whether the total computing resources are sufficient for processing the data to be scanned based on the number of the second tasks and the number of the first tasks, including:

[0095] It is determined whether the numerical value of the second task number is greater than the numerical value of the first task number. If not, the total computing resources are sufficient.

[0096] In step S33 or step S43, the calculation module 13 determines whether the total computing resources are sufficient for processing the data to be scanned based on the number of the second tasks and the number of the first tasks, and further includes:

[0097] A determination is made as to whether the value of the second number of tasks is greater than the value of the first number of tasks. If so, total computing resources are insufficient, and the value of the first number of tasks is used as the target value. Thus, when the total computing resources of the database system are insufficient, this embodiment of the present invention uses the smaller value of the first number of tasks calculated as the target value to avoid issuing too many scanning tasks and to allocate multiple data shards to be processed to at least one or more scanning tasks, thereby reducing the task size and improving data scanning efficiency.

[0098] To address the long-tail problem that arises during data scanning, allocation module 14, when assigning data slices to be processed to a target number of scanning tasks, determines whether the data volume of the data slices already received by the scanning task exceeds a preset threshold. If so, it then sends the data slices with a data volume less than the threshold to the scanning task. Thus, allocation module 14 provided in this embodiment of the present invention can dynamically allocate data slices based on the actual operating conditions of each scanning task during the execution of the scanning task, ensuring that the resources of each scanning task are fully utilized and reducing the risk of long-tail problems.

[0099] When the number of data shards is particularly large or the calculated number of first tasks is relatively large, the calculation module 13 provided in this embodiment of the present invention is further configured to obtain a pre-set upper limit for the number of scanning tasks, determine whether the target number is greater than the upper limit, and if so, use the upper limit as the target number. Thus, this embodiment of the present invention uses the set upper limit as the concurrency granularity to avoid the costs of heartbeat maintenance, task management, monitoring statistics, and the like caused by too many scanning tasks.

[0100] The third embodiment of the present invention further provides a database system, which is used to execute the data scanning method provided by the first embodiment of the present invention. For details of the data scanning method, please refer to the content provided by the first embodiment of the present invention, and the embodiment of the present invention will not be repeated here.

[0101] The fourth embodiment of the present invention further provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the data scanning method are implemented. For details of the data scanning method, please refer to the content provided in the first embodiment of the present invention. The embodiment of the present invention will not be repeated here.

[0102] The fifth embodiment of the present invention further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps of the data scanning method are implemented. For details of the data scanning method, please refer to the content provided in the first embodiment of the present invention, and the embodiments of the present invention will not be repeated here.

[0103] The sixth embodiment of the present invention further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps of the data scanning method are implemented. For details of the data scanning method, please refer to the content provided in the first embodiment of the present invention, and the embodiment of the present invention will not be repeated here.

[0104] The seventh embodiment of the present invention provides a specific application embodiment of an adaptive data scanning method, which solves the problem of huge task scale in data scanning and makes further optimization to solve the long-tail problem commonly seen in the data scanning process.

[0105] In this embodiment of the present invention, a single scan task supports processing multiple data splits. An adaptive scanning algorithm is introduced to determine the total number of scan tasks and the number of data splits to be processed by each scan task based on data statistics of the data to be scanned, the total available computing resources of the database system, and the estimated resource requirements for the scan tasks. Total computing resources include processor CPU resources, memory resources, and other resources.

[0106] First, after the data to be scanned is divided into data slices, information such as the amount of data corresponding to each data slice, the total amount of data to be scanned, and the total number of data slices will be recorded in the data statistics information.

[0107] According to the total computing resources, resource estimates and data statistics, the total number of scanning tasks is adaptively calculated (corresponding to Figure 6 The value of M in , where M is the target value) and the number of data shards to be processed by each scanning task.

[0108] The total number of scan tasks is determined based on the data statistics of the data to be scanned, the total available computing resources of the database system, and the estimated resource requirements of the scan tasks. Specifically, the following are included:

[0109] First, get the following parameters:

[0110] 1) The total number of data shards A;

[0111] 2) The total amount of data in the total data shard: B;

[0112] 3) Available resources of the database system: C, that is, the total computing resources, the value unit of total computing resources is GB;

[0113] 4) The estimated resources occupied by each scanning task: i.e., the estimated resource amount D;

[0114] 5) The amount of data recommended to be processed by each scanning task: that is, the data processing volume E;

[0115] 6) The number of data shards recommended for each scanning task: the number of processing F

[0116] Calculation method of target value:

[0117] 1) First, calculate the maximum number of first tasks G that can be issued based on the total computing resources: G = C / D;

[0118] 2) Calculate the maximum number of second tasks H required to process all splits based on the recommended values:

[0119] a11. If the total amount of data B is provided, the calculation is based on the total amount of data: H = B / E;

[0120] b11. If the total data volume B is not provided, calculate based on the total number of data shards: H = A / F;

[0121] 3) There may be two situations:

[0122] a21.H<=G (resources are sufficient, but it is necessary to ensure that each scanning task processes at least one data shard). At this time, the target value is equal to Math.min(A, G), that is, the smaller value of A or G is taken.

[0123] b22.H>G: The target value is G.

[0124] Therefore, in the embodiment of the present invention, when the number of data shards is small enough, the total number of data shards A will be used as the concurrency granularity, and A scanning tasks will be issued to process A data shards. When the number of data shards is large, a fixed value G will be used as the concurrency granularity, and G scanning tasks will be issued to process A data shards. At this time, while ensuring that each scanning task processes at least one data shard, the excess data shards will be dynamically allocated to at least one scanning task for processing. When the target number value finally obtained is large, there will be an upper limit value as the concurrency granularity to avoid the cost of heartbeat maintenance, task management, monitoring statistics, etc. caused by too many scanning tasks. If each scanning task processes about 128MB of data, it is better and resources can be fully utilized. The number of concurrencies generated adaptively is shown in the following table:

[0125]

[0126]

[0127] Compared with the existing scanning task that only processes one data slice, a scanning task in the embodiment of the present invention will process multiple data slices. In addition to reducing the number of scanning tasks and reducing the task scale, multiple data slices on a scanning task also support concurrent execution, which speeds up the entire scanning execution speed.

[0128] Secondly, the present invention also addresses the common long-tail problem in data scanning by dynamically assigning data shards to scanning tasks. Specifically, to address the long-tail problem, data shards are dynamically allocated during the execution of scanning tasks based on their actual performance, ensuring that the resources of each task node are fully utilized and reducing the risk of long-tail problems.

[0129] The process of dynamically allocating and sending data shards to scanning tasks is as follows: Figure 7 As shown:

[0130] For example, if a split with a large amount of data has been sent in the scanning task Task 1, a split with a smaller amount of data will be sent next as much as possible to ensure overall balance and reduce the long-tail risk caused by computing skew.

[0131] Therefore, the present invention addresses the issue of excessive scan tasks by enabling the concurrent processing of multiple data shards within a single scan task. This capability adaptively calculates a target number of scan tasks based on total computing resources, resource estimates, and data statistics, while also respecting a set upper limit. Furthermore, it supports more flexible allocation of data shards to be processed based on the actual operating status of the scan task, addressing the common long-tail problem in data scanning.

[0132] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0133] The above are merely embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included within the scope of the claims of the present application.

Claims

1. A data scanning method, characterized in that: The method is applied to a database system and includes: Obtaining the total unoccupied computing resources of the database system and an estimated amount of resources required for the scanning task to process data slices, where the data slices are datasets obtained by segmenting the data to be scanned; Obtaining data statistical information of the data to be scanned, wherein the data statistical information is information used to describe the total amount of data of the data to be scanned and the total number of the data fragments; Calculating a first task number of the scanning task according to the total computing resources and the estimated amount of resources, where the first task number is equal to a ratio of the total computing resources to the estimated amount of resources; Determine whether the numerical value of the first task number is less than the numerical value of the total number; If so, the value of the first number of tasks is used as the value of the target number, and the data slices to be processed are allocated to the scanning tasks of the target number to implement data scanning, and multiple data slices are allocated to at least one scanning task; If not, the value of the total number is taken as the value of the target number, and for the scanning tasks of the target number, the data slices to be processed are allocated to realize data scanning, and the corresponding number of scanning tasks are issued with the total number of the data slices as the concurrency granularity.

2. The method according to claim 1, characterized in that Before determining whether the numerical value of the first number of tasks is less than the numerical value of the total number of tasks, the method further includes: Obtaining a pre-configured data processing volume of the scanning task; Calculating the second task number of the scanning task according to the total amount of data and the data processing amount; Based on the second number of tasks and the first number of tasks, determine whether the total computing resources are sufficient when processing the data to be scanned. If so, determine whether the numerical value of the first number of tasks is less than the numerical value of the total number.

3. The method according to claim 1, characterized in that Before determining whether the numerical value of the first number of tasks is less than the numerical value of the total number of tasks, the method further includes: Obtaining a pre-configured processing number for the scanning task, where the processing number is the number of data shards that the scanning task can process; Calculating a second task number of the scanning task according to the total number and the processed number; Based on the second number of tasks and the first number of tasks, determine whether the total computing resources are sufficient when processing the data to be scanned. If so, determine whether the numerical value of the first number of tasks is less than the numerical value of the total number.

4. The method according to claim 2 or 3, characterized in that The step of determining whether the total computing resources are sufficient when processing the data to be scanned based on the second number of tasks and the first number of tasks includes: Determine whether the numerical value of the second number of tasks is greater than the numerical value of the first number of tasks. If so, the total computing resources are insufficient. At this time, the numerical value of the first number of tasks is used as the value of the target number.

5. The method according to claim 1, wherein In the process of allocating the data slices to be processed for the target number of scanning tasks, the method further includes: It is determined whether the data volume of the data slices received by the scanning task is greater than a preset threshold. If so, the data slices with a data volume less than the threshold are sent to the scanning task.

6. The method according to claim 1, characterized in that The method further comprises: Obtain a preset upper limit value for the number of scanning tasks, determine whether the value of the target number is greater than the upper limit value, and if so, use the upper limit value as the value of the target number.

7. A database system, characterized in that: The database system is used to execute the data scanning method according to any one of claims 1 to 6.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method steps described in any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps of any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the method steps of any one of claims 1 to 6 are implemented.

Images