Database-based big data set computing method, device, equipment and medium

Abstract

The present invention discloses a method, device, equipment and medium for large data set operation based on a database. The method includes: dividing the operation data set according to the data volume of the operation data set, a first data operation threshold and a preset hash function to obtain a division result; determining the data set operation subtasks for performing the set operation according to the division result, classifying the data set operation subtasks according to the data volume of each data set operation subtask and the second data operation threshold; and performing the data set operation of each data set operation subtask in the set operation classification in parallel based on the maximum number of parallel threads corresponding to the data set operation subtask and the preset set operation operation. The embodiment of the present invention can accurately classify the subtasks and automatically select a better set operation method through the above technical scheme, so as to solve the problems of excessive performance, high memory resource occupation and inability to parallel process encountered in database set operation when the data scale is extremely large.

Description

Technical Field

[0001] The present invention relates to the field of database technology, and in particular to a method, device, equipment and medium for computing large data sets based on a database. Background Art

[0002] In the modern information society, as data continues to grow in size and complexity, data processing and analysis tasks are becoming increasingly complex. As one of the core functions of database management systems, database set operations play a vital role in data query, analysis, and mining. However, traditional database set operation implementations require constructing a hash table using the full data set of the first dataset. When the data scale is very large, the hash table consumes a large amount of memory and is prone to out-of-memory errors. Furthermore, when traversing the data set of the second dataset and querying the hash table, the large data volume also increases the probability of hash collisions, and the CPU consumption associated with hash collision resolution is also high. Existing database set operations are only suitable for small data sets and face performance bottlenecks and resource constraints when processing very large data sets. Furthermore, the entire process of traditional database set operations is difficult to parallelize. Therefore, a new set operation method for large data sets is urgently needed to address the performance, memory resource usage, and parallel processing challenges encountered by database set operations on very large data sets. Summary of the Invention

[0003] In view of this, the present invention provides a database-based big data set operation method, device, equipment and medium, which can accurately classify subtasks and automatically select a more optimal set operation method to solve the problems of performance, excessive memory resource usage and poor parallel processing encountered in database set operations when the data scale is extremely large.

[0004] According to one aspect of the present invention, an embodiment of the present invention provides a database-based large data set operation method, which is applied to a database node. The method includes:

[0005] Get the operation data sets corresponding to the left child node and the right child node respectively;

[0006] Slicing the computational data set according to the data size of the computational data set, a preset first data computation threshold, and a preset hash function to obtain corresponding slicing results; wherein the slicing results include at least: at least two slice data files of each of the computational data sets, a slicing number, and the data size corresponding to each of the slice data files;

[0007] Determine a data set operation subtask for performing a set operation according to the segmentation result, and classify the data set operation subtasks according to the data size of each data set operation subtask and a preset second data operation threshold;

[0008] Based on the maximum number of parallel threads corresponding to the data set operation subtask and the preset set operation operation, the data set operation of each data set operation subtask in the set operation classification is executed in parallel, and each of the data set operations is merged to obtain a target data operation result set.

[0009] According to another aspect of the present invention, an embodiment of the present invention further provides a database-based large data set operation device, which is applied to a database node, and includes:

[0010] An acquisition module is used to obtain the operation data sets corresponding to the left child node and the right child node respectively;

[0011] a segmentation module, configured to segment the computational data set according to the data size of the computational data set, a preset first data computation threshold, and a preset hash function to obtain corresponding segmentation results; wherein the segmentation results include at least: at least two slice data files for each data segmentation of the computational data set, a segmentation number, and the data size corresponding to each slice data file;

[0012] a classification module, configured to determine a data set operation subtask to perform a set operation on according to the segmentation result, and to classify each of the data set operation subtasks into a set operation according to the data size of each of the data set operation subtasks and a preset second data operation threshold;

[0013] An operation module is used to execute the data set operation of each data set operation subtask in the set operation classification in parallel based on the maximum number of parallel threads corresponding to the data set operation subtask and the preset set operation operation, and merge the data set operations to obtain the target data operation result set.

[0014] According to another aspect of the present invention, an embodiment of the present invention further provides an electronic device, comprising:

[0015] at least one processor; and

[0016] a memory communicatively connected to the at least one processor; wherein,

[0017] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the database-based large data set operation method described in any embodiment of the present invention.

[0018] According to another aspect of the present invention, an embodiment of the present invention further provides a computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable a processor to implement the database-based large data set operation method described in any embodiment of the present invention when executed.

[0019] The technical solution of the embodiment of the present invention is to divide the operation data set into corresponding division results by the data volume of the operation data set, a preset first data operation threshold and a preset hash function; when the size of two data sets exceeds the threshold, the data can be adaptively divided to facilitate the processing of subsequent data set operations; the data set operation subtasks for performing set operations are determined according to the division results, and the set operations are classified for each data set operation subtask according to the data volume of each data set operation subtask and the preset second data operation threshold, and the divided subtasks can be classified again to automatically select a better set operation method based on the data skew scenario; the data set operations of each data set operation subtask in the set operation classification are performed in parallel based on the maximum number of parallel threads corresponding to the data set operation subtask and the preset set operation operation, and the target data operation result set is obtained by merging each data set operation, so that the subtasks can be accurately classified and a better set operation method can be automatically selected to solve the problems of performance, excessive memory resource usage and poor parallel processing encountered in database set operations when the data scale is extremely large.

[0020] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present invention, nor is it intended to limit the scope of the present invention. Other features of the present invention will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 A flowchart of a large data set operation method based on a database provided by one embodiment of the present invention;

[0023] Figure 2 A flowchart of another database-based large data set operation method provided by one embodiment of the present invention;

[0024] Figure 3 A schematic diagram of a method for automatic data sharding and segmentation according to an embodiment of the present invention;

[0025] Figure 4 A diagram showing the principle of dividing data set operation subtasks according to an embodiment of the present invention;

[0026] Figure 5 A diagram showing the principle of classifying database set operation subtasks provided by one embodiment of the present invention;

[0027] Figure 6 A schematic diagram of a hash-based sharding set operation principle provided by one embodiment of the present invention;

[0028] Figure 7 A schematic diagram of a shard set operation based on external sorting provided by one embodiment of the present invention;

[0029] Figure 8 A schematic diagram of a set operation subtask based on external sorting provided by an embodiment of the present invention

[0030] Figure 9 A schematic diagram of a data set operation for parallel execution task merging provided by one embodiment of the present invention;

[0031] Figure 10 A block diagram of a large data set computing device based on a database according to an embodiment of the present invention;

[0032] Figure 11 A schematic structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0033] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0034] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0035] In one embodiment, Figure 1 A flowchart of a database-based large data set operation method provided for one embodiment of the present invention. This embodiment is applicable to the situation when a set operation is performed on a certain database node in a distributed database. The method can be executed by a database-based large data set operation device. The database-based large data set operation device can be implemented in the form of hardware and / or software. The database-based large data set operation device can be configured in an electronic device.

[0036] like Figure 1 As shown, in this embodiment, a large data set operation method based on a database is applied to a database node. The method specifically includes the following steps:

[0037] S110 , obtaining the operation data sets corresponding to the left child node and the right child node respectively.

[0038] The left child node and the right child node can be understood as a node in a distributed database, and each node corresponds to a corresponding operation data set, that is, an operation data set for performing set operations.

[0039] In this embodiment, the operation data set includes: a first operation data set corresponding to the left child node and a second operation data set corresponding to the right child node. In this embodiment, the first operation data set corresponding to the left child node and the second operation data set corresponding to the right child node may include but are not limited to a table format with multiple rows and columns. The operation data sets corresponding to the left child node and the right child node may be queried respectively. When the data scale threshold is met, the corresponding set operation operations are performed on the queried first operation data set and the second operation data set. This embodiment does not impose any restrictions here. For example, the first operation data set is represented as t1.a, t1.b; the second operation data set is represented as t2.a, t2.b.

[0040] S120 , dividing the operation data set according to the data size of the operation data set, a preset first data operation threshold, and a preset hash function to obtain corresponding division results.

[0041] The data size refers to the data size corresponding to the first operation data set corresponding to the left child node and the second operation data set corresponding to the right child node, respectively. The preset first data operation threshold can be understood as the data operation threshold corresponding to the first operation data set and the second operation data set when performing set operations. The preset first data operation threshold can be manually defined according to needs or based on experience, and this embodiment does not impose any restrictions on this.

[0042] In this embodiment, the number of data set shards corresponding to the first operation data set and the number of data set shards corresponding to the second operation data set can be determined by presetting a fixed number of shards, or by dynamically evaluating the number of shards. The first operation data set and the second operation data set are segmented according to the number of data set shards corresponding to the first operation data set and the second operation data set, respectively, and the preset hash function to obtain corresponding segmentation results. In this embodiment, the corresponding segmentation results obtained by segmenting the operation data set include at least: at least two slice data files for data segmentation of each operation data set, the segmentation number, and the data size corresponding to each of the slice data files. It should be noted that the preset hash function used when segmenting the first operation data set and the second operation data set is the same hash function, which can be understood as using the same preset hash function to segment the first operation data set and the second operation data set, respectively.

[0043] S130 , determining data set operation subtasks for performing set operations according to the segmentation results, and classifying the data set operation subtasks according to the data size of each data set operation subtask and a preset second data operation threshold.

[0044] The data set operation subtask refers to the split numbers of at least two slice data files corresponding to the splitting of the first and second operation data sets, respectively. Slice data files with the same split number are identified as one data set operation subtask. This can be understood as follows: splitting the first operation data set corresponds to the first split number of at least two slice data files, and splitting the second operation data set corresponds to the second split number of at least two slice data files. Slice data files with the same split number between the first and second split numbers are identified as one data set operation subtask.

[0045] In this embodiment, each data set operation subtask corresponds to a corresponding data volume size, and the data volume size corresponds to a corresponding preset second data operation threshold value, and the data set operation subtask can be classified into set operations by the preset second data operation threshold value. In some embodiments, the categories of set operation classification may include: hash set operation queue and external sorting set operation queue; different set operation classification categories correspond to different set operation methods. Among them, the first data set operation and the second data set operation are data set operation subtasks executed in parallel. In this embodiment, the preset second data operation threshold value is smaller than the preset first data operation threshold value, and the second data operation threshold value can be set manually according to needs, and this embodiment does not limit it here.

[0046] In this embodiment, shard data files with the same sharding number can be searched in the sharding results corresponding to the first operation data set and the second operation data set respectively, and the shard data files with the same sharding number can be assigned to one data set operation subtask; by counting the total data volume of each data set operation subtask, each data set operation subtask is classified into a set operation according to the comparison result between the total data volume and the preset second data operation threshold; specifically, when the total data volume exceeds the preset second data operation threshold, the data set operation subtask that exceeds the preset second data operation threshold is divided into an external sorting set operation queue, waiting for the execution of the first data set operation; the data set operation subtask whose total data volume does not exceed the preset second data operation threshold is divided into a hash set operation queue, waiting for the execution of the second data set operation.

[0047] S140 , executing data set operations of each data set operation subtask in the set operation classification in parallel based on the maximum number of parallel threads corresponding to the data set operation subtask and the preset set operation operation, and merging each data set operation to obtain a target data operation result set.

[0048] The maximum number of parallel threads refers to the maximum number of threads that can execute each data set operation subtask in parallel. This maximum number of parallel threads can be set as needed and is not limited in this embodiment. The preset set operation refers to an operation related to performing set operations, which may include but is not limited to any of the following: union (UNION), intersection (INTERSECT), and difference (MINUS / EXCEPT).

[0049] In this embodiment, different set operation modes are selected for performing corresponding data set operations according to different set operation categories. In some embodiments, when the set operation classification category is a hash set operation queue, a hash table can be pre-built using data elements in one of the slice data files for the same number in each data set operation subtask, and each data element in the other slice data file can be traversed to determine the number of occurrences of each data element in the hash table. Then, a data set operation is performed on the first slice data file and the second slice data file in the data set operation subtask based on the number of occurrences and a preset set operation. In other embodiments, for the slice data files for the same number in each data set operation subtask, the slice data files are multi-way split to obtain corresponding split data sub-files, each slice data sub-file is multi-way sorted according to a preset sorting order, and the sorted new first slice data sub-file and the new second slice data sub-file are multi-way sorted to generate corresponding target slice data files, the data elements in the target slice data files are traversed, and the number of data elements in the target slice data files is determined; the data elements and the number of data elements are written into a pre-configured memory hash table; and the data set operation is performed on the first slice data file and the second slice data file in the data set operation subtask based on the memory hash table and the preset set operation. This embodiment is not limited here.

[0050] The technical solution of the embodiment of the present invention is to divide the operation data set into corresponding division results by dividing the operation data set according to the data volume of the operation data set, a preset first data operation threshold and a preset hash function, determine the data set operation subtask for performing the set operation according to the division results, and classify the data set operation subtasks into set operations according to the data volume of each data set operation subtask and the preset second data operation threshold; based on the maximum number of parallel threads corresponding to the data set operation subtask and the preset set operation operation, the data set operation of each data set operation subtask in the set operation classification is performed in parallel, and by merging each data set operation to obtain a target data operation result set, the subtasks can be accurately classified and a better set operation method can be automatically selected to solve the problems of performance, excessive memory resource usage and poor parallel processing encountered in database set operations when the data scale is extremely large.

[0051] In one embodiment, Figure 2A flowchart of another database-based large data set operation method provided for one embodiment of the present invention. Based on the above embodiments, this embodiment divides the operation data set according to the data size of the operation data set, a preset first data operation threshold and a preset hash function to obtain corresponding division results; determines the data set operation subtasks for performing the set operation according to the division results, and classifies the data set operation subtasks according to the data size of each data set operation subtask and the preset second data operation threshold, and further refines the data set operation of each data set operation subtask in the set operation classification in parallel based on the maximum number of parallel threads corresponding to the data set operation subtask and the preset set operation operation.

[0052] like Figure 2 As shown, the database-based large data set operation method in this embodiment may specifically include the following steps:

[0053] S210 , obtaining the operation data sets corresponding to the left child node and the right child node respectively.

[0054] S220: Determine the number of data set shards corresponding to the first operation data set and the second operation data set respectively; wherein determining the number of data set shards includes: presetting a fixed number of shards, or dynamically evaluating the number of shards.

[0055] In this embodiment, the preset fixed number of shards can be understood as a fixed number of shards set manually, that is, the fixed number of shards is directly set by the user according to the needs, and the query result sets corresponding to the first operation data set and the second operation data set can be divided into P parts according to the user's preset fixed number of shards; the dynamic evaluation of the number of shards can be understood as evaluation by the data node based on the estimated size of the SELECT query result sets on both sides, the maximum number of parallel threads of the set operation operator and the preset row number threshold.

[0056] In one embodiment, when the number of data set shards is a dynamically evaluated number of shards, determining the number of data set shards corresponding to the first operation data set and the second operation data set, respectively, includes:

[0057] Evaluate the data volumes corresponding to the first operation data set and the second operation data set respectively;

[0058] Determining the maximum data size when performing the set operation based on the data size of the first operation data set and the data size of the second operation data set;

[0059] Determining a first ratio between a preset first data operation threshold and a preconfigured maximum number of parallel threads for collective operations;

[0060] The number of data set shards corresponding to the first operation data set and the second operation data set is determined according to the maximum data size and the first ratio.

[0061] In this embodiment, the amount of data on the data set operated by each parallel thread in each data set shard satisfies a preset first data operation threshold.

[0062] In this embodiment, the data sizes corresponding to the first operation data set and the second operation data set are evaluated respectively; the maximum data size when performing set operations is determined based on the data size of the first operation data set and the data size of the second operation data set; and the number of data set shards corresponding to the first operation data set and the second operation data set are determined based on a first ratio between a preset first data operation threshold and a pre-configured maximum number of parallel threads for set operations.

[0063] Specifically, while determining the number of dataset shards includes setting a fixed number of shards, it also supports dynamic shard evaluation. If the number of dataset shards is fixed and set directly by the user, the query result sets of the first and second computation datasets can be divided into P parts based on the user's preset value. If the number of dataset shards is automatically evaluated, the evaluation is based on the estimated SELECT query result set sizes of the first and second computation datasets, the maximum number of parallel threads for the set operation operator, and a preset row count threshold. Specifically, the automatic shard evaluation rule is as follows: Based on the data sizes of the first and second computation datasets, assuming the estimated SELECT query result set sizes of M and N rows, respectively, if the algorithm exceeds threshold 1, the algorithm calculates the number of shard files into which the two tables' data should be split. Threshold 1 in the algorithm represents the maximum amount of data allowed for each set operation. For example, if the row count threshold is set to n, n rows of data are allowed to participate in each set operation. Assuming the maximum number of parallel threads is p, each thread is only allowed to perform set operations on n / p rows of data. We need to split the data of the two tables into P shard files to meet the threshold requirements under the condition of maximum parallelism, where P = MAX(M,N) / (n / p) rounded up. To better understand the principle diagram of automatic data sharding and segmentation, Figure 3 A schematic diagram of a method for automatically sharding and partitioning data, provided in one embodiment of the present invention. In this embodiment, the data is partitioned using the same hash function, which maps the data to fixed-length output values ​​and is typically used to divide data into different partitions, shards, or buckets.

[0064] S230 , dividing the first operation data set and the second operation data set according to the number of data set fragments and a preset hash function to obtain corresponding division results.

[0065] In this embodiment, after determining the number of data set shards corresponding to the first operation data set and the second operation data set respectively, the first operation data set and the second operation data set can be segmented using the same preset hash function to obtain corresponding segmentation results. In this embodiment, the segmentation results include at least: at least two slice data files for data segmentation of each operation data set, the segmentation number, and the data size corresponding to each slice data file.

[0066] In one embodiment, S230 may include: S2301-S2303:

[0067] S2301 : Encode the first operation data set and the second operation data set into binary character string forms respectively to obtain the encoded first encoded operation data set and the second encoded operation data set.

[0068] In this embodiment, the first operation data set and the second operation data set can be respectively encoded into binary character strings to obtain a first encoded operation data set corresponding to the encoded first operation data set and a second encoded operation data set corresponding to the second operation data set.

[0069] S2302: Map the first encoding operation data set and the second encoding operation data set into corresponding hash values ​​according to a preset hash function.

[0070] In this embodiment, the first encoding operation data set and the second encoding operation data set can be mapped to corresponding hash values ​​according to a preset hash function; wherein the preset hash function used when mapping the first encoding operation data set and the second encoding operation data set is the same hash function.

[0071] S2303. Perform a modulo operation on the number of data set slices corresponding to the first coding operation data set and the second coding operation data set according to the hash value, thereby dividing the first coding operation data set and the second coding operation data set into P hash buckets to obtain at least two slice data files; and number the slice data files respectively, and count the data size corresponding to each slice data file.

[0072] Where P represents the number of data set shards corresponding to the first encoding operation data set and the second encoding operation data set; each hash bucket corresponds to a disk shard file.

[0073] In this embodiment, a modulo operation can be performed on the number of data set shards corresponding to the first encoding operation data set and the second encoding operation data set according to the hash value, so that the first encoding operation data set and the second encoding operation data set are respectively divided into P hash buckets to obtain at least two slice data files; and the slice data files are numbered respectively, and the data size corresponding to each slice data file is counted.

[0074] Exemplarily, the slice data files are P, the t1 table data is the first operation data set in the above embodiment, and the t2 table data is the second operation data set in the above embodiment. According to the P calculated by automatic evaluation or the P set by the user, the t1 table data is loaded, and the hash function A is used to divide the data (t1.at1.b) into P hash buckets, numbered from 0 to P-1 respectively. Each hash bucket corresponds to a shard file on the disk, and the data size of each shard file is counted. At the same time, the t2 table data uses the same hash function A to divide the data (t2.a t2.b) into P hash buckets, numbered from 0 to P-1 respectively. The data in each hash bucket is written to the corresponding shard file, and the data size of each shard file is counted.

[0075] S240 , searching for a shard data file having the same segmentation number in the segmentation results corresponding to the first operation data set and the second operation data set, respectively, according to the segmentation number.

[0076] In this embodiment, the shard data files with the same shard number are searched in the shard results corresponding to the first operation data set and the second operation data set according to the shard number. In this embodiment, the categories of set operations include: hash set operation queue and external sort set operation queue.

[0077] S250: Assign the shard data files with the same segmentation number to one data set operation subtask.

[0078] In this embodiment, in the segmentation results corresponding to the first operation data set and the segmentation results corresponding to the second operation data set, the shard data files with the same segmentation number are assigned to one data set operation subtask.

[0079] The example line is for better understanding of the principle diagram of data set operation subtask division. Figure 4 A diagram showing the principle of dividing data set operation subtasks provided by an embodiment of the present invention is shown as follows: Figure 4 As shown in the figure, after the SELECT query result sets on the left and right sides are sharded, the shards with the same shard number are grouped into one set operation subtask.

[0080] S260 , counting the total data size of each data set operation subtask, and determining whether the total data size exceeds a preset second data operation threshold, if so, executing S270 ; if not, executing S280 .

[0081] In this embodiment, the total data size of each data set operation subtask is counted, and it is determined whether the total data size exceeds the preset second data operation threshold. If so, the data set operation subtask that exceeds the preset second data operation threshold is divided into the external sorting set operation queue, waiting for the execution of the first data set operation; if not, the data set operation subtask that does not exceed the preset second data operation threshold is divided into the hash set operation queue, waiting for the execution of the second data set operation.

[0082] S270 , assigning data set operation subtasks exceeding a preset second data operation threshold to an external sorted set operation queue, waiting for execution of a first data set operation.

[0083] S280: Assign the data set operation subtasks that do not exceed the preset second data operation threshold to a hash set operation queue to wait for the execution of the second data set operation.

[0084] The first data set operation and the second data set operation are data set operation subtasks executed in parallel.

[0085] For example, to better understand the classification principle diagram of database set operation subtasks, Figure 5 A schematic diagram of a database set operation subtask classification principle provided by an embodiment of the present invention, such as Figure 5 As shown, the data set operation subtasks that exceed the preset second data operation threshold are divided into the external sorting set operation queue, waiting for the execution of the first data set operation, and the data set operation subtasks that do not exceed the preset second data operation threshold are divided into the hash set operation queue, waiting for the execution of the second data set operation.

[0086] S290: For the first slice data file and the second slice data file in each data set operation subtask, pre-build a hash table using the first data element in the first slice data file, and traverse each second data element in the second slice data file to determine the number of times each second data element appears in the hash table.

[0087] The first slice data file corresponds to the first operation data set; the second slice data file corresponds to the second operation data set.

[0088] In this embodiment, when the category of the set operation classification is a hash set operation queue, for the first slice data file and the second slice data file in each data set operation subtask, a hash table is pre-built using the first data element in the first slice data file, and each second data element in the second slice data file is traversed to determine the number of times each second data element appears in the hash table.

[0089] S2100. Perform a data set operation on the first slice data file and the second slice data file in the data set operation subtask according to the number of times the second data element appears in the hash table and the preset set operation operation, and merge the data set operations to obtain a target data operation result set.

[0090] The preset set operation includes at least intersection, union and difference.

[0091] In this embodiment, the first slice data file and the second slice data file in the data set operation subtask are subjected to data set operation according to the number of times the second data element appears in the hash table and the preset set operation operation. For example, to facilitate a better understanding of the principle diagram of the hash-based shard set operation, Figure 6 A schematic diagram of a hash-based sharding set operation principle provided by one embodiment of the present invention. Figure 6 As shown in the figure, this is an optional method for performing set operations based on hashing. This method uses a hash table to count the number of times each element appears on both the left and right sides, then scans the hash table and completes the set operation according to the specified set operation. Specifically, Figure 6 As shown, for the intersection, if (6, 'a') appears on both the left and right sides, it is added to the result set; for the difference, if (6, 'a') appears more times on the left than on the right, it is added to the result set; for the union, if (6, 'a') appears on the left or right, it is added to the result set.

[0092] S2110 , for the first slice data file and the second slice data file in each data set operation subtask, perform multi-way splitting on the first slice data file and the second slice data file to obtain corresponding first slice data sub-files and second slice data sub-files.

[0093] In this embodiment, when the category of the set operation classification is the external sorted set operation queue, for the first slice data file and the second slice data file in each data set operation subtask, the first slice data file and the second slice data file are multi-way split to obtain the corresponding first slice data sub-file and second slice data sub-file.

[0094] S2120 , performing multi-way sorting on each of the first slice data sub-files and the second slice data sub-files according to a preset sorting order to obtain sorted new first slice data sub-files and new second slice data sub-files.

[0095] The preset sorting order is ascending or descending, and the preset sorting order corresponding to the first slice data sub-file and the second slice data sub-file is the same; the multi-way sorting is a parallel or non-parallel method.

[0096] In this embodiment, each first slice data subfile and second slice data subfile is multi-way sorted in ascending or descending order to obtain a new first slice data subfile and a new second slice data subfile after sorting. It should be noted that the preset sorting order corresponding to the first slice data subfile and the second slice data subfile is the same. It can be understood that if the order of the multi-way sorting of each first slice data subfile is ascending, the second slice data subfile is also sorted in ascending order; if the order of the multi-way sorting of each first slice data subfile is descending, the second slice data subfile is also sorted in descending order.

[0097] S2130 : Perform multi-way merging on the new first slice data sub-file and the new second slice data sub-file to generate corresponding first target slice data sub-file and second target slice data sub-file.

[0098] In this embodiment, multiplexing is performed on the new first slice data sub-file and the new second slice data sub-file to generate the corresponding first target slice data sub-file and second target slice data sub-file.

[0099] S2140 , traverse the data elements in the first target slice data sub-file and the second target slice data sub-file, and determine the number of data elements existing in the first target slice data sub-file and the second target slice data sub-file respectively.

[0100] In this embodiment, the data elements in the first target slice data sub-file and the second target slice data sub-file are traversed, and the number of data elements in the first target slice data sub-file and the second target slice data sub-file is determined.

[0101] S2150. Write the data elements and the number of existing ones into a pre-configured memory hash table.

[0102] In this embodiment, the data elements and the number of occurrences are written into a pre-configured in-memory hash table.

[0103] S2160 , performing a data set operation on the first slice data file and the second slice data file in the data set operation subtask according to the memory hash table and the preset set operation operation, and merging the data set operations to obtain a target data operation result set.

[0104] In this embodiment, a data set operation is performed on the first slice data file and the second slice data file in the data set operation subtask according to the memory hash table and the preset set operation operation, and each data set operation is merged to obtain a target data operation result set.

[0105] For example, to better understand the principle diagram of shard set operation based on external sorting, Figure 7 A schematic diagram of a shard set operation based on external sorting provided by one embodiment of the present invention. Figure 8 A schematic diagram of a set operation subtask based on external sorting is provided for one embodiment of the present invention. In this embodiment, a multi-way merge sorting algorithm is adopted. The fragmented files on both sides of the subtask are first segmented separately, and the memory sorting is completed for the segmented small files. Then, multiple new sorted small files are generated. Finally, multiple small files are multi-way merged to generate the final ordered fragmented files. When multi-way merge sorting is performed on the fragmented files on the left and right sides of the subtask, a certain number of threads can be started to perform multi-way merge sorting in parallel. The subtask that performs set operations based on sorting can control the memory hash table because it is ordered data. For example, for the tuple (15'b'), since the data is ordered, it is easy to determine whether the element exists on the left and right sides, and the corresponding number of elements, and then perform set operations. After the set operation is completed, the result of the row can be directly output, or the hash table can be generated. After accumulating multiple rows of data, the set operation results of these rows of data are output together. After completing the calculation of part of the number of rows, the memory hash table can be cleared and the memory can be reused. Figure 7 As shown, for the intersection, (15, 'b') only appears on the left, so the row is not in the result set; for the difference, (15, 'b') appears more times on the left than on the right, so it is added to the result set; for the union, (15, 'b') appears on the left or the right, so it is added to the result set.

[0106] In order to better understand the parallel execution of tasks for each data set operation, the target data operation result set is obtained by merging the tasks. Figure 9 A schematic diagram of a data set operation for parallel execution task merging provided by one embodiment of the present invention.

[0107] The above technical solution of the embodiment of the present invention determines the number of data set shards corresponding to the first operation data set and the second operation data set respectively, performs data set sharding on the first operation data set and the second operation data set according to the number of data set shards and a preset hash function to obtain corresponding sharding results, searches for shard data files with the same sharding number in the sharding results corresponding to the first operation data set and the second operation data set respectively according to the sharding number; assigns shard data files with the same sharding number to one data set operation subtask; counts the total data volume of each data set operation subtask, and determines whether the total data volume exceeds a preset second data operation threshold; if it exceeds the preset second data operation threshold, the data set operation that exceeds the preset second data operation threshold is performed. The operator task is divided into the external sorted set operation queue, waiting for the execution of the first data set operation; if it does not exceed the preset second data operation threshold, the data set operation subtask that does not exceed the preset second data operation threshold is divided into the hash set operation queue, waiting for the execution of the second data set operation. Based on the hash set operation queue or the external sorted set operation queue, as well as the maximum number of parallel threads corresponding to the data set operation subtask and the preset set operation operation, the data set operation of each data set operation subtask is executed in parallel, which can further accurately classify the subtasks and automatically select a better set operation method, so as to solve the problems of performance, high memory resource usage and inability to be processed in parallel when the database set operation is extremely large.

[0108] For example, in order to better understand the operation method of large data set operation based on the database, in this embodiment, the result sets of two SELECT queries are assumed to be subjected to set operation, select t1.a, t1.b from t1 setopselect t2.a, t2.b from t2 as an example for explanation, where setop is a set operation, which can be any one of union (UNION), intersection (INTERSECT), and difference (MINUS / EXCEPT).

[0109] a1. Assume the estimated result sets of the left and right SELECT queries are M and N rows, respectively. The row count threshold is set to n, and the number of parallel threads for set operations is set to p. Each thread is only allowed to perform set operations on n / p rows of data. To meet the threshold requirement while maintaining maximum parallelism, we need to split the data in both tables into P shard files, where P = MAX(M,N) / (n / p), rounded up.

[0110] a2. Load the data from table t1 based on the obtained P. Use hash function A to partition the data (t1.a t1.b) into P hash buckets, numbering them from 0 to P-1. Each hash bucket corresponds to a shard file on disk. The algorithm calculates the data size of each shard file. Use the same hash function A to partition the data (t2.a t2.b) from table t2 into P hash buckets, numbering them from 0 to P-1. Write the data in each hash bucket to the corresponding shard file, and calculate the data size of each shard file.

[0111] a3. Count the data size of each shard file and categorize the subtasks (a subtask is a shard file with the same number (same hash bucket) on both sides). If the data size of a subtask exceeds the threshold, the subtask is added to the external sorted set operation task queue. If it does not exceed the preset threshold, the subtask is added to the hash-based set operation task queue.

[0112] a4. Based on the preset maximum number of parallel threads p for the set operation operator, start p threads, read tasks from the hash-based and external-sorted task queues, and update the total scale of the set operation in progress. In this embodiment, the set operation classification category is the external-sorted set operation queue, and the set operation classification category is the hash set operation queue.

[0113] After a5, p worker threads have subtasks that complete the collective operation, they directly output the results upstream, reducing the total data size involved in the collective operation. Outputting the results upstream immediately after the subtasks complete can reduce the memory usage of the collective operation operator and shorten the time the upstream operator waits for the results.

[0114] This technical solution is mainly for distributed databases, specifically for a certain database node. When the set operation starts, if the data scale exceeds a given threshold, the same hash function is used to split the two data sets involved in the set operation into multiple independent subtasks to ensure that the data scale of each subtask meets the threshold we set as much as possible. During the data segmentation process, the segmented data is written to the disk to reduce memory usage. Since data segmentation is not necessarily uniform in actual production, and may not always meet the set threshold, the algorithm will classify the multiple generated subtasks according to the data volume and the set threshold. By automatically completing data segmentation, multiple independent subtasks are generated, and the subtasks can be processed in parallel. The subtasks can automatically select a relatively better set operation algorithm based on their classification to complete the entire set operation.

[0115] In one embodiment, Figure 10This is a block diagram of a database-based large data set operation device according to one embodiment of the present invention. This device is suitable for performing set operations on a specific database node in a distributed database. The device can be implemented in hardware or software. It can be configured in an electronic device to implement a database-based large data set operation processing method according to an embodiment of the present invention.

[0116] like Figure 10 As shown, the device is applied to a database node and includes: an acquisition module 1010 , a segmentation module 1020 , a classification module 1030 and a calculation module 1040 .

[0117] The acquisition module 1010 is used to acquire the operation data sets corresponding to the left child node and the right child node respectively;

[0118] A segmentation module 1020 is configured to segment the operation data set according to the data size of the operation data set, a preset first data operation threshold, and a preset hash function to obtain corresponding segmentation results;

[0119] a classification module 1030 for determining data set operation subtasks for performing set operations based on the segmentation results, and performing set operation classification on each of the data set operation subtasks based on the data size of each of the data set operation subtasks and a preset second data operation threshold;

[0120] The operation module 1040 is used to execute the data set operation of each data set operation subtask in the set operation classification in parallel based on the maximum number of parallel threads corresponding to the data set operation subtask and the preset set operation operation, and merge each of the data set operations to obtain a target data operation result set.

[0121] In an embodiment of the present invention, a segmentation module segments an operation data set according to the data volume of the operation data set, a preset first data operation threshold and a preset hash function to obtain corresponding segmentation results; a classification module determines a data set operation subtask for performing set operation according to the segmentation result, and performs set operation classification on each data set operation subtask according to the data volume of each data set operation subtask and a preset second data operation threshold; an operation module performs data set operation of each data set operation subtask in the set operation classification in parallel based on the maximum number of parallel threads corresponding to the data set operation subtask and the preset set operation operation, and obtains a target data operation result set by merging each data set operation, thereby accurately classifying the subtasks and automatically selecting a more optimal set operation method, thereby solving the problems of performance, excessive memory resource usage and poor parallel processing encountered in database set operation when the data scale is extremely large.

[0122] In one embodiment, the operation data set includes: a first operation data set corresponding to the left child node and a second operation data set corresponding to the right child node; accordingly, the segmentation module 1020 includes:

[0123] A shard number determination unit, configured to determine the number of dataset shards corresponding to the first operation dataset and the second operation dataset, respectively; wherein determining the number of dataset shards includes: presetting a fixed number of shards, or dynamically evaluating the number of shards;

[0124] A result determination unit, configured to perform data set segmentation on the first operation data set and the second operation data set according to the number of data set shards and the preset hash function to obtain corresponding segmentation results;

[0125] The segmentation result at least includes: at least two slice data files for each of the operation data sets to be segmented, a segmentation number, and a data size corresponding to each of the slice data files.

[0126] In one embodiment, when the number of data set shards is the number of dynamic evaluation shards, the result determination unit includes:

[0127] an evaluation subunit, configured to evaluate the data amounts corresponding to the first operation data set and the second operation data set respectively;

[0128] A first determining subunit is configured to determine a maximum data size when performing a set operation based on a data size of the first operation data set and a data size of the second operation data set;

[0129] A second determining subunit is configured to determine a first ratio between a preset first data operation threshold and a pre-configured maximum number of parallel threads for collective operations;

[0130] The shard number determination subunit is used to determine the number of data set shards corresponding to the first operation data set and the second operation data set respectively according to the maximum data size and the first ratio.

[0131] In one embodiment, the result determination unit includes:

[0132] an encoding subunit, configured to encode the first operation data set and the second operation data set into binary character string forms to obtain encoded first encoded operation data set and second encoded operation data set;

[0133] a mapping subunit, configured to map the first encoding operation data set and the second encoding operation data set into corresponding hash values ​​according to the preset hash function; wherein the preset hash function used when mapping the first encoding operation data set and the second encoding operation data set is the same hash function;

[0134] The partitioning subunit is configured to perform a modulo operation on the number of dataset shards corresponding to the first encoding operation dataset and the second encoding operation dataset, respectively, according to the hash value, thereby partitioning the first encoding operation dataset and the second encoding operation dataset into P hash buckets to obtain at least two slice data files; and to number the slice data files respectively, and to count the data size corresponding to each slice data file; wherein P represents the number of dataset shards corresponding to the first encoding operation dataset and the second encoding operation dataset; and each hash bucket corresponds to a disk shard file.

[0135] In one embodiment, the categories of the set operation classification include: hash set operation queue and external sorting set operation queue; accordingly, the classification module 1030 includes:

[0136] a search unit, configured to search for a shard data file having the same shard number in the shard results corresponding to the first operation data set and the second operation data set respectively according to the shard number;

[0137] a determining unit, configured to assign the shard data files with the same shard number to one data set operation subtask;

[0138] a judging unit, configured to count the total data size of each of the data set operation subtasks, and judge whether the total data size exceeds a preset second data operation threshold;

[0139] a first partitioning unit, configured to partition the data set operation subtask exceeding the preset second data operation threshold into the outer sorted set operation queue to wait for execution of the first data set operation if the data set operation subtask exceeds the preset second data operation threshold;

[0140] The second division unit is used to divide the data set operation subtask that does not exceed the preset second data operation threshold into the hash set operation queue to wait for the execution of the second data set operation if the data set operation does not exceed the preset second data operation threshold; wherein the first data set operation and the second data set operation are data set operation subtasks executed in parallel.

[0141] In one embodiment, the category of the set operation classification is a hash set operation queue; accordingly, the operation module 1040 includes:

[0142] a traversal unit, configured to, for each of the first slice data files and the second slice data files in each of the data set operation subtasks, pre-build a hash table using the first data element in the first slice data file, and traverse each second data element in the second slice data file to determine the number of occurrences of each second data element in the hash table; wherein the first slice data file corresponds to the first operation data set; and the second slice data file corresponds to the second operation data set;

[0143] An operation unit is used to perform a data set operation on the first slice data file and the second slice data file in the data set operation subtask according to the number of times the second data element appears in the hash table and a preset set operation operation; wherein the preset set operation operation includes at least intersection, union and difference.

[0144] In one embodiment, the category of the set operation classification is an external sorted set operation queue; accordingly, the operation module 1040 further includes:

[0145] a splitting unit, configured to perform multi-way splitting on the first slice data file and the second slice data file in each of the data set operation subtasks to obtain corresponding first slice data subfiles and second slice data subfiles;

[0146] a sorting unit, configured to perform multi-way sorting on each of the first slice data subfiles and the second slice data subfiles according to a preset sorting order to obtain sorted new first slice data subfiles and new second slice data subfiles; wherein the preset sorting order is ascending or descending, and the preset sorting order corresponding to the first slice data subfile and the second slice data subfile is the same; and the multi-way sorting is performed in parallel or non-parallel manner;

[0147] A generating unit, configured to perform multi-way merging on the new first slice data subfile and the new second slice data subfile to generate corresponding first target slice data subfile and second target slice data subfile;

[0148] a determining unit, configured to traverse the data elements in the first target slice data subfile and the second target slice data subfile, and determine the number of the data elements existing in the first target slice data subfile and the second target slice data subfile respectively;

[0149] a writing unit, configured to write the data element and the existing quantity into the preconfigured memory hash table;

[0150] The operation unit is used to perform a data set operation on the first slice data file and the second slice data file in the data set operation subtask according to the memory hash table and a preset set operation.

[0151] The database-based big data set operation device provided in the embodiment of the present invention can execute the database-based big data set operation processing method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0152] In one embodiment, Figure 11 A schematic diagram of the structure of an electronic device provided for an embodiment of the present invention. The electronic device 10 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0153] like Figure 11 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., which is communicatively connected to the at least one processor 11. The memory stores a computer program that can be executed by the at least one processor. The processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 into the random access memory (RAM) 13. Various programs and data required for the operation of the electronic device 10 can also be stored in the RAM 13. The processor 11, ROM 12, and RAM 13 are connected to each other via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0154] Multiple components in the electronic device 10 are connected to the I / O interface 15, including an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a magnetic disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0155] The processor 11 can be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as a database-based large data set operation method.

[0156] In some embodiments, the database-based large data set operation processing method can be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as the storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the database-based large data set operation method described above can be performed. Alternatively, in other embodiments, the processor 11 can be configured to perform the database-based large data set operation method in any other appropriate manner (for example, by means of firmware).

[0157] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0158] Computer programs for implementing the methods of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable database-based large data set computing device, so that when the computer program is executed by the processor, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The computer program can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0159] In the context of the present invention, computer-readable storage media can be tangible media that can contain or store a computer program for use with an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. Computer-readable storage media can include but are not limited to electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, computer-readable storage media can be machine-readable signal media. More specific examples of machine-readable storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0160] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0161] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0162] A computing system may include clients and servers. The clients and servers are typically remote from each other and typically interact via a communication network. This client-server relationship arises through computer programs running on the respective computers, creating a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host. This server is a hosting product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosting and VPS services.

[0163] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in the present invention can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved. This is not limited herein.

[0164] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A large data set computing method based on a database, characterized in that: Applied to a database node, the method comprises: Obtaining operation data sets corresponding to the left child node and the right child node respectively, wherein the operation data sets include: a first operation data set corresponding to the left child node and a second operation data set corresponding to the right child node; The method comprises: determining the number of data set fragments corresponding to the first operation data set and the second operation data set respectively; and performing data set fragmentation on the first operation data set and the second operation data set according to the number of data set fragments and the preset hash function to obtain corresponding fragmentation results; Determine a data set operation subtask for performing set operation according to the segmentation result, and classify each data set operation subtask by set operation according to the data amount of each data set operation subtask and a preset second data operation threshold; The categories of the set operation classification include: hash set operation queue and external sorting set operation queue; accordingly, determining the data set operation subtask to perform the set operation according to the segmentation result, and classifying the data set operation subtask according to the data volume of each data set operation subtask and the preset second data operation threshold, including: Counting the total data size of each of the data set operation subtasks, and determining whether the total data size exceeds a preset second data operation threshold; If the preset second data operation threshold is exceeded, the data set operation subtask exceeding the preset second data operation threshold is divided into the external sorting set operation queue to wait for the execution of the first data set operation; If the preset second data operation threshold is not exceeded, the data set operation subtask that does not exceed the preset second data operation threshold is divided into the hash set operation queue to wait for the execution of the second data set operation; Based on the maximum number of parallel threads corresponding to the data set operation subtask and the preset set operation operation, the data set operation of each data set operation subtask in the set operation classification is executed in parallel, and each of the data set operations is merged to obtain a target data operation result set.

2. The method according to claim 1, characterized in that Determining the number of shards of the data set includes: presetting a fixed number of shards, or dynamically evaluating the number of shards; The segmentation result at least includes: at least two slice data files for data segmentation of each of the operation data sets, a segmentation number, and a data volume size corresponding to each of the slice data files.

3. The method according to claim 2, characterized in that In a case where the number of data set shards is the number of dynamically evaluated shards, determining the number of data set shards corresponding to the first operation data set and the second operation data set respectively includes: Evaluating the data volumes corresponding to the first operation data set and the second operation data set respectively; Determining a maximum data size when performing a set operation according to the data size of the first operation data set and the data size of the second operation data set; Determine a first ratio between a preset first data operation threshold and a preconfigured maximum number of parallel threads for collective operations; The number of data set fragments corresponding to the first operation data set and the second operation data set respectively is determined according to the maximum data amount and the first ratio.

4. The method according to claim 2, characterized in that: The step of performing data set segmentation on the first operation data set and the second operation data set according to the data set segmentation number and the preset hash function to obtain corresponding segmentation results includes: Encoding the first operation data set and the second operation data set into binary character string forms to obtain a first encoded operation data set and a second encoded operation data set; Mapping the first encoding operation data set and the second encoding operation data set to corresponding hash values ​​according to the preset hash function; wherein the preset hash function used when mapping the first encoding operation data set and the second encoding operation data set is the same hash function; According to the hash value, a modulo operation is performed on the number of data set shards corresponding to the first coding operation data set and the second coding operation data set, so that the first coding operation data set and the second coding operation data set are respectively divided into P hash buckets to obtain at least two slice data files; and the slice data files are numbered respectively, and the data amount corresponding to each slice data file is counted; wherein each hash bucket corresponds to a disk shard file.

5. The method according to claim 2, characterized in that: Searching for a shard data file having the same shard number in the shard results corresponding to the first operation data set and the second operation data set respectively according to the shard number; Assign the shard data files with the same shard number to one data set operation subtask; The first data set operation and the second data set operation are data set operation subtasks executed in parallel.

6. The method according to claim 1, characterized in that The category of the set operation classification is a hash set operation queue; accordingly, the data set operation of each data set operation subtask in the set operation classification is performed in parallel based on the maximum number of parallel threads corresponding to the data set operation subtask and the preset set operation operation, including: For the first slice data file and the second slice data file in each of the data set operation subtasks, a hash table is pre-built using the first data element in the first slice data file, and each second data element in the second slice data file is traversed to determine the number of times each second data element appears in the hash table; wherein the first slice data file corresponds to the first operation data set; and the second slice data file corresponds to the second operation data set; The first slice data file and the second slice data file in the data set operation subtask are subjected to data set operation according to the number of times the second data element appears in the hash table and the preset set operation operation; wherein the preset set operation operation includes at least intersection, union and difference.

7. The method according to claim 1, characterized in that The category of the set operation classification is an external sorted set operation queue; accordingly, the data set operation of each data set operation subtask in the set operation classification is performed in parallel based on the maximum number of parallel threads corresponding to the data set operation subtask and the preset set operation operation, including: For the first slice data file and the second slice data file in each of the data set operation subtasks, respectively performing multi-way segmentation on the first slice data file and the second slice data file to obtain corresponding first slice data sub-files and second segmentation data sub-files; Perform multi-way sorting on each of the first slice data sub-files and the second slice data sub-files according to a preset sorting order to obtain sorted new first slice data sub-files and new second slice data sub-files; Performing multiplexing on the new first slice data subfile and the new second slice data subfile to generate corresponding first target slice data subfile and second target slice data subfile; Traversing the data elements in the first target slice data subfile and the second target slice data subfile, and determining the number of the data elements existing in the first target slice data subfile and the second target slice data subfile respectively; Writing the data element and the number of occurrences into a preconfigured in-memory hash table; A data set operation is performed on the first slice data file and the second slice data file in the data set operation subtask according to the memory hash table and the preset set operation.

8. A large data set computing device based on a database, characterized in that: Applied to a database node, the device comprises: An acquisition module, used to acquire operation data sets corresponding to a left child node and a right child node respectively, wherein the operation data sets include: a first operation data set corresponding to the left child node and a second operation data set corresponding to the right child node; A segmentation module, configured to segment the operation data set according to the data volume of the operation data set, a preset first data operation threshold and a preset hash function to obtain a corresponding segmentation result; The segmentation module includes: A shard number determining unit, used to determine the number of data set shards corresponding to the first operation data set and the second operation data set respectively; A result determination unit, configured to perform data set segmentation on the first operation data set and the second operation data set according to the data set segmentation number and the preset hash function to obtain corresponding segmentation results; A classification module, used for determining the data set operation subtasks for performing set operation according to the segmentation result, and performing set operation classification on each of the data set operation subtasks according to the data volume of each of the data set operation subtasks and a preset second data operation threshold, wherein the set operation classification categories include: hash set operation queue and external sorting set operation queue; The classification module comprises: A judging unit, used to count the total data size of each of the data set operation subtasks, and judge whether the total data size exceeds a preset second data operation threshold; A first partitioning unit is used to partition the data set operation subtask exceeding the preset second data operation threshold to the external sorting set operation queue to wait for the execution of the first data set operation if the data set operation subtask exceeds the preset second data operation threshold; A second division unit is used to divide the data set operation subtask that does not exceed the preset second data operation threshold into the hash set operation queue to wait for the execution of the second data set operation if the data set operation subtask does not exceed the preset second data operation threshold; The operation module is used to execute the data set operation of each data set operation subtask in the set operation classification in parallel based on the maximum number of parallel threads corresponding to the data set operation subtask and the preset set operation operation, and merge each data set operation to obtain the target data operation result set.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the database-based large data set operation method described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the database-based large data set operation method described in any one of claims 1 to 7 when executed.

Images