Data processing method and device, equipment, storage medium and program product

Abstract

The embodiment of the invention provides a data processing method and device, equipment, a storage medium and a program product. The method comprises the following steps: acquiring first data belonging to a target partition from a first data table; utilizing a Hash connection algorithm to generate a first table item in a Hash relation table in a memory for the first data; when a failure is caused by insufficient memory residual space when a second table item corresponding to second data belonging to the target partition in the first data table is to be generated, switching to a sorting merging connection algorithm; sorting the data belonging to the target partition obtained from the Hash relation table and the residual data belonging to the target partition obtained from the first data table by using a sorting, merging and connecting algorithm to obtain a first sorting result, and sorting the data belonging to the target partition obtained from the second data table to obtain a second sorting result; and based on the first sorting result and the second sorting result, executing a connection operation by using a sorting merging connection algorithm.

Description

Technical Field

[0001] This specification relates to the field of computer technology, and in particular to a data processing method, apparatus, device, storage medium, and program product. Background Technology

[0002] In Structured Query Language (SQL), the join operation is one of the core operations.

[0003] Currently, the commonly used join algorithm is the hash join algorithm. Hash join builds a hash table based on data from one side of the table and completes the connection by probing data from the other side. Its advantage is fast matching speed, but its disadvantage is high memory consumption. When memory is insufficient, the hash table cannot be built, leading to task failure.

[0004] Therefore, a solution is needed to address the task failure caused by hash table construction failure. Summary of the Invention

[0005] This specification provides a data processing method, apparatus, device, storage medium, and program product in several aspects to effectively utilize the technical advantages of hash join and sort-merge join algorithms to ensure the performance and stability of the query engine.

[0006] The first aspect of this specification provides a data processing method, including: Retrieve the first data belonging to the target partition from the first data table; Using a hash join algorithm, the first entry in a hash table is generated in memory for the first data; If the process of generating a corresponding second table entry for the second data belonging to the target partition in the first data table fails due to insufficient remaining memory, the algorithm is switched to the sort-merge-join algorithm. Using the sort-merge-join algorithm, the data belonging to the target partition obtained from the hash table and the remaining data belonging to the target partition obtained from the first data table are sorted to obtain a first sorting result. The data belonging to the target partition obtained from the second data table are then sorted to obtain a second sorting result. Based on the first sorting result and the second sorting result, the sorting-merge-join algorithm is used to perform a join operation.

[0007] A second aspect of this specification provides a data processing apparatus, comprising: The first acquisition module is used to acquire the first data belonging to the target partition from the first data table; The generation module is used to generate the first entry in a hash table in memory for the first data using a hash join algorithm. The switching module is used to switch to the sort-merge-join algorithm when the failure occurs due to insufficient remaining memory space when generating a corresponding second table entry for the second data belonging to the target partition in the first data table. The sorting module is used to sort the data belonging to the target partition obtained from the hash table and the remaining data belonging to the target partition obtained from the first data table using the sorting-merge-join algorithm to obtain a first sorting result, and to sort the data belonging to the target partition obtained from the second data table to obtain a second sorting result. An execution module is used to perform a join operation based on the first sorting result and the second sorting result using the sorting-merge-join algorithm.

[0008] A third aspect of this specification provides an electronic device, comprising: a memory and a processor, wherein, The memory is used to store programs; The processor, coupled to the memory, is configured to execute the program stored in the memory to implement the method described in any of the preceding embodiments.

[0009] A fourth aspect of this specification provides a computer-readable storage medium storing a computer program that, when executed by a computer, enables the implementation of any of the methods described above.

[0010] A fifth aspect of this specification provides a computer program product, including a computer program that, when executed by a processor, implements the method described in any of the preceding descriptions.

[0011] In the technical solution provided in the embodiments of this specification, when insufficient memory prevents the continued construction of entries in the hash table, the algorithm switches from hash join to sort-merge join to leverage the low memory consumption of sort-merge join. After switching to sort-merge join, data belonging to the target partition is retrieved from the hash table, and the remaining data belonging to the target partition is retrieved from the database, thus obtaining the complete data of the target partition. This ensures that sort-merge join can complete the join operation normally. Therefore, the technical solution provided in the embodiments of this specification only requires fault recovery within the problematic partition, avoiding the resource waste caused by large-scale rollback. Attached Figure Description

[0012] The accompanying drawings, which are provided to further illustrate this specification, form part of this specification.

[0013] Figure 1 This specification provides a schematic diagram of the structure of a data processing system as an exemplary embodiment. Figure 2 A flowchart illustrating a data processing method provided in an exemplary embodiment of this specification; Figure 3 A flowchart illustrating a data processing method provided in an exemplary embodiment of this specification; Figure 4 This is a schematic diagram of the structure of an electronic device provided as another exemplary embodiment of this specification. Detailed Implementation

[0014] To make the objectives, technical solutions, and advantages of this specification clearer, the technical solutions of this specification will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments, and not all of the embodiments. Based on these embodiments, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this specification.

[0015] It should be noted that, in the cases involving user information in the embodiments of this specification, the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of this specification are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse. In addition, the various models involved in this specification (including but not limited to language models or large models) comply with relevant laws and standards.

[0016] Before introducing the specific details of each embodiment in this specification, a brief explanation of the technical terms mentioned herein will be provided.

[0017] Join operations are fundamental relational database query operations, retrieving information that meets certain conditions from two different relations. There are many methods for implementing joins, among which hash joins have proven to have good performance; however, existing hash joins suffer from partition overflow. If partition overflow occurs during the join process, it severely degrades the algorithm's efficiency. Partition overflow occurs when the size of the partitioned data after hash partitioning is still greater than the available memory size. The main cause is data skew, and if partition overflow occurs, it will significantly reduce the efficiency of the join.

[0018] The shuffle hash join algorithm is a join algorithm used in distributed computing systems. The core idea of ​​the shuffle hash join algorithm is to "shuffle" two large datasets, sending data with the same join key to the same computing node, and then independently performing a standard hash join on each computing node. It uses the join key to hash partition the construction table (usually the smaller table in the two tables to be joined) and the probe table (usually the larger table in the two tables to be joined), assigning data with the same join key value to the same partition, allowing the hash join algorithm to be performed independently in each partition. Within each partition, a hash relation table is built based on the data from the construction table. Specifically, a hash function is used to calculate a hash value for the join key value from the construction table, and this hash value is used as the key, with the record as the value, and the key-value pair is inserted into the hash relation table. Then, the hash value of the join key value is calculated for the data from the probe table, and a lookup and matching is performed with the hash relation table in memory to find records that meet the join conditions and generate the merged result. The shuffle hash join algorithm has the technical advantage of high join efficiency. The shuffle hash join algorithm has an additional shuffling (i.e., partitioning) step compared to the hash join algorithm. In this article, the data in the data table refers to records.

[0019] The BytesToBytesMap data structure is used to build or store hash join tables. After mapping the join keys from the data in the built table to a hash value using a hash function, the record is inserted into the BytesToBytesMap, where the key is the hash value and the value is the data or a pointer to that data. This data structure allows for fast storage and retrieval of data in memory using key-value pairs, optimizing data access speed and improving the efficiency of hash join operations.

[0020] The core idea of ​​the Sort Merge Join algorithm is as follows: It uses a join key to perform hash partitioning on the two tables to be joined (assuming it's the first and second tables), ensuring that data with the same join key value is assigned to the same partition. Within each partition, the data from the first table is sorted according to the join key, resulting in sorted result A, and the data from the second table is sorted, resulting in sorted result B. Then, a first iterator is created for sorted result A, pointing to the beginning of sorted result A, to traverse sorted result A; a second iterator is created for sorted result B, pointing to the beginning of sorted result B, to traverse sorted result B. Based on these two iterators, a merge join is performed.

[0021] The code generation path is the execution path in SQL based on code generation. It compiles SQL operations into efficient Java bytecode, which is dynamically generated and executed at runtime, thereby reducing virtual machine overhead and improving computing performance.

[0022] Adaptive Query Execution (AQE) framework: used to dynamically optimize the execution plan based on data statistics at runtime, thereby improving query performance.

[0023] An iterator is a tool used to traverse data within a container. Its core function is to provide a unified data access interface. In a connected context, a container can be understood as a collection of data. A key characteristic of iterators is that data that has already been iterated over will not be iterated over again.

[0024] In Java, the remaining iterator usually refers to the forEachRemaining method of the Iterator class, which is used to iterate over the remaining data in a collection (i.e., untraveled or unprocessed data).

[0025] The sort-merge-join algorithm consists of two phases: the sorting phase and the merging phase.

[0026] 1. Sorting stage First, within the same partition, the data from the first data table and the data from the second data table need to be sorted based on the join key.

[0027] Memory consumption: Sorting algorithms (such as quicksort and mergesort) use external sorting. This means that the database will utilize available memory as much as possible. It reads a batch of data that can fit into memory, sorts it in memory, and then writes it back to disk as an ordered temporary block. This process is repeated until the entire partition is divided into multiple ordered blocks. Finally, mergesort is used to combine these ordered blocks on disk into a final ordered table.

[0028] Key point: The sorting phase does not require loading the entire partition's data into memory at once. It only needs a fixed-size memory buffer. Regardless of the amount of data in the partition, the sorting is completed in batches by reading and writing data to the disk. Memory consumption is controllable and relatively fixed.

[0029] 2. Merger Phase Once the data from the first data table and the data from the second data table within the same partition are sorted by the join key, it can be done as if merging two sorted linked lists. Simultaneously, scan from the beginning of both sorted linked lists, compare the join keys of the current row, and if they are equal, output the join result; otherwise, move the pointer of the table with the smaller key forward.

[0030] Memory consumption: The merging phase is a streaming process, which only requires storing a small portion of the data from the first data table and a small portion of the data from the second data table in memory.

[0031] The memory requirement is approximately equal to the size of the fixed sort buffer, and is independent of the table size. However, because the sort-merge-join algorithm requires more disk I / O, its efficiency is lower than that of the hash-join algorithm.

[0032] The hash join algorithm requires one-time memory; if a 10GB hash table needs to be built, at least 10GB of memory is required.

[0033] Sort-merge-join has "stream-controlled" memory requirements. It processes data in batches, uses a fixed memory window, and automatically overflows to disk when memory is full. Regardless of the size of the partition, it only uses 100MB of memory. The core of sort-merge-join is the outer sort-merge algorithm, specifically designed for handling large datasets.

[0034] In big data join query scenarios, although the shuffle hash join algorithm has the advantage of high efficiency, the preconditions for using the shuffle hash join algorithm are very strict. For example, the average data size from the building table in the same partition does not exceed the threshold parameter used to control whether to enable the broadcast join algorithm, and the size of the table on one side must be significantly smaller than the size of the table on the other side. The definition of significantly smaller is that three times the size of the table on one side must be less than the size of the table on the other side.

[0035] If the mixed shuffle hash join algorithm is forced to improve performance, it will easily lead to an Out Of Memory (OOM) error when building the hash table, affecting the stability of the production environment.

[0036] In an adaptive query execution framework, if a stage failure is detected, a fallback approach of "re-optimizing the physical execution plan and resubmitting" will have the following problems: 1. Wide range of impact.

[0037] For example, if a certain stage includes 100 tasks, and 99 tasks succeed while 1 task fails, the entire stage needs to be rerun. 2. Serious waste of resources.

[0038] Completed calculation results are discarded.

[0039] 3. Data consistency issues.

[0040] If the failed phase involves writing data, for example, if the phase requires performing a join operation and outputting the join result, dirty data may have already been written.

[0041] 4. Complex framework modifications To avoid failures, which would require significant modifications to the adaptive query execution framework, this specification provides a new data join method. The core idea of ​​this method is: for tasks performing join operations, a hash join algorithm is used by default. If insufficient memory is encountered, the algorithm is switched to a sort-merge join algorithm. Successful tasks are unaffected; that is, successful tasks do not need to be recalculated.

[0042] Before introducing the data processing method provided in the embodiments of this specification, the architecture of the distributed query system involved in this method will be described. For example... Figure 1 As shown, the system architecture includes a driver 1 and multiple executors 2.

[0043] Driver 1 is used to: parse the SQL used for join queries into a logical plan; generate a physical execution plan based on the logical plan; divide the physical execution plan into multiple stages, wherein the stages are separated by shuffle boundaries; further divide each stage to obtain multiple tasks, wherein the number of tasks in each stage is equal to the number of partitions; and distribute the tasks to multiple executors 2.

[0044] Executor 2 is used to execute the assigned task.

[0045] The aforementioned stages include a target stage, which performs the join operation on the first and second data tables. This target stage is divided into multiple tasks, the number of which equals the number of partitions in the first or second data table. Each partition corresponds one-to-one with a task, and each task performs the join operation on the target partition corresponding to that task. For example, if the number of partitions is 100, then the number of tasks is 100.

[0046] In the embodiments described in this specification, the driver is used to schedule tasks, control progress, collect results, and ensure that each executor works together to complete the Join query. Multiple executors are used to perform tasks in a distributed manner.

[0047] Figure 2This is a flowchart illustrating a data processing method provided in one embodiment of this specification. The execution entities of this data processing method include the aforementioned... Figure 1 The actuator in the process. For example... Figure 2 As shown, the method includes: 100. Retrieve the first data belonging to the target partition from the first data table.

[0048] 102. Using a hash join algorithm, generate the first entry in a hash table in memory for the first data.

[0049] 104. If the process of generating a corresponding second table entry for the second data belonging to the target partition in the first data table fails due to insufficient remaining memory, switch to the sort-merge-join algorithm.

[0050] 106. Using the sort-merge-join algorithm, sort the data belonging to the target partition obtained from the hash table and the remaining data belonging to the target partition obtained from the first data table to obtain a first sorting result, and sort the data belonging to the target partition obtained from the second data table to obtain a second sorting result.

[0051] 108. Based on the first sorting result and the second sorting result, perform a join operation using the sorting-merge-join algorithm.

[0052] The first and second data tables mentioned above are stored in the database, and they are two tables that need to be joined. Optionally, the first data table has less data than the second data table.

[0053] In step 100 above, data from the first data table is retrieved from the database, and the partition to which the data belongs is determined based on a preset hash function. For example, the hash value of the join key in the data can be calculated based on the preset hash function, and the partition to which the data belongs can be determined based on the hash value.

[0054] For example, a data source operator (or scan operator) can be used to read data from the first data table stored in the database to obtain data belonging to the first data table. A shuffling operator is then used to calculate the hash value of the join key of this data based on a preset hash function. Based on the hash value, the partition to which the data belongs is determined, and the data is transmitted to the join operator responsible for that partition. The join operator then uses a hash join algorithm to generate an entry in a hash table for this data in memory. The aforementioned join operator runs on the executor.

[0055] In the technical solution provided in this embodiment, there are multiple partitions, each corresponding to a different join operator, and these different join operators can be executed in parallel. The join operator responsible for a certain partition is used to perform the join operation between data belonging to that partition in the first data table and data belonging to that partition in the second data table.

[0056] In step 102 above, the hash join algorithm is used to generate the first entry in the hash table in memory for the first data.

[0057] The first entry may include a key-value pair, wherein the value in the key-value pair includes the first data, and the key in the key-value pair includes the hash value of the concatenation key of the first data.

[0058] In step 104 above, when attempting to generate a corresponding second table entry for the second data belonging to the target partition in the first data table, insufficient remaining memory prevents the creation of the second table entry. At this point, the algorithm switches from hash linking to sort-merge-join.

[0059] In some optional implementations, after successfully generating the first entry, free memory space is requested; if the request fails, the algorithm switches to the sort-merge-join algorithm. Assume that after successfully generating the first entry, the requested memory space for storing the hash table entries has been used up. Therefore, the executor needs to request free memory space from the memory management module again. Upon receiving this request, the memory management module determines whether the remaining memory space of the executor is less than or equal to a preset threshold. If the remaining memory space is less than or equal to the preset threshold, the memory management module refuses to provide the executor with memory space for storing the hash table entries. For the executor, the memory space request fails, and therefore, the algorithm switches to the sort-merge-join algorithm. The aforementioned preset threshold is greater than 0.

[0060] In step 106 above, when a failure occurs due to insufficient remaining memory, the hash table already stores multiple entries. These entries correspond one-to-one with multiple processed data (processed records) belonging to the target partition in the first data table. Processed data refers to data in the first data table belonging to the target partition that has a corresponding entry generated in the hash table. Each entry in the hash table is a key-value pair. In the key-value pair corresponding to a specific piece of processed data, the key is the hash value of the concatenation key of that processed data, and the value is the processed data itself. In other words, at this point, the hash table stores a portion of the data from the first data table belonging to the target partition.

[0061] In practical applications, the data source operator retrieves data from the first data table through traversal. When insufficient memory causes a failure, some data in the first data table has already been traversed by the data source operator. Following its original traversal mechanism, the data source operator can only continue traversing the remaining data in the first data table, and cannot repeat traversing already traversed data. Therefore, after switching to the sort-merge-join algorithm, the data source operator cannot provide complete data to the join operator responsible for the target partition. To address the data gap issue, the join operator responsible for the target partition retrieves the data belonging to the target partition from the first data table from a hash table stored in memory. This data is the part that the data source operator cannot provide to the join operator responsible for the target partition.

[0062] During the construction of the hash table, the data source operator has not yet traversed the second data table. Therefore, after switching to the sort-merge-join algorithm, the data source operator can be used to obtain the complete data belonging to the target partition from the second data table.

[0063] Using the sort-merge-join algorithm, the data belonging to the target partition obtained from the hash table and the remaining data belonging to the target partition obtained from the first data table are sorted to obtain a first sorting result.

[0064] The remaining data belonging to the target partition in the first data table refers to the data in the first data table that belongs to the target partition and has not yet generated corresponding entries in the hash table.

[0065] The sort-merge-join algorithm is used to sort the data belonging to the target partition obtained from the second data table to obtain a second sorting result.

[0066] In practical applications, the above sort-merge-join algorithm can utilize the current remaining memory space and the external sorting techniques described above to sort the data, obtaining the first sorting result and the second sorting result.

[0067] In some embodiments of the above 108, the connection result can be output by comparing the connection keys of the corresponding rows of the first sorting result and the second sorting result.

[0068] You can use two cursors to traverse the first and second sort results respectively, comparing the join keys row by row. When the join keys of two rows are equal, output the join result; if they are not equal, move the cursor with the smaller join key forward until the traversal is complete.

[0069] In the technical solution provided in the embodiments of this specification, when insufficient memory prevents the continued construction of entries in the hash table, the algorithm switches from hash join to sort-merge join to leverage the low memory consumption of sort-merge join. After switching to sort-merge join, data belonging to the target partition is retrieved from the hash table, and the remaining data belonging to the target partition is retrieved from the database, thus obtaining the complete data of the target partition. This ensures that sort-merge join can complete the join operation normally. Therefore, the technical solution provided in the embodiments of this specification only requires fault recovery within the problematic partition, avoiding the resource waste caused by large-scale rollback.

[0070] In some embodiments, the step 106 above, "using the sorting-merge-join algorithm to sort the data belonging to the target partition obtained from the hash table and the remaining data belonging to the target partition obtained from the first data table to obtain a first sorting result," can be implemented using the following steps: 1060. Determine the amount of data that can be accommodated based on the remaining space in the memory.

[0071] 1061. According to the amount of data that can be accommodated, obtain the data belonging to the target partition from the hash table and / or obtain the remaining data belonging to the target partition from the first data table to obtain the first batch of data.

[0072] 1062. Sort the first batch of data and store the sorting results on the disk to free up memory space.

[0073] 1063. Obtain the data belonging to the target partition from the hash table and / or obtain the remaining data belonging to the target partition from the first data table to obtain the second batch of data.

[0074] 1064. Sort the second batch of data and store the sorting results of the second batch on the disk until the data belonging to the target partition in the hash table and the remaining data belonging to the target partition in the first data table are all obtained and sorted.

[0075] 1065. Merge sort the multiple batches of sorting results on the disk to obtain the first sorting result.

[0076] In the aforementioned 1060, the amount of data that can be accommodated can be less than the size of the remaining space.

[0077] In the above 1061 to 1064, the amount of data in each batch is equal to or less than the amount of data that can be accommodated.

[0078] In one alternative approach, data belonging to the target partition can be retrieved first from the hash table. After all the data belonging to the target partition in the hash table has been retrieved, the remaining data belonging to the target partition can then be retrieved from the first data table. In another alternative approach, the remaining data belonging to the target partition can be retrieved first from the first data table. After all the remaining data belonging to the target partition in the first data table has been retrieved, the data belonging to the target partition can then be retrieved from the hash table.

[0079] For example, according to the stated data capacity, data belonging to the target partition is retrieved from the hash table to obtain the first batch of data. The first batch of data is sorted, and the sorting results are stored on disk to free up memory space. Starting from the data after the first batch, according to the stated data capacity, data belonging to the target partition is retrieved from the hash table to obtain the second batch of data. The second batch of data is sorted, and the sorting results are stored on disk, until all data belonging to the target partition in the hash table has been retrieved and sorted. Next, according to the stated data capacity, the remaining data belonging to the target partition is retrieved from the database to obtain the Nth batch of data. The Nth batch of data is sorted, and the sorting results are stored on disk, until all remaining data belonging to the target partition in the database has been retrieved and sorted. Here, N is a positive integer.

[0080] In step 1065 above, multiple batches of sorting results on the disk are merged to obtain the first sorting result. Merge sort achieves overall sorting by merging ordered segments.

[0081] In the embodiments described in this specification, the sort-merge-join algorithm utilizes external sorting techniques to achieve data sorting with a relatively small memory space.

[0082] In some embodiments, after obtaining the first sorting result, the hash table is deleted from the memory, thereby freeing up memory space.

[0083] The above embodiments describe the handling methods when the hash table construction fails. In other embodiments, when the hash table construction is successful, the join operation can be directly implemented using a hash join algorithm. Specifically, the above methods may further include: 110. When it is determined that all data belonging to the target partition in the first data table has completed the generation of corresponding table entries, the data belonging to the target partition is read row by row from the second data table and matched with the table entries in the hash table to output the connection result.

[0084] When all data belonging to the target partition in the first data table has completed the generation of corresponding table entries, it indicates that the corresponding relational table has been constructed. Therefore, the join operation of the target partition can be performed based on the hash join algorithm. Specifically, the data belonging to the target partition is read row by row from the second data table, and the data is matched with the table entries in the hash relational table. If a match is found, the two matching data are joined to obtain the join result, and the join result is output.

[0085] In the embodiments of this specification, the hash join algorithm is executed first by default. If successful, this can effectively improve processing efficiency. If failure occurs due to memory overflow, the sort-merge join algorithm is used. The technical solution provided in the embodiments of this specification dynamically judges based on actual memory usage, eliminating the need for complex parameter tuning and enabling adaptive performance optimization. Furthermore, the user is unaware of the rollback process and does not need to modify the SQL. In addition, this solution does not write dirty data before failure due to insufficient memory, thus ensuring complete consistency of the final query results.

[0086] In some embodiments, after switching to the sort-merge-join algorithm, the following steps can be used to achieve sorting: S11. For the currently constructed partial hash table, construct the target iterator.

[0087] S12. Concatenate the target iterator with the remaining iterators of the first data table to obtain the concatenated iterator.

[0088] S13. Based on the concatenated iterator, sorting is performed using a sorting-merging-connecting algorithm to obtain the first sorting result.

[0089] In S11 above, the target iterator is used to traverse the hash table to obtain the data belonging to the target partition of the first data table.

[0090] The target iterator is used to traverse the key-value pairs in this part of the hash table. Since the value in the key-value pair is the data of the first data table belonging to the target partition, the target iterator can retrieve the data of the first data table belonging to the target partition by traversing the key-value pairs in this part of the hash table.

[0091] The target iterator points to the starting position of the hash table. For example, the hash table includes key-value pairs from the 0th to the 100th key-value pair, and the target iterator points to the 0th key-value pair. The target iterator is used to traverse key-value pairs from the 0th to the 100th key-value pair.

[0092] To free up memory for subsequent join operations based on the sort-merge-join algorithm, the target iterator may include a destructive iterator. This destructive iterator deletes the hash table from memory after traversing a portion of it.

[0093] In S12 above, the remaining iterator of the first data table is used to traverse the untraversed data of the first data table in order to obtain the remaining data of the first data table belonging to the target partition.

[0094] The target iterator is concatenated with the remaining iterator of the target partition (using the ++ operator) to obtain the concatenated iterator. This concatenated iterator is used to first retrieve the data belonging to the target partition from the hash table, and then retrieve the remaining data belonging to the target partition from the first data table.

[0095] In S13 above, based on the concatenated iterator, the data belonging to the target partition is first obtained from the hash table, and then the remaining data belonging to the target partition is obtained from the first data table. These data are then sorted to obtain the first sorting result.

[0096] A first sorter can be created, which obtains the processed data and the unprocessed data of the target partition through a concatenated iterator. The concatenated iterator provides data to the first sorter row by row, and the first sorter sorts the data provided by the iterator.

[0097] The first sorter uses an external sorting mechanism to sort the processed data (data with corresponding entries in the hash table) and unprocessed data (data without corresponding entries in the hash table) of the target partition. In the external sorting mechanism, data is first loaded into memory in batches for sorting, then the ordered small batches of data are spilled to disk to generate multiple ordered files. Finally, a multi-way merge algorithm is used to read the multiple ordered files and merge them into a globally ordered result, thus efficiently completing large-scale data sorting with limited memory.

[0098] In the technical solution provided in the embodiments of this specification, a target iterator is constructed for the hash table in memory. This iterator is then concatenated with the remaining iterators of the first data table. In this way, the concatenated iterator can obtain the processed data (data for which corresponding entries have been built in the hash table) and the unprocessed data (data for which corresponding entries have not been built in the hash table) of the target partition. The sorting-merge-join algorithm can then conveniently and quickly complete the sorting based on this concatenated iterator. The iterator provides a unified interface for accessing data, eliminating the need to concern oneself with the specific implementation details of the underlying data structures (such as linked lists, arrays, etc.), thus reducing programming complexity and allowing developers to focus more on the implementation of the scenario logic.

[0099] In some embodiments, after switching to the sort-merge-join algorithm, data belonging to the target partition can be retrieved from the second data table based on an iterator of the second data table.

[0100] A second sorter can be created, which retrieves data from the target partition by iterating through an iterator of a second data table. Data belonging to the target partition from the second data table is provided to the second sorter row by row using the iterator of the second data table, and the second sorter sorts the data provided by the iterator.

[0101] In the embodiments of this specification, the data belonging to the target partition in the first data table and the data belonging to the target partition in the second data table are both sorted externally. This sorting process is not limited by memory size, and subsequent merging and joining are performed based on the sorting results.

[0102] In some optional implementations, the step S14 above, "performing a join operation using a sort-merge-join algorithm based on the first sorting result and the second sorting result," can be implemented using the following steps: S140. Construct a first ordered iterator for the first sorting result.

[0103] S142. Construct a second ordered iterator for the second sorting result.

[0104] S144. Based on the first ordered iterator and the second ordered iterator, perform the join operation using the sort-merge-join algorithm.

[0105] In S140 above, the first ordered iterator is used to traverse the first sorted result according to the sorting order of the first sorted result. The first ordered iterator points to the starting position of the first sorted result.

[0106] In S142 above, the second ordered iterator is used to traverse the second sorted result according to the sort order of the second sorted result. The second ordered iterator points to the starting position of the second sorted result.

[0107] In S144 above, the first ordered iterator and the second ordered iterator are the cursors (or pointers) mentioned above. For a detailed explanation of the implementation of the sort-merge-join algorithm for performing the join operation between target partitions, please refer to the section above introducing the working principle of the sort-merge-join algorithm; it will not be elaborated upon here.

[0108] In some embodiments, when the hash join algorithm fails to execute the task corresponding to the target partition, the algorithm is switched to sort-merge join to execute the task corresponding to the target partition. The execution results of the tasks corresponding to other partitions (successful tasks) can continue to be used to generate the final result, thereby avoiding the performance loss caused by large-scale recalculation. Therefore, this scheme can reduce redundant calculations, thereby reducing resource waste and improving system performance and stability.

[0109] In practical applications, the physical execution plan generated by the driver contains multiple operators, such as Scan, Filter, Shuffle, and Join. The physical execution plan is divided into stages; specifically, it is split at the Shuffle operator level, forming multiple stages. Optionally, whole-stage code generation is performed on the operator chain within each stage to obtain the corresponding Java bytecode. Each stage is compiled into a TaskSet and distributed to the executor for execution. The TaskSet includes multiple tasks derived from the stage, and each task contains the bytecode.

[0110] In some embodiments, the bytecode contained in the task included in the target phase described above is used to implement... Figure 2 The corresponding data processing method. In the embodiments of this specification, the bytecode can implement both the doConsume mode and the doProduce mode. That is to say, the bytecode can achieve fast execution of both hash algorithms and sort-merge algorithms. In this way, the high-performance characteristics of the code generation path can be maintained, and seamless algorithm switching can be achieved.

[0111] In other embodiments, code generation technology is used to generate Java bytecode for the hash join logic of the target partition. A conditional statement is generated during the code compilation phase to determine whether the system is in fallback mode. If fallback mode is not enabled, the pre-generated Java bytecode is executed to implement a fast hash join operation. If fallback mode is enabled, an "Operation Not Supported" exception is thrown to explicitly indicate that the current context does not support the execution of this code path. Through this mechanism, the system can dynamically select the execution path at runtime based on actual environmental conditions: using Java bytecode when conditions permit, and automatically switching to interpreted execution or other alternative execution modes when conditions are not met, while ensuring that the generated compiled code is not incorrectly invoked in inappropriate scenarios. This design achieves a balance between execution efficiency and system robustness, fully leveraging the performance advantages of code generation technology while preventing the erroneous execution of incompatible paths through exception protection mechanisms, thus improving system stability and maintainability.

[0112] The following will combine Figure 3 The data processing methods provided in the embodiments of this specification are described in detail. For example... Figure 3 As shown, this method includes the following steps: Step 201: Construct a hash table.

[0113] The target phase includes Task A, which performs join operations on data belonging to the target partition in the first data table and data belonging to the target partition in the second data table. Upon receiving Task A, the executor, by default, first performs the join operation on the target partition using a hash algorithm. Specifically, the executor constructs a hash table for the target partition.

[0114] Step 202: Check for memory overflow issues.

[0115] During the process of the executor building the hash table for the target partition, it checks for insufficient memory. If insufficient memory is detected, steps 203-206 are executed; if no insufficient memory is detected, it means the hash table is built successfully, and therefore, step 207 is executed.

[0116] Step 203: Construct the unfinished unsafe hash relation table.

[0117] The incomplete underlying hash table includes a target iterator built for the hash table and a remaining iterator for the first data table. The target iterator is used to traverse the hash table to obtain processed data belonging to the target partition in the first data table. The remaining iterator is used to traverse the first data table to obtain unprocessed data belonging to the target partition.

[0118] Step 204: Sort the target partitions.

[0119] The target iterator is concatenated with the remaining iterators to obtain a concatenated iterator. This concatenated iterator is used to first retrieve the processed data and then the unprocessed data. A first sorter is created, and based on the concatenated iterator, the first sorter is used to sort the processed and unprocessed data belonging to the target partition in the first data table to obtain a first sorting result.

[0120] Create a second sorter, based on the iterator of the second data table, and use the second sorter to sort the data in the second data table that belong to the target partition to obtain the second sorting result.

[0121] Step 205: Generate an ordered iterator.

[0122] For the first sorted result, create a first ordered iterator; for the second sorted result, create a second ordered iterator. The first ordered iterator is used to traverse the first sorted result according to its sorting order, and the second ordered iterator is used to traverse the second sorted result according to its sorting order.

[0123] Step 206: Perform sort-merge join.

[0124] Use the first and second ordered iterators described above to traverse the two sorted results respectively to perform a merge join.

[0125] Step 207: Perform hash join.

[0126] Once the hash table is constructed, the target partition can be joined based on the hash join algorithm.

[0127] Step 208: Output the results.

[0128] Once the connection is established, output the connection result.

[0129] Table 1 lists the performance metrics obtained by using the shuffle hash join algorithm (SHJ) alone, the performance metrics obtained by using the sort merge algorithm (SMJ) alone, and the performance metrics obtained by using the technical solution provided in the embodiments of this specification (i.e., SHJ + rollback SMJ). These performance metrics are based on proof-of-concept tests. Tests have shown that the improved technical solutions in the embodiments of this specification can effectively solve the OOM problem, improve system stability, increase success rate, and reduce resource consumption.

[0130] This specification also provides a data processing apparatus according to one embodiment. The apparatus includes: The first acquisition module is used to acquire the first data belonging to the target partition from the first data table; The generation module is used to generate the first entry in a hash table in memory for the first data using a hash join algorithm. The switching module is used to switch to the sort-merge-join algorithm when the failure occurs due to insufficient remaining memory space when generating a corresponding second table entry for the second data belonging to the target partition in the first data table. The sorting module is used to sort the data belonging to the target partition obtained from the hash table and the remaining data belonging to the target partition obtained from the first data table using the sorting-merge-join algorithm to obtain a first sorting result, and to sort the data belonging to the target partition obtained from the second data table to obtain a second sorting result. An execution module is used to perform a join operation based on the first sorting result and the second sorting result using the sorting-merge-join algorithm.

[0131] It should be noted that the apparatus provided in the above embodiments can implement the technical solutions described in the corresponding method embodiments above. The specific implementation principles and corresponding beneficial effects of each module or unit can be found in the corresponding content of each method embodiment above, and will not be repeated here.

[0132] This specification also provides an electronic device according to one embodiment. For example... Figure 4 As shown, the electronic device includes a processor 42 and a memory 41. The memory 41 stores one or more computer programs (or instructions); the processor 42 is coupled to the memory 41 and is used for the at least one or more computer programs to implement the steps in the methods provided in the embodiments of this specification.

[0133] Furthermore, the electronic device also includes other components such as a communication component 43, a display 44, a power supply component 45, and an audio component 46. Only some components are shown here for illustrative purposes, and it is not intended that the electronic device contains only these components.

[0134] The methods in this specification can be implemented, in whole or in part, by software, hardware, firmware, or any combination thereof. When implemented in software, they can be implemented, in whole or in part, as a computer program product. Therefore, this specification also provides a computer program product. This computer program product includes a computer program / instructions that, when executed by an electronic component such as a processor, can perform, in whole or in part, the steps or functions of the methods provided in the embodiments of this specification. The computer can be a general-purpose computer, a special-purpose computer, a computer network, network equipment, user equipment, core network equipment, or other programmable device.

[0135] The aforementioned memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random-Access Memory (SRAM), Electrically Erasable Programmable Read Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0136] The aforementioned display includes a screen, which may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a Touch Panel, the screen can be implemented as a touchscreen to receive input signals from the user. The Touch Panel includes one or more touch sensors to sense touches, swipes, and gestures on the Touch Panel. The touch sensors can sense not only the boundaries of touch or swipe actions but also the duration and pressure associated with the touch or swipe operation.

[0137] The aforementioned power supply components provide power to various components within the device in which they reside. These power supply components may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power to the device in which they reside.

[0138] The aforementioned audio component can be configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC) configured to receive external audio signals when the device containing the audio component is in an operating mode, such as call mode, recording mode, or voice recognition mode. The received audio signals can be further stored in memory or transmitted via a communication component. In some embodiments, the audio component also includes a speaker for outputting audio signals.

[0139] Accordingly, embodiments of this specification also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, enables the processor to implement the steps in the above-described method embodiments. The computer-readable storage medium includes volatile or non-volatile components, or a combination thereof, and can be removable or non-removable. Examples of computer-readable storage media include, but are not limited to, phase-change random access memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random-access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), flash memory or other memory technologies, CD-ROM, Digital Video Disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transfer medium.

[0140] Accordingly, embodiments of this specification also provide a computer program product, which includes a computer program or instructions that, when executed by a processor, cause the processor to implement the steps in the above-described method embodiments. It should be understood that each step or combination of steps in the above-described method flow can be implemented by the computer program or instructions. Furthermore, these computer programs or instructions can be applied to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device, enabling the processor of the general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device to function as an apparatus for implementing the corresponding functions in the above-described method embodiments.

[0141] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0142] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for embodiments such as devices, electronic devices, storage media, and program products, since they are basically similar to the method embodiments, the descriptions are relatively simple, and relevant parts can be referred to the descriptions of the method embodiments.

[0143] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0144] The above are merely embodiments of this specification and are not intended to limit this specification. Various modifications and variations can be made to this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of the claims of this application.

Claims

1. A data processing method, characterized by, The method comprises the following steps: obtaining first data belonging to a target partition from a first data table; generating first table entries in a hash relation table in memory for the first data by using a hash join algorithm; switching to a sort-merge join algorithm in case of failure due to insufficient remaining memory space when generating corresponding second table entries for second data belonging to the target partition in the first data table; sorting data belonging to the target partition obtained from the hash relation table and remaining data belonging to the target partition obtained from the first data table by using the sort-merge join algorithm to obtain a first sorting result, and sorting data belonging to the target partition obtained from a second data table to obtain a second sorting result; performing a join operation based on the first sorting result and the second sorting result by using the sort-merge join algorithm.

2. The method of claim 1, wherein, The method for sorting data belonging to the target partition obtained from the hash relation table and remaining data belonging to the target partition obtained from the first data table by using the sort-merge join algorithm to obtain a first sorting result comprises the following steps: determining the amount of data that can be accommodated according to the remaining memory space; obtaining data belonging to the target partition from the hash relation table and / or remaining data belonging to the target partition from the first data table according to the amount of data that can be accommodated to obtain a first batch of data; sorting the first batch of data and storing the first batch of sorting results in a disk to release memory space; obtaining data belonging to the target partition from the hash relation table and / or remaining data belonging to the target partition from the first data table to obtain a second batch of data; sorting the second batch of data and storing the second batch of sorting results in the disk until data belonging to the target partition in the hash relation table and remaining data belonging to the target partition in the database are all obtained and sorted; performing merge sorting on the multiple batches of sorting results in the disk to obtain a first sorting result.

3. The method of claim 2, wherein, The method further comprises the following steps: after obtaining the first sorting result, deleting the hash relation table in the memory.

4. The method according to any one of claims 1 to 3, characterized in that, The method for performing a join operation based on the first sorting result and the second sorting result by using the sort-merge join algorithm comprises the following steps: comparing the join keys of corresponding rows in the first sorting result and the second sorting result to output a join result.

5. The method according to any one of claims 1 to 3, characterized in that, The method further comprises the following steps: when all data belonging to the target partition in the first data table is determined to have completed the generation of corresponding table entries, reading data belonging to the target partition from the second data table row by row, and matching the table entries in the hash relation table to output a join result.

6. The method according to any one of claims 1 to 3, characterized in that, The first data table and the second data table are stored in a database; The amount of data in the first data table is less than the amount of data in the second data table.

7. The method according to any one of claims 1 to 3, characterized in that, The method for switching to the sort-merge join algorithm in case of failure due to insufficient remaining memory space when generating corresponding second table entries for second data belonging to the target partition in the first data table comprises the following steps: requesting idle memory space after successfully generating the first table entries; switching to the sort-merge join algorithm when the request fails.

8. A data processing apparatus, characterized by, The method comprises the following steps: The first obtaining module is configured to obtain first data belonging to a target partition from a first data table; The generating module is configured to generate, in memory, a first entry in a hash relation table for the first data by using a hash join algorithm; The switching module is configured to switch to a sort-merge join algorithm in a case where a failure occurs due to insufficient remaining memory space when generating a corresponding second entry for second data belonging to the target partition in the first data table; The sorting module is configured to sort, by using the sort-merge join algorithm, data belonging to the target partition obtained from the hash relation table and remaining data belonging to the target partition obtained from the first data table to obtain a first sorting result, and sort data belonging to the target partition obtained from a second data table to obtain a second sorting result; The executing module is configured to perform a join operation by using the sort-merge join algorithm based on the first sorting result and the second sorting result.

9. An electronic device, comprising: comprise: a memory and a processor, wherein the memory is configured to store a program; the processor, coupled to the memory, is configured to execute the program stored in the memory to implement the method in any one of claims 1 to 7.

10. A computer readable storage medium storing a computer program, characterized in that, The computer program is executed by a computer to implement the method in any one of claims 1 to 7.

11. A computer program product comprising a computer program, characterized in that, The computer program is executed by a processor to implement the method in any one of claims 1 to 7.

Images