Table connection method and device and computer equipment
By determining the minimum memory footprint of a data table in Spark SQL and selecting a target local shared memory pool, the inefficiency of existing table join methods is resolved, enabling efficient table join operations.
Patent Information
- Application Number
- CN202511718073.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-21
- Publication Date
- 2026-03-06
AI Technical Summary
Existing table join methods in Spark SQL require sending the data table to the compute node as either the complete table data or table partition data, which consumes network transmission resources and is inefficient.
By determining the minimum memory usage between the first and second data tables, a target local shared memory pool is selected, and the data table corresponding to the minimum memory usage is sent to the shared pool, enabling each computing node to retrieve the table locally for table join operations.
It effectively saves network bandwidth usage during table joins and improves the efficiency of establishing table joins.
Smart Images
Figure CN121614291A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data processing technology, and in particular to a table joining method, apparatus, and computer device. Background Technology
[0002] A table join, also known as a JOIN operation, is an operation that combines two or more data tables (i.e., data sets) into a new table according to certain conditions. In Spark SQL (Spark Structured Query Language), the join operation is similar to the join operation in traditional databases, allowing data tables to be combined using one or more join keys. The join operation is the most complex and costly operation model in SQL queries, and it is also a relatively common operation used in OLAP (Online Analytical Processing) scenarios.
[0003] Currently, common table join methods include broadcast join, shuffle hash join, and sort merge join. However, each of these table join methods requires sending at least one table to the computing node in the form of complete table data or table partition data, which not only consumes network transmission resources but also makes the establishment of table joins inefficient. Summary of the Invention
[0004] Therefore, it is necessary to provide a table connection method, device, and computer equipment that can improve the satisfaction of flight attendants, addressing the aforementioned technical problems.
[0005] Firstly, this application provides a table join method. The method includes:
[0006] In response to a table join processing request for the first data table and the second data table, determine the first memory usage of the first data table and the second memory usage of the second data table;
[0007] The target local shared memory pool is determined based on the minimum memory usage between the first and second memory usages.
[0008] The data table corresponding to the minimum memory usage is sent to the target local memory shared pool, so that each computing node can retrieve the data table corresponding to the minimum memory usage from the target local memory shared pool and perform table join operations on the first data table and the second data table.
[0009] In one embodiment, determining the target local shared memory pool based on the minimum memory usage between the first and second memory usages includes:
[0010] The memory pool usage method of the candidate local memory pool is determined based on the minimum memory usage between the first and second memory usages and the memory capacity of the candidate local memory pool.
[0011] When the memory pool is used as shared memory, the target local shared memory pool is selected from the candidate local memory pools based on the minimum memory usage between the first and second occupied memory, the table join processing requests, and the performance indicators of the memory devices corresponding to the candidate local memory pools.
[0012] In one embodiment, the memory pool usage method of the candidate local memory pool is determined based on the minimum memory usage between the first and second occupied memory and the memory capacity of the candidate local memory pool, including:
[0013] Sort the candidate local memory pools by memory capacity and determine the maximum memory capacity;
[0014] If the minimum memory usage between the first and second memory usages is less than the maximum memory capacity, the memory pool usage mode of the candidate local memory pool is determined to be used as shared memory.
[0015] If the minimum memory usage between the first and second memory usages is not less than the maximum memory capacity, the candidate local memory pool is determined to be used as a cache.
[0016] In one embodiment, selecting a target local shared memory pool from the candidate local memory pools based on the minimum memory usage between the first and second memory usages, the table join processing request, and the performance metrics of the memory devices corresponding to the candidate local memory pools includes:
[0017] The candidate local memory pool with a memory capacity greater than the minimum memory usage is selected as the first local memory pool;
[0018] The performance score of the first local memory pool is determined based on the performance metrics corresponding to the first local memory pool.
[0019] Based on the task priority of the table join request and the performance score of the first local memory pool, select the target local shared memory pool from the first local memory pool.
[0020] In one embodiment, selecting a target local shared memory pool from the first local memory pool based on the task priority of the table join request and the performance score of the first local memory pool includes:
[0021] If there is only one first local memory pool, then the first local memory pool will be used as the target local shared memory pool.
[0022] When there are at least two first local shared memory pools, the performance level of each first local memory pool is determined based on the performance score of each first local memory pool.
[0023] The first local memory pool whose performance level matches the task priority of the table join request is used as the target local shared memory pool.
[0024] In one embodiment, the method further includes:
[0025] When the memory usage mode is cached, the first data table and the second data table are partitioned to obtain the first table partition corresponding to the first data table and the second table partition corresponding to the second data table.
[0026] The first table partition and the second table partition are transferred to each compute node through the candidate local memory pool, so that each compute node can perform table join operations on the first data table and the second data table based on the first table group and the second table partition.
[0027] Secondly, this application also provides a table connection device. The device includes:
[0028] The first determining module is configured to determine the first memory occupied by the first data table and the second memory occupied by the second data table in response to a table join processing request for the first data table and the second data table.
[0029] The second determining module is used to determine the target local shared memory pool based on the minimum memory usage between the first and second occupied memory.
[0030] The first sending module is used to send the data table corresponding to the minimum memory usage to the target local memory shared pool, so that each computing node can retrieve the data table corresponding to the minimum memory usage from the target local memory shared pool and perform table join operations on the first data table and the second data table.
[0031] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:
[0032] In response to a table join processing request for the first data table and the second data table, determine the first memory usage of the first data table and the second memory usage of the second data table;
[0033] The target local shared memory pool is determined based on the minimum memory usage between the first and second memory usages.
[0034] The data table corresponding to the minimum memory usage is sent to the target local memory shared pool, so that each computing node can retrieve the data table corresponding to the minimum memory usage from the target local memory shared pool and perform table join operations on the first data table and the second data table.
[0035] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps:
[0036] In response to a table join processing request for the first data table and the second data table, determine the first memory usage of the first data table and the second memory usage of the second data table;
[0037] The target local shared memory pool is determined based on the minimum memory usage between the first and second memory usages.
[0038] The data table corresponding to the minimum memory usage is sent to the target local memory shared pool, so that each computing node can retrieve the data table corresponding to the minimum memory usage from the target local memory shared pool and perform table join operations on the first data table and the second data table.
[0039] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, performs the following steps:
[0040] In response to a table join processing request for the first data table and the second data table, determine the first memory usage of the first data table and the second memory usage of the second data table;
[0041] The target local shared memory pool is determined based on the minimum memory usage between the first and second memory usages.
[0042] The data table corresponding to the minimum memory usage is sent to the target local memory shared pool, so that each computing node can retrieve the data table corresponding to the minimum memory usage from the target local memory shared pool and perform table join operations on the first data table and the second data table.
[0043] The aforementioned table join method, apparatus, and computer device, in response to a table join processing request for a first data table and a second data table, determine a first memory footprint for the first data table and a second memory footprint for the second data table. A target local shared memory pool is determined based on the minimum memory footprint between the first and second memory footprints. The data table corresponding to the minimum memory footprint is sent to the target local shared memory pool, enabling each computing node to retrieve the data table corresponding to the minimum memory footprint from the target local shared memory pool and perform a table join operation for the first and second data tables. In this application, when each computing node establishes a table join for the first and second data tables, each computing node can retrieve the data table corresponding to the minimum memory footprint from the target local shared memory pool. Therefore, the data table corresponding to the minimum memory footprint only needs to be transmitted once and does not need to be sent separately to each computing node, effectively saving network bandwidth occupied by data table transmission during table join establishment. Furthermore, since each computing node can obtain the data table corresponding to the minimum memory footprint locally, the efficiency of table join establishment is effectively improved. Attached Figure Description
[0044] Figure 1 This is a diagram illustrating the application environment of the table join method provided in this embodiment.
[0045] Figure 2 This is a flowchart illustrating the first table join method provided in this embodiment;
[0046] Figure 3 This is a schematic diagram of the first method for selecting a target local shared memory pool provided in this embodiment;
[0047] Figure 4 This is a schematic diagram of the table partitioning principle corresponding to the first and second data tables provided in this embodiment;
[0048] Figure 5 This is a schematic diagram of the second method for selecting a target local shared memory pool provided in this embodiment;
[0049] Figure 6 This is a flowchart illustrating the second table join method provided in this embodiment;
[0050] Figure 7 This is a structural block diagram of a table connection device provided in this embodiment;
[0051] Figure 8 This is an internal structural diagram of the computer device provided in this embodiment. Detailed Implementation
[0052] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.
[0053] A table join, also known as a JOIN operation, is an operation that combines two or more data tables (i.e., data sets) into a new table according to certain conditions. In Spark SQL (Spark Structured Query Language), the join operation is similar to the join operation in traditional databases, allowing data tables to be combined using one or more join keys. The join operation is the most complex and costly operation model in SQL queries, and it is also a relatively common operation used in OLAP (Online Analytical Processing) scenarios.
[0054] Currently, common table join methods include broadcast join, shuffle hash join, and sort merge join. However, each of these table join methods requires sending at least one table to the computing node in the form of complete table data or table partition data, which not only consumes network transmission resources but also makes the establishment of table joins inefficient.
[0055] To address the aforementioned technical problems, the table join method provided in this application can be applied to, for example... Figure 1 In the application environment shown, Figure 1In this architecture, the Host represents the compute node, and the CXL (Compute Express Link) Switch is the core network device enabling the scalable expansion of the CXL protocol. GFAM (Global Fabric Attached Memory) is a new memory expansion technology designed to further enhance memory access and processing capabilities in data centers and high-performance computing environments. GFAM allows the system to connect more memory resources through the CXL interface, thereby breaking through the memory capacity limitations of a single processor or memory controller. GFAM is designed to provide a scalable memory architecture to meet the growing demands of data-intensive applications; the target local shared memory pool is created based on the GFAM device. Specifically, in response to a table join request for a first data table and a second data table, the control device determines the first memory footprint of the first data table and the second memory footprint of the second data table. The control device determines the target local shared memory pool based on the minimum memory footprint between the first and second memory footprints. The control device sends the data table corresponding to the minimum memory footprint to the target local shared memory pool, enabling each compute node to retrieve the data table corresponding to the minimum memory footprint from the target local shared memory pool and perform table join operations for the first and second data tables.
[0056] Among them, control equipment refers to equipment applied to... Figure 1 The global controller in the system is responsible for the overall management of global memory. The control device can be integrated into... Figure 1 The controller in the CXL Switch.
[0057] In one embodiment, such as Figure 2 As shown, a table join method is provided, which can be applied to... Figure 1 Taking the control equipment in the example, the explanation includes the following steps:
[0058] S201, in response to a table join processing request for the first data table and the second data table, determine the first memory occupied by the first data table and the second memory occupied by the second data table.
[0059] In this context, the first and second data tables refer to the two tables that need to be linked. A data table also refers to a collection of data. A table join establishes a relationship between data tables; it's also called a Join operation, which combines two or more data tables (i.e., data collections) into a new table based on certain conditions. In Spark SQL, the join operation is similar to the join operation in traditional databases, allowing data tables to be merged using one or more join keys. The join operation is the most complex and costly operation model in SQL queries, and it's also one of the more frequently used operations in OLAP scenarios. The first memory usage refers to the memory required by the first data table. The second memory usage refers to the memory required by the second data table.
[0060] As an optional implementation of this application, in response to a table join processing request for a first data table and a second data table, the amount of data contained in the first data table and the second data table is obtained; based on the amount of data contained in the first data table, a first memory occupation corresponding to the first data table is determined; and based on the amount of data contained in the second data table, a second memory occupation corresponding to the second data table is determined.
[0061] Another optional implementation of this application involves parsing the table join request to obtain the first memory usage of the first data table and the second memory usage of the second data table. The table join request carries the first memory usage of the first data table and the second memory usage of the second data table.
[0062] S202, determine the target local shared memory pool based on the minimum memory usage between the first and second occupied memory.
[0063] The minimum memory usage refers to the smaller of the first and second memory usage values. The target local shared memory pool refers to a memory area configured locally that can be accessed by all compute nodes.
[0064] As an optional implementation of this application, a target local shared memory pool is selected from each candidate local memory pool based on the minimum memory usage between the first and second memory usages, and the memory capacity of each candidate local memory pool. Specifically, the candidate local memory pool whose memory capacity is closest to and greater than the minimum memory usage is selected as the target local shared memory pool. Optionally, in this embodiment, the candidate local memory pool can be... Figure 1 The GFAM devices in the pool can be multiple, and each GFAM device can serve as a candidate local memory pool.
[0065] S203, the data table corresponding to the minimum memory usage is sent to the target local memory shared pool, so that each computing node can retrieve the data table corresponding to the minimum memory usage from the target local memory shared pool and perform table join operations on the first data table and the second data table.
[0066] Optionally, in this embodiment, the maximum memory usage between the first and second memory usages is determined; the data table corresponding to the maximum memory usage is partitioned to obtain a third table partition; the third table partition is then transmitted to each computing node through the first local memory pool, for example, as... Figure 1 As shown Figure 1 In each Host, T1-1, T1-2, and T1-N are all third-table partitions, where N is the number of third-table partitions. The first local memory pool is the local memory pool among the candidate local memory pools, excluding the target local shared memory pool. The data table corresponding to the minimum memory usage is sent to the target local shared memory pool, so that each compute node retrieves the data table corresponding to the minimum memory usage from the target local shared memory pool and performs a table join operation on the first data table and the second data table. Specifically, each compute node retrieves the data table corresponding to the minimum memory usage from the target local shared memory pool, performs a table join operation on this data table and the corresponding third-table partition, thereby realizing the table join operation on the first data table and the second data table. Figure 1 In this context, T2 represents the data table with the minimum memory usage. It should be noted that the Join operation between the third table partition and the data table with the minimum memory usage performed by each compute node is a standard operation; the specific details will not be elaborated here.
[0067] In this embodiment, in response to a table join request for a first data table and a second data table, a first memory footprint for the first data table and a second memory footprint for the second data table are determined. A target local shared memory pool is determined based on the minimum memory footprint between the first and second memory footprints. The data table corresponding to the minimum memory footprint is sent to the target local shared memory pool, enabling each compute node to retrieve the data table corresponding to the minimum memory footprint from the target local shared memory pool and perform a table join operation for the first and second data tables. In this application, when establishing a table join between the first and second data tables, each compute node can retrieve the data table corresponding to the minimum memory footprint from the target local shared memory pool. Therefore, the data table corresponding to the minimum memory footprint only needs to be transmitted once and does not need to be sent separately to each compute node, effectively saving network bandwidth occupied by data table transmission during table join establishment. Furthermore, since each compute node can obtain the data table corresponding to the minimum memory footprint locally, the efficiency of table join establishment is effectively improved.
[0068] In one embodiment, to more accurately determine the target local shared memory pool and improve the utilization efficiency of the shared memory pool, such as Figure 3 As shown, in an optional implementation of S202, it includes:
[0069] S301, determine the memory pool usage method of the candidate local memory pool based on the minimum memory usage between the first and second occupied memory and the memory capacity of the candidate local memory pool.
[0070] Among them, memory capacity refers to the maximum total amount of data that the candidate local memory pool can hold.
[0071] Optionally, in this embodiment, the memory capacities of the candidate local memory pools are sorted to determine the maximum memory capacity. If the minimum memory usage between the first and second occupied memory is less than the maximum memory capacity, the candidate local memory pool is determined to be used as shared memory. If the minimum memory usage between the first and second occupied memory is not less than the maximum memory capacity, the candidate local memory pool is determined to be used as a cache.
[0072] Based on the above embodiments, when the candidate local memory pool is determined to be used as a cache, an optional implementation of the table join method is as follows: When the memory usage is cached, the first data table and the second data table are partitioned to obtain a first table partition corresponding to the first data table and a second table partition corresponding to the second data table. The first table partition and the second table partition are then transmitted to each compute node through the candidate local memory pool, so that each compute node performs a table join operation on the first data table and the second table based on the first table partition and the second table partition. Figure 4 As shown, T1 and T2 are the first and second data tables, respectively; T1-1, T1-2, and T1-N are the first table partitions corresponding to the first data table; and T2-2, T2-3, and T2-N are the second table partitions corresponding to the second data table. Optionally, in this embodiment, partitioning the first and second data tables to obtain the first table partitions corresponding to the first data table and the second table partitions corresponding to the second data table can be implemented by shuffling the two tables separately, and then using the shuffle results, i.e., the table partitions generated according to the join key, as a candidate local memory pool as a cache. The table partitions no longer transmit data over the network, but instead transmit data via CXL. The first and second table partitions are first stored in the candidate local memory pool. When a certain value is reached or the candidate local memory pool has no space, they are transmitted to the corresponding node, which greatly reduces the waiting overhead.
[0073] S302, when the memory pool is used as shared memory, the target local shared memory pool is selected from the candidate local memory pools based on the minimum memory usage between the first and second occupied memory, the table join processing request, and the performance indicators of the memory device corresponding to the candidate local memory pool.
[0074] Optionally, in this embodiment, a candidate local memory pool whose content capacity is greater than the minimum memory usage and whose performance indicators meet the performance requirements carried in the table join processing request is selected as the target local shared memory pool.
[0075] In this embodiment, the memory pool usage mode of the candidate local memory pool is determined based on the minimum memory usage between the first and second occupied memory and the memory capacity of the candidate local memory pool. When the memory pool is used as shared memory, a target local shared memory pool is selected from the candidate local memory pools based on the minimum memory usage between the first and second occupied memory, table join processing requests, and the performance indicators of the memory devices corresponding to the candidate local memory pools. This embodiment considers not only the minimum memory usage but also the table join processing requests and the performance indicators of the memory devices corresponding to the candidate local memory pools when determining the target local shared memory pool, providing a more comprehensive consideration and ensuring that the determined target local shared memory pool better meets user needs.
[0076] In one embodiment, to more accurately select the target local shared memory pool, such as Figure 5 As shown, an optional implementation of selecting a target local shared memory pool from candidate local memory pools based on the minimum memory usage between the first and second memory usages, table join processing requests, and performance metrics of the memory devices corresponding to candidate local memory pools includes:
[0077] S501 selects the candidate local memory pool with a memory capacity greater than the minimum memory usage as the first local memory pool.
[0078] Optionally, in this embodiment, the content capacity of the candidate local memory pools is sorted, and the candidate local memory pools with memory capacity greater than the minimum memory occupied are all regarded as the first local memory pool.
[0079] S502 determines the performance score of the first local memory pool based on the performance indicators corresponding to the first local memory pool.
[0080] Optionally, the performance metrics in this embodiment include at least one of bandwidth, latency, and frequency.
[0081] As an optional implementation of this application, the performance index corresponding to the first local memory pool is input into the trained neural network model, and the neural network model outputs the performance score of the first local memory pool.
[0082] Another optional implementation of this application involves standardizing the performance metrics corresponding to the first local memory pool to obtain metric values for each performance metric. The metric values for each performance metric are then fused to obtain a fusion result, which is used as the performance score of the first local memory pool. The fusion processing methods include, but are not limited to, summation, product, and weighted summation.
[0083] S503, selects the target local shared memory pool from the first local memory pool based on the task priority of the table join request and the performance score of the first local memory pool.
[0084] Among them, task priority refers to the execution priority of the task corresponding to the table join request.
[0085] Optionally, in this embodiment, the table join request carries the task priority.
[0086] Optionally, in this embodiment, if there is only one first local memory pool, the first local memory pool is used as the target local shared memory pool. If there are at least two first local shared memory pools, the performance level of each first local memory pool is determined based on its performance score. The first local memory pool whose performance level matches the task priority of the table join request is used as the target local shared memory pool.
[0087] Optionally, in this embodiment, an optional implementation method for determining the performance level of each first local memory pool based on its performance score is to determine the performance level of each first local memory pool based on its performance score and each candidate score interval; wherein each candidate score interval has a corresponding interval level. Alternatively, another optional implementation method for determining the performance level of each first local memory pool based on its performance score and each candidate score interval is to, for each first local memory pool, determine the candidate score interval in which its performance score falls, and use the interval level corresponding to that candidate score interval as the performance level of that first local memory pool.
[0088] In this embodiment, candidate local memory pools with memory capacity greater than the minimum memory usage are designated as the first local memory pool. A performance score for the first local memory pool is determined based on its corresponding performance metrics. A target local shared memory pool is selected from the first local memory pools based on the task priority of the table join request and the performance score of the first local memory pool. The target local shared memory pool determined in this embodiment ensures that task priorities match the performance scores of each first local memory pool, achieving priority response for high-priority tasks and timely response for lower-priority tasks. This satisfies both task requirements and user needs, improving user satisfaction.
[0089] In one embodiment, such as Figure 6 As shown, another optional implementation of a table join method includes:
[0090] S601, in response to a table join processing request for the first data table and the second data table, determines the first memory occupied by the first data table and the second memory occupied by the second data table.
[0091] S602 sorts the memory capacities of candidate local memory pools and determines the maximum memory capacity.
[0092] S603, if the minimum memory usage between the first and second memory usages is less than the maximum memory capacity, determine that the candidate local memory pool should be used as shared memory.
[0093] S604, if the minimum memory usage between the first and second memory usages is not less than the maximum memory capacity, determines that the candidate local memory pool is used as a cache.
[0094] S605, when the memory pool is used as shared memory, selects the candidate local memory pool with a memory capacity greater than the minimum memory usage as the first local memory pool.
[0095] S606 determines the performance score of the first local memory pool based on the performance indicators corresponding to the first local memory pool.
[0096] S607, when the number of first local memory pools is one, uses the first local memory pool as the target local shared memory pool.
[0097] S608, when the number of first local shared memory pools is at least two, determines the performance level of each first local memory pool based on the performance score of each first local memory pool.
[0098] S609 selects the first local memory pool whose performance level matches the task priority of the table join request as the target local shared memory pool.
[0099] S610, the data table corresponding to the minimum memory usage is sent to the target local memory shared pool, so that each computing node can retrieve the data table corresponding to the minimum memory usage from the target local memory shared pool and perform table join operations on the first data table and the second data table.
[0100] S611, when the memory usage mode is cache usage, performs partitioning processing on the first data table and the second data table to obtain the first table partition corresponding to the first data table and the second table partition corresponding to the second data table.
[0101] S612, the first table partition and the second table partition are transferred to each compute node through the candidate local memory pool, so that each compute node can perform table join operations on the first data table and the second data table based on the first table group and the second table partition.
[0102] In response to a table join request for a first data table and a second data table, this application determines the first memory footprint of the first data table and the second memory footprint of the second data table. Based on the minimum memory footprint between the first and second memory footprints, a target local shared memory pool is determined. The data table corresponding to the minimum memory footprint is sent to the target local shared memory pool, enabling each compute node to retrieve the data table corresponding to the minimum memory footprint from the target local shared memory pool and perform a table join operation for the first and second data tables. In this application, when establishing a table join between the first and second data tables, each compute node can retrieve the data table corresponding to the minimum memory footprint from the target local shared memory pool. Therefore, the data table corresponding to the minimum memory footprint only needs to be transmitted once, without needing to be sent separately to each compute node, effectively saving network bandwidth occupied by data table transmission during table join establishment. Furthermore, since each compute node can obtain the data table corresponding to the minimum memory footprint locally, the efficiency of table join establishment is effectively improved.
[0103] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.
[0104] Based on the same inventive concept, this application also provides a table join apparatus for implementing the table join method described above. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, the specific limitations in one or more table join apparatus embodiments provided below can be found in the limitations of the table join method described above, and will not be repeated here.
[0105] In one embodiment, such as Figure 7 As shown, a table connection device 1 is provided, including: a first determining module 10, a second determining module 20, and a first sending module 30, wherein:
[0106] The first determining module is configured to determine the first memory occupied by the first data table and the second memory occupied by the second data table in response to a table join processing request for the first data table and the second data table.
[0107] The second determining module is used to determine the target local shared memory pool based on the minimum memory usage between the first and second occupied memory.
[0108] The first sending module is used to send the data table corresponding to the minimum memory usage to the target local memory shared pool, so that each computing node can retrieve the data table corresponding to the minimum memory usage from the target local memory shared pool and perform table join operations on the first data table and the second data table.
[0109] In one embodiment, the second determining module is further specifically used for:
[0110] The memory pool usage method of the candidate local memory pool is determined based on the minimum memory usage between the first and second memory usages and the memory capacity of the candidate local memory pool.
[0111] When the memory pool is used as shared memory, the target local shared memory pool is selected from the candidate local memory pools based on the minimum memory usage between the first and second occupied memory, the table join processing requests, and the performance indicators of the memory devices corresponding to the candidate local memory pools.
[0112] In one embodiment, the second determining module is further specifically used for:
[0113] Sort the candidate local memory pools by memory capacity and determine the maximum memory capacity;
[0114] If the minimum memory usage between the first and second memory usages is less than the maximum memory capacity, the memory pool usage mode of the candidate local memory pool is determined to be used as shared memory.
[0115] If the minimum memory usage between the first and second memory usages is not less than the maximum memory capacity, the candidate local memory pool is determined to be used as a cache.
[0116] In one embodiment, the second determining module is further specifically used for:
[0117] The candidate local memory pool with a memory capacity greater than the minimum memory usage is selected as the first local memory pool;
[0118] The performance score of the first local memory pool is determined based on the performance metrics corresponding to the first local memory pool.
[0119] Based on the task priority of the table join request and the performance score of the first local memory pool, select the target local shared memory pool from the first local memory pool.
[0120] In one embodiment, the second determining module is further specifically used for:
[0121] If there is only one first local memory pool, then the first local memory pool will be used as the target local shared memory pool.
[0122] When there are at least two first local shared memory pools, the performance level of each first local memory pool is determined based on the performance score of each first local memory pool.
[0123] The first local memory pool whose performance level matches the task priority of the table join request is used as the target local shared memory pool.
[0124] In one embodiment, a table connection device 1 further includes:
[0125] The partitioning module is used to partition the first data table and the second data table when the memory usage mode is cache usage, to obtain the first table partition corresponding to the first data table and the second table partition corresponding to the second data table.
[0126] The second sending module is used to transmit the first table partition and the second table partition to each computing node through the candidate local memory pool, so that each computing node can perform table join operations on the first data table and the second data table based on the first table group and the second table partition.
[0127] Each module in the aforementioned table connection device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.
[0128] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 8 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores table-related information. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When executed by the processor, the computer program implements a table join method.
[0129] Those skilled in the art will understand that Figure 8 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.
[0130] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:
[0131] In response to a table join processing request for the first data table and the second data table, determine the first memory usage of the first data table and the second memory usage of the second data table;
[0132] The target local shared memory pool is determined based on the minimum memory usage between the first and second memory usages.
[0133] The data table corresponding to the minimum memory usage is sent to the target local memory shared pool, so that each computing node can retrieve the data table corresponding to the minimum memory usage from the target local memory shared pool and perform table join operations on the first data table and the second data table.
[0134] In one embodiment, when the processor executes a computer program, it further performs the following steps: determining a target local shared memory pool based on the minimum memory usage between a first occupied memory and a second occupied memory, including:
[0135] The memory pool usage method of the candidate local memory pool is determined based on the minimum memory usage between the first and second memory usages and the memory capacity of the candidate local memory pool.
[0136] When the memory pool is used as shared memory, the target local shared memory pool is selected from the candidate local memory pools based on the minimum memory usage between the first and second occupied memory, the table join processing requests, and the performance indicators of the memory devices corresponding to the candidate local memory pools.
[0137] In one embodiment, when the processor executes a computer program, it further performs the following steps: determining the memory pool usage of the candidate local memory pool based on the minimum memory usage between the first occupied memory and the second occupied memory and the memory capacity of the candidate local memory pool, including:
[0138] Sort the candidate local memory pools by memory capacity and determine the maximum memory capacity;
[0139] If the minimum memory usage between the first and second memory usages is less than the maximum memory capacity, the memory pool usage mode of the candidate local memory pool is determined to be used as shared memory.
[0140] If the minimum memory usage between the first and second memory usages is not less than the maximum memory capacity, the candidate local memory pool is determined to be used as a cache.
[0141] In one embodiment, when the processor executes a computer program, it further performs the following steps: selecting a target local shared memory pool from the candidate local memory pools based on the minimum memory usage between the first and second occupied memory, the table join processing request, and the performance metrics of the memory devices corresponding to the candidate local memory pools, including:
[0142] The candidate local memory pool with a memory capacity greater than the minimum memory usage is selected as the first local memory pool;
[0143] The performance score of the first local memory pool is determined based on the performance metrics corresponding to the first local memory pool.
[0144] Based on the task priority of the table join request and the performance score of the first local memory pool, select the target local shared memory pool from the first local memory pool.
[0145] In one embodiment, when the processor executes the computer program, it further performs the following steps: selecting a target local shared memory pool from the first local memory pool based on the task priority of the table join request and the performance score of the first local memory pool, including:
[0146] If there is only one first local memory pool, then the first local memory pool will be used as the target local shared memory pool.
[0147] When there are at least two first local shared memory pools, the performance level of each first local memory pool is determined based on the performance score of each first local memory pool.
[0148] The first local memory pool whose performance level matches the task priority of the table join request is used as the target local shared memory pool.
[0149] In one embodiment, the processor, when executing a computer program, also performs the following steps:
[0150] When the memory usage mode is cached, the first data table and the second data table are partitioned to obtain the first table partition corresponding to the first data table and the second table partition corresponding to the second data table.
[0151] The first table partition and the second table partition are transferred to each compute node through the candidate local memory pool, so that each compute node can perform table join operations on the first data table and the second data table based on the first table group and the second table partition.
[0152] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:
[0153] In response to a table join processing request for the first data table and the second data table, determine the first memory usage of the first data table and the second memory usage of the second data table;
[0154] The target local shared memory pool is determined based on the minimum memory usage between the first and second memory usages.
[0155] The data table corresponding to the minimum memory usage is sent to the target local memory shared pool, so that each computing node can retrieve the data table corresponding to the minimum memory usage from the target local memory shared pool and perform table join operations on the first data table and the second data table.
[0156] In one embodiment, when the computer program is executed by a processor, it further performs the following steps: determining a target local shared memory pool based on the minimum memory usage between a first occupied memory and a second occupied memory, including:
[0157] The memory pool usage method of the candidate local memory pool is determined based on the minimum memory usage between the first and second memory usages and the memory capacity of the candidate local memory pool.
[0158] When the memory pool is used as shared memory, the target local shared memory pool is selected from the candidate local memory pools based on the minimum memory usage between the first and second occupied memory, the table join processing requests, and the performance indicators of the memory devices corresponding to the candidate local memory pools.
[0159] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: determining the memory pool usage of the candidate local memory pool based on the minimum memory usage between the first and second occupied memory and the memory capacity of the candidate local memory pool, including:
[0160] Sort the candidate local memory pools by memory capacity and determine the maximum memory capacity;
[0161] If the minimum memory usage between the first and second memory usages is less than the maximum memory capacity, the memory pool usage mode of the candidate local memory pool is determined to be used as shared memory.
[0162] If the minimum memory usage between the first and second memory usages is not less than the maximum memory capacity, the candidate local memory pool is determined to be used as a cache.
[0163] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: selecting a target local shared memory pool from the candidate local memory pools based on the minimum memory usage between the first and second occupied memory, the table join processing request, and the performance metrics of the memory devices corresponding to the candidate local memory pools, including:
[0164] The candidate local memory pool with a memory capacity greater than the minimum memory usage is selected as the first local memory pool;
[0165] The performance score of the first local memory pool is determined based on the performance metrics corresponding to the first local memory pool.
[0166] Based on the task priority of the table join request and the performance score of the first local memory pool, select the target local shared memory pool from the first local memory pool.
[0167] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: selecting a target local shared memory pool from the first local memory pool based on the task priority of the table join request and the performance score of the first local memory pool, including:
[0168] If there is only one first local memory pool, then the first local memory pool will be used as the target local shared memory pool.
[0169] When there are at least two first local shared memory pools, the performance level of each first local memory pool is determined based on the performance score of each first local memory pool.
[0170] The first local memory pool whose performance level matches the task priority of the table join request is used as the target local shared memory pool.
[0171] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:
[0172] When the memory usage mode is cached, the first data table and the second data table are partitioned to obtain the first table partition corresponding to the first data table and the second table partition corresponding to the second data table.
[0173] The first table partition and the second table partition are transferred to each compute node through the candidate local memory pool, so that each compute node can perform table join operations on the first data table and the second data table based on the first table group and the second table partition.
[0174] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, performs the following steps:
[0175] In response to a table join processing request for the first data table and the second data table, determine the first memory usage of the first data table and the second memory usage of the second data table;
[0176] The target local shared memory pool is determined based on the minimum memory usage between the first and second memory usages.
[0177] The data table corresponding to the minimum memory usage is sent to the target local memory shared pool, so that each computing node can retrieve the data table corresponding to the minimum memory usage from the target local memory shared pool and perform table join operations on the first data table and the second data table.
[0178] In one embodiment, when the computer program is executed by a processor, it further performs the following steps: determining a target local shared memory pool based on the minimum memory usage between a first occupied memory and a second occupied memory, including:
[0179] The memory pool usage method of the candidate local memory pool is determined based on the minimum memory usage between the first and second memory usages and the memory capacity of the candidate local memory pool.
[0180] When the memory pool is used as shared memory, the target local shared memory pool is selected from the candidate local memory pools based on the minimum memory usage between the first and second occupied memory, the table join processing requests, and the performance indicators of the memory devices corresponding to the candidate local memory pools.
[0181] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: determining the memory pool usage of the candidate local memory pool based on the minimum memory usage between the first and second occupied memory and the memory capacity of the candidate local memory pool, including:
[0182] Sort the candidate local memory pools by memory capacity and determine the maximum memory capacity;
[0183] If the minimum memory usage between the first and second memory usages is less than the maximum memory capacity, the memory pool usage mode of the candidate local memory pool is determined to be used as shared memory.
[0184] If the minimum memory usage between the first and second memory usages is not less than the maximum memory capacity, the candidate local memory pool is determined to be used as a cache.
[0185] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: selecting a target local shared memory pool from the candidate local memory pools based on the minimum memory usage between the first and second occupied memory, the table join processing request, and the performance metrics of the memory devices corresponding to the candidate local memory pools, including:
[0186] The candidate local memory pool with a memory capacity greater than the minimum memory usage is selected as the first local memory pool;
[0187] The performance score of the first local memory pool is determined based on the performance metrics corresponding to the first local memory pool.
[0188] Based on the task priority of the table join request and the performance score of the first local memory pool, select the target local shared memory pool from the first local memory pool.
[0189] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: selecting a target local shared memory pool from the first local memory pool based on the task priority of the table join request and the performance score of the first local memory pool, including:
[0190] If there is only one first local memory pool, then the first local memory pool will be used as the target local shared memory pool.
[0191] When there are at least two first local shared memory pools, the performance level of each first local memory pool is determined based on the performance score of each first local memory pool.
[0192] The first local memory pool whose performance level matches the task priority of the table join request is used as the target local shared memory pool.
[0193] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:
[0194] When the memory usage mode is cached, the first data table and the second data table are partitioned to obtain the first table partition corresponding to the first data table and the second table partition corresponding to the second data table.
[0195] The first table partition and the second table partition are transferred to each compute node through the candidate local memory pool, so that each compute node can perform table join operations on the first data table and the second data table based on the first table group and the second table partition.
[0196] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.
[0197] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0198] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.
Claims
1. A table join method, characterized by, The method comprises: in response to a table connection processing request for a first data table and a second data table, determining a first occupied memory of the first data table and a second occupied memory of the second data table; determining a target local shared memory pool according to a minimum occupied memory between the first occupied memory and the second occupied memory; sending a data table corresponding to the minimum occupied memory to the target local memory shared pool, so that each computing node retrieves the data table corresponding to the minimum occupied memory from the target local memory shared pool, and performs a table connection operation on the first data table and the second data table.
2. The method of claim 1, wherein, The method comprises: determining a memory pool usage mode of the candidate local memory pool according to the minimum occupied memory between the first occupied memory and the second occupied memory and a memory capacity of the candidate local memory pool; in a case where the memory pool usage mode is used as shared memory, selecting a target local shared memory pool from the candidate local memory pool according to the minimum occupied memory between the first occupied memory and the second occupied memory, the table connection processing request, and a performance index of a memory device corresponding to the candidate local memory pool.
3. The method of claim 2, wherein, The method comprises: sorting the memory capacity of the candidate local memory pool to determine a maximum memory capacity; in a case where the minimum occupied memory between the first occupied memory and the second occupied memory is less than the maximum memory capacity, determining that the memory pool usage mode of the candidate local memory pool is used as shared memory; in a case where the minimum occupied memory between the first occupied memory and the second occupied memory is not less than the maximum memory capacity, determining that the memory pool usage mode of the candidate local memory pool is used as cache.
4. The method of claim 2, wherein, The method comprises: selecting a target local shared memory pool from the candidate local memory pool according to the minimum occupied memory between the first occupied memory and the second occupied memory, the table connection processing request, and a performance index of a memory device corresponding to the candidate local memory pool. selecting a candidate local memory pool with a memory capacity greater than the minimum occupied memory as a first local memory pool; determining a performance score of the first local memory pool according to a performance index corresponding to the first local memory pool; 5. The method of claim 4, wherein, selecting a target local shared memory pool from the first local memory pool according to a task priority of the table connection request and the performance score of the first local memory pool. The method comprises: in a case where the number of the first local memory pools is one, selecting the first local memory pool as the target local shared memory pool; In a case where the number of the first local shared memory pools is at least two, performance levels of the first local memory pools are determined according to performance scores of the first local memory pools; The first local memory pool, whose performance level matches the task priority of the table join request, is matched as the target local shared memory pool.
6. The method of claim 3, wherein, The method further comprises: In a case where the memory usage mode is cache usage, the first data table and the second data table are partitioned to obtain a first table partition corresponding to the first data table and a second table partition corresponding to the second data table; The first table partition and the second table partition are transmitted to each computing node through the candidate local memory pool, so that each computing node performs a table join operation on the first data table and the second data table based on the first table partition and the second table partition.
7. A table joining device, characterized by Comprise: A first determination module configured to determine a first occupied memory of the first data table and a second occupied memory of the second data table in response to a table join processing request for the first data table and the second data table; A second determination module configured to determine a target local shared memory pool according to a minimum occupied memory between the first occupied memory and the second occupied memory; A first sending module configured to send a data table corresponding to the minimum occupied memory to the target local memory shared pool, so that each computing node retrieves the data table corresponding to the minimum occupied memory from the target local memory shared pool and performs a table join operation on the first data table and the second data table. 8.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is configured to perform the method according to any one of claims 1-7. The processor executes the computer program to realize the steps of the table join method in any one of claims 1 to 6.
9. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to realize the steps of the table join method in any one of claims 1 to 6.
10. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to realize the steps of the table join method in any one of claims 1 to 6.