Real-time database distributed query optimization method based on memory calculation

Abstract

The invention relates to the technical field of distributed database optimization, and discloses a real-time database distributed query optimization method based on memory computing, comprising the following steps: modeling query delays, the query delays comprising computing delays and communication delays; and based on the modeling result of the query delay, the query progress and the resource allocation strategy of each node are determined, each node is a single calculation unit in the distributed database, the calculation delay of the node is inversely proportional to the resource allocation and query load of the node, and the communication delay between the nodes is determined by the data transmission quantity and the bandwidth between the nodes. Through an optimal control theory, a game theory model and a self-adaptive control mechanism, dynamic allocation of calculation and communication resources, collaborative optimization of resources among nodes, self-adaptive adjustment of query loads and resource scheduling optimization driven by historical data are realized, the query execution efficiency is effectively improved, and the overall query delay is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of distributed database optimization, and specifically to a distributed query optimization method for a real-time database based on in-memory computing. Background Art

[0002] In the context of the growing demands for large-scale data processing and real-time querying, distributed databases have become an important infrastructure to support modern information systems. Whether in application scenarios such as financial transaction analysis, e-commerce data retrieval, or large-scale data stream processing in the Internet of Things, the query response speed directly affects the service quality of the system. However, due to the huge amount of data, query tasks often need to be executed in parallel among multiple computing nodes, and the computational complexity, data transmission requirements, and resource consumption of different queries are all different.

[0003] The prior art has made certain progress in distributed query optimization. Some optimization methods are based on the static analysis of query plans. By means of query statement parsing, index optimization, etc., unnecessary data scans and computational amounts are reduced, thereby improving the query execution efficiency. In addition, some technical solutions adopt task scheduling optimization strategies. According to the load conditions of computing nodes, query tasks are assigned to different nodes to improve the utilization rate of computing resources. At the same time, in order to reduce the network communication cost, some solutions also optimize data sharding and local computing to reduce cross-node data transmission and improve the query execution speed.

[0004] However, there are still many deficiencies in the prior art, resulting in difficulties in further improving query performance. First of all, the existing static query optimization strategies lack the ability of dynamic adjustment. During the execution of query tasks, the load and bandwidth conditions of computing nodes are likely to change at any time, and fixed resource allocation strategies are difficult to adapt to the real-time fluctuations of query requirements, resulting in overload of some nodes and longer query response times. Secondly, the allocation of computing resources and network bandwidth lacks a global optimization mechanism. Each computing node usually performs resource scheduling independently and lacks an effective coordination strategy. This way is likely to cause some query tasks to occupy too much computing resources, while other query tasks are blocked for a long time, affecting the system throughput. In addition, the flexibility of resource scheduling is insufficient and cannot adapt to the changes in query load. When some query tasks involve a large amount of computing or cross-node data transmission, the system lacks an effective feedback mechanism and cannot adjust computing or communication resources according to the query execution progress, resulting in large fluctuations in query performance. Summary of the Invention

[0005] Aiming at the deficiencies of the prior art, the present invention provides a distributed query optimization method for a real-time database based on in-memory computing, which solves the problems of fixed resource allocation methods, lack of cooperation among computing nodes, lag in adjustment when query load changes, and inability to optimize resource scheduling based on historical data in the prior art.

[0006] To achieve the above objectives, the present invention is realized through the following technical solutions: A real-time database distributed query optimization method based on in-memory computing, comprising the following steps:

[0007] Model the query latency, where the query latency includes computational latency and communication latency;

[0008] Based on the modeling results of the query latency, determine the query progress and resource allocation strategy for each node. Each node is a single computing unit in the distributed database, and the computational latency of a node is inversely proportional to its resource allocation and query load. The communication latency between nodes is determined by the data transmission volume and the bandwidth between nodes;

[0009] Utilize the optimal control theory to optimize the resource allocation strategy for each node to minimize the total query latency. Among them, the resource allocation strategy for each node is dynamically adjusted based on the load status and communication requirements of the node;

[0010] Based on the game theory model, consider the resource competition and cooperation between each node. By adjusting the resource allocation strategy of the node, minimize the overall query latency;

[0011] By monitoring the query progress, load information, and network status of each node, adopt an adaptive control mechanism to dynamically adjust the resource allocation strategy of each node to maintain the optimal query performance.

[0012] Preferably, the modeling of the query latency includes:

[0013] Construct a computational model of the node to represent the processing time of the node's computational tasks. The computational model associates the resource allocation of the node with its computational load;

[0014] Model the communication latency between nodes, where the communication latency depends on the load information of each node and the bandwidth between nodes;

[0015] Minimize the sum of the query latency as the objective function. Among them, the computational latency and the communication latency respectively occupy certain weight coefficients to optimize the query performance.

[0016] Preferably, the optimal control theory includes:

[0017] Optimize the resource allocation for each node. The optimization steps determine the optimal resource allocation strategy for each node through the optimal control algorithm to minimize the query latency;

[0018] Among them, the resource allocation strategy of the node is dynamically adjusted according to the query load, network bandwidth, and the load status of the node;

[0019] The optimal control algorithm is adaptively adjusted based on the real-time status of nodes to ensure the optimal execution of query tasks under different load conditions.

[0020] Preferably, the game theory model includes:

[0021] Analyze the resource competition and cooperation relationships among nodes. The game theory model reflects the mutual influence among nodes by setting a competition coefficient;

[0022] The game theory model minimizes the global query latency by optimizing the resource allocation strategy of each node;

[0023] Under this game model, nodes perform resource scheduling according to global information to achieve the optimal balance of overall performance.

[0024] Preferably, the dynamic adjustment includes:

[0025] In the case of high query load, resources are preferentially allocated to high-priority query tasks to ensure low-latency execution of important query tasks;

[0026] Adjust the resource configuration of nodes according to the real-time load information and resource requirements among nodes to ensure balanced load and efficient operation of the system;

[0027] Nodes dynamically select resource allocation strategies according to the weights of communication latency and computing latency to optimize the overall query latency.

[0028] Preferably, the further applications of the game theory model include:

[0029] When performing resource scheduling among nodes, based on the dual objectives of load balancing and latency minimization, a cooperative game mode is adopted to coordinate the resource allocation among nodes to avoid excessive competition among nodes and improve the global query efficiency;

[0030] Each node actively adjusts its resource usage strategy based on the global resource situation, and uses the cooperative game strategy to relieve the resource bottleneck of a single node, thereby enhancing the stability of the system and the query response speed;

[0031] The game theory model balances the resource competition among nodes by calculating the revenue function of each node, further optimizes the query latency, and ensures the efficient operation of the system in a multi-node environment.

[0032] Preferably, the optimization steps further include:

[0033] According to historical query data, predict the query load and node status, and then adjust the resource allocation strategy of each node so that the system can proactively respond to potential high-load situations;

[0034] For scenarios with large fluctuations in query load, adjust the allocation ratio of computing resources and bandwidth resources to better adapt to load changes and ensure that the query latency remains within the expected range;

[0035] By sharing load predictions and resource allocation information among each node, further improve the efficiency of resource scheduling.

[0036] Preferably, the modeling of the query latency further includes:

[0037] Through historical data analysis and real-time monitoring, combined with query load prediction and computing resource evaluation, adjust the resource allocation strategy of the nodes so that resources can adapt dynamically under different query loads to optimize query performance;

[0038] When considering the load information of each node, dynamically adjust the weights of communication latency and computing latency, thereby more precisely controlling the latency minimization strategy to ensure fast response under high load;

[0039] The update and feedback mechanism of the query latency model can calibrate the prediction results in real time, improving the accuracy of resource scheduling and query execution efficiency.

[0040] Preferably, the adaptive adjustment includes:

[0041] In the case of high fluctuations in query load, by dynamically adjusting the allocation ratio of computing resources and bandwidth resources to meet the low-latency requirements of query tasks;

[0042] Through the analysis of historical data and real-time feedback mechanism, adjust the resource allocation strategy of each node so that resource configuration can anticipate potential high-load situations;

[0043] The adjustment process uses a dynamic feedback algorithm to minimize computing latency and optimize resource configuration based on the real-time state of the system, keeping the query latency within a predetermined range.

[0044] Preferably, the resource allocation for the query further includes:

[0045] According to historical query data and real-time load prediction information, pre-configure resources for each node to ensure that the query latency can still be maintained within an acceptable range under high load;

[0046] In the case of large fluctuations in query load, dynamically adjust the allocation ratio of computing resources and bandwidth resources to optimize the resource utilization efficiency of the nodes;

[0047] By sharing the load prediction and resource allocation information of each node, further improve the overall resource scheduling efficiency of the system to optimize query performance.

[0048] The present invention provides a distributed query optimization method for a real-time database based on in-memory computing, which has the following beneficial effects:

[0049] 1. The present invention adopts a resource allocation optimization strategy based on the optimal control theory, achieving the technical effect of dynamically adjusting computing and communication resources. Compared with the technical solutions in the prior art where static resource allocation strategies are difficult to adapt to the dynamic changes of query loads, it solves the deficiencies that the utilization rates of computing resources and network bandwidth are low during the execution of query tasks, resulting in unpredictable query response times.

[0050] 2. The present invention uses a game theory model to analyze the resource competition and cooperation mechanism among nodes, achieving the technical effect of optimizing global resource allocation and reducing unnecessary delays caused by resource contention. Compared with the technical solutions in the prior art where the resources of each node are independently scheduled and the global optimization ability is insufficient, it solves the problem that due to the ineffective coordination of the competition relationship among computing nodes, some nodes are overloaded and some nodes waste resources, affecting the query throughput.

[0051] 3. The present invention adopts an adaptive control mechanism combined with real-time feedback to adjust the resource scheduling strategy, achieving the technical effect of dynamically matching computing requirements and network conditions during the execution of queries. Compared with the technical solutions in the prior art where there is a lack of real-time load monitoring and resource adaptive adjustment and it is unable to flexibly optimize when query loads fluctuate, it solves the deficiencies that high-load query tasks occupy resources for a long time and low-load query tasks do not receive sufficient computing support, resulting in unstable query response times.

[0052] 4. The present invention adopts a load prediction method based on historical query data, achieving the technical effect of optimizing computing resources and bandwidth allocation in advance and improving the execution efficiency of query tasks. Compared with the technical solutions in the prior art where query resource scheduling completely depends on the real-time load situation and lacks the ability of early warning and optimization, it solves the problem that resource contention is serious during peak queries and resources are wasted during off-peak queries, resulting in a decrease in the overall throughput of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 It is a schematic flowchart of the method of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0054] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0055] Please refer to the attached Figure 1, the embodiments of the present invention provide a distributed query optimization method for a real-time database based on in-memory computing, including the following steps:

[0056] S1. Model the query latency, where the query latency includes computational latency and communication latency;

[0057] In the process of query optimization in a distributed database, query latency is one of the key factors affecting system performance. Query latency mainly consists of computational latency and communication latency. Computational latency comes from the execution time of query tasks on nodes, while communication latency is related to data interaction between nodes. Generally, query optimization needs to comprehensively consider computing power, bandwidth resources, and load balancing to ensure that query tasks can be completed efficiently. In the present invention, the modeling of query latency not only involves the analysis of computing resources and data transmission characteristics but also combines the optimal control theory and game theory model to dynamically optimize the resource allocation of nodes, improve query efficiency, and reduce latency. In particular, the present invention further realizes real-time monitoring of node load and network status based on an adaptive control mechanism, so as to dynamically adjust the resource allocation strategy during query execution to adapt to changes in query load.

[0058] In this embodiment, modeling the query latency specifically includes two main parts: computational latency and communication latency.

[0059] Generally, computational latency is determined by the computational complexity of the query task, the resource allocation of the node, and the load status. Computational latency can be expressed as:

[0060]

[0061] where T comp represents computational latency; L represents the computational load of the query task, and this load can be determined by factors such as the amount of data involved in the query, index usage, and the complexity of query logical operations; C n is the computational resource capacity of node n, including CPU clock frequency, number of computing cores, and memory bandwidth, etc.

[0062] In a possible implementation, the computing power of a node is limited by its currently allocated resources. Therefore, when the load is high, the computing power may decrease, thereby increasing query latency. In some embodiments, the computational resource allocation of the node can be dynamically adjusted through an optimal control algorithm to reduce computational latency and improve query execution efficiency.

[0063] On the other hand, communication latency is mainly determined by the data transmission volume involved in the query task and the bandwidth between nodes. Generally, communication latency can be expressed as:

[0064]

[0065] Among them, T comm represents the communication delay; D represents the amount of data that the query task needs to transmit between nodes, which usually depends on the execution method of the query statement, such as whether it involves table joins (JOIN), data aggregations (AGGREGATION), etc.; B is the available bandwidth between nodes, which is limited by the physical network environment, concurrent data traffic, and bandwidth allocation policies, etc.

[0066] In some embodiments, to reduce the communication delay, the allocation strategy of the query task can be adjusted to minimize the data transfer between multiple nodes as much as possible. For example, when performing a distributed query, nodes with better data locality can be preferentially selected to execute the computing task to reduce remote data transmission, thereby reducing the value of D and improving the query efficiency.

[0067] As an option, the optimization of the query delay not only depends on the static computation and communication delay model, but also incorporates an adaptive control mechanism that enables the system to dynamically adjust the allocation of computing resources and communication resources during operation. Specifically, the system can monitor the load information of each node, the execution progress of the current query task, and the network status in real time, and adjust the resource allocation based on the feedback control strategy. When the load is high, the system can increase the allocation of computing resources for high-priority query tasks while reducing the resource occupancy of low-priority tasks to ensure that the query response time is within an acceptable range.

[0068] In a possible implementation, the optimization of the query delay also incorporates a game theory model to balance the resource competition and cooperation between nodes. Specifically, when each node executes a query task, it needs to trade off between computing resources and communication resources. If all nodes compete for computing resources first, it may lead to insufficient bandwidth utilization, while if the bandwidth resources are over-occupied, it may cause network congestion. Therefore, in the present invention, the game theory model is used to analyze the competition relationship between nodes and dynamically adjust the resource allocation strategy to optimize the overall query delay.

[0069] In addition, in this embodiment, the modeling of the query delay also considers historical query data and load prediction factors. Generally, there are significant fluctuations in the computing and communication requirements of different query tasks. Therefore, the system can predict the future query load through historical data analysis and adjust the resource allocation strategy in advance. For example, in some embodiments, machine learning methods such as long short-term memory networks (LSTM) or reinforcement learning algorithms can be used to analyze the query patterns in the past period of time, predict the future query load trend, and accordingly adjust the computing and communication resources of the nodes to reduce the query delay.

[0070] S2. Based on the modeling results of query latency, determine the query progress and resource allocation strategy for each node. Each node is a single computing unit in the distributed database, and the computing latency of the node is inversely proportional to its resource allocation and query load. The communication latency between nodes is determined by the data transmission volume and the bandwidth between nodes.

[0071] In a distributed database environment, the execution of a query often involves multiple computing nodes. The system needs to reasonably allocate computing and communication resources among different nodes to optimize query latency. Based on the aforementioned query latency modeling, the present invention further determines the query progress and resource allocation strategy for each node, making the allocation of computing tasks more in line with the global optimization goal of the system. Generally, the computing power of a computing node is limited, and its computing latency is inversely proportional to the resource allocation and query load, that is, the more computing resources are allocated, the larger the amount of query tasks that can be processed per unit time, and the corresponding computing latency is reduced. In addition, the data interaction between nodes is subject to bandwidth constraints, and the communication latency is mainly determined by the data transmission volume and the bandwidth size. Therefore, in the actual query optimization process, it is necessary to comprehensively consider the allocation of node computing power and communication bandwidth resources to ensure the efficiency of overall query execution. In the present invention, by analyzing the modeling results of query latency, the query task progress of each node is dynamically adjusted, and the allocation of computing and communication resources is reasonably planned to minimize query latency while ensuring system load balancing.

[0072] In this embodiment, based on the query latency modeling results, determine the query progress and resource allocation strategy for each node.

[0073] Generally, the query computing power of a computing node is affected by the amount of resources it can use. When the query load is high, insufficient computing resources will lead to an increase in query latency. To quantify this relationship, the time for computing node n to process a query task can be defined as:

[0074]

[0075] where, T calc,n is the computing latency required for computing node n to execute the query task; W n is the workload of the query task allocated to node n, which is usually determined by factors such as the amount of data involved in the query, the index usage, and the query complexity; R n represents the computing resources allocated to node n, including CPU processing power, memory bandwidth, etc., which can be dynamically adjusted.

[0076] As an option, the system can dynamically adjust R n during the query execution process through a load balancing strategy to reduce the computing latency of high-load nodes. For example, during the execution of a query task, if the computing load of a certain node is much higher than that of other nodes, its query task volume W can be reduced.n , and assign some tasks to the nodes with lighter loads to achieve balanced distribution of computing tasks, thereby optimizing the overall query performance.

[0077] On the other hand, communication latency is also a major component of query execution time, affecting the data transfer efficiency between nodes. Generally, the transfer latency of data between two nodes i and j can be expressed as:

[0078]

[0079] where T comm,ij is the data transfer time between node i and node j; S ij is the amount of data transferred between nodes i and j, which is usually related to the size of the tables involved in the query, the type of query, and the data aggregation method; B ij is the effective bandwidth between nodes i and j, which is affected by the network topology, bandwidth sharing situation, and bandwidth management strategy.

[0080] Specifically, communication latency can be optimized by reducing the amount of data transferred S ij or increasing the available bandwidth B ij . In a possible implementation, data transfer can be reduced through local computing strategies. For example, when a query task involves a join operation (JOIN) of multiple tables, the join calculation can be preferentially executed on the local node where the data is stored to reduce the need for cross-node data transfer, thereby reducing communication latency.

[0081] In some embodiments, the task scheduling order of nodes can also be dynamically adjusted based on the execution progress of the query. For example, during the query execution process, if the computing tasks of some nodes have not been completed while the computing tasks of other nodes have been completed, the execution order of subsequent query tasks can be appropriately adjusted to avoid an increase in the overall query latency caused by the asynchronous execution of computing tasks. This dynamic scheduling method can ensure smoother execution of query tasks and improve system resource utilization.

[0082] In the implementation of the present invention, in order to further optimize the execution efficiency of query tasks, the system uses the optimal control theory to dynamically adjust the query progress and resource allocation strategy. Generally, the optimal execution time of a query task can be expressed as minimizing the objective function:

[0083] min∑ n (T calc,n +∑jT comm,nj );

[0084] where ∑ n (T calc,n ) represents the total computing latency of all computing nodes; ∑ j Tcomm,nj Denote the data transmission delay of node n among all relevant nodes; min represents minimizing the objective function, that is, optimizing the query execution process to minimize the total delay.

[0085] In some embodiments, the execution progress of the query task can be adjusted based on a feedback control mechanism. For example, when the computing load of a certain node exceeds the threshold, the system can temporarily reduce its query task allocation, reduce its computing delay, and at the same time adjust the resource utilization of other nodes to optimize the overall query execution time.

[0086] As an optimization method, the system can also combine game theory models to coordinate the resource competition among nodes. In a distributed database environment, each node hopes to maximize its computing efficiency while avoiding resource bottlenecks affecting the query task execution. In the game theory model, the resource allocation of each node can be regarded as a strategy optimization problem, whose goal is to minimize the global query delay while maximizing the resource utilization. Specifically, each node n selects a resource allocation strategy and adjusts according to the strategies of other nodes, so as to finally form a global optimization solution:

[0087]

[0088] where is the optimal resource allocation strategy of node n, which needs to comprehensively consider the computing delay, communication delay, and the overall query execution situation of the system, ∑ n (T calc,n ) represents the total computing delay of all computing nodes; ∑ j T comm,nj denotes the data transmission delay of node n among all relevant nodes; argmin represents finding the optimal parameter value that minimizes the objective function, that is, finding the optimal resource allocation to minimize the sum of computing and communication delays.

[0089] In some embodiments, the system can also combine machine learning techniques. By analyzing historical query data, it can predict the load situation of future query tasks and adjust the query progress and resource allocation in advance. For example, a time series prediction model (such as an LSTM network) can be used to analyze the query task execution data in the past period and predict the possible load fluctuations in the future, so as to adjust the computing resources and bandwidth allocation in advance to make the execution of query tasks more efficient.

[0090] S3. Use optimal control theory to optimize the resource allocation strategy of each node to minimize the total query delay, where the resource allocation strategy of each node is dynamically adjusted based on the load status and communication requirements of the node;

[0091] In the process of query optimization in a distributed database, the reasonable allocation of resources is a key factor affecting query latency. Since both computing resources and network bandwidth are limited resources, the system needs to perform efficient resource scheduling among different computing nodes to ensure that query tasks can be executed in an optimal manner. Generally, the resource allocation of each computing node not only affects its own computing latency but also indirectly affects the overall communication overhead of the system. Therefore, relying solely on static resource allocation strategies is difficult to meet the real-time optimization requirements of distributed queries, and dynamic adjustment of resource allocation is required through optimal control theory. Based on optimal control theory, this invention monitors the node load status and communication requirements in real time during the execution of query tasks, and optimizes the allocation of computing resources and network bandwidth accordingly, so that the system can reduce unnecessary data transmission overhead while ensuring computing power, thereby minimizing the total query latency. During the optimization process, not only the computing power of a single node is considered, but also the global query execution situation is combined to improve the overall throughput and response speed of the system.

[0092] In this embodiment, the optimal control theory is used to optimize the resource allocation strategy of each node to minimize the total query latency, and the resource allocation strategy of each node is dynamically adjusted based on its current load status and communication requirements.

[0093] Generally, the computing latency of query tasks is greatly affected by computing resources. When computing resources are sufficient, the computing time of a single query task is short, while in the case of tight computing resources, the query execution time will increase accordingly. To quantify this relationship, the computing latency of node n processing query tasks can be defined as follows:

[0094]

[0095] where T c,n is the computing latency of node n; Q n represents the query load assigned to node n, which is usually determined by factors such as the data volume involved in the query, the index structure, and the complexity of query operations; R n represents the computing resources allocated to node n, including CPU computing power, memory usage, and cache resources, etc.; α is the computing efficiency coefficient, and its value is affected by the hardware capabilities of the node and the current load level.

[0096] In a possible implementation, the system dynamically adjusts the allocation strategy of R n according to the load status of the computing node, so that high-load nodes can obtain more computing resources, while low-load nodes reduce resource occupancy to achieve balanced scheduling of computing tasks. This adaptive adjustment strategy can effectively reduce computing latency and improve query execution efficiency.

[0097] On the other hand, communication latency is also crucial for query optimization, especially in cases where multiple nodes collaborate to process query tasks. The data transfer time between nodes is usually determined by bandwidth resources and the amount of data. The communication latency can be expressed as follows:

[0098]

[0099] where T comm,n is the communication latency of node n; D n is the amount of data that needs to be transferred for the query task at node n, which is usually determined by factors such as the size of the data tables involved in the query, the way of join operations (JOIN), and data aggregation (AGGREGATION); B n represents the effective bandwidth allocated to node n, which is affected by physical network conditions, current bandwidth usage, and bandwidth scheduling policies; β is the bandwidth utilization efficiency factor, which is used to measure the actual data transfer efficiency.

[0100] Specifically, in some embodiments, the system dynamically adjusts the data transfer strategy between nodes according to the communication requirements of the current query task. For example, when the bandwidth resources of a certain node are tight, the system can reduce the frequency of remote data access for that node and try to execute computational tasks locally to reduce communication latency. In addition, if a certain query task involves a large amount of data transfer, the system can preferentially allocate higher bandwidth resources to this query task to ensure that it can complete data interaction faster.

[0101] In one possible implementation, the system uses an optimal control algorithm to minimize the total query latency, and the optimization objective can be defined as follows:

[0102]

[0103] where the optimal control variables include computational resources R n and bandwidth resources B n , and the goal is to dynamically adjust the allocation of these resources during the execution of the query task to minimize the overall query latency; min represents minimizing the objective function, that is, optimizing the query execution process to minimize the total latency; Q n represents the query task load that node n needs to process, and its value is related to factors such as the computational complexity and data volume of the query; α is the computational efficiency coefficient, and its value is affected by the hardware capabilities of the node and the current load level; R n represents the computational resources allocated to node n, including CPU computing power, memory usage, and cache resources, etc.; D n is the amount of data that needs to be transferred for the query task at node n; β is the bandwidth utilization efficiency factor, which is used to measure the actual data transfer efficiency; B nRepresents the effective bandwidth allocated to node n, and this value is affected by physical network conditions, current bandwidth usage, and bandwidth scheduling policies.

[0104] As an option, the system can also utilize a feedback control mechanism to monitor the query execution status in real time and adjust the resource allocation strategy based on historical query data. For example, during query execution, if the execution time of a computing task on a certain node is too long, the system can immediately increase its computing resource allocation or reduce its query load Q n , to optimize the query execution time. Similarly, if the data transmission time of a certain node exceeds the threshold, the system can adjust the bandwidth allocation B n , or adopt data compression technology to reduce the amount of data transmitted D n , thereby reducing communication latency.

[0105] In some embodiments, the optimal control theory also combines a game theory model to coordinate the resource competition relationships among different nodes. In a distributed environment, each node hopes to maximize its own computing power while reducing data transmission overhead. Therefore, the system needs to reasonably allocate computing resources and bandwidth resources among different nodes to ensure global query optimization. In the game theory model, the resource allocation strategy of each node can be regarded as an optimization problem, and its goal is to ensure that all nodes can achieve a balance between computing power and communication capabilities, making the overall query performance of the system optimal.

[0106] In a possible implementation, the system is based on a reinforcement learning method. By analyzing the execution situations of historical query tasks, it predicts future query load changes and adjusts the resource allocation strategy of computing resources and bandwidth resources in advance. For example, using time series analysis methods, the system can predict the peak of query requests in a future time period and pre-allocate more computing resources to key query tasks before the load peak arrives, thereby reducing query latency during the peak period and improving system stability.

[0107] S4. Consider the resource competition and cooperation among each node based on the game theory model, and minimize the overall query latency by adjusting the resource allocation strategy of the nodes;

[0108] In a distributed database system, each computing node is an independent computing unit, and query tasks are executed in parallel among multiple nodes. Due to limited computing resources and bandwidth resources, resource competition is bound to occur among multiple nodes during the execution of query tasks. Unreasonable resource allocation may lead to overload of some nodes while underutilization of resources in other nodes, thus affecting the overall query efficiency. Therefore, it is difficult to achieve globally optimal query performance only through static resource allocation strategies, and it is necessary to analyze the resource competition relationship among different nodes based on game theory models and optimize the global resource utilization efficiency through cooperation. The present invention utilizes a game theory model to adjust the resource allocation strategies of nodes, enabling all nodes to achieve a globally optimal balance under the constraint of limited resources, so as to minimize the overall query latency. In this process, the system comprehensively considers the computing task load, network bandwidth occupancy, and complexity of query tasks, making the resource allocation more reasonable and improving the overall stability of query execution.

[0109] In this embodiment, based on the game theory model, the resource competition and cooperation among each node are considered, and by adjusting the resource allocation strategies of nodes, the overall query latency is minimized.

[0110] Generally, when each node n executes a query task, it will first maximize its own computing resource utilization rate and at the same time minimize the communication latency caused by data transmission. However, if all nodes only optimize resource usage from their own perspectives, it may lead to problems such as bandwidth contention and uneven distribution of computing resources, thereby increasing the overall query latency of the system. To quantify this process, this embodiment uses a non - cooperative game model to describe the resource competition among nodes and defines the revenue function of each node n as follows:

[0111]

[0112] where, U n represents the revenue of node n under the current resource allocation scheme; B n represents the effective bandwidth allocated to node n, and this value is affected by physical network conditions, current bandwidth usage, and bandwidth scheduling strategies; Q n represents the current query task load of this node; C n is the communication overhead between nodes, usually determined by the data transmission volume and bandwidth occupancy; γ and δ are weight parameters for resource allocation, used to balance the impacts of computing resources and communication resources.

[0113] In game - theoretic analysis, the resource competition among nodes can be achieved by adjusting R n and C n to make the overall revenue reach an equilibrium state. Specifically, the system needs to find a resource allocation strategy to make the revenue U n of all nodes reach the optimal globally.

[0114] As an option, the present invention further introduces a cooperative game model to achieve resource collaboration among nodes. Generally, if some nodes have fewer query tasks while other nodes have heavier tasks, then through resource sharing, some computing tasks can be assigned to low-load nodes to reduce the computing latency of high-load nodes. At the same time, the bandwidth resources can also be dynamically adjusted among different nodes to reduce the bandwidth occupancy of tasks with high communication requirements. The cooperative game model can be defined as the following optimization objective:

[0115] max∑ n log(U n );

[0116] where min represents minimizing the objective function, that is, optimizing the query execution process to minimize the total latency; log(U n ) represents the logarithm of the benefit function of node n. The logarithmic form is used to ensure the fairness of resource allocation and optimize the benefits of all nodes evenly.

[0117] Among them, by taking the logarithm of the revenue functions of all nodes and summing them, it can be ensured that the resource utilization rates of all nodes are evenly distributed globally, avoiding excessive resource occupancy by a single node and affecting the overall query performance.

[0118] Specifically, in a possible implementation, the system will monitor the resource usage of each node in real time during the query task execution process and adjust the resource allocation based on the cooperative game theory. For example, if the execution time of a certain node's computing task is too long, the system can reduce its query task load and increase the computing tasks of other low-load nodes to reduce the query latency. At the same time, if a certain node's network bandwidth is overloaded, the system can, if possible, reduce the remote data access of this node and increase the proportion of local computing tasks to optimize the communication latency.

[0119] In some embodiments, the reinforcement learning algorithm can also be combined. By analyzing historical query data, predicting the change trend of future query loads, and adjusting the resource allocation strategy in advance. For example, the reinforcement learning method based on the Markov decision process (MDP) can be used to analyze the impact of different resource allocation strategies on the query latency and dynamically optimize the resource allocation based on historical experience to make the query latency reach the optimal state.

[0120] In the further optimization of the game theory model, the present invention also considers the fairness issue of resource competition. Generally, if some high-priority query tasks occupy a large amount of computing resources and bandwidth resources, it may cause longer delays for low-priority tasks and affect the overall stability of the query system. To solve this problem, the present invention adopts a resource allocation strategy based on fairness constraints and defines the constrained optimization objective as follows:

[0121]

[0122] Among them, T c,n and T comm,n respectively represent the computing latency and the communication latency; min represents minimizing the objective function, that is, optimizing the query execution process to minimize the total latency; T c,n is the computing latency of node n; its value is affected by the computing resource allocation R n and the query task load Q n ; T comm,n represents the communication latency of node n, which is affected by the data transfer volume, the bandwidth resource B n and the network condition.

[0123] is a resource fairness constraint condition to ensure that the computing resources R m of all computing nodes n, m are close to the ratio with their query task loads Q n , Q m to achieve reasonable resource allocation and avoid resource overload or idleness of some nodes; indicates that the constraint condition applies to all computing nodes n and m to ensure the balance of global resource allocation.

[0124] The constraint condition ensures that the computing resource allocation of all nodes is proportional to their query loads, thus ensuring the fairness of resource allocation.

[0125] In some embodiments, the system can adjust the resource sharing ratio between nodes in real time based on a feedback control mechanism to ensure that the query task can still maintain stable query performance in a dynamic environment. For example, if the computing latency of a certain node increases due to an increase in computing tasks, the system can detect this situation through a real-time feedback mechanism and increase the computing resource allocation of this node to reduce the query latency. At the same time, if the communication latency of a certain node increases due to a large data transfer volume, the system can appropriately adjust the bandwidth resource allocation or reduce unnecessary data transfer to optimize the overall query efficiency.

[0126] S5. By monitoring the query progress, load information, and network condition of each node, an adaptive control mechanism is used to dynamically adjust the resource allocation strategy of each node to maintain the optimal query performance.

[0127] In the process of query optimization in a distributed database, the query progress, load conditions, and network status of nodes are all key factors affecting query efficiency. Since the occupancy of computing resources may change with load fluctuations during the execution of a query task, and the availability of network bandwidth may also be affected by concurrent queries or data transmission requirements, static resource allocation strategies are difficult to adapt to the dynamic execution requirements of query tasks. Generally, a distributed system needs to dynamically adjust the resource allocation strategy by monitoring the query execution status of each node in real time and combining the usage of computing resources and communication resources to optimize query performance. The present invention adopts an adaptive control mechanism. By monitoring the execution progress of the query task, the node load status, and the network bandwidth situation, it continuously optimizes the allocation of computing and communication resources during the query execution process, enabling the system to maintain optimal performance under different query load conditions. The present invention not only combines the operating status of computing nodes but also utilizes a feedback control mechanism to enable the system to adjust the resource allocation strategy in real time to ensure that the query task can be executed as efficiently as possible in a resource-constrained environment.

[0128] In this embodiment, by monitoring the query progress, load information, and network status of each node, an adaptive control mechanism is used to dynamically adjust the resource allocation strategy of each node to maintain optimal query performance.

[0129] Generally, the execution progress of a query task can be measured by the task completion ratio. Define the query progress P n as the completion degree of node n in executing the query task, expressed as follows:

[0130]

[0131] where P n is the execution progress of the current query task of node n; represents the workload of the query task that node n has executed; represents the total workload of the query task of node n.

[0132] As an option, the system can dynamically adjust the computing resource allocation of nodes according to the change trend of the query progress P n . For example, when the execution speed of a query task of a certain node is significantly slower than that of other nodes, its computing resource R n can be temporarily increased to enable its query task to be completed faster and reduce the overall query latency of the system. At the same time, if the query progress of some nodes is much faster than that of other nodes, the computing resource allocation of these nodes can be appropriately reduced to ensure system load balancing.

[0133] On the other hand, load information is an important factor affecting query performance, and the system needs to monitor the computing load situation of nodes in real time during the query execution process. Define the real-time computing load L of node nn As follows:

[0134]

[0135] Among them, L n represents the current computing load of node n; Q n represents the query task load assigned to node n; C n is the currently available computing resource of node n.

[0136] Specifically, during the query execution, if the computing load L n of a certain node exceeds the set threshold L max , the system can reduce its query task load Q n or increase its computing resource C n to reduce the computing latency and improve the query execution efficiency. Similarly, if the computing load of a certain node is too low, that is, L n <L min , the query task load can be appropriately increased to improve the utilization rate of computing resources and make the overall resource utilization of the system more balanced.

[0137] In a possible implementation, the system also monitors the network condition and adjusts the allocation of bandwidth resources according to the data transmission situation between nodes. Define the data transmission rate R ij between node i and node j as follows:

[0138]

[0139] Among them, R ij is the real-time data transmission rate between node i and node j; S ij represents the amount of data sent from node i to node j; T ij is the time interval of this data transmission.

[0140] As an optimization method, the system can dynamically adjust the data scheduling strategy of query tasks based on the changing trend of the network condition. For example, when the transmission rate R ij of a certain node is lower than the preset threshold R min , it may indicate that the network bandwidth is limited. At this time, cross-node data interaction can be reduced and the proportion of local computing tasks can be increased to reduce the communication latency. In addition, if the data transmission rate of a certain node is much higher than that of other nodes, its bandwidth allocation can be increased to ensure more efficient data transmission.

[0141] In some embodiments, the system adopts a feedback control mechanism to enable the resource allocation to be dynamically adjusted with the change of the query execution state. Define the adaptive control adjustment amount ΔR n as:

[0142]

[0143] Among them, ΔR n is the adjustment amount of the computing resources of node n; P opt is the expected query progress of the system; k p and k d are control gain parameters used to adjust the control strength; represents the change rate of the query progress and is used to reflect the dynamic trend of the query task execution.

[0144] Specifically, during the execution of the query task, the system will calculate ΔR n in real time and adjust the allocation of the computing resources R n so that the query task can be executed according to the optimal progress. If the query progress of a certain node lags behind the expected value, the system will increase its computing resources to speed up the query execution speed; if the query progress is too fast, its computing resources may be reduced to optimize the overall query scheduling and make the load more balanced.

[0145] In another possible implementation, the system can also combine the analysis of historical query data to predict the change trend of future query loads and adjust the resource allocation in advance. For example, use time series prediction methods (such as the ARIMA model) to analyze the past query execution data, predict the query loads in a future time period, and pre-allocate the computing resources and network bandwidth in advance to improve the execution efficiency of the query task.

[0146] Although the embodiments of the present invention have been shown and described, for those of ordinary skill in the art, it can be understood that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the present invention, and the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for optimizing distributed queries of a real-time database based on in-memory computing, characterized in that, The following steps are involved: Modeling query latency, wherein the query latency includes computational latency and communication latency; Based on the query latency modeling results, determine the query progress and resource allocation strategy for each node. Each node is a single computing unit in a distributed database, and the computing latency of a node is inversely proportional to its resource allocation and query load. The communication latency between nodes is determined by the data transmission volume and the inter-node bandwidth. Optimal control theory is used to optimize the resource allocation strategy of each node to minimize the total query latency. The resource allocation strategy of each node is dynamically adjusted based on the node's load status and communication requirements. Based on the game theory model, the resource competition and collaboration between each node are considered, and the overall query latency is minimized by adjusting the node resource allocation strategy. By monitoring the query progress, load information and network status of each node, an adaptive control mechanism is used to dynamically adjust the resource allocation strategy of each node to maintain optimal query performance.

2. The real-time database distributed query optimization method based on in-memory computing according to claim 1, wherein The query latency modeling includes: Representing the processing time of a node's computing task by constructing a computing model of the node, wherein the computing model relates the node's resource allocation to its computing load; Modeling inter-node communication delays, which depend on the load information of each node and the bandwidth between nodes; The sum of the query delays is minimized as an objective function, wherein the computation delay and the communication delay each have a certain weight coefficient to optimize the query performance.

3. The real-time database distributed query optimization method based on memory computing according to claim 1 is characterized in that: The optimal control theory includes: Optimizing resource allocation for each node, wherein the optimization step determines the optimal resource allocation strategy for each node through an optimal control algorithm to minimize query latency; Among them, the node resource allocation strategy is dynamically adjusted according to the query load, network bandwidth and node load status; The optimal control algorithm performs adaptive adjustments based on the real-time status of the nodes to ensure optimal execution of query tasks under different load conditions.

4. The real-time database distributed query optimization method based on in-memory computing according to claim 1, wherein The game theory model includes: Analyze the resource competition and collaboration relationship between nodes. The game theory model reflects the mutual influence between nodes by setting the competition coefficient; The game theory model minimizes global query latency by optimizing the resource allocation strategy of each node; Under this game model, nodes schedule resources based on global information to achieve the optimal balance of overall performance.

5. The real-time database distributed query optimization method based on memory computing according to claim 1 is characterized in that: The dynamic adjustment includes: Under high query load conditions, resources are allocated first to high-priority query tasks to ensure low-latency execution of important query tasks; Adjust node resource allocation based on real-time load information and resource requirements between nodes to ensure balanced load and efficient operation of the system; The node dynamically selects the resource allocation strategy based on the weight of communication delay and computation delay to optimize the overall query delay.

6. The method for optimizing distributed query of a real-time database based on in-memory computing according to claim 4, wherein Further applications of the game theory model include: When scheduling resources between nodes, a collaborative game model is used to coordinate resource allocation between nodes based on the dual goals of load balancing and delay minimization to avoid excessive competition between nodes and improve global query efficiency. Each node actively adjusts its resource usage strategy based on the global resource situation, and uses cooperative game strategies to alleviate the resource bottleneck of a single node, thereby enhancing the stability of the system and the query response speed; The game theory model balances the resource competition among nodes by calculating the revenue function of each node, further optimizing the query latency, and ensuring the efficient operation of the system in a multi-node environment.

7. The real-time database distributed query optimization method based on memory computing according to claim 3 is characterized in that: The optimization steps further include: According to historical query data, predict the query load and node status, and then adjust the resource allocation strategy of each node, enabling the system to proactively respond to potential high-load situations; For scenarios with large fluctuations in query load, adjust the allocation ratio of computing resources and bandwidth resources to better adapt to load changes and ensure that the query latency remains within the expected range; By sharing the load prediction and resource allocation situation among each node, further improve the efficiency of resource scheduling.

8. The real-time database distributed query optimization method based on in-memory computing according to claim 2, characterized in that The modeling of the query latency further includes: Through historical data analysis and real-time monitoring, combined with query load prediction and computing resource evaluation, adjust the resource allocation strategy of the node, enabling the resources to adapt dynamically under different query loads to optimize the query performance; When considering the load information of each node, dynamically adjust the weights of communication latency and computing latency, thereby more precisely controlling the latency minimization strategy to ensure fast response under high load; The update and feedback mechanism of the query latency model can calibrate the prediction results in real time, improving the accuracy of resource scheduling and the query execution efficiency.

9. The real-time database distributed query optimization method based on in-memory computing according to claim 5, characterized in that The adaptive adjustment includes: In the case of high fluctuations in query load, dynamically adjust the allocation ratio of computing resources and bandwidth resources to meet the low-latency requirements of query tasks; Through the analysis of historical data and the real-time feedback mechanism, adjust the resource allocation strategy of each node, enabling the resource configuration to anticipate potential high-load situations; The adjustment process uses a dynamic feedback algorithm to minimize the computing latency and optimize the resource configuration based on the real-time state of the system, keeping the query latency within a predetermined range.

10. The real-time database distributed query optimization method based on memory computing according to claim 9 is characterized in that: The resource allocation of the query further includes: According to historical query data and real-time load prediction information, pre-configure resources for each node to ensure that the query latency can still be maintained within an acceptable range under high load; In the case of large fluctuations in query load, dynamically adjust the allocation ratio of computing resources and bandwidth resources to optimize the resource utilization efficiency of the node; By sharing the load prediction and resource allocation information of each node, further improve the overall resource scheduling efficiency of the system to optimize the query performance.

Images