Query request processing method and device, electronic equipment, medium and program product

Abstract

This application discloses a method, apparatus, electronic device, medium, and program product for processing query requests, relating to the field of data query technology. The method includes: parsing a query statement from a received query request; determining the query type of the query request based on the query statement, where the query type includes one of the following: a first type for single-table queries and a second type for multi-table queries; obtaining node status evaluation values ​​of multiple storage nodes through a node status evaluation model; determining path nodes from the multiple storage nodes to execute the query request based on the query type and the node status evaluation values ​​of the multiple storage nodes, and constructing a query execution path based on the path nodes; and sequentially sending query instructions to all path nodes in the query execution path according to the execution order to control all path nodes to collaboratively execute the query instructions.

Description

Technical Field

[0001] This application relates to the field of data query technology, and in particular to a method, apparatus, electronic device, medium, and program product for processing query requests. Background Technology

[0002] Currently, distributed storage systems face numerous challenges when performing data queries. Traditional methods often require traversing all storage nodes in a query request. On the one hand, as the scale of data continues to expand, the complexity of query requests also increases, leading to longer query times. On the other hand, traditional query methods can only use fixed query rules, resulting in some nodes becoming overloaded while others remain idle when there are multiple query requests, thus affecting query efficiency.

[0003] In summary, the low efficiency of data retrieval is a technical problem stemming from the inefficient processing of query requests. Currently, no effective solution has been proposed to address this issue.

[0004] Therefore, it is necessary to improve the relevant technology to overcome the aforementioned defects. Summary of the Invention

[0005] This application provides a method, apparatus, electronic device, medium, and program product for processing query requests, to at least solve the technical problem of low data query efficiency caused by low query request processing efficiency in related technologies.

[0006] This application provides a query request processing method applied to a management node of a distributed storage system, wherein the distributed storage system further deploys multiple storage nodes, and a data transmission channel has been established between the multiple storage nodes. The method includes: parsing a query statement from a received query request; determining the query type of the query request based on the query statement, wherein the query type includes one of the following: a first type for single-table queries, and a second type for multi-table queries; obtaining node status evaluation values ​​of the multiple storage nodes through a node status evaluation model; determining path nodes for executing the query request from the multiple storage nodes based on the query type of the query request and the node status evaluation values ​​of the multiple storage nodes, and constructing a query execution path based on the path nodes; and sequentially sending query instructions to all path nodes in the query execution path according to the execution order to control all path nodes to collaboratively execute the query instructions.

[0007] This application also provides a query request processing apparatus, comprising: a parsing module, configured to parse a query statement from a received query request, and determine the query type of the query request based on the query statement, wherein the query type includes one of the following: a first type for performing a single-table query, and a second type for performing a multi-table query; an acquisition module, configured to acquire node status evaluation values ​​of the plurality of storage nodes through a node status evaluation model; a determination module, configured to determine path nodes for executing the query request from the plurality of storage nodes based on the query type of the query request and the node status evaluation values ​​of the plurality of storage nodes, and construct a query execution path based on the path nodes; and a sending module, configured to send query instructions sequentially to all path nodes in the query execution path according to the execution order, so as to control all path nodes to collaboratively execute the query instructions.

[0008] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of any of the above-described query request processing methods.

[0009] This application also provides a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, it implements the steps of any of the above-described query request processing methods.

[0010] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above-described query request processing methods.

[0011] This application describes a method where, when a management node receives a query request, it first parses the query statement and then determines the query type: a simple query targeting a single data table (Type 1) or a complex query involving multiple data tables (Type 2). Next, it intelligently evaluates the node status of storage nodes using a node status assessment model. This evaluation, combined with the query type, determines the query execution path, significantly improving query efficiency. For example, for single-table queries, the management node prioritizes nodes storing the target data block with high performance evaluation values, avoiding unnecessary data transfer and increasing query speed. For multi-table queries, a heuristic search algorithm is used to determine the query execution path while considering node status. This method addresses the problem of low data query efficiency caused by inefficient query request processing, improving query request processing efficiency, significantly shortening the response time of complex queries, enhancing real-time data query performance, and rationally allocating tasks to execution nodes along the query execution path. This avoids node overload, effectively balances system load, and improves resource utilization. Attached Figure Description

[0012] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a hardware structure block diagram of a server device for a query request processing method according to an embodiment of this application.

[0014] Figure 2 This is a flowchart of a query request processing method according to an embodiment of this application;

[0015] Figure 3 This is a schematic diagram of the data query process according to an embodiment of this application;

[0016] Figure 4 This is a schematic diagram of the node state evaluation calculation process according to an embodiment of this application;

[0017] Figure 5 This is a schematic diagram of a query request processing apparatus according to an embodiment of this application;

[0018] Figure 6 This is a computer system architecture block diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0019] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0020] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0021] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0022] The distributed storage system upon which the query request processing method relies is described here. The distributed storage system deploys a management node and multiple storage nodes, and data transmission channels have been established between these storage nodes.

[0023] The Management Node (MN) is the central hub of the distributed storage system, responsible for global coordination, monitoring, and query optimization. Specifically, it performs the following functions: monitoring the status of storage nodes, including hardware resources (CPU, memory, disk) and load; providing directory services for data blocks, i.e., mapping the logical view of data to its physical location; intelligently allocating query requests based on the real-time status of each node to avoid overloading certain nodes; parsing query requests, selecting the optimal query execution path, and designing efficient collaboration methods between nodes.

[0024] In a distributed storage system, storage nodes (SNs) are responsible for storing and retrieving data, with each node holding a portion of the data. Data replication or striping is used across multiple storage nodes to improve redundancy and availability. Specifically, the system performs the following functions: storing data blocks on local storage hardware (such as hard drives); responding to data read or write requests from the management node or other storage nodes; and automatically replicating data to other nodes according to policies, or migrating data in case of node failure. In this application, storage nodes are simply referred to as nodes.

[0025] The data transmission channel can be understood as a high-speed data exchange channel between storage nodes and between storage nodes and management nodes, and is the cornerstone of the efficient operation of a distributed storage system. Based on technologies such as high-speed Ethernet, it ensures low-latency and high-bandwidth data transmission. Furthermore, it uses network protocols such as TCP / IP, UDP, and RDMA for packet encapsulation and decapsulation to achieve secure and efficient data transmission. During data transmission, the optimal data transmission path and timing are dynamically selected based on the network topology and node status; for example, intelligent routing based on network traffic or data migration during periods of low node load.

[0026] The methods and embodiments provided in this application can be executed on a server device or similar computing device where a distributed storage system resides. Taking running on a server device as an example, Figure 1 This is a hardware structure block diagram of a server device for a query request processing method according to an embodiment of this application. For example... Figure 1 As shown, the server device may include one or more ( Figure 1Only one is shown in the image. A processor 102 (which may include, but is not limited to, a microprocessor MPU or a programmable logic device FPGA, etc.) and a memory 104 for storing data are also shown. The server device may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the server equipment described above. For example, the server equipment may also include components that are more... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0027] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the query request processing method in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thus implementing the above-described method. The memory 104 may include high-speed random access memory and non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to server devices via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0028] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by a communication provider for the server device. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0029] This application provides an embodiment of a query request processing method, which will be described in detail below with reference to the execution flow of the query request processing method. Figure 2 As shown, the specific steps are as follows:

[0030] Step S202: Parse the query statement from the received query request, and determine the query type of the query request based on the query statement. The query type includes one of the following: a first type for performing a single table query, and a second type for performing a multi-table query.

[0031] In this step, semantic parsing is performed on the query request to extract the query statement and obtain key information such as data tables, fields, query conditions, and join relationships. Then, based on the structure and complexity of the query statement, the query requests are categorized to facilitate the selection of subsequent query execution paths.

[0032] Step S204: Obtain the node status evaluation values ​​of the multiple storage nodes through the node status evaluation model;

[0033] It should be noted that a node status assessment model can also be pre-built. For example, real-time data can be collected on the storage node's performance parameters such as CPU (Central Processing Unit) utilization, memory utilization, network bandwidth utilization, disk I / O rate, and disk I / O bandwidth utilization, as well as hardware configuration information (i.e., hardware configuration parameters) such as CPU model, number of cores, memory capacity, memory bandwidth, disk type, disk rotation speed, and disk I / O throughput, and the data block storage location. Then, a node status assessment model can be built by weighted summation of these three types of data.

[0034] Performance metrics are used to assess the current load of a node. For example, when a node's CPU utilization consistently exceeds 80% and its memory utilization is close to 90%, it can be determined that the node is under high load. Hardware configuration information is used to assess the node's processing capabilities. For example, nodes with high-performance CPUs and large-capacity memory have a relatively stronger ability to handle complex computational tasks. The data block storage location represents the information about the data blocks stored on each node and the data distribution, i.e., the mapping relationship between data blocks and nodes, which can be used to quickly locate the node containing the data block to be queried.

[0035] Compared to existing methods that only consider a single or a few performance metrics for node status evaluation, the node status evaluation model in this step can comprehensively consider the performance metrics of storage nodes (such as CPU utilization and memory utilization), hardware configuration (such as CPU model and memory capacity), and data block storage location. Based on this, it calculates evaluation values ​​to reflect the health status, load capacity, and operating status of nodes in real time, thereby avoiding the allocation of query tasks to nodes with high loads and improving the overall processing efficiency of query requests.

[0036] Step S206: Determine the path node for executing the query request from the multiple storage nodes according to the query type of the query request and the node status evaluation value of the multiple storage nodes, and construct the query execution path based on the path node;

[0037] It should be noted that the query execution path constructed in this step can utilize nodes with low load, high processing power, and close data storage locations to execute the query task, while meeting the query requirements. For example, for simple queries, if the data blocks involved in the query conditions have replicas on a certain node, and the node's state evaluation value is low (i.e., low load and high processing power), then that node is directly selected to execute the query. For complex queries, a heuristic search algorithm is used to comprehensively consider node load, processing power, data storage location, and network connectivity to determine the set of nodes containing the relevant data blocks, and then nodes are selected step by step to construct the query execution path.

[0038] Step S208: Send query instructions to all path nodes in the query execution path in the order of execution to control all path nodes to cooperate in executing the query instructions.

[0039] In this step, for query execution paths involving multiple path nodes, these nodes collaborate in a pipelined manner to complete the query task. For example, in a multi-table join query, after the first path node completes its initial query processing of its local data block, it immediately transmits the result to the next path node. The next path node, upon receiving the data, can immediately begin its next processing operation without waiting for the previous path node to complete all operations. This minimizes the waiting time between nodes and improves the parallelism of query execution. For each path node's query process, it can first check if relevant data or results exist in the local cache. If they exist, the cached data is returned directly, avoiding duplicate queries and data transfers.

[0040] Through the above steps, when the management node receives a query request, it first parses the query statement and then determines the query type, i.e., whether the query is a simple query targeting a single data table (Type 1) or a complex query involving multiple data tables (Type 2). Then, it intelligently evaluates the node status assessment value of the storage nodes using a node status assessment model. Finally, it uses the node status assessment value and the query type to determine the query execution path, significantly improving query efficiency. For example, for single-table queries, the management node prioritizes nodes storing the target data block with high performance assessment values, avoiding unnecessary data transfer and improving query speed. For multi-table queries, while considering node status, a heuristic search algorithm is used to determine the query execution path. This method solves the problem of low data query efficiency caused by low query request processing efficiency, improves query request processing efficiency, significantly shortens the response time of complex queries, improves the real-time performance of data queries, and rationally allocates tasks to execution nodes along the query execution path, avoiding node overload, effectively balancing system load, and improving resource utilization.

[0041] In an exemplary embodiment, the process of determining the query type of the query request based on the query statement includes: splitting the query statement into multiple strings, wherein the multiple strings include at least one of the following: a first string, a second string, a third string, and a fourth string, wherein the first string represents the data table corresponding to the query request, the second string represents the query fields of the query request, the third string represents the query conditions corresponding to the query request, and the fourth string represents the connection relationship of the data tables corresponding to the query request; if a first condition is met, the query type is determined to be the first type, wherein the first condition includes: the number of data tables is less than a first preset value; the number of query fields is less than a second preset value; and the number of query conditions is less than a third preset value; or, if a second condition is met, the query type is determined to be the second type, wherein the second condition includes: the number of data tables is greater than a first preset value; the number of query fields is greater than a second preset value; the number of query conditions is greater than a third preset value; and the connection relationship of the data tables is a connection relationship of multiple data tables. Alternatively, the number of query fields may also be equal to the second preset value.

[0042] By carefully analyzing the query statements, this embodiment can accurately determine the type of query, providing a basis for subsequent node selection and path construction.

[0043] For example, a query statement "SELECT name FROM table1 WHERE id=123" will be broken down into strings representing the data table (table1), the query field (name), and the query condition (id=123). Since it only involves a single data table, it will be classified as the first type, namely a single-table query.

[0044] For the query statement "SELECT column1 FROM table1 JOIN table2 ON table1.id=table2.id WHERE table1.id=123", the join relationship between the data tables will be identified, classifying it as a second type, namely a multi-table query. By setting different preset values ​​(such as a first preset value of 2 for the number of data tables, a second preset value of 3 for the number of query fields, and a third preset value of 2 for the number of query conditions), the complexity of the query can be flexibly differentiated, thus adopting a more appropriate processing strategy.

[0045] For single-table queries, high-performance nodes are prioritized for execution; for multi-table queries, an optimal path is constructed to ensure efficient data transmission. By rationally classifying and allocating tasks, this method improves query efficiency and overall system performance.

[0046] In an exemplary embodiment, before parsing the query statement from the received query request, the received query request may be further validated to obtain a validation result, wherein the validation includes syntax checking and permission validation; if it is determined that the validation result indicates that the query request is grammatically correct and the target object sending the query request has query permission, then the query request is determined to have passed validation; if it is determined that either the validation result indicates that the query request is grammatically correct or the target object sending the query request has query permission is not true, then the query request is determined to have failed validation, and validation failure information is sent to the target object.

[0047] This embodiment enhances the security and data integrity of distributed storage systems by validating query requests. For example, when a user submits the query "SELECT name FROM users WHERE email='example@example.com'", the system first checks for syntax correctness, i.e., compliance with SQL syntax rules, and then verifies the user's permission to access the "users" table. Only when the query syntax is correct and the user's permission verification passes will the query request be further processed. This verification process prevents the impact of illegal or erroneous query commands, ensuring secure data access. By performing rigorous syntax and permission checks before query execution, this embodiment effectively prevents data leakage and system failures, enhancing system security.

[0048] In an exemplary embodiment, the technical solution for obtaining the node status evaluation values ​​of the plurality of storage nodes through a node status evaluation model may include: obtaining the performance index parameters, hardware configuration parameters, and data block storage locations of the storage nodes from the monitoring data of the storage nodes; inputting a first evaluation score corresponding to the performance index parameters, a second evaluation score corresponding to the hardware configuration parameters, and a third evaluation score corresponding to the data block storage locations into the node status evaluation model, and obtaining the node status evaluation value output by the node status evaluation model, wherein the node status evaluation value represents the weighted sum of the first evaluation score, the second evaluation score, and the third evaluation score. Calculating the node status evaluation value for node status evaluation is beneficial for optimizing query paths and improving query efficiency. For example, in a distributed storage system with multiple storage nodes, monitoring performance metrics such as CPU utilization, memory utilization, network bandwidth utilization, and disk bandwidth utilization of each node, along with hardware configuration information such as CPU model, memory capacity, and disk type, as well as the storage location of data blocks on the node, allows for the weighted summation of this information using a node status evaluation model. This results in an evaluation score that comprehensively reflects the node's performance and load, providing a scientific basis for selecting paths for query requests. Intelligent node status evaluation avoids assigning query tasks to overloaded or underperforming nodes, thereby improving query efficiency and resource utilization.

[0049] In an exemplary embodiment, the process of obtaining the performance index parameters, hardware configuration parameters, and data block storage location of the storage node from monitoring data of the storage node may include: obtaining the initialization configuration parameters of the storage node during the initialization phase and the operation configuration parameters of the storage node during the operation phase from the monitoring data, and determining the hardware configuration parameters based on the initialization configuration parameters and the operation configuration parameters; obtaining the data block number corresponding to the node number of the storage node from the monitoring data, and determining the data block storage location of the data block on the storage node if the data block indicated by the data block number is already stored on the storage node; obtaining the CPU utilization rate, memory utilization rate, network bandwidth utilization rate, and disk bandwidth utilization rate of the storage node from the monitoring data, and determining the performance index parameters based on the CPU utilization rate, memory utilization rate, network bandwidth utilization rate, and disk bandwidth utilization rate. This embodiment can monitor the hardware configuration information of the storage node during initialization, such as initialization configuration parameters like CPU model, memory capacity, and disk type, as well as dynamically changing performance indicators during operation, such as operating configuration parameters like CPU utilization, memory utilization, network bandwidth utilization, and disk bandwidth utilization, ensuring the accuracy and timeliness of node status assessment.

[0050] During node operation, if the hardware configuration changes (such as adding memory or replacing disks), the changed hardware information can be reported to the management node in a timely manner, and the management node will update the corresponding records in the database.

[0051] Specifically, a data block location table can be created in the database. This table records the storage location information of data blocks on storage nodes and can include data block number, storage node number, data block size, storage path, etc. When data blocks are migrated or replicas are updated, the management node updates the data block location information in a timely manner.

[0052] In an exemplary embodiment, the hardware configuration parameters include at least one of the following: CPU model, memory capacity, and disk type. Before inputting the first evaluation score corresponding to the performance indicator parameter, the second evaluation score corresponding to the hardware configuration parameter, and the third evaluation score corresponding to the data block storage location into the node state evaluation model, the performance score corresponding to the CPU model in its respective performance score range can be determined, the first capacity score corresponding to the memory capacity in its respective memory capacity score range can be determined, and the disk score corresponding to the disk type can be determined. A first weighted sum value is calculated among the performance score, the first capacity score, and the disk score. The first weighted sum value is normalized according to the preset evaluation score corresponding to the hardware configuration parameter to obtain the second evaluation score. In this embodiment, the processing of hardware configuration parameters is an indispensable part of node state evaluation. For example, for a storage node, based on hardware configuration information such as CPU model, memory capacity, and disk type, and then based on preset performance score ranges, memory capacity score ranges, and disk type scores, performance scores, capacity scores, and disk scores are obtained, and then a second evaluation score is obtained through weighted summation. In this embodiment, the normalization process ensures the comparability of different hardware configuration parameters, making the evaluation results more fair and accurate. This process quantifies hardware configuration information, providing a more comprehensive reference for path selection in query requests, and improving query efficiency and resource utilization.

[0053] Optionally, determining the performance score of the CPU model within its performance score range may include: obtaining a base score (i.e., the minimum value of the range) from the performance score range to which the CPU model belongs, and adding a score corresponding to the number of CPU cores to the base score, which is the performance score.

[0054] Optionally, determining the capacity score of the memory capacity within its corresponding capacity score range may include:

[0055] The base score (i.e. the minimum value of the range) is obtained from the memory capacity within its respective capacity score range. The score corresponding to the memory frequency is added to the base score, which is the first capacity score.

[0056] Optionally, determining the disk score corresponding to the disk type may include: determining the base score corresponding to the protocol type used by the disk, and adding a score corresponding to the disk rotation speed to the base score to obtain the disk score.

[0057] It is understood that the process of normalizing the first weighted sum to obtain the second evaluation score is also applicable to normalizing the third weighted sum to obtain the first evaluation score and normalizing the second weighted sum to obtain the third evaluation score, which will not be elaborated here.

[0058] It should be noted that the method of handling hardware configuration parameters in this embodiment is not only applicable to distributed storage systems, but also includes, but is not limited to, other distributed computing platforms, cloud computing resource scheduling, and other systems that require evaluation of node hardware capabilities, thereby optimizing system performance and resource allocation.

[0059] In an exemplary embodiment, before inputting the first evaluation score corresponding to the performance index parameter, the second evaluation score corresponding to the hardware configuration parameter, and the third evaluation score corresponding to the data block storage location into the node status evaluation model, it is further possible to determine the number of times the data block is accessed within the access cycle, determine the access score corresponding to the number of accesses in the relevant score range, determine the second capacity score corresponding to the block capacity in the relevant block capacity score range, determine the network transmission parameters between the storage node and the sending node that sends the query request, and determine the transmission score corresponding to the network transmission parameters; and determine the third evaluation score based on the second weighted sum of the access score, the second capacity score, and the transmission score. Through this embodiment, by calculating the weighted sum of the access score, the second capacity score, and the transmission score, the impact of the data block storage location on query efficiency can be comprehensively evaluated, solving the problem of quantifying data block location information in query path selection, ensuring high efficiency of data transmission and balanced load between nodes, and improving query efficiency and overall system performance.

[0060] This involves determining the number of times a data block is accessed within an access period, and then determining the access score corresponding to that number of accesses within its respective score range. For example, the number of accesses to each data block within a certain time period is counted, and different ranges are divided based on the number of accesses, with each range having a corresponding score. For instance, if a data block was accessed 100 times in the past 24 hours, and the score range for 100 accesses is [10, 20], then the score for 100 accesses would be 10.

[0061] Among these, network transmission parameters can be understood as the number of network hops or network latency between the storage node and the sending node, used to measure the distance between the storage node containing the data block to be queried and the sending node. The fewer the network hops or the lower the network latency, the higher the transmission score. For example, a network hop count of 1 or a network latency of less than 1ms results in a score of 90 to 100; a network hop count of 2 to 3 or a network latency of 1 to 5ms results in a transmission score of 70 to 89.

[0062] In an exemplary embodiment, the process of determining the performance indicator parameters based on the CPU utilization, memory utilization, network bandwidth utilization, and disk bandwidth utilization includes: calculating a third weighted sum among the CPU utilization, memory utilization, network bandwidth utilization, and disk bandwidth utilization in each computation cycle; and determining the third weighted sum as the first evaluation score. In this embodiment, by calculating a comprehensive performance evaluation score (i.e., the first evaluation score) through weighted summation based on real-time performance indicators such as CPU utilization, memory utilization, network bandwidth utilization, and disk bandwidth utilization of the storage node in each computation cycle, the first evaluation score can reflect the node's load and processing capacity in real time, providing a scientific basis for path selection for query requests. By dynamically adjusting the weighting coefficients, this solution can flexibly respond to performance requirements in different scenarios, efficiently execute query tasks, and improve query efficiency and resource utilization.

[0063] It should be noted that the performance indicator parameter determination technology is not only applicable to distributed storage systems, but also to other systems that need to evaluate the real-time performance of nodes, such as real-time data analysis platforms and network monitoring systems.

[0064] Based on the above embodiments, it can be seen that this application assigns different weights to performance index parameters, hardware configuration parameters, and data block storage locations based on the node status evaluation model, and then calculates the node status evaluation value by weighted average. In this process, firstly, the load is obtained by quantifying the performance index parameters, and secondly, the processing capacity is obtained by quantifying the hardware configuration parameters from the perspectives of CPU processing capacity, memory processing capacity, and disk processing capacity. Then, the node status evaluation value is calculated by combining the data block storage location with preset weights, thereby accurately reflecting the actual operating status of the node, providing a scientific basis for the selection of subsequent query execution paths, and improving query efficiency and resource utilization.

[0065] Furthermore, the weight allocation can be dynamically adjusted based on the actual application scenario and the degree of impact of each parameter on query performance. For example, in scenarios dominated by computationally intensive queries, the weight of CPU utilization can be appropriately increased; in scenarios with limited network bandwidth, the weight of network bandwidth utilization should be increased.

[0066] In an exemplary embodiment, the scheme for determining the path node for executing the query request from the plurality of storage nodes based on the query type of the query request and the node status evaluation values ​​of the plurality of storage nodes includes: for the first type, determining a first node from the plurality of storage nodes, wherein the first node stores a first data block or a copy of the first data block, and the first data block or the copy of the first data block satisfies the query statement; if the first node's corresponding first node status evaluation value is lower than a first threshold, the first node is determined as the path node. This embodiment proposes a method for determining the path node in a single-table query, ensuring that the query task is assigned to the most suitable node, avoiding unnecessary data transmission and node overload, and improving query efficiency and resource utilization. For example, for a query statement "SELECT name FROM table1 WHERE id=123", the system searches for nodes that store the table1 data block or a copy thereof, and evaluates the status of these nodes. If the node's status evaluation value is lower than a preset first threshold (e.g., 80 points), the node is considered to have good performance and can efficiently execute the query task, thus being determined as a path node.

[0067] The first threshold can be adjusted based on the actual state, or it can be set as the average of the node state evaluation values ​​of multiple storage nodes.

[0068] In an exemplary embodiment, the process of determining a path node for executing the query request from the plurality of storage nodes based on the query type of the query request and the node status evaluation values ​​of the plurality of storage nodes, and constructing a query execution path based on the path node, includes: for a second type, determining a second node from the plurality of storage nodes, and obtaining a node set based on the second node, wherein the second node stores a second data block or a copy of the second data block, and the second data block or the copy of the second data block satisfies the query statement; obtaining a third node from the node set whose node status evaluation value is lower than a second threshold; selecting a starting node from the plurality of third nodes corresponding to the first data table in the query statement, and sorting the remaining nodes using a heuristic search algorithm to obtain sorted nodes, wherein the remaining nodes are the nodes other than the starting node among the plurality of third nodes, the network bandwidth between every two adjacent nodes in the sorted nodes is lower than a bandwidth threshold, and the data transmission delay between every two adjacent nodes is lower than a delay threshold; determining the path node based on the starting node and the remaining nodes, and constructing a query execution path based on the starting node and the sorted nodes. This embodiment proposes a method for determining path nodes in multi-table queries. This process not only considers node performance and load but also optimizes data transmission paths, ensuring efficient execution of query tasks and improving query efficiency and resource utilization. For example, for a JOIN query involving table1 and table2, nodes storing these two tables or copies thereof are searched, forming a node set. Then, the status of these nodes is evaluated, and nodes with status evaluation values ​​below a second threshold (e.g., 70 points) are selected as candidate nodes. Using a heuristic search algorithm, a starting node is selected from the candidate nodes, and the remaining nodes are sorted to ensure that the network bandwidth and data transmission latency between any two adjacent nodes meet a preset threshold, thereby constructing the optimal query execution path.

[0069] Optionally, the first data table mentioned above may be determined based on the query order, such as the first data table to be queried, or it may be randomly selected from multiple data tables to be queried. This application does not impose any restrictions on the selection of data tables.

[0070] Compared to existing technologies that use fixed order or random allocation to determine query execution paths, the query execution path construction method proposed in the above embodiments of this application can dynamically adjust according to the type of query request and the status of storage nodes. When dealing with queries involving multi-table joins and complex condition filtering, considering the actual distribution of data blocks, it can make full use of nodes with strong processing capabilities, reduce waste of node resources and network transmission latency, and improve query performance.

[0071] Furthermore, by rationally selecting the node containing the data block or its replica based on the query request type and node status, the utilization efficiency of the replica is improved. For example, when querying hot data, priority can be given to nodes that store replicas of hot data blocks, nodes with lower load and closer proximity, i.e., nodes with more access frequency, larger data block capacity, fewer network hops, or lower network latency, thus improving query efficiency.

[0072] In an exemplary embodiment, the scheme of sending query instructions sequentially to all path nodes in the query execution path according to the execution order to control all path nodes to collaboratively execute the query instructions includes: sending the query instruction to the first path node; determining the execution result of the first path node executing the query instruction, wherein the first execution node is the starting node; determining the execution order according to the sorted order of the nodes; and sending the execution result and the query instruction to the next path node according to the execution order, so that the next path node continues to execute the query instruction according to the execution result. This embodiment proposes a pipelined method for executing query instructions, which improves query efficiency. In the constructed query execution path, the query instruction is first sent to the starting node. After the starting node completes execution and returns the execution result, the execution result and the query instruction are sent to the next node according to the sorted order of the path nodes, and so on, until all nodes complete the query task. This process is similar to a pipeline operation, which can fully utilize the processing capacity of each node, avoid waiting time between nodes, and improve query efficiency. By introducing pipelined processing, this embodiment can effectively handle the execution of complex queries, shorten response time, and improve resource utilization.

[0073] Optionally, in one embodiment, frequently queried data blocks and query results can be stored by setting up cache spaces on each node. During the query process, a node first checks whether relevant data or results exist in its local cache. If they exist, the cached data is used directly, avoiding duplicate queries and data transmission. For a frequently queried hot data block, copies of this hot data block can be stored in the caches of multiple nodes. When this hot data block is queried again, the node closest to the sending node that has cached a copy of this hot data block can provide the query results and query data, improving query response speed. This distributed caching mechanism proposed in this embodiment shortens the response time of complex queries, further improving query response speed and overall system performance. In practical applications, this method can significantly improve the real-time performance of data queries while effectively balancing system load and enhancing system stability and scalability.

[0074] Furthermore, based on Figure 3The data query process illustrated illustrates the data query process surrounding a distributed storage system, including system architecture setup, node status assessment, and query request processing. System architecture setup provides the foundational environment for data storage and querying; node status assessment is a crucial prerequisite for query optimization, providing a basis for query execution path selection; and query request processing concretely implements the optimization strategy into the query task execution. These three elements work together to ensure the effective implementation of the technical solution presented in this application and improve the data query performance of the distributed storage system.

[0075] I. Establishing the system architecture. This includes the following steps.

[0076] Step a: Storage Node Deployment. Deploy multiple storage nodes in the distributed storage system, equipping each node with corresponding hardware devices such as servers, storage disks, and network interface cards. The server configuration is selected based on actual needs. For example, for nodes with high processing power requirements, servers with high-performance CPUs and large-capacity memory can be selected. For nodes with large storage capacity requirements, more storage disks can be provided.

[0077] Step b: Network Connection Setup. Connect the various storage nodes through a high-speed network to form a distributed storage network. Network connection methods can include Ethernet, Fibre Channel, etc., ensuring sufficient network bandwidth between nodes to meet data transmission requirements. Simultaneously, to improve network reliability, a redundant network topology, such as dual-link redundant connections, is adopted to avoid system performance degradation due to the failure of a single network link.

[0078] Step c: Management Node Configuration. Configure one or more management nodes to manage and monitor the entire distributed storage system. Management nodes are responsible for collecting status information from each storage node, including node load, processing capacity, and data storage location, and executing query optimization strategies based on this information. Management nodes are also responsible for receiving user query requests, parsing and classifying them, and allocating the optimal execution path and node collaboration method for each query request based on the query execution path selection algorithm.

[0079] 2. Node status assessment. Details are as follows.

[0080] Step 1: Performance metrics collection.

[0081] Install performance monitoring software on each storage node to periodically collect performance metrics such as CPU utilization, memory utilization, network bandwidth utilization, and disk I / O bandwidth utilization.

[0082] The sampling frequency can be adjusted according to actual needs. It is generally recommended to set it to once every 10 seconds or 30 seconds to ensure that the real-time status of the node can be reflected in a timely manner.

[0083] For example, CPU usage and memory usage can be obtained through the system's built-in performance monitoring tools (such as the top command and vmstat command in Linux systems);

[0084] Obtain network bandwidth utilization through network monitoring tools;

[0085] Obtain disk I / O speed using disk monitoring tools.

[0086] Step 2: Obtain hardware information.

[0087] During the node initialization phase, the node's hardware configuration information is obtained, including CPU model, number of cores, memory capacity, disk type and speed, etc., and this information is stored in the database of the management node.

[0088] During node operation, if the hardware configuration changes (such as adding memory, replacing disks, etc.), the node promptly reports the changed hardware information to the management node, and the management node updates the corresponding records in the database.

[0089] Hardware information will serve as the basis for node status assessment. For example, nodes with high-performance CPUs and large memory capacity will be assigned higher baseline values ​​when calculating the weights of processing capability-related indicators, thus affecting the final status assessment value.

[0090] Step 3: Record the location of the data block.

[0091] When a data block is stored on a node, the management node records the mapping relationship between the data block and the node and stores the relevant information in the database. Simultaneously, when a data block is migrated or its replica is updated, the management node promptly updates the data block's location information. For example, a data block location table is created in the database, containing fields such as data block ID, storage node ID, data block size, and storage path, to record the storage location information of the data blocks.

[0092] Data block location information affects the data access cost factor in node status evaluation. If a node stores a large number of frequently accessed data blocks, the weight related to data storage location will be appropriately increased when calculating the status evaluation value, thereby increasing the priority of the node in query path selection.

[0093] Step 4: Node status assessment calculation.

[0094] like Figure 4As shown, the node status evaluation calculation process first involves data acquisition, specifically including collecting real-time performance indicators such as CPU utilization and memory utilization, obtaining hardware configuration information, and recording data block location information. Next, score calculation is performed, specifically including calculating the real-time performance indicator score S1, the comprehensive hardware configuration score S2, and the data block location score S3 according to rules. Finally, the evaluation value is calculated by weighting and summing the real-time performance indicator score (corresponding to the first evaluation score), the comprehensive hardware configuration score (corresponding to the second evaluation score), and the comprehensive data block location score (corresponding to the third evaluation score) according to different weights to obtain the node status evaluation value.

[0095] In the process of calculating the evaluation value, it is assumed that the total weight of the real-time performance index is W1, the weight of the comprehensive score of hardware configuration is W2, and the weight of the comprehensive score of data block location is W3, and that W1+W2+W3=1.

[0096] The node status evaluation value S is calculated using the formula S = S1 × W1 + S2 × W2 + S3 × W3 = 1. Where S1 is the real-time performance score, S2 is the overall hardware configuration score, and S3 is the overall data block location score.

[0097] Depending on the specific application scenario, such as in big data analytics where computational power is extremely demanding, W1=0.4, W2=0.4, and W3=0.2 can be set. In scenarios with frequent data access and sensitivity to data transmission latency, W1=0.4, W2=0.2, and W3=0.4 can be adjusted.

[0098] It should be noted that the score calculation process will be explained in conjunction with the following steps.

[0099] To determine the overall hardware configuration score, you can perform the following steps.

[0100] Step 4.1: Determine the scoring criteria for each hardware parameter.

[0101] (1) CPU related: Based on the performance ranking of CPU models in the market, they are divided into different performance ranges (i.e., the performance score range of the CPU model) and assigned corresponding scores.

[0102] For example, high-performance CPUs like the Intel Core i9-13900K can be scored between 85 and 100 points; mid-to-high-end CPUs like the Intel Core i7-13700K can be scored between 70 and 84 points; and low-to-mid-range CPUs like the Intel Core i3-13100 can be scored between 50 and 69 points. The number of CPU cores also affects the score; each additional core adds a certain number of points to the base score (corresponding to the base value), such as 2 points per core. Assuming a CPU model at a certain node has a base score of 85 and 16 cores, then the CPU score = 85 + 16 × 2 = 117 points.

[0103] (2) Memory-related: different capacity ranges are divided according to the size of the memory capacity (i.e., the memory capacity score range to which the memory capacity belongs) for scoring.

[0104] For example, memory capacity of 128GB and above scores 90-100 points; 64GB-127GB scores 75-89 points; and 32GB-63GB scores 60-74 points. Furthermore, considering factors such as memory frequency, higher-frequency memory can receive additional points on top of the base capacity score; for example, 1 point is added for every 100MHz increase in memory frequency. If a node has 128GB of memory, corresponding to a base score of 90 points, and a memory frequency of 3600MHz, which is 400MHz higher than the base frequency of 3200MHz, then the memory score = 90 + 4 = 94 points.

[0105] (3) Regarding disks, solid-state drives (SSDs) are assigned different scores based on different protocol standards (e.g., NVMe SSDs outperform SATA SSDs) and read / write speed metrics. Read / write speed metrics include, for example, disk rotation speed.

[0106] For example, high-performance NVMe SSDs can score 80-100 points, SATA SSDs 60-79 points, and hard drives 40-59 points. Disk rotation speed is an important scoring criterion for HDDs; the higher the rotation speed, the higher the score. For example, a 7200 RPM HDD adds 10 points to the base type score, and a 5400 RPM HDD adds 5 points. If a node uses a high-performance NVMe SSD with a base type score of 90, then the disk portion score will also be 90 points.

[0107] Step 4.2: Calculate the overall score of hardware configuration.

[0108] The scores of various hardware parameters such as CPU, memory, and disk are weighted and summed according to certain weights to obtain the final score of hardware configuration.

[0109] Assuming the CPU score has a weight of 0.4, the memory score has a weight of 0.3, and the disk score has a weight of 0.3, then the hardware configuration score = CPU score × 0.4 + Memory score × 0.3 + Disk score × 0.3. Taking the previously calculated result as an example, the hardware configuration score = 117 × 0.4 + 94 × 0.3 + 90 × 0.3 = 46.8 + 28.2 + 27 = 102 points.

[0110] However, considering the actual scoring range, the calculation results can be normalized to fall within the 0-100 score range. For example, using the formula: Final hardware configuration score = 100 × (102 / theoretical maximum hardware configuration score), assuming the theoretical maximum hardware configuration score (corresponding to the above preset evaluation score) is 130, then the final hardware configuration score = 100 × (102 / 130) ≈ 78.46 points.

[0111] The following steps can be performed to determine the overall score based on the location of data blocks.

[0112] Step 4.3: Determine the scoring criteria for data block location. The data block location score is mainly based on factors such as the access frequency of the data block, the size of the data block, and the distance between the data block and the query initiation point.

[0113] The data block size can also be understood as the block capacity of the data block.

[0114] Access frequency: This counts the number of times each data block is accessed within a certain time period (e.g., the past 24 hours), and scores are assigned based on the number of accesses. For example, 1000 or more accesses receive a score of 80 to 100; 500 to 999 accesses receive a score of 60 to 79; and 100 to 499 accesses receive a score of 40 to 59, etc.

[0115] Similarly, data block size is scored according to different ranges. For example, data block sizes of 1GB and above are scored 70 to 100 points; 500MB to 999MB are scored 50 to 69 points; 100MB to 499MB are scored 30 to 49 points, and so on.

[0116] Finally, the distance to the query initiator is measured based on the number of network hops or network latency between the node and the query initiator. The fewer the network hops or the lower the network latency, the higher the score. For example, a network hop count of 1 or a network latency of less than 1ms results in a score of 90 to 100; a network hop count of 2 to 3 or a network latency of 1 to 5ms results in a score of 70 to 89, and so on.

[0117] Step 4.4: Calculate the overall score for the location of the data block.

[0118] The scores of various influencing factors, such as access frequency, data block size, and distance from the query initiator, are weighted and summed according to certain weights to obtain the final score of data block location.

[0119] Assuming the access frequency score has a weight of 0.4, the data block size score has a weight of 0.3, and the distance from the query initiator score has a weight of 0.3, then the data block location score = access frequency score × 0.4 + data block size score × 0.3 + distance from the query initiator score × 0.3. Taking the previously calculated result as an example, the data block location score = 90 × 0.4 + 85 × 0.3 + 95 × 0.3 = 36 + 25.5 + 28.5 = 90 points.

[0120] To determine the real-time performance metric score, the following steps can be performed.

[0121] Step 4.5: Real-time performance metrics score for computing nodes.

[0122] The management node calculates the real-time performance score of the node by weighting and summing these real-time performance indicators every minute or every 5 minutes based on the performance indicators collected in real time.

[0123] For example, assuming that the weight of CPU utilization is 0.3, the weight of memory utilization is 0.3, the weight of network bandwidth utilization is 0.2, and the weight of disk I / O rate is 0.1 in the node state evaluation model, for node A, its CPU utilization is 50%, memory utilization is 60%, network bandwidth utilization is 30%, and disk I / O bandwidth utilization is 40%, then the real-time performance index score of node A = 0.4×50 + 0.3×60 + 0.2×30 + 0.1×40 = 48.

[0124] Compared with traditional node evaluation methods that only consider a single or a few indicators, this application comprehensively considers real-time performance indicators such as node CPU utilization, memory utilization, network bandwidth utilization, and disk I / O rate, as well as node hardware configuration information and data storage location information. The node status evaluation value calculated by weighted summation of these information can accurately reflect the actual operating status of the node. Thus, by comprehensively evaluating the node status based on the node's load capacity, processing capacity, and data storage location, it avoids blindly selecting nodes and causing low query efficiency, and selects the best execution path for subsequent query requests.

[0125] III. Query Request Processing.

[0126] Step 5: Query request received.

[0127] Users send query requests to the distributed block storage system through a client. The query request first reaches the management node. The management node performs preliminary verification on the received query request, including syntax checking and permission verification.

[0128] If the query request has a syntax error or the user does not have the corresponding query permission, the management node returns an error message to the client; if the query request is verified, it proceeds to the next processing step.

[0129] Step 6: Query request parsing and classification.

[0130] The management node uses a query parser to perform syntactic and semantic parsing on query requests, extract key information from the query, and classify query requests according to query complexity and resource requirements.

[0131] For example, for an SQL query request, the query parser can use techniques such as lexical analysis and syntax analysis to break down the query statement into its components, extracting information such as the data tables, fields, query conditions, and join relationships involved in the query. For instance, for an SQL query statement "SELECT column1, column2 FROM table1 WHERE condition1 AND condition2 JOIN table2 ON table1.id=table2.id", parsing can clearly identify that the tables involved in the query are table1 and table2, the query conditions are condition1 and condition2, and the join relationship is an inner join based on the id field.

[0132] Then, based on predefined query classification rules, it is determined whether the query is a simple query or a complex query.

[0133] For example, a simple query can be defined as a query involving only a few fields of a single table and with simple query conditions, such as "SELECT column1 FROM table1 WHERE column2=value";

[0134] Complex queries include multi-table joins, complex condition filtering queries, and aggregation queries, such as “SELECT COUNT (*), SUM (column1) FROM table1 JOIN table2 ON table1.id=table2.id WHEREcondition1 OR condition2 GROUP BY column3”.

[0135] Step 7: Select the execution path.

[0136] The management node selects the best execution path for the query request based on the classification results of the query request and the status evaluation value of each node using a query execution path selection algorithm.

[0137] For simple queries, the appropriate node is selected to execute the query directly based on the location information of the data block and the node status evaluation value. For example, if the query statement is "SELECT column1 FROM table1 WHERE column2=value", and the data block replicas in table1 that meet the condition are stored on node A, and node A has a CPU utilization of 30%, a memory utilization of 40%, a network bandwidth utilization of 20%, and a disk I / O bandwidth utilization of 10%, and the node status evaluation value calculated by the node status evaluation model is low, then node A is selected first to execute the query.

[0138] For complex queries, a heuristic search algorithm is used to construct the optimal query execution path based on factors such as node load, processing power, data storage location, and network connectivity.

[0139] First, based on the data tables and fields involved in the query, determine the set of nodes that may contain relevant data blocks.

[0140] Then, from this set of nodes, nodes are selected step by step according to certain rules (such as prioritizing nodes with lower state evaluation values), and a query execution path is constructed. During the path construction process, network connectivity and data transmission latency between nodes are considered to minimize data transmission across nodes.

[0141] For example, in a join query involving three tables, suppose the data blocks of the first table are distributed across nodes A, B, and C; the data blocks of the second table are distributed across nodes B, D, and E; and the data blocks of the third table are distributed across nodes C, E, and F. Using a node state evaluation model, it is found that the state evaluation values ​​of nodes A, B, and E are low, and the network bandwidth between nodes A and B, and between nodes B and E, is high, resulting in low data transmission latency.

[0142] A query execution path can be constructed starting from node A and passing through nodes B and E in sequence. On this path, the relevant data block of the first table is first obtained on node A, then transmitted to node B via the network, and joined with the relevant data block of the second table on node B. Finally, the result is transmitted to node E, and joined and queried with the relevant data block of the third table on node E.

[0143] Based on the above steps, a query execution path selection algorithm based on node state evaluation values ​​and query request types is proposed. This algorithm can select the optimal query execution path using different strategies for different types of query requests (simple queries and complex queries). For example, for simple single-table queries, a suitable node is selected directly based on the data block location and node state. For complex multi-table queries, a heuristic search algorithm is used to construct the path by comprehensively considering node load, processing power, data storage location, and network connectivity. Compared with the traditional fixed or random path selection method, this effectively reduces cross-node data transmission and improves the efficiency of complex queries.

[0144] Step 8: Node collaboration and query execution.

[0145] After determining the query execution path, the management node sends query task instructions to the relevant storage nodes. Each storage node then collaborates to complete the query task according to the instructions.

[0146] During query execution, a pipelined processing approach is adopted to improve query efficiency. For example, after the first storage node completes the initial processing of its local data block, it transmits the result to the next node via the network. The next node then begins the next processing operation upon receiving the data. Simultaneously, a distributed caching mechanism is introduced, setting up cache space on each node to store frequently queried data blocks and query results. When a query request arrives, each node, during the query process, first checks whether relevant data or results exist in its local cache. If they exist, the cached data is used directly, avoiding duplicate queries and data transfers. For example, for a frequently queried hot data block, copies of the data block are stored in the caches of multiple nodes. When a query request involves this data block, the node closest to the query initiator and with the data block in its cache can directly return the data, significantly improving query response speed.

[0147] This step innovatively introduces a pipelined processing approach and a distributed caching mechanism. Pipeline processing breaks the limitations of traditional sequential processing, enabling parallel processing and transmission of data between nodes. Distributed caching dynamically stores hot data and query results, reducing redundant queries and data transmission. These two strategies work together to significantly improve node collaboration efficiency and query performance.

[0148] Step 9: Return Query Results. After all storage nodes have completed the collaborative query task, they return the final query results to the management node. The management node organizes and encapsulates the query results, and then returns them to the client. Upon receiving the query results, the client performs appropriate display or further processing.

[0149] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0150] Embodiments of this application also provide a query request processing apparatus, such as... Figure 5 As shown, it specifically includes:

[0151] The parsing module 52 is used to parse the query statement from the received query request and determine the query type of the query request based on the query statement. The query type includes one of the following: a first type for performing a single table query and a second type for performing a multi-table query.

[0152] The acquisition module 54 is used to acquire the node status evaluation values ​​of the plurality of storage nodes through the node status evaluation model;

[0153] The determining module 56 is used to determine the path node for executing the query request from the multiple storage nodes according to the query type of the query request and the node status evaluation value of the multiple storage nodes, and to construct a query execution path based on the path node;

[0154] The sending module 58 is used to send query instructions to all path nodes in the query execution path in the order of execution, so as to control all path nodes to cooperate in executing the query instructions.

[0155] Through the modules described above, when the management node receives a query request, it first parses the query statement and then determines the query type: whether it's a simple query targeting a single data table (Type 1) or a complex query involving multiple data tables (Type 2). Then, it intelligently evaluates the node status of the storage nodes using a node status evaluation model. This evaluation, combined with the query type, determines the query execution path, significantly improving query efficiency. For example, for single-table queries, the management node prioritizes nodes storing the target data block with high performance evaluation values, avoiding unnecessary data transfer and improving query speed. For multi-table queries, considering node status while using a heuristic search algorithm to determine the execution path, this method solves the problem of low data query efficiency caused by inefficient query request processing. It improves query request processing efficiency, significantly shortens the response time of complex queries, enhances the real-time performance of data queries, and rationally allocates tasks to execution nodes along the query execution path, avoiding node overload, effectively balancing system load, and improving resource utilization.

[0156] In an exemplary embodiment, the parsing module is further configured to: split the query statement into multiple strings, wherein the multiple strings include at least one of the following: a first string, a second string, a third string, and a fourth string, wherein the first string represents the data table corresponding to the query request, the second string represents the query field of the query request, the third string represents the query condition corresponding to the query request, and the fourth string represents the connection relationship of the data tables corresponding to the query request; and, if a first condition is met, determine the query type as the first type, wherein the first condition includes: the number of data tables is less than a first preset value; the number of query fields is less than a second preset value; and the number of query conditions is less than a third preset value; or, if a second condition is met, determine the query type as the second type, wherein the second condition includes: the number of data tables is greater than a first preset value; the number of query fields is greater than a second preset value; the number of query conditions is greater than a third preset value; and the connection relationship of the data tables is a connection relationship of multiple data tables.

[0157] In an exemplary embodiment, the parsing module is further configured to: verify the received query request before parsing the query statement from the received query request, and obtain a verification result, wherein the verification includes syntax checking and permission verification; if it is determined that the verification result indicates that the query request is grammatically correct and the target object sending the query request has query permission, then the query request is determined to pass the verification; if it is determined that either the verification result indicates that the query request is grammatically correct or the target object sending the query request has query permission is not true, then the query request is determined to fail the verification, and a verification failure message is sent to the target object.

[0158] In an exemplary embodiment, the acquisition module is further configured to: acquire the performance index parameters of the storage node, the hardware configuration parameters of the storage node, and the data block storage location of the storage node from the monitoring data of the storage node; input the first evaluation score corresponding to the performance index parameters, the second evaluation score corresponding to the hardware configuration parameters, and the third evaluation score corresponding to the data block storage location into the node status evaluation model, and obtain the node status evaluation value output by the node status evaluation model, wherein the node status evaluation value represents the weighted sum of the first evaluation score, the second evaluation score, and the third evaluation score.

[0159] In an exemplary embodiment, the acquisition module is further configured to: acquire initialization configuration parameters of the storage node during the initialization phase and operation configuration parameters of the storage node during the operation phase from the monitoring data, and determine the hardware configuration parameters based on the initialization configuration parameters and the operation configuration parameters; acquire a data block number corresponding to the node number of the storage node from the monitoring data, and determine the data block storage location of the data block on the storage node if the data block indicated by the data block number has been stored on the storage node; acquire the CPU utilization rate, memory utilization rate, network bandwidth utilization rate, and disk bandwidth utilization rate of the storage node from the monitoring data, and determine the performance index parameters based on the CPU utilization rate, memory utilization rate, network bandwidth utilization rate, and disk bandwidth utilization rate.

[0160] In an exemplary embodiment, the hardware configuration parameters include at least one of the following: CPU model, memory capacity, and disk type. The acquisition module is further configured to: determine the performance score corresponding to the CPU model in its respective performance score range, determine the first capacity score corresponding to the memory capacity in its respective memory capacity score range, and determine the disk score corresponding to the disk type; calculate a first weighted sum among the performance score, the first capacity score, and the disk score; and normalize the first weighted sum according to a preset evaluation score corresponding to the hardware configuration parameters to obtain a second evaluation score.

[0161] In an exemplary embodiment, the acquisition module is further configured to: determine the number of times the data block is accessed within an access period, and determine the access score corresponding to the number of accesses in the corresponding access score range; determine the second capacity score corresponding to the block capacity of the data block in the corresponding block capacity score range; determine the network transmission parameters between the storage node and the sending node that sends the query request; and determine the transmission score corresponding to the network transmission parameters; and determine the third evaluation score based on a second weighted sum of the access score, the second capacity score, and the transmission score.

[0162] In an exemplary embodiment, the acquisition module is further configured to: calculate a third weighted sum among the CPU utilization, the memory utilization, the network bandwidth utilization, and the disk bandwidth utilization in each computing cycle; and determine the third weighted sum as the first evaluation score.

[0163] In an exemplary embodiment, the determining module is further configured to: for the first type, determine a first node from the plurality of storage nodes, wherein the first node stores a first data block or a copy of the first data block, and the first data block or the copy of the first data block satisfies the query statement; and determine the first node as the path node if the first node status evaluation value corresponding to the first node is lower than a first threshold.

[0164] In an exemplary embodiment, the acquisition module is further configured to: for the second type, determine a second node from the plurality of storage nodes, and obtain a node set based on the second node, wherein the second node stores a second data block or a copy of the second data block, and the second data block or the copy of the second data block satisfies the query statement; acquire a third node from the node set whose node status evaluation value is lower than a second threshold; select a starting node from the plurality of third nodes corresponding to the first data table in the query statement, and sort the remaining nodes using a heuristic search algorithm to obtain sorted nodes, wherein the remaining nodes are nodes other than the starting node among the plurality of third nodes, the network bandwidth between each two adjacent nodes in the sorted nodes is lower than a bandwidth threshold, and the data transmission delay between each two adjacent nodes is lower than a delay threshold; determine the path node based on the starting node and the remaining nodes, and construct a query execution path based on the starting node and the sorted nodes.

[0165] In an exemplary embodiment, the acquisition module is further configured to: send the query instruction to the first path node, determine the execution result of the query instruction executed by the first path node, wherein the first execution node is the starting node; determine the execution order according to the sorted order of the nodes; and send the execution result and the query instruction to the next path node according to the execution order, so that the next path node continues to execute the query instruction according to the execution result.

[0166] For a description of the features in the embodiment corresponding to the query request processing device, please refer to the relevant description in the embodiment corresponding to the query request processing method, which will not be repeated here.

[0167] Embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the steps in any of the above-described query request processing method embodiments.

[0168] Optionally, Figure 6 This is a computer system architecture block diagram of an electronic device according to an embodiment of this application. For example... Figure 6As shown, the computer system 600 includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 602 or programs loaded from storage section 608 into random access memory (RAM). The random access memory 603 also stores various programs and data required for system operation. The CPU 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output interface 605 (I / O interface) is also connected to the bus 604.

[0169] The following components are connected to the input / output interface 605: an input section 606 including a keyboard, mouse, etc.; an output section 607 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a local area network card, modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the input / output interface 605 as needed. A removable medium 611, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 610 as needed so that computer programs read from it can be installed into the storage section 608 as needed.

[0170] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above-described query request processing method embodiments when running.

[0171] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0172] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described query request processing method embodiments.

[0173] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described query request processing method embodiments.

[0174] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0175] The foregoing has provided a detailed description of a query request processing method, apparatus, electronic device, medium, and program product provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and core ideas of this application. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A method for processing query requests, characterized in that, A management node applied to a distributed storage system, wherein the distributed storage system further comprises multiple storage nodes, and data transmission channels have been established between the multiple storage nodes, including: The query statement is parsed from the received query request, and the query type of the query request is determined based on the query statement. The query type includes one of the following: a first type for performing a single table query, and a second type for performing a multi-table query. The node status evaluation values ​​of the multiple storage nodes are obtained through a node status evaluation model; Based on the query type of the query request and the node status evaluation value of the plurality of storage nodes, a path node for executing the query request is determined from the plurality of storage nodes, and a query execution path is constructed based on the path node; Query instructions are sent sequentially to all path nodes in the query execution path according to the execution order, so as to control all path nodes to cooperate in executing the query instructions in a pipelined processing manner; Each storage node has a cache space set up, and the query result corresponding to the query instruction is obtained first from the cache space corresponding to each path node; In the case of querying hot data, the node with the lowest load and the closest distance to the cache space is selected from the multiple storage nodes as the path node. The node status evaluation model is constructed based on performance index parameters, hardware configuration parameters, and data block storage location. The node status evaluation values ​​of the multiple storage nodes are obtained through a node status evaluation model, including: The performance metrics, hardware configuration parameters, and data block storage locations of the storage nodes are obtained from the monitoring data of the storage nodes. The first evaluation score corresponding to the performance index parameter, the second evaluation score corresponding to the hardware configuration parameter, and the third evaluation score corresponding to the data block storage location are input into the node status evaluation model to obtain the node status evaluation value output by the node status evaluation model, wherein the node status evaluation value represents the weighted sum of the first evaluation score, the second evaluation score, and the third evaluation score; The hardware configuration parameters include at least one of the following: CPU model, memory capacity, and disk type; the method further includes: Add a score corresponding to the number of CPU cores to the base score of the performance score range to obtain the performance score of the CPU model in the performance score range. Add a score corresponding to the memory frequency to the base score of the capacity score range to which the memory capacity belongs, to obtain the first capacity score corresponding to the memory capacity in the memory capacity score range; Add a score corresponding to the disk rotation speed to the base score corresponding to the protocol type used by the disk to obtain the disk score corresponding to the disk type. Calculate the first weighted sum of the performance score, the first capacity score, and the disk score; The first weighted sum is normalized based on the preset evaluation score corresponding to the hardware configuration parameters to obtain the second evaluation score.

2. The query request processing method according to claim 1, characterized in that, The query type of the query request is determined based on the query statement, including: The query statement is split into multiple strings, wherein the multiple strings include at least one of the following: a first string, a second string, a third string, and a fourth string, wherein the first string is used to represent the data table corresponding to the query request, the second string is used to represent the query field of the query request, the third string is used to represent the query condition corresponding to the query request, and the fourth string is used to represent the connection relationship of the data table corresponding to the query request; If the first condition is met, the query type is determined to be the first type, wherein the first condition includes: the number of data tables is less than a first preset value; the number of query fields is less than a second preset value; and the number of query conditions is less than a third preset value. Alternatively, if the second condition is met, the query type is determined to be the second type, wherein the second condition includes: the number of data tables is greater than a first preset value; the number of query fields is greater than a second preset value; the number of query conditions is greater than a third preset value; and the data table connection relationship is a connection relationship of multiple data tables.

3. The query request processing method according to claim 1, characterized in that, Before parsing the query statement from the received query request, the method further includes: The received query request is verified to obtain a verification result, wherein the verification includes syntax checking and permission verification; If the verification result indicates that the query request is syntactically correct and the target object sending the query request has query permissions, then the query request is determined to have passed verification. If it is determined that the verification result indicates that the query request is grammatically correct, and the target object sending the query request has query permissions, then the query request is determined to have failed verification, and a verification failure message is sent to the target object.

4. The query request processing method according to claim 1, characterized in that, The performance metrics, hardware configuration parameters, and data block storage locations of the storage nodes are obtained from the monitoring data of the storage nodes, including: The initialization configuration parameters of the storage node during the initialization phase and the operation configuration parameters of the storage node during the operation phase are obtained from the monitoring data, and the hardware configuration parameters are determined based on the initialization configuration parameters and the operation configuration parameters. Obtain the data block number corresponding to the node number of the storage node from the monitoring data; if the data block indicated by the data block number has been stored in the storage node, determine the data block storage location of the data block on the storage node. The CPU utilization, memory utilization, network bandwidth utilization, and disk bandwidth utilization of the storage node are obtained from the monitoring data, and the performance index parameters are determined based on the CPU utilization, memory utilization, network bandwidth utilization, and disk bandwidth utilization.

5. The query request processing method according to claim 4, characterized in that, Before inputting the first evaluation score corresponding to the performance index parameter, the second evaluation score corresponding to the hardware configuration parameter, and the third evaluation score corresponding to the data block storage location into the node state evaluation model, the method further includes: Determine the number of times the data block is accessed within the access period, and determine the access score corresponding to the number of accesses in the relevant score range; Determine the second capacity score corresponding to the block capacity of the data block within its respective block capacity score range; Determine the network transmission parameters between the storage node and the sending node that sent the query request; and determine the transmission score corresponding to the network transmission parameters; The third evaluation score is determined based on a second weighted sum of the access score, the second capacity score, and the transmission score.

6. The query request processing method according to claim 4, characterized in that, The performance metric parameters are determined based on the CPU utilization, memory utilization, network bandwidth utilization, and disk bandwidth utilization, including: Within each computation cycle, a third weighted sum is calculated among the CPU utilization, memory utilization, network bandwidth utilization, and disk bandwidth utilization. The third weighted sum is determined as the first evaluation score.

7. The query request processing method according to claim 1, characterized in that, Based on the query type of the query request and the node status evaluation values ​​of the plurality of storage nodes, a path node for executing the query request is determined from the plurality of storage nodes, including: For the first type, a first node is determined from the plurality of storage nodes, wherein the first node stores a first data block or a copy of the first data block, and the first data block or the copy of the first data block satisfies the query statement; If the state evaluation value of the first node corresponding to the first node is lower than the first threshold, the first node is determined as the path node.

8. The query request processing method according to claim 1, characterized in that, Based on the query type of the query request and the node status evaluation values ​​of the multiple storage nodes, a path node for executing the query request is determined from the multiple storage nodes, and a query execution path is constructed based on the path node, including: For the second type, a second node is determined from the plurality of storage nodes, and a node set is obtained based on the second node, wherein the second node stores a second data block or a copy of the second data block, and the second data block or the copy of the second data block satisfies the query statement; Obtain the third node whose node state evaluation value is lower than the second threshold from the node set; Select the starting node corresponding to the first data table in the query statement from multiple third nodes, and sort the remaining nodes using a heuristic search algorithm to obtain sorted nodes. The remaining nodes are the nodes other than the starting node among the multiple third nodes. The network bandwidth between any two adjacent nodes in the sorted nodes is lower than the bandwidth threshold, and the data transmission delay between any two adjacent nodes is lower than the delay threshold. The path nodes are determined based on the starting node and the remaining nodes, and a query execution path is constructed based on the starting node and the sorted nodes.

9. The query request processing method according to claim 8, characterized in that, Query instructions are sent sequentially to all path nodes in the query execution path according to the execution order, so as to control all path nodes to cooperate in executing the query instructions, including: Send the query instruction to the first path node, and determine the execution result of the query instruction by the first path node, wherein the first execution node is the starting node; The execution order is determined based on the sorted order of the nodes. The execution result and the query instruction are sent to the next path node according to the execution order, so that the next path node continues to execute the query instruction based on the execution result.

10. A query request processing apparatus, characterized in that, include: The parsing module is used to parse the query statement from the received query request and determine the query type of the query request based on the query statement. The query type includes one of the following: a first type for performing a single table query and a second type for performing a multi-table query. The acquisition module is used to obtain the node status evaluation values ​​of multiple storage nodes through the node status evaluation model. The determination module is used to determine the path node for executing the query request from the plurality of storage nodes according to the query type of the query request and the node status evaluation value of the plurality of storage nodes, and to construct a query execution path based on the path node; The sending module is used to send query instructions sequentially to all path nodes in the query execution path according to the execution order, so as to control all path nodes to cooperate in executing the query instructions in a pipelined processing manner; wherein, each storage node has a cache space set up, and the query results corresponding to the query instructions are preferentially obtained from the cache space corresponding to each path node; wherein, in the case of performing hot data query, the path node is selected from the multiple storage nodes whose cache space stores a copy of the hot data block, and whose load is lower and closer. The acquisition module is further configured to: construct the node status evaluation model based on performance index parameters, hardware configuration parameters, and data block storage location; acquire the performance index parameters, hardware configuration parameters, and data block storage location of the storage node from the monitoring data of the storage node; input the first evaluation score corresponding to the performance index parameters, the second evaluation score corresponding to the hardware configuration parameters, and the third evaluation score corresponding to the data block storage location into the node status evaluation model, and obtain the node status evaluation value output by the node status evaluation model, wherein the node status evaluation value represents the weighted sum of the first evaluation score, the second evaluation score, and the third evaluation score; The acquisition module, wherein the hardware configuration parameters include at least one of the following: CPU model, memory capacity, and disk type, is further configured to: add a score corresponding to the number of CPU cores to the base score of the performance score range to which the CPU model belongs, to obtain a performance score corresponding to the CPU model in the performance score range; add a score corresponding to the memory frequency to the base score of the capacity score range to which the memory capacity belongs, to obtain a first capacity score corresponding to the memory capacity in the memory capacity score range; add a score corresponding to the disk rotation speed to the base score corresponding to the protocol type used by the disk, to obtain a disk score corresponding to the disk type; calculate a first weighted sum among the performance score, the first capacity score, and the disk score; and normalize the first weighted sum according to the preset evaluation score corresponding to the hardware configuration parameters to obtain a second evaluation score.

11. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the query request processing method according to any one of claims 1 to 9 when executing the computer program.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, it implements the steps of the query request processing method according to any one of claims 1 to 9.

13. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the query request processing method according to any one of claims 1 to 9.

Images