Query request processing method and device, electronic equipment, medium and program product

Abstract

The invention discloses a query request processing method and device, electronic equipment, a medium and a program product, and relates to the technical field of data query, the query request processing method comprises the following steps: parsing a query statement from a received query request, and determining a query type of the query request according to the query statement, the query type comprising one of the following: a first type for performing single-table query, a second type for performing single-table query, and a third type for performing single-table query; the second type is used for carrying out multi-table query; obtaining node state evaluation values of the plurality of storage nodes through a node state evaluation model; determining a path node for executing the query request from the plurality of storage nodes according to the query type of the query request and the node state evaluation values of the plurality of storage nodes, and constructing a query execution path based on the path node; and sending a query instruction to all the path nodes in the query execution path in sequence according to the execution sequence so as to control all the path nodes to cooperatively execute the query instruction.

Description

Technical Field

[0001] The present application relates to the field of data query technology, and in particular to a query request processing method, device, electronic device, medium and program product. Background Art

[0002] Currently, distributed storage systems face numerous challenges when performing data queries. Traditional methods often require traversing all storage nodes in a query request. This, on the one hand, increases the complexity of query requests as data scale continues to grow, leading to longer query times. On the other hand, traditional query methods rely solely on fixed query rules. When multiple queries are received, some nodes may become overloaded while others remain idle, compromising query efficiency.

[0003] In summary, in the related art, the technical problem of low data query efficiency due to low query request processing efficiency has not yet been proposed to effectively solve this technical problem.

[0004] Therefore, it is necessary to improve the related technology to overcome the above-mentioned defects in the related technology. Summary of the Invention

[0005] The present application provides a query request processing method, device, electronic device, medium and program product to at least solve the technical problem in the related art that data query efficiency is low due to low query request processing efficiency.

[0006] The present application provides a method for processing query requests, which is applied to a management node of a distributed storage system, wherein the distributed storage system is further deployed with multiple storage nodes, and a data transmission channel has been established between the multiple storage nodes. The method comprises: parsing a query statement from a received query request, determining a query type of the query request based on the query statement, wherein the query type includes one of the following: a first type for performing a single-table query, and a second type for performing a multi-table query; obtaining node status evaluation values ​​of the multiple storage nodes through a node status evaluation model; determining a path node for executing the query request from the multiple storage nodes based on the query type of the query request and the node status evaluation values ​​of the multiple storage nodes, and constructing a query execution path based on the path node; and sending query instructions to all path nodes in the query execution path in order of execution to control all path nodes to collaboratively execute the query instructions.

[0007] The present application also provides a query request processing device, including: a parsing module, used to parse a query statement from a received query request, and determine the query type of the query request based on the query statement, the query type including one of the following: a first type for performing a single-table query, a second type for performing a multi-table query; an acquisition module, used to obtain node status evaluation values ​​of the multiple storage nodes through a node status evaluation model; a determination module, used to determine a path node for executing the query request from the multiple storage nodes based on the query type of the query request and the node status evaluation values ​​of the multiple storage nodes, and construct a query execution path based on the path node; a sending module, used to send query instructions to all path nodes in the query execution path in order of execution, so as to control all path nodes to collaboratively execute the query instructions.

[0008] The present application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the steps of any of the above-mentioned query request processing methods when executing the computer program.

[0009] The present application also provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned query request processing methods are implemented.

[0010] The present application also provides a computer program product, including a computer program, which implements the steps of any of the above-mentioned query request processing methods when the computer program is executed by a processor.

[0011] Through this application, when a management node receives a query request, it first parses the query statement and then determines the query type of the query request, that is, whether the query is a simple query targeting a single data table (first type) or a complex query involving multiple data tables (second type). It then uses a node status evaluation model to intelligently evaluate the node status evaluation value of the storage node, and then uses the node status evaluation value and the query type of the query request to determine the query execution path, thereby significantly improving query efficiency. For example, for single-table queries, the management node prioritizes nodes that store the target data block and have a high performance evaluation value for querying, avoiding unnecessary data transmission and increasing query speed. For multi-table queries, the query execution path is determined by a heuristic search algorithm while considering the node status. This method solves the problem of low data query efficiency caused by inefficient query request processing, improves query request processing efficiency, significantly shortens the response time of complex queries, and improves the real-time performance of data queries. At the same time, it rationally allocates tasks to the execution nodes under the query execution path, avoiding node overload, effectively balancing system load, and improving resource utilization. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0013] Figure 1 This is a hardware structure block diagram of a server device for a method for processing a query request according to an embodiment of the present application;

[0014] Figure 2 is a flowchart of a method for processing a query request according to an embodiment of the present application;

[0015] Figure 3 is a schematic diagram of a data query process according to an embodiment of the present application;

[0016] Figure 4 is a schematic diagram of a node status evaluation calculation process according to an embodiment of the present application;

[0017] Figure 5 is a schematic diagram of a query request processing device according to an embodiment of the present application;

[0018] Figure 6 It is a block diagram of the computer system structure of the electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0019] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0020] It should be noted that, in the description of this application, the terms "comprises," "includes," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. The terms "first," "second," etc., in this application are used to distinguish similar objects, and are not used to describe a particular order or sequence.

[0021] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0022] The distributed storage system is described here in conjunction with the distributed storage system on which the execution of the query request processing method depends. The distributed storage system is deployed with a management node and multiple storage nodes, and data transmission channels are established between the multiple storage nodes.

[0023] The Management Node (MN) is the central hub of the distributed storage system, responsible for global coordination, monitoring, and query optimization. It performs the following functions: monitoring storage node status, including hardware resources (CPU, memory, disk) and load; providing directory services for data blocks, mapping the logical view of data to its physical location; intelligently distributing query requests based on the real-time status of each node to avoid overloading certain nodes; parsing query requests, selecting the optimal query execution path, and designing efficient collaboration between nodes.

[0024] Storage nodes (SNs) are responsible for data storage and retrieval, with each node storing a portion of the data. In a distributed storage system, data replicas or striped storage are created across multiple storage nodes to improve data redundancy and availability. Specifically, they implement the following functions: They store data blocks based on local storage hardware (such as hard drives). They respond to data read or write requests from management nodes or other storage nodes. They automatically replicate data to other nodes based on policies, or migrate data in the event of a node failure. In this application, storage nodes are simply referred to as nodes.

[0025] Data transmission channels, which can be understood as high-speed data exchange channels between storage nodes and between storage nodes and management nodes, are the cornerstone of efficient distributed storage systems. Leveraging technologies such as high-speed Ethernet, they ensure low-latency and high-bandwidth data transmission. They also utilize network protocols such as TCP / IP, UDP, and RDMA for data packet encapsulation and decapsulation, ensuring secure and efficient data transmission. During data transmission, the optimal data transmission path and timing are dynamically selected based on network topology and node status. For example, this involves intelligent routing based on network traffic or data migration during periods of low node load.

[0026] The method embodiments provided in the embodiments of the present application can be executed in a server device or similar computing device where a distributed storage system is located. Taking running on a server device as an example, Figure 1 This is a hardware structure diagram of a server device for a query request processing method according to an embodiment of the present application. Figure 1 As shown, the server device may include one or more ( Figure 1Only one is shown) a processor 102 (the processor 102 may include but is not limited to a microprocessor MPU or a programmable logic device FPGA) and a memory 104 for storing data. The server device may also include a transmission device 106 and an input / output device 108 for communication functions. It will be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above server device. Figure 1 More or fewer components than shown, or with Figure 1 Different configurations shown.

[0027] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the query request processing method in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, implementing the above-mentioned method. The memory 104 may include a high-speed random access memory and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely located relative to the processor 102, and these remote memories can be connected to the server device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0028] Transmission device 106 is used to receive or transmit data via a network. A specific example of the aforementioned network may include a wireless network provided by a communication provider of the server device. In one embodiment, transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In another embodiment, transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0029] The embodiment of the present application provides a method for processing a query request, which is described in detail in conjunction with the execution flow of the method for processing a query request. Figure 2 The specific steps are as follows:

[0030] Step S202: parse a query statement from the received query request, and determine a query type of the query request based on the query statement, where the query type includes one of the following: a first type for performing a single-table query, and a second type for performing a multi-table query;

[0031] In this step, the query request is semantically parsed, the query statement is extracted, and key information about the query statement is obtained, such as data tables, fields, query conditions, and connection relationships. The query request is then classified into different categories based on the structure and complexity of the query statement, facilitating the selection of subsequent query execution paths.

[0032] Step S204: obtaining node status evaluation values ​​of the plurality of storage nodes through a node status evaluation model;

[0033] It should be noted that a node status assessment model can also be built in advance. For example, performance indicator parameters such as the CPU (Central Processing Unit) usage, memory usage, network bandwidth occupancy, disk I / O rate, and disk I / O bandwidth occupancy of the storage node can be collected in real time. Hardware configuration information (i.e., hardware configuration parameters) such as the CPU model, number of cores, memory capacity, memory bandwidth, disk type, disk rotation speed, and disk I / O throughput of the storage node can also be collected, as well as the data block storage location. The node status assessment model can then be constructed by taking a weighted sum of these three types of data.

[0034] Performance indicator parameters are used to assess the current load of a node. For example, if a node's CPU utilization consistently exceeds 80% and its memory utilization approaches 90%, the node can be considered to be under high load. Hardware configuration information is used to assess a node's processing power. For example, nodes with high-performance CPUs and large memory capacities are relatively capable of handling complex computing tasks. The data block storage location represents the data block information and data distribution stored on each node, specifically the mapping between data blocks and nodes. This can be used to quickly locate the node where the data block being queried resides.

[0035] Compared with existing node status evaluation methods that only consider a single or a few performance indicators, the node status evaluation model in this step can comprehensively consider the performance indicators of the storage node (such as CPU utilization, memory utilization), hardware configuration (such as CPU model, memory capacity) and data block storage location, and then calculate the evaluation value based on this, reflecting the node's health status, load capacity and operating status in real time, thereby avoiding assigning query tasks to nodes with higher loads and improving the overall processing efficiency of query requests.

[0036] Step S206: determining a path node for executing the query request from the plurality of storage nodes according to the query type of the query request and the node status evaluation values ​​of the plurality of storage nodes, and constructing a query execution path based on the path node;

[0037] It should be noted that the query execution path constructed in this step can execute the query task using nodes with low load, high processing power, and close data storage locations, while meeting the query requirements. For example, for simple queries, if the data block involved in the query conditions has a copy on a certain node and the node's status assessment value is low (i.e., low load and high processing power), then that node is directly selected for query execution. For complex queries, a heuristic search algorithm is used to comprehensively consider node load, processing power, data storage location, and network connectivity to determine the set of nodes containing the relevant data blocks, and then gradually select nodes to construct the query execution path.

[0038] Step S208 : Sending query instructions to all path nodes in the query execution path in sequence according to the execution order, so as to control all path nodes to collaboratively execute the query instructions.

[0039] In this step, for query execution paths that include multiple path nodes, multiple path nodes collaborate to complete the query task using a pipelined processing method. For example, in a multi-table association query, after the first path node completes the preliminary query processing of its local data block, it immediately transmits the result to the next path node. The next path node can start the next processing operation while receiving the data without waiting for the previous path node to complete all operations. This can minimize the waiting time between nodes and improve the parallelism of query execution. For the query process of each path node, you can first check whether there is relevant data or results in the local cache. If so, the data in the cache is directly returned, avoiding repeated queries and data transmission.

[0040] Through the above steps, when a management node receives a query request, it first parses the query statement and then determines the query type of the query request: whether it is a simple query targeting a single data table (the first type) or a complex query involving multiple data tables (the second type). It then uses a node status assessment model to intelligently evaluate the node status evaluation values ​​of storage nodes. The node status evaluation values ​​and the query type of the query request are then used to determine the query execution path, significantly improving query efficiency. For example, for single-table queries, the management node prioritizes nodes that store the target data block and have high performance evaluation values, avoiding unnecessary data transmission and increasing query speed. For multi-table queries, the query execution path is determined using a heuristic search algorithm while considering node status. This approach addresses the problem of inefficient data query processing caused by inefficient query request processing, improves query request processing efficiency, significantly shortens the response time of complex queries, and enhances the real-time performance of data queries. It also rationally allocates tasks to execution nodes along the query execution path, avoiding node overload, effectively balancing system load, and improving resource utilization.

[0041] In an exemplary embodiment, the process of determining the query type of the query request based on the query statement includes: splitting the query statement to obtain multiple character strings, wherein the multiple character strings include at least one of the following: a first character string, a second character string, a third character string, and a fourth character string, wherein the first character string is used to represent a data table corresponding to the query request, the second character string is used to represent a query field of the query request, the third character string is used to represent a query condition corresponding to the query request, and the fourth character string is used to represent a connection relationship between the data tables corresponding to the query request; if a first condition is met, determining the query type to be the first type, wherein the first condition includes: the number of data tables is less than a first preset value; the number of query fields is less than a second preset value; and the number of query conditions is less than a third preset value; or if a second condition is met, determining the query type to be the second type, wherein the second condition includes: the number of data tables is greater than a first preset value; the number of query fields is greater than a second preset value; the number of query conditions is greater than a third preset value; and the connection relationship between the data tables is a connection relationship between multiple data tables. Alternatively, the number of query fields may be equal to the second preset value.

[0042] By carefully analyzing the query statement, this embodiment can accurately determine the type of query, providing a basis for subsequent node selection and path construction.

[0043] For example, a query statement "SELECT name FROM table1 WHERE id=123" will be split into strings representing the data table (table1), the query field (name), and the query condition (id=123). Since only a single data table is involved, it will be classified as the first type, that is, a single-table query.

[0044] For the query "SELECT column1 FROM table1 JOIN table2 ON table1.id=table2.id WHERE table1.id=123," the table join relationship is identified and classified as the second type, a multi-table query. By setting different preset values ​​(such as the first preset value for the number of tables is 2, the second preset value for the number of query fields is 3, and the third preset value for the number of query conditions is 2), you can flexibly distinguish query complexity and adopt more appropriate processing strategies.

[0045] For single-table queries, nodes with good performance are prioritized for direct execution. For multi-table queries, an optimal path is constructed to ensure efficient data transmission. By rationally classifying and allocating tasks, this method improves query efficiency and overall system performance.

[0046] In an exemplary embodiment, before parsing the query statement from the received query request, the received query request may further be verified to obtain a verification result, wherein the verification includes syntax checking and permission verification; if it is determined that the verification result is used to indicate that the syntax of the query request is correct and the target object sending the query request has query permission, then it is determined that the query request has passed the verification; if it is determined that either of the verification result is used to indicate that the syntax of the query request is correct and the target object sending the query request has query permission is not true, then it is determined that the query request has not passed the verification, and a verification failure message is sent to the target object.

[0047] Through this embodiment, query requests are verified, which can improve the security and data integrity of the distributed storage system. For example, when a user submits the query "SELECT name FROM users WHERE email='example@example.com'", it first checks whether the syntax is correct, that is, whether it complies with the SQL syntax rules, and then verifies whether the user has permission to access the "users" table. Only when the query syntax is correct and the user permission verification is passed will the query request be further processed. This verification process avoids the impact of illegal or erroneous query instructions and ensures secure access to data. By performing strict syntax and permission checks before query execution, this embodiment effectively prevents data leakage and system failures and enhances the security of the system.

[0048] In an exemplary embodiment, a technical solution for obtaining node status evaluation values ​​of the multiple storage nodes through a node status evaluation model may include: obtaining performance indicator parameters of the storage node, hardware configuration parameters of the storage node, and data block storage locations of the storage node from monitoring data of the storage node; inputting a first evaluation score corresponding to the performance indicator parameters, a second evaluation score corresponding to the hardware configuration parameters, and a third evaluation score corresponding to the data block storage location into the node status evaluation model, and obtaining the node status evaluation value output by the node status evaluation model, wherein the node status evaluation value represents a weighted sum of the first evaluation score, the second evaluation score, and the third evaluation score. Performing node status evaluation by calculating the node status evaluation value is beneficial to optimizing the query path and improving query efficiency. For example, in a distributed storage system consisting of multiple storage nodes, performance indicators such as CPU utilization, memory utilization, network bandwidth utilization, and disk bandwidth utilization are monitored for each node. Hardware configuration information such as CPU model, memory capacity, and disk type is also obtained, as well as the storage location of data blocks on the node. This information is weighted and summed using a node status assessment model and converted into an evaluation score, which comprehensively reflects the node's performance and load, providing a scientific basis for path selection for query requests. By intelligently assessing node status, query tasks can be avoided from being assigned to overloaded or underperforming nodes, thereby improving query efficiency and resource utilization.

[0049] In an exemplary embodiment, the process of obtaining the performance indicator parameters of the storage node, the hardware configuration parameters of the storage node and the data block storage location of the storage node from the monitoring data of the storage node may include: obtaining the initialization configuration parameters of the storage node in the initialization stage and the operation configuration parameters of the storage node in the operation stage from the monitoring data, and determining the hardware configuration parameters based on the initialization configuration parameters and the operation configuration parameters; obtaining the data block number corresponding to the node number of the storage node from the monitoring data, and determining the data block storage location of the data block on the storage node when the data block indicated by the data block number has been stored in the storage node; obtaining the central processing unit CPU usage, memory usage, network bandwidth occupancy and disk bandwidth occupancy of the storage node from the monitoring data, and determining the performance indicator parameters based on the CPU usage, the memory usage, the network bandwidth occupancy and the disk bandwidth occupancy. This embodiment can monitor the hardware configuration information of the storage node during initialization, such as initialization configuration parameters such as CPU model, memory capacity, disk type, and performance indicators that change dynamically during operation, such as CPU utilization, memory utilization, network bandwidth utilization, and disk bandwidth utilization. Operational configuration parameters can ensure the accuracy and timeliness of node status assessment.

[0050] During the operation of the node, if the hardware configuration changes (such as adding memory, replacing the disk, etc.), the changed hardware information can be reported to the management node in a timely manner, and the management node will update the corresponding records in the database.

[0051] A data block location table can be created in the database to record the storage location information of data blocks on storage nodes. The table can include the data block number, storage node number, data block size, storage path, etc. When a data block is migrated or a replica is updated, the management node promptly updates the data block location information.

[0052] In an exemplary embodiment, the hardware configuration parameters include at least one of the following: CPU model, memory capacity, and disk type. Before inputting the first evaluation score corresponding to the performance indicator parameter, the second evaluation score corresponding to the hardware configuration parameter, and the third evaluation score corresponding to the data block storage location into the node status assessment model, the performance score corresponding to the CPU model in the performance score range to which it belongs, the first capacity score corresponding to the memory capacity in the memory capacity score range to which it belongs, and the disk score corresponding to the disk type can be determined; a first weighted sum value between the performance score, the first capacity score, and the disk score is calculated; and the first weighted sum value is normalized according to the preset evaluation score corresponding to the hardware configuration parameter to obtain the second evaluation score. The processing of hardware configuration parameters in this embodiment is an indispensable part of node status assessment. For example, for a storage node, based on hardware configuration information such as the CPU model, memory capacity, and disk type, the performance score, capacity score, and disk score are converted based on the preset performance score range, memory capacity score range, and disk type score, and then the second evaluation score is obtained by weighted sum. In this embodiment, the normalization process ensures comparability between different hardware configuration parameters, making the evaluation results more fair and accurate. This process quantifies the hardware configuration information, providing a more comprehensive reference for query request path selection, and improving query efficiency and resource utilization.

[0053] Optionally, determining the performance score of the CPU model in the performance score range to which it belongs may include: obtaining a basic score (i.e., the minimum value of the range) from the performance score range to which the CPU model belongs, and adding a score corresponding to the number of CPU cores to the basic score, which is the performance score.

[0054] Optionally, determining the capacity score of the memory capacity in the capacity score range may include:

[0055] A basic score (ie, the minimum value of the interval) is obtained from the capacity score interval to which the memory capacity belongs, and a score corresponding to the memory frequency is added to the basic score to obtain the first capacity score.

[0056] Optionally, determining the disk score corresponding to the disk type may include: determining a basic score corresponding to the protocol type used by the disk, and adding a score corresponding to the disk rotation speed to the basic score to obtain the disk score.

[0057] It can be understood that the process of normalizing the first weighted sum value to obtain the second evaluation score is also applicable to normalizing the third weighted sum value to obtain the first evaluation score and normalizing the second weighted sum value to obtain the third evaluation score, and will not be repeated here.

[0058] It should be noted that the processing method of hardware configuration parameters in this embodiment is not only applicable to distributed storage systems, but also includes but is not limited to other distributed computing platforms, cloud computing resource scheduling, and other systems that require evaluation of node hardware capabilities to optimize system performance and resource allocation.

[0059] In an exemplary embodiment, before inputting the first evaluation score corresponding to the performance indicator parameter, the second evaluation score corresponding to the hardware configuration parameter, and the third evaluation score corresponding to the data block storage location into the node status evaluation model, further, the number of accesses to the data block within the access cycle can be determined, and the access score corresponding to the access number in the corresponding number score interval can be determined; the second capacity score corresponding to the block capacity of the data block in the corresponding block capacity score interval can be determined; the network transmission parameters between the storage node and the sending node that sends the query request can be determined; and the transmission score corresponding to the network transmission parameters can be determined; and the third evaluation score can be determined based on a second weighted sum of the access score, the second capacity score, and the transmission score. Through this embodiment, the weighted sum of the access score, the second capacity score, and the transmission score is calculated, which can comprehensively evaluate the impact of the data block storage location on the query efficiency, solve the problem of quantifying the data block location information in the query path selection, ensure the efficiency of data transmission and the balanced load between nodes, and improve the query efficiency and overall system performance.

[0060] The number of accesses to the data block within an access period is determined, and the access score corresponding to the access count within the corresponding score interval is determined. For example, the number of accesses to each data block within a certain period of time is counted, and different intervals are divided based on the number of accesses, with each interval having a corresponding score. For example, if there were 100 accesses in the past 24 hours, and the score interval corresponding to 100 accesses is [10, 20], then the score for 100 accesses is 10.

[0061] The network transmission parameter, which can be understood as the number of network hops or network latency between the storage node and the sending node, measures the distance between the storage node where the data block being queried resides and the sending node. The fewer network hops or the lower the network latency, the higher the transmission score. For example, a network hop of 1 or a latency of less than 1ms would result in a score of 90 to 100 points; a network hop of 2 to 3 or a latency of 1 to 5ms would result in a transmission score of 70 to 89 points.

[0062] In an exemplary embodiment, the implementation process of determining the performance indicator parameters based on the CPU usage, the memory usage, the network bandwidth occupancy, and the disk bandwidth occupancy includes: calculating a third weighted sum of the CPU usage, the memory usage, the network bandwidth occupancy, and the disk bandwidth occupancy within each calculation cycle; and determining the third weighted sum as the first evaluation score. In this embodiment, a comprehensive performance evaluation score (i.e., the first evaluation score) is obtained by performing a weighted sum calculation based on the real-time performance indicators of the storage node, such as the CPU usage, memory usage, network bandwidth occupancy, and disk bandwidth occupancy within each calculation cycle. The first evaluation score can be used to reflect the node's load and processing capacity in real time, providing a scientific basis for path selection for query requests. By dynamically adjusting the weighting coefficient, this solution can flexibly respond to performance requirements in different scenarios, efficiently execute query tasks, and improve query efficiency and resource utilization.

[0063] It should be noted that the performance indicator parameter determination technology is not only applicable to distributed storage systems, but also to other systems that need to evaluate the real-time performance of nodes, such as real-time data analysis platforms and network monitoring systems.

[0064] Based on the above embodiments, it can be known that the present application assigns different weights to performance indicator parameters, hardware configuration parameters and data block storage locations based on the node status evaluation model, and then calculates the node status evaluation value by weighted average. In this process, first, the performance indicator parameters are quantified to obtain the load situation, and second, the hardware configuration parameters are quantified from the perspective of CPU processing power, memory processing power and disk processing power to obtain the processing power, and then the node status evaluation value is calculated according to the preset weights in combination with the data block storage location, thereby accurately reflecting the actual operating status of the node, providing a scientific basis for the selection of subsequent query execution paths, and improving query efficiency and resource utilization.

[0065] Furthermore, the weight distribution can be dynamically adjusted based on the actual application scenario and the impact of each parameter on query performance. For example, in scenarios with computationally intensive queries, the weight of CPU usage can be appropriately increased; in scenarios with limited network bandwidth, the weight of network bandwidth utilization should be increased.

[0066] In one exemplary embodiment, a method for determining a path node for executing a query request from among the multiple storage nodes based on the query type of the query request and the node status evaluation values ​​of the multiple storage nodes includes: for the first type, determining a first node from the multiple storage nodes, wherein the first node stores a first data block or a replica of the first data block, and the first data block or the replica of the first data block satisfies the query statement; and determining the first node as the path node if the first node status evaluation value corresponding to the first node is below a first threshold. This embodiment provides a method for determining path nodes for single-table queries, ensuring that query tasks are assigned to the most appropriate nodes, avoiding unnecessary data transmission and node overload, and improving query efficiency and resource utilization. For example, for a query statement "SELECT name FROM table1 WHERE id=123," nodes storing the table1 data block or its replica are searched and their status is evaluated. If the node's status evaluation value is below a preset first threshold (e.g., 80 points), the node is considered to have good performance and can efficiently execute the query task, and is thus determined as a path node.

[0067] The first threshold may be adjusted based on the actual status, or may be set to an average of node status evaluation values ​​of multiple storage nodes.

[0068] In an exemplary embodiment, a process of determining path nodes for executing the query request from the plurality of storage nodes based on the query type of the query request and the node status evaluation values ​​of the plurality of storage nodes, and constructing a query execution path based on the path nodes includes: for a second type, determining a second node from the plurality of storage nodes, and obtaining a node set based on the second node, wherein the second node stores a second data block or a copy of the second data block that satisfies the query statement; obtaining a third node from the node set whose node status evaluation value is lower than a second threshold; selecting a starting node corresponding to the first data table in the query statement from the plurality of third nodes, and sorting the remaining nodes using a heuristic search algorithm to obtain sorted nodes, wherein the remaining nodes are nodes in the plurality of third nodes other than the starting node, and the network bandwidth between each two adjacent nodes in the sorted nodes is lower than a bandwidth threshold, and the data transmission delay between each two adjacent nodes is lower than a delay threshold; determining the path nodes based on the starting node and the remaining nodes, and constructing a query execution path based on the starting node and the sorted nodes. This embodiment proposes a method for determining path nodes for multi-table queries. This process not only considers node performance and load but also optimizes data transmission paths, ensuring efficient query execution and improving query efficiency and resource utilization. For example, for a JOIN query involving table1 and table2, nodes storing these two tables or their replicas are searched to form a node set. The status of these nodes is then evaluated, and nodes with status evaluation values ​​below a second threshold (e.g., 70 points) are selected as candidate nodes. A heuristic search algorithm is used to select a starting node from the candidate nodes, and the remaining nodes are sorted to ensure that the network bandwidth and data transmission latency between each two adjacent nodes meet preset thresholds, thereby constructing the optimal query execution path.

[0069] Optionally, the first data table is determined based on a query order, such as the first data table to be queried, or may be randomly selected from multiple data tables to be queried. This application does not impose any restrictions on the selection of data tables.

[0070] Compared to existing methods that use a fixed order or random allocation to determine the query execution path, the method for constructing the query execution path proposed in the above embodiments of the present application can dynamically adjust based on the type of query request and the status of the storage node. When faced with queries involving multi-table joins and complex conditional filtering, the actual distribution of data blocks can be considered, fully utilizing nodes with higher processing capabilities, reducing node resource waste and network transmission delays, and improving query performance.

[0071] Furthermore, the node where the data block or data block replica resides is properly selected for query based on the query request type and node status, improving the efficiency of replica utilization. For example, when querying hot data, nodes with low load and closer proximity to the hot data block can be prioritized. This means nodes with high access counts, large data block capacity, few network hops, or low network latency can be prioritized, improving query efficiency.

[0072] In one exemplary embodiment, a scheme for sequentially sending query instructions to all path nodes in a query execution path in an execution order to control all path nodes to collaboratively execute the query instruction includes: sending the query instruction to a first path node, determining the execution result of the query instruction by the first path node, wherein the first execution node is the starting node; determining the execution order based on the sorted order of the sorted nodes; and sending the execution result and the query instruction to the next path node in the execution order, so that the next path node continues to execute the query instruction based on the execution result. This embodiment proposes a pipelined method for executing query instructions, which improves query efficiency. In the constructed query execution path, the query instruction is first sent to the starting node. After the starting node completes execution and returns an execution result, the execution result and the query instruction are sent to the next node in the sorted order of the path nodes, and so on, until all nodes complete the query task. This processing process is similar to a pipeline operation, which can fully utilize the processing power of each node, avoid waiting time between nodes, and improve query efficiency. By introducing pipeline processing, this embodiment can effectively handle the execution of complex queries, shorten response time, and improve resource utilization.

[0073] Optionally, in one embodiment, frequently queried data blocks and query results can be stored by setting up cache space on each node. During the query process, the node will first check whether there is relevant data or results in the local cache. If so, the data in the cache will be used directly to avoid repeated queries and data transmission. For a frequently queried hot data block, if a copy of this hot data block is stored in the cache of multiple nodes. When this hot data block is queried again, the query results and query data can be fed back by the node that is closest to the sending node and has cached a copy of this hot data block, thereby improving the query response speed. The distributed caching mechanism proposed in this embodiment shortens the response time of complex queries, further improving the query response speed and the overall system performance. In practical applications, this method can significantly improve the real-time performance of data queries, while effectively balancing the system load and enhancing the stability and scalability of the system.

[0074] Further, based on Figure 3The data query process shown here illustrates the data query process for a distributed storage system, including the implementation steps of building a system architecture, node status assessment, and query request processing. Building a system architecture provides a foundational environment for data storage and querying; node status assessment is a key prerequisite for query optimization, providing a basis for query execution path selection; and query request processing implements optimization strategies into query task execution. These three elements work together to ensure the effective implementation of this application's technical solution and improve the data query performance of the distributed storage system.

[0075] 1. Build the system architecture. The specific steps include the following.

[0076] Step a: Deploy storage nodes. Deploy multiple storage nodes in the distributed storage system, equipping each node with the appropriate hardware, including servers, storage disks, and network interface cards. Server configurations are selected based on actual needs. For example, for nodes requiring high processing power, servers with high-performance CPUs and large memory capacities can be selected. For nodes requiring larger storage capacities, additional storage disks can be added.

[0077] Step b: Network connection setup. Connect each storage node via a high-speed network to form a distributed storage network. Network connections can use Ethernet, Fibre Channel, or other methods to ensure sufficient network bandwidth between nodes to meet data transmission requirements. To improve network reliability, employ a redundant network topology, such as dual-link redundant connections, to prevent system performance degradation caused by a single network link failure.

[0078] Step c: Management Node Configuration. One or more management nodes are set up to manage and monitor the entire distributed storage system. Management nodes are responsible for collecting status information from each storage node, including node load, processing capacity, and data storage location, and implementing query optimization strategies based on this information. Management nodes are also responsible for receiving user query requests, parsing and classifying them, and assigning the optimal execution path and node collaboration method based on the query execution path selection algorithm.

[0079] 2. Node status assessment. Details are as follows.

[0080] Step 1: Performance indicator collection.

[0081] Install performance monitoring software on each storage node to regularly collect performance indicators such as the node's CPU usage, memory usage, network bandwidth usage, and disk I / O bandwidth usage.

[0082] The collection frequency can be adjusted according to actual needs. It is generally recommended to set it to collect data every 10 seconds or 30 seconds to ensure that the real-time status of the node can be reflected in a timely manner.

[0083] For example, use the system's built-in performance monitoring tools (such as the top command and vmstat command in Linux) to obtain CPU usage and memory usage;

[0084] Obtain network bandwidth usage through network monitoring tools;

[0085] Use disk monitoring tools to obtain disk I / O rates.

[0086] Step 2: Obtain hardware information.

[0087] During the node initialization phase, the node's hardware configuration information is obtained, including CPU model, number of cores, memory capacity, disk type and rotation speed, and the information is stored in the management node's database.

[0088] During the operation of the node, if the hardware configuration changes (such as adding memory, replacing the disk, etc.), the node will promptly report the changed hardware information to the management node, and the management node will update the corresponding records in the database.

[0089] Hardware information will serve as the basic parameter for node status evaluation. For example, nodes with high-performance CPUs and large-capacity memory will be given a higher baseline value when calculating the weights of processing capacity-related indicators, which will affect the final status evaluation value.

[0090] Step 3: Data block location record.

[0091] When a data block is stored on a node, the management node records the mapping between the data block and the node and stores the relevant information in the database. Furthermore, when a data block is migrated or a replica is updated, the management node promptly updates the data block's location information. For example, this can be achieved by creating a data block location table in the database containing fields such as the data block ID, storage node ID, data block size, and storage path to record the data block's storage location information.

[0092] Data block location information will affect the data access cost factor in node status evaluation. If a node stores a large number of frequently accessed data blocks, the weight related to the data storage location will be appropriately increased when calculating the status evaluation value, thereby improving the priority of the node in query path selection.

[0093] Step 4: Node status evaluation calculation.

[0094] like Figure 4As shown in the figure, in the node status evaluation calculation process, data collection begins. Specifically, this includes collecting real-time performance indicators such as CPU usage and memory usage, obtaining hardware configuration information, and recording data block location information. Next, score calculation is performed, specifically calculating the real-time performance indicator score S1 according to the rules, calculating the hardware configuration score S2, and calculating the data block location score S3. Finally, the evaluation value is calculated. Specifically, the real-time performance indicator score (corresponding to the first evaluation score), the hardware configuration score (corresponding to the second evaluation score), and the data block location score (corresponding to the third evaluation score) are weighted and summed to obtain the node status evaluation value.

[0095] In the evaluation value calculation process, it is assumed that the total weight of the real-time performance index is W1, the weight of the hardware configuration comprehensive score is W2, and the weight of the data block location comprehensive score is W3, and W1+W2+W3=1 is satisfied.

[0096] The calculation formula for the node status evaluation value S is S = S1 × W1 + S2 × W2 + S3 × W3 = 1. Where S1 is the real-time performance index score, S2 is the hardware configuration comprehensive score, and S3 is the data block location comprehensive score.

[0097] Depending on the actual application scenario, for example, in a big data analysis scenario with extremely high computing power requirements, W1=0.4, W2=0.4, and W3=0.2 can be set. In a scenario with frequent data access and sensitivity to data transmission delays, W1=0.4, W2=0.2, and W3=0.4 can be adjusted.

[0098] It should be noted that the score calculation process is described in conjunction with the following steps.

[0099] To calculate the comprehensive score of your hardware configuration, perform the following steps.

[0100] Step 4.1: Determine the scoring criteria for each hardware parameter.

[0101] (1) CPU related: According to the performance ranking of CPU models in the market, they are divided into different performance ranges (i.e., the performance score range of the CPU models) and assigned corresponding scores.

[0102] For example, high-performance CPU models, such as the Intel Core i9-13900K, can be assigned a score range of 85-100 points; mid-to-high-end CPU models, such as the Intel Core i7-13700K, can be assigned a score range of 70-84 points; and mid-to-low-end CPU models, such as the Intel Core i3-13100, can be assigned a score range of 50-69 points. The number of CPU cores also affects the score. Each additional core adds a certain number of points to the base model score (corresponding to the base score), such as 2 points per core. For example, if a node's CPU model has a base score of 85 and 16 cores, the CPU score = 85 + 16 × 2 = 117 points.

[0103] (2) Memory related: the memory capacity is divided into different capacity intervals (i.e. the memory capacity score interval to which the memory capacity belongs) for scoring.

[0104] For example, memory capacity of 128GB and above is scored 90-100 points; 64GB-127GB is scored 75-89 points; 32GB-63GB is scored 60-74 points, etc. Furthermore, factors such as memory frequency are taken into account. High-frequency memory can be given additional points on top of the base capacity score, such as adding 1 point for every 100MHz increase in memory frequency. For example, if a node has 128GB of memory, the corresponding base score is 90 points, and the memory frequency is 3600MHz, a 400MHz increase from the base frequency of 3200MHz. The memory score = 90 + 4 = 94 points.

[0105] (3) Disk-related: In terms of disk type, solid-state drives (SSDs) are assigned different scores based on different protocol standards (e.g., SSDs using the NVMe protocol perform better than SSDs using the SATA protocol) and read / write speed indicators. An example of a read / write speed indicator is the disk rotation speed.

[0106] For example, a high-performance NVMe SSD can score 80-100 points, a SATA SSD can score 60-79 points, and a hard disk can score 40-59 points. Disk speed is an important factor in HDD scoring; higher speeds earn higher scores. For example, a 7200 rpm HDD can add 10 points to the base type score, while a 5400 rpm HDD can add 5 points. If a node uses a high-performance NVMe SSD and its base type score is 90, the disk score will equal 90 points.

[0107] Step 4.2: Calculate the comprehensive score of hardware configuration.

[0108] The scores of various hardware parameters such as CPU, memory, and disk are weighted and summed according to certain weights to obtain the final score of the hardware configuration.

[0109] Assuming the CPU score has a weight of 0.4, the memory score has a weight of 0.3, and the disk score has a weight of 0.3, the hardware configuration score = CPU score × 0.4 + Memory score × 0.3 + Disk score × 0.3. Using the previously calculated result as an example, the hardware configuration score = 117 × 0.4 + 94 × 0.3 + 90 × 0.3 = 46.8 + 28.2 + 27 = 102 points.

[0110] However, considering the actual scoring range setting, the calculation results can be normalized so that they fall within the range of 0-100 points. For example, through the formula: Final hardware configuration score = 100×(102 / theoretical maximum value of hardware configuration score). Assuming that the theoretical maximum value of the hardware configuration score (corresponding to the above preset evaluation score) is 130, the final hardware configuration score = 100×(102 / 130)≈78.46 points.

[0111] For the comprehensive score of the data block position, the following steps can be performed.

[0112] Step 4.3: Determine the scoring criteria for the data block location score. The data block location score is mainly based on factors such as the access frequency of the data block, the size of the data block, and the distance between the data block and the query initiator.

[0113] The data block size can also be understood as the block capacity of the data block.

[0114] Access frequency: Counts the number of accesses to each data block within a certain period (e.g., the past 24 hours), and then divides the data into different intervals for scoring. For example, 1,000 or more accesses are scored 80 to 100 points; 500 to 999 accesses are scored 60 to 79 points; 100 to 499 accesses are scored 40 to 59 points, and so on.

[0115] Similarly, data block size is divided into different ranges for scoring. For example, data block size of 1GB and above is scored from 70 to 100 points; 500MB to 999MB is scored from 50 to 69 points; 100MB to 499MB is scored from 30 to 49 points, etc.

[0116] Finally, the distance from the query originator is measured based on the number of network hops or network latency between the node and the query originator. Fewer network hops or lower network latency result in a higher score. For example, a network hop of 1 or latency under 1ms results in a score of 90 to 100; a network hop of 2 to 3 or latency of 1 to 5ms results in a score of 70 to 89, and so on.

[0117] Step 4.4: Calculate the comprehensive score of the data block position.

[0118] The scores of various influencing factors such as access frequency, data block size, and distance from the query initiator are weighted and summed according to certain weights to obtain the final score of the data block location.

[0119] Assuming the access frequency score is weighted 0.4, the data block size score is weighted 0.3, and the distance score from the query initiator is weighted 0.3, then the data block location score = access frequency score × 0.4 + data block size score × 0.3 + distance score from the query initiator × 0.3. Using the previously calculated result as an example, the data block location score = 90 × 0.4 + 85 × 0.3 + 95 × 0.3 = 36 + 25.5 + 28.5 = 90 points.

[0120] To calculate the real-time performance score, perform the following steps:

[0121] Step 4.5: Calculate the node real-time performance index score.

[0122] The management node calculates the performance indicators collected in real time every minute or every 5 minutes, and calculates the real-time performance indicator score of the node by weighted summation of these real-time performance indicators.

[0123] For example, assuming that the weight of CPU usage, memory usage, network bandwidth utilization, and disk I / O rate in the node status evaluation model is 0.3, 0.3, 0.2, and 0.1, for node A, its CPU usage is 50%, memory usage is 60%, network bandwidth utilization is 30%, and disk I / O bandwidth utilization is 40%. Then, the real-time performance indicator score of node A = 0.4 × 50 + 0.3 × 60 + 0.2 × 30 + 0.1 × 40 = 48.

[0124] Compared with the traditional node evaluation method that only considers a single or a few indicators, this application comprehensively considers the node's real-time performance indicators such as CPU utilization, memory utilization, network bandwidth occupancy, disk I / O rate, as well as the node's hardware configuration information and data storage location information. The node status evaluation value calculated by weighted summation of this information can accurately reflect the actual operating status of the node, thereby avoiding blindly selecting nodes and resulting in low query efficiency, and selecting the best execution path for subsequent query requests by comprehensively evaluating the node status based on the node's load capacity, processing power and data storage location.

[0125] 3. Processing of query requests.

[0126] Step 5: Query request received.

[0127] When a user sends a query request to the distributed block storage system through a client, the query request first reaches the management node. The management node performs preliminary verification on the received query request, including syntax checking and permission verification.

[0128] If the query request syntax is incorrect or the user does not have the corresponding query permission, the management node returns an error message to the client; if the query request verification is successful, the next processing step is entered.

[0129] Step 6: Query request parsing and classification.

[0130] The management node uses the query parser to perform syntax and semantic analysis on the query request, extract the key information of the query, and classify the query request according to the complexity and resource requirements of the query.

[0131] For example, for a SQL query request, the query parser can use lexical analysis, syntax analysis, and other techniques to break the query statement into its components, extracting information such as the data tables, fields, query conditions, and join relationships involved in the query. For example, for the SQL query statement "SELECT column1, column2 FROM table1 WHERE condition1 AND condition2 JOIN table2 ON table1.id=table2.id," parsing clearly indicates that the tables involved in the query are table1 and table2, the query conditions are condition1 and condition2, and the join relationship is an inner join based on the id field.

[0132] Then, based on predefined query classification rules, it is determined whether the query is a simple query or a complex query.

[0133] For example, a simple query can be defined as a query involving only a small number of fields in a single table and with simple query conditions, such as "SELECT column1 FROM table1 WHERE column2=value";

[0134] Complex queries include multi-table join queries, complex condition filtering queries, and aggregate queries, such as "SELECTCOUNT (*), SUM (column1) FROM table1 JOIN table2 ON table1.id=table2.id WHEREcondition1 OR condition2 GROUP BY column3".

[0135] Step 7: Query execution path selection.

[0136] The management node uses the query execution path selection algorithm to select the best execution path for the query request based on the classification results of the query request and the status evaluation value of each node.

[0137] For simple queries, the appropriate node is selected for query execution based on the data block location information and node status evaluation value. For example, if the query statement is "SELECT column1 FROM table1 WHERE column2=value", and the data block replica that meets this condition in table1 is stored on node A, the CPU usage of node A is 30%, the memory usage is 40%, the network bandwidth usage is 20%, and the disk I / O bandwidth usage is 10%, and the node status evaluation model calculates that the status evaluation value of node A is lower, then node A is selected first to execute the query.

[0138] For complex queries, the optimal query execution path is constructed through heuristic search algorithms, taking into account factors such as node load, processing power, data storage location, and network connectivity.

[0139] First, based on the data tables and fields involved in the query, determine the node set that may contain related data blocks.

[0140] Then, from this set of nodes, nodes are gradually selected according to certain rules (such as prioritizing nodes with low status evaluation values) to build a query execution path. During the path construction process, the network connectivity and data transmission latency between nodes are taken into consideration to minimize cross-node data transmission.

[0141] For example, in a join query involving three tables, assume that the data blocks of the first table are distributed on nodes A, B, and C, the data blocks of the second table are distributed on nodes B, D, and E, and the data blocks of the third table are distributed on nodes C, E, and F. The node status assessment model finds that the status assessment values ​​of nodes A, B, and E are low, and the network bandwidth between nodes A and B and between B and E is high, resulting in low data transmission delay.

[0142] A query execution path can be constructed starting from node A and passing through nodes B and E in sequence. On this path, the relevant data block of the first table is first obtained on node A, then transmitted to node B through the network, and associated with the relevant data block of the second table on node B. Finally, the result is transmitted to node E, and the final association and query operation is performed with the relevant data block of the third table on node E.

[0143] Through the above steps, we propose a query execution path selection algorithm based on node status evaluation and query request type. This algorithm can select the optimal query execution path for different query types (simple and complex). For example, for simple single-table queries, the algorithm directly selects the appropriate node based on the data block location and node status. For complex multi-table queries, a heuristic search algorithm is used to construct a path based on node load, processing power, data storage location, and network connectivity. Compared to traditional fixed or random path selection methods, this algorithm effectively reduces cross-node data transmission and improves the efficiency of complex queries.

[0144] Step 8: Node collaboration and query execution.

[0145] After determining the query execution path, the management node sends query task instructions to the relevant storage nodes. Each storage node cooperates to complete the query task according to the instruction requirements.

[0146] During the query execution process, a pipelined processing approach is adopted to improve query execution efficiency. For example, after the first storage node completes the initial processing of the local data block, it transmits the result to the next node through the network. The next node begins the next processing operation while receiving the data. At the same time, a distributed cache mechanism is introduced, and cache space is set up on each node to store frequently queried data blocks and query results. When a query request arrives, each node will first check whether there is relevant data or results in the local cache during the query process. If so, the data in the cache is directly used to avoid repeated queries and data transmission. For example, for a frequently queried hot data block, copies of the data block are stored in the caches of multiple nodes. When a query request involves this data block, the node closest to the query initiator and with the data block in the cache can directly return the data, greatly improving the query response speed.

[0147] In this step, we innovatively introduce pipeline processing and distributed caching mechanisms. Pipeline processing breaks the limitations of traditional sequential processing and achieves parallel data processing and transmission between nodes. Distributed caching dynamically stores hot data and query results, reducing repeated queries and data transmission. These two strategies work together to significantly improve node collaboration efficiency and query performance.

[0148] Step 9: Return query results. After all storage nodes complete the collaborative query task, they return the final query results to the management node. The management node organizes and packages the query results and returns them to the client. After receiving the query results, the client displays or processes them accordingly.

[0149] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method.

[0150] The embodiment of the present application also provides a query request processing device, such as Figure 5 As shown, specifically including:

[0151] A parsing module 52 is configured to parse a query statement from a received query request and determine a query type of the query request based on the query statement, wherein the query type includes one of the following: a first type for performing a single-table query and a second type for performing a multi-table query;

[0152] An acquisition module 54 is configured to acquire node status evaluation values ​​of the plurality of storage nodes through a node status evaluation model;

[0153] a determination module 56 configured to determine a path node for executing the query request from the plurality of storage nodes according to a query type of the query request and node status evaluation values ​​of the plurality of storage nodes, and to construct a query execution path based on the path node;

[0154] The sending module 58 is configured to send query instructions to all path nodes in the query execution path in sequence according to an execution order, so as to control all path nodes to collaboratively execute the query instructions.

[0155] Through the above module, when a management node receives a query request, it first parses the query statement and then determines the query type of the query request: whether it is a simple query targeting a single data table (the first type) or a complex query involving multiple data tables (the second type). It then uses a node status assessment model to intelligently evaluate the node status evaluation values ​​of storage nodes. This, combined with the node status evaluation values ​​and the query type of the query request, determines the query execution path, significantly improving query efficiency. For example, for single-table queries, the management node prioritizes nodes that store the target data block and have high performance evaluation values, avoiding unnecessary data transmission and increasing query speed. For multi-table queries, the query execution path is determined using a heuristic search algorithm while considering node status. This approach addresses the problem of inefficient data query processing caused by inefficient query request processing, improves query request processing efficiency, significantly shortens the response time of complex queries, and enhances the real-time performance of data queries. It also rationally allocates tasks to execution nodes along the query execution path, avoiding node overload, effectively balancing system load, and improving resource utilization.

[0156] In an exemplary embodiment, the parsing module is further used to: split the query statement to obtain multiple strings, wherein the multiple strings include at least one of the following: a first string, a second string, a third string and a fourth string, the first string is used to represent the data table corresponding to the query request, the second string is used to represent the query field of the query request, the third string is used to represent the query condition corresponding to the query request, and the fourth string is used to represent the connection relationship of the data tables corresponding to the query request; if the first condition is met, determine that the query type is the first type, wherein the first condition includes: the number of data tables is less than a first preset value; the number of query fields is less than a second preset value; the number of query conditions is less than a third preset value; or, if the second condition is met, determine that the query type is the second type, wherein the second condition includes: the number of data tables is greater than a first preset value; the number of query fields is greater than a second preset value; the number of query conditions is greater than a third preset value; and the connection relationship between the data tables is a connection relationship between multiple data tables.

[0157] In an exemplary embodiment, the parsing module is further used to: before parsing the query statement from the received query request, verify the received query request to obtain a verification result, wherein the verification includes syntax checking and authority verification; if it is determined that the verification result is used to indicate that the syntax of the query request is correct and the target object sending the query request has query authority, then it is determined that the query request has passed the verification; if it is determined that either of the verification result is used to indicate that the syntax of the query request is correct and the target object sending the query request has query authority is not true, then it is determined that the query request has not passed the verification, and a verification failure message is sent to the target object.

[0158] In an exemplary embodiment, the acquisition module is further used to: obtain the performance indicator parameters of the storage node, the hardware configuration parameters of the storage node and the data block storage location of the storage node from the monitoring data of the storage node; input the first evaluation score corresponding to the performance indicator parameters, the second evaluation score corresponding to the hardware configuration parameters and the third evaluation score corresponding to the data block storage location into the node status assessment model, and obtain the node status assessment value output by the node status assessment model, wherein the node status assessment value represents the weighted sum of the first evaluation score, the second evaluation score and the third evaluation score.

[0159] In an exemplary embodiment, the acquisition module is also used to: obtain the initialization configuration parameters of the storage node in the initialization stage and the operating configuration parameters of the storage node in the operation stage from the monitoring data, and determine the hardware configuration parameters based on the initialization configuration parameters and the operating configuration parameters; obtain the data block number corresponding to the node number of the storage node from the monitoring data, and determine the data block storage location of the data block on the storage node when the data block indicated by the data block number has been stored in the storage node; obtain the central processing unit CPU usage, memory usage, network bandwidth occupancy and disk bandwidth occupancy of the storage node from the monitoring data, and determine the performance indicator parameters based on the CPU usage, the memory usage, the network bandwidth occupancy and the disk bandwidth occupancy.

[0160] In an exemplary embodiment, the hardware configuration parameters include at least one of the following: CPU model, memory capacity and disk type, and the acquisition module is further used to: determine the performance score corresponding to the CPU model in the performance score range to which it belongs, determine the first capacity score corresponding to the memory capacity in the memory capacity score range to which it belongs, and determine the disk score corresponding to the disk type; calculate a first weighted sum value between the performance score, the first capacity score and the disk score; and normalize the first weighted sum value according to the preset evaluation score corresponding to the hardware configuration parameter to obtain the second evaluation score.

[0161] In an exemplary embodiment, the acquisition module is further used to: determine the number of accesses to the data block within an access cycle, and determine the access score corresponding to the number of accesses in the corresponding number score interval; determine the second capacity score corresponding to the block capacity of the data block in the corresponding block capacity score interval; determine the network transmission parameters between the storage node and the sending node that sends the query request; and determine the transmission score corresponding to the network transmission parameters; and determine the third evaluation score based on a second weighted sum value among the access score, the second capacity score and the transmission score.

[0162] In an exemplary embodiment, the acquisition module is further used to: calculate a third weighted sum of the CPU usage, the memory usage, the network bandwidth occupancy and the disk bandwidth occupancy in each calculation cycle; and determine the third weighted sum as the first evaluation score.

[0163] In an exemplary embodiment, the determination module is further used to: for the first type, determine a first node from the multiple storage nodes, wherein a first data block or a copy of the first data block is stored on the first node, and the first data block or the copy of the first data block satisfies the query statement; when the first node status evaluation value corresponding to the first node is lower than a first threshold, determine the first node as the path node.

[0164] In an exemplary embodiment, the acquisition module is further used to: for the second type, determine a second node from the multiple storage nodes, and obtain a node set based on the second node, wherein the second node stores a second data block or a copy of the second data block that satisfies the query statement; obtain a third node from the node set whose node status evaluation value is lower than a second threshold; select a starting node corresponding to the first data table in the query statement from multiple third nodes, and use a heuristic search algorithm to sort the remaining nodes to obtain sorted nodes, wherein the remaining nodes are nodes other than the starting node in the multiple third nodes, and the network bandwidth between each two adjacent nodes in the sorted nodes is lower than the bandwidth threshold, and the data transmission delay between each two adjacent nodes is lower than the delay threshold; determine the path node based on the starting node and the remaining nodes, and construct a query execution path based on the starting node and the sorted nodes.

[0165] In an exemplary embodiment, the acquisition module is further used to: send the query instruction to the first path node, determine the execution result of the query instruction executed by the first path node, wherein the first execution node is the starting node; determine the execution order according to the sorting order of the sorted nodes; and send the execution result and the query instruction to the next path node in accordance with the execution order, so that the next path node continues to execute the query instruction according to the execution result.

[0166] For the description of the features in the embodiment corresponding to the query request processing device, please refer to the relevant description of the embodiment corresponding to the query request processing method, which will not be repeated here.

[0167] An embodiment of the present application further provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any of the above-mentioned query request processing method embodiments.

[0168] Optionally, Figure 6 : is a block diagram of the computer system structure of the electronic device according to the embodiment of the present application. Figure 6As shown, computer system 600 includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 602 or programs loaded from storage 608 into random access memory (RAM) 803. RAM 603 also stores various programs and data required for system operation. CPU 601, ROM 602, and RAM 603 are interconnected via bus 604. Input / output interface 605 (I / O interface) is also connected to bus 604.

[0169] The following components are connected to the input / output interface 605: an input section 606 including a keyboard, mouse, and the like; an output section 607 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and speakers; a storage section 608 including a hard disk; and a communication section 609 including a network interface card such as a local area network card or a modem. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the input / output interface 605 as needed. Removable media 611, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 610 as needed, so that computer programs read from the media can be installed in the storage section 608 as needed.

[0170] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above-mentioned query request processing method embodiments when running.

[0171] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0172] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the computer program implements the steps in any of the above-mentioned query request processing method embodiments.

[0173] An embodiment of the present application further provides another computer program product, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in any of the above-mentioned query request processing method embodiments are implemented.

[0174] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0175] The above is a detailed introduction to the query request processing method, device, electronic device, medium and program product provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core ideas of the present application. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of the present application.

Claims

1. A method for processing a query request, characterized in that: A management node applied to a distributed storage system, wherein the distributed storage system is further deployed with multiple storage nodes and data transmission channels have been established between the multiple storage nodes, including: Parsing a query statement from the received query request, and determining a query type of the query request according to the query statement, the query type including one of the following: a first type for performing a single-table query, and a second type for performing a multi-table query; Obtaining node status evaluation values ​​of the plurality of storage nodes through a node status evaluation model; determining, from the plurality of storage nodes, a path node for executing the query request according to a query type of the query request and node status evaluation values ​​of the plurality of storage nodes, and constructing a query execution path based on the path node; The query instruction is sent to all path nodes in the query execution path in sequence according to the execution order, so as to control all path nodes to collaboratively execute the query instruction.

2. The method for processing a query request according to claim 1, wherein: Determining the query type of the query request according to the query statement includes: Splitting the query statement to obtain multiple character strings, wherein the multiple character strings include at least one of the following: a first character string, a second character string, a third character string, and a fourth character string, wherein the first character string is used to represent a data table corresponding to the query request, the second character string is used to represent a query field of the query request, the third character string is used to represent a query condition corresponding to the query request, and the fourth character string is used to represent a connection relationship between the data tables corresponding to the query request; If a first condition is met, determining that the query type is the first type, wherein the first condition includes: the number of data tables is less than a first preset value; the number of query fields is less than a second preset value; and the number of query conditions is less than a third preset value; Alternatively, when the second condition is met, the query type is determined to be the second type, wherein the second condition includes: the number of data tables is greater than a first preset value; the number of query fields is greater than a second preset value; the number of query conditions is greater than a third preset value; and the connection relationship between the data tables is a connection relationship between multiple data tables.

3. The method for processing a query request according to claim 1, wherein: Before parsing the query statement from the received query request, the method further includes: Verifying the received query request to obtain a verification result, wherein the verification includes syntax checking and authority verification; If it is determined that the verification result indicates that the syntax of the query request is correct and the target object sending the query request has query authority, then it is determined that the query request passes the verification; If it is determined that the verification result indicates that the syntax of the query request is correct and that the target object sending the query request has query authority, it is determined that the query request has failed verification and verification failure information is sent to the target object.

4. The method for processing a query request according to claim 1, wherein: Obtaining node status evaluation values ​​of the plurality of storage nodes through a node status evaluation model includes: Acquire performance indicator parameters of the storage node, hardware configuration parameters of the storage node, and data block storage locations of the storage node from monitoring data of the storage node; The first evaluation score corresponding to the performance indicator parameter, the second evaluation score corresponding to the hardware configuration parameter, and the third evaluation score corresponding to the data block storage location are input into the node status assessment model, and the node status assessment value output by the node status assessment model is obtained, wherein the node status assessment value represents a weighted sum of the first evaluation score, the second evaluation score, and the third evaluation score.

5. The method for processing a query request according to claim 4, wherein: Acquiring performance indicator parameters of the storage node, hardware configuration parameters of the storage node, and data block storage locations of the storage node from monitoring data of the storage node, including: Acquire initialization configuration parameters of the storage node in the initialization phase and operation configuration parameters of the storage node in the operation phase from the monitoring data, and determine the hardware configuration parameters according to the initialization configuration parameters and the operation configuration parameters; Acquire a data block number corresponding to the node number of the storage node from the monitoring data, and determine the data block storage location of the data block on the storage node if the data block indicated by the data block number has been stored in the storage node; The central processing unit CPU usage, memory usage, network bandwidth occupancy and disk bandwidth occupancy of the storage node are obtained from the monitoring data, and the performance indicator parameters are determined according to the CPU usage, the memory usage, the network bandwidth occupancy and the disk bandwidth occupancy.

6. The method for processing a query request according to claim 5, wherein: The hardware configuration parameters include at least one of the following: CPU model, memory capacity, and disk type. Before inputting the first evaluation score corresponding to the performance indicator parameter, the second evaluation score corresponding to the hardware configuration parameter, and the third evaluation score corresponding to the data block storage location into the node status assessment model, the method further includes: Determine a performance score corresponding to the CPU model in the corresponding performance score interval, determine a first capacity score corresponding to the memory capacity in the corresponding memory capacity score interval, and determine a disk score corresponding to the disk type; Calculating a first weighted sum of the performance score, the first capacity score, and the disk score; The first weighted sum value is normalized according to the preset evaluation score corresponding to the hardware configuration parameter to obtain the second evaluation score.

7. The method for processing a query request according to claim 5, wherein: Before inputting the first evaluation score corresponding to the performance indicator parameter, the second evaluation score corresponding to the hardware configuration parameter, and the third evaluation score corresponding to the data block storage location into the node status assessment model, the method further includes: Determine the number of accesses to the data block within an access period, and determine an access score corresponding to the number of accesses in a corresponding number score interval; Determining a second capacity score corresponding to the block capacity of the data block in the corresponding block capacity score interval; Determining a network transmission parameter between the storage node and the sending node that sends the query request; and determining a transmission score corresponding to the network transmission parameter; The third evaluation score is determined according to a second weighted sum value among the access score, the second capacity score, and the transmission score.

8. The method for processing a query request according to claim 5, wherein: Determining the performance indicator parameter according to the CPU usage, the memory usage, the network bandwidth occupancy, and the disk bandwidth occupancy includes: In each calculation cycle, calculating a third weighted sum of the CPU usage, the memory usage, the network bandwidth occupancy, and the disk bandwidth occupancy; The third weighted sum value is determined as the first evaluation score.

9. The method for processing a query request according to claim 1, wherein: Determining a path node for executing the query request from the plurality of storage nodes according to the query type of the query request and the node status evaluation values ​​of the plurality of storage nodes includes: For the first type, determining a first node from the multiple storage nodes, wherein the first node stores a first data block or a copy of the first data block, and the first data block or the copy of the first data block satisfies the query statement; When the first node state evaluation value corresponding to the first node is lower than a first threshold, the first node is determined as the path node.

10. The method for processing a query request according to claim 1, wherein: Determining a path node for executing the query request from the plurality of storage nodes according to the query type of the query request and the node status evaluation values ​​of the plurality of storage nodes, and constructing a query execution path based on the path node, including: For the second type, determining a second node from the multiple storage nodes, and obtaining a node set based on the second node, wherein the second node stores a second data block or a copy of the second data block, and the second data block or the copy of the second data block satisfies the query statement; Acquire a third node from the node set, the node state evaluation value of which is lower than a second threshold; Selecting a starting node corresponding to the first data table in the query statement from the plurality of third nodes, and sorting the remaining nodes using a heuristic search algorithm to obtain sorted nodes, wherein the remaining nodes are nodes in the plurality of third nodes other than the starting node, and a network bandwidth between every two adjacent nodes in the sorted nodes is lower than a bandwidth threshold, and a data transmission delay between every two adjacent nodes is lower than a delay threshold; The path nodes are determined according to the starting node and the remaining nodes, and a query execution path is constructed according to the starting node and the sorted nodes.

11. The method for processing a query request according to claim 10, wherein: Sending query instructions to all path nodes in the query execution path in order of execution to control all path nodes to collaboratively execute the query instructions, including: Sending the query instruction to a first path node, and determining an execution result of the query instruction executed by the first path node, wherein the first execution node is the starting node; Determining the execution order according to the sorted order of the sorted nodes; The execution result and the query instruction are sent to the next path node in the execution order, so that the next path node continues to execute the query instruction according to the execution result.

12. A query request processing device, characterized in that: include: a parsing module, configured to parse a query statement from a received query request, and determine a query type of the query request based on the query statement, wherein the query type includes one of the following: a first type for performing a single-table query, and a second type for performing a multi-table query; An acquisition module, configured to acquire node status evaluation values ​​of multiple storage nodes through a node status evaluation model; a determination module, configured to determine a path node for executing the query request from the plurality of storage nodes according to a query type of the query request and node status evaluation values ​​of the plurality of storage nodes, and to construct a query execution path based on the path node; The sending module is used to send query instructions to all path nodes in the query execution path in sequence according to the execution order, so as to control all path nodes to collaboratively execute the query instructions.

13. An electronic device, characterized in that: include: memory for storing computer programs; A processor, configured to implement the steps of the method for processing a query request according to any one of claims 1 to 11 when executing the computer program.

14. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the steps of the method for processing a query request according to any one of claims 1 to 11 are implemented.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method for processing a query request according to any one of claims 1 to 11 are implemented.

Images