Distributed query method and related equipment

Abstract

The embodiment of the invention provides a distributed query method and related equipment. The distributed query method comprises the following steps: in response to an SQL task submitted by a submitting end, splitting the SQL task to obtain a plurality of SQL sub-tasks; executing each SQL sub-task in a distributed manner through each cluster node to obtain a corresponding sub-task data result, the cluster nodes being nodes of the computer cluster; and aggregating the subtask results to obtain a final query result. According to the technical scheme, the SQL query task submitted by the submitting end is intelligently split, the complex SQL query task is decomposed into a plurality of subtasks capable of being processed in parallel, and the query speed is remarkably increased through the parallel computing capacity of a computer cluster. When the subtasks are executed, the split SQL subtasks are distributed to a plurality of computing nodes to be executed, parallel execution mechanisms of some cluster frames can process a plurality of tasks at the same time, and therefore the overall processing time is shortened.

Description

Technical Field

[0001] This application relates to the fields of computer and communication technology, and more specifically, to a distributed query method and related equipment. Background Technology

[0002] In current technologies, centralized processing systems may become bottlenecks when handling large-scale security data, leading to longer response times. This is especially true for long-running and complex security data analysis tasks, where latency increases significantly. Furthermore, in centralized systems, computing resources (such as CPU and memory) may be unevenly distributed. For example, some tasks may consume excessive resources, while others may become inefficient due to insufficient resources; this imbalance further impacts overall processing efficiency. Summary of the Invention

[0003] The embodiments of this application provide a distributed query method and related equipment, which can at least to some extent overcome the problem of low processing efficiency of the centralized processing system in the prior art.

[0004] Other features and advantages of this application will become apparent from the following detailed description, or may be learned in part from practice of this application.

[0005] According to one aspect of the embodiments of this application, a distributed query method is provided, applied in a computer cluster. The distributed query method includes: in response to an SQL task submitted by a submitting end, splitting the SQL task into multiple SQL subtasks; distributively executing each of the SQL subtasks through each cluster node to obtain corresponding subtask data results, wherein the cluster node is a node of the computer cluster; and aggregating the results of each subtask to obtain a final query result.

[0006] In some embodiments of this application, the SQL task is split into multiple SQL subtasks, specifically including: preprocessing the SQL task into SQL statement text; inputting the SQL statement text into a task segmentation model to obtain multiple SQL subtasks.

[0007] In some embodiments of this application, the SQL task is split into multiple SQL subtasks, specifically including: obtaining task information of the SQL task; if the task information meets a first splitting condition, splitting the SQL task into multiple SQL subtasks using a first splitting mode; if the task information meets a second splitting condition, splitting the SQL task into multiple SQL subtasks using a second splitting mode.

[0008] In some embodiments of this application, the task information includes the time range of the requested query. If the task information satisfies the first splitting condition, the SQL task is split into multiple SQL subtasks using the first splitting mode. Specifically, this includes: if the time range of the requested query exceeds a predetermined time range threshold, then the first request condition is satisfied; based on the time range of the requested query, the SQL task is split into multiple SQL subtasks in units of time periods.

[0009] In some embodiments of this application, the task information includes the length of the filtering conditions. If the task information satisfies the second splitting condition, the SQL task is split into multiple SQL subtasks using the second splitting mode. Specifically, this includes: if the length of the filtering conditions exceeds a predetermined length threshold, then the second request condition is satisfied; and the SQL task is split into multiple SQL subtasks based on the length of the filtering conditions and the number of cluster nodes.

[0010] In some embodiments of this application, the step of distributing the execution of each SQL subtask across cluster nodes to obtain the corresponding subtask data results specifically includes: analyzing the SQL subtask using an SQL parser to obtain parsing information; converting the parsing information into an executable API call; calling the API interface to directly obtain data; and obtaining the corresponding subtask data results.

[0011] In some embodiments of this application, the step of using an SQL parser to analyze the SQL subtask and obtain parsing information specifically includes: using an SQL parser to analyze the SQL subtask and generate an abstract syntax tree; extracting query information from the abstract syntax tree to form parsing information.

[0012] In some embodiments of this application, the step of converting the parsed information into an executable API call, calling the API interface to directly obtain data, and obtaining the corresponding subtask data result specifically includes: constructing an API call request based on the parsed information, wherein the API call request contains query parameters; sending the API call request to the target API interface to obtain data, and obtaining the corresponding subtask data result.

[0013] In some embodiments of this application, the step of aggregating the results of each subtask to obtain the final query result specifically includes: pre-aggregating the results of each subtask to obtain the pre-aggregated result corresponding to each subtask; and aggregating the pre-aggregated results corresponding to each subtask to obtain the final query result.

[0014] In some embodiments of this application, after the distributed execution of each of the SQL subtasks to obtain the corresponding subtask data results, the method further includes: caching the subtask data results.

[0015] In some embodiments of this application, after aggregating the results of each subtask to obtain the final query result, the method further includes: saving the final query result.

[0016] According to one aspect of the embodiments of this application, a distributed query device is provided, the distributed query device comprising: a task splitting module, configured to split the SQL task in response to a submitted SQL task from a submitting end, to obtain multiple SQL subtasks; a task execution module, configured to execute each of the SQL subtasks in a distributed manner through each cluster node to obtain corresponding subtask data results, wherein the cluster node is a node of the computer cluster; and a result aggregation module, configured to aggregate the results of each subtask to obtain a final query result.

[0017] In some embodiments of this application, the task splitting module specifically includes: a text preprocessing submodule, used to preprocess the SQL task into SQL statement text; and a model segmentation submodule, used to input the SQL statement text into a task segmentation model to obtain multiple SQL subtasks.

[0018] In some embodiments of this application, the task splitting module specifically includes: an information acquisition submodule, used to acquire task information of the SQL task; a first splitting submodule, used to split the SQL task into multiple SQL subtasks in a first splitting mode if the task information meets a first splitting condition; and a second splitting submodule, used to split the SQL task into multiple SQL subtasks in a second splitting mode if the task information meets a second splitting condition.

[0019] In some embodiments of this application, the task information includes the time range of the requested query, and the first splitting submodule specifically includes: a first condition unit, used to satisfy a first request condition if the time range of the requested query exceeds a predetermined time range threshold; and a first splitting unit, used to split the SQL task into multiple SQL subtasks based on the time range of the requested query, using time periods as units.

[0020] In some embodiments of this application, the task information includes the length of the filtering conditions, and the second splitting submodule specifically includes: a second condition unit, used to satisfy the second request condition if the length of the filtering conditions exceeds a predetermined length threshold; and a second splitting unit, used to split the SQL task into multiple SQL subtasks according to the length of the filtering conditions and the number of cluster nodes.

[0021] In some embodiments of this application, the task execution module specifically includes: a parsing and analysis submodule, used to analyze the SQL subtask using an SQL parser to obtain parsing information; and an interface call submodule, used to convert the parsing information into an executable API call, call the API interface to directly obtain data, and obtain the corresponding subtask data result.

[0022] In some embodiments of this application, the parsing and analysis submodule specifically includes: a task parsing unit, used to analyze the SQL subtask using an SQL parser to generate an abstract syntax tree; and an information extraction unit, used to extract query information from the abstract syntax tree to form parsing information.

[0023] In some embodiments of this application, the interface call submodule specifically includes: a request construction unit, used to construct an API call request based on the parsed information, wherein the API call request includes query parameters; and a request sending unit, used to send the API call request to the target API interface to obtain data and obtain the corresponding subtask data result.

[0024] In some embodiments of this application, the result aggregation module specifically includes: a result pre-aggregation module, used to pre-aggregate the results of each subtask to obtain the pre-aggregated result corresponding to each subtask; and a result aggregation module, used to aggregate the pre-aggregated results corresponding to each subtask to obtain the final query result.

[0025] In some embodiments of this application, the distributed query device further includes a result caching unit for caching the subtask data results.

[0026] In some embodiments of this application, the distributed query device further includes a result storage unit for storing the final query result.

[0027] According to one aspect of the embodiments of this application, a computer-readable medium is provided having a computer program stored thereon, which, when executed by a processor, implements the distributed query method as described in the above embodiments.

[0028] According to one aspect of the embodiments of this application, an electronic device is provided, including: one or more processors; and a storage device for storing one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement the distributed query method as described in the above embodiments.

[0029] According to one aspect of the embodiments of this application, a computer program product is provided, including one or more computer programs that, when executed by one or more processors, implement the steps of the distributed query method as described in the above embodiments.

[0030] In some embodiments of this application, the technical solutions intelligently split the SQL query tasks submitted by the submitting end. Complex SQL query tasks are decomposed into multiple subtasks that can be processed in parallel, thereby significantly improving query speed by leveraging the parallel computing capabilities of computer clusters. When executing subtasks, the split SQL subtasks are distributed across multiple computing nodes for execution. The parallel execution mechanisms of some cluster frameworks can handle multiple tasks simultaneously, thereby shortening the overall processing time. Simultaneously, utilizing distributed processing with computer clusters allows for dynamic allocation of computing resources and flexible adjustment of resource allocation, improving resource utilization. Therefore, the technical solutions of this application solve the problem of low processing efficiency in existing centralized processing systems.

[0031] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0032] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort. In the drawings:

[0033] Figure 1 An exemplary implementation environment diagram is shown, in which the technical solutions of the embodiments of this application can be applied.

[0034] Figure 2 A flowchart illustrating a distributed query method provided in an embodiment of this application is shown.

[0035] Figure 3 It shows that according to Figure 2 A flowchart illustrating a specific implementation of step S100 in the distributed query method shown in the corresponding embodiment.

[0036] Figure 4 It shows that according to Figure 3 A flowchart illustrating a specific implementation of step S200 in the distributed query method shown in the corresponding embodiment.

[0037] Figure 5 A schematic diagram of the structure of a distributed query device provided in an embodiment of this application is shown.

[0038] Figure 6 A schematic diagram of the structure of a computer system for an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0039] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided to make this application more comprehensive and complete, and to fully convey the concept of the exemplary embodiments to those skilled in the art.

[0040] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this application. However, those skilled in the art will recognize that the technical solutions of this application can be practiced without one or more of the specific details, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this application.

[0041] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0042] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0043] Figure 1 A schematic diagram of an exemplary system architecture to which the technical solutions of the embodiments of this application can be applied is shown.

[0044] like Figure 1 As shown, the system architecture may include terminal devices (such as...) Figure 1 The device shown includes one or more of a smartphone 101, tablet 102, and portable computer 103 (which could also be a desktop computer, etc.), a network 104, and a server 105. The network 104 serves as a medium for providing a communication link between the terminal device and the server 105. The network 104 can include various connection types, such as wired communication links, wireless communication links, etc.

[0045] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included. For example, server 105 could be a server cluster composed of multiple servers. Generally, this server cluster is a local cluster.

[0046] Users can interact with server 105 via network 104 using terminal devices to receive or send messages, etc. Server 105 can be a server providing various services. For example, a user can upload an SQL task to server 105 using terminal device 103 (or terminal device 101 or 102). Server 105 can respond to the submitted SQL task by splitting it into multiple SQL subtasks; executing each SQL subtask in a distributed manner through cluster nodes, where the cluster nodes are the nodes of the computer cluster; and aggregating the results of each subtask to obtain the final query result.

[0047] It should be noted that the distributed query method provided in this application embodiment is generally executed by server 105, and correspondingly, the distributed query device is generally set in server 105. However, in other embodiments of this application, the terminal device may also have similar functions to the server, thereby executing the distributed query scheme provided in this application embodiment.

[0048] The implementation details of the technical solutions in the embodiments of this application are described in detail below:

[0049] Figure 2 A flowchart of a distributed query method according to an embodiment of this application is shown. This distributed query method can be executed by a server, which may be... Figure 1 The server shown. (Refer to...) Figure 2 As shown, this distributed query method includes at least:

[0050] S100, in response to the SQL task submitted by the submitting end, the SQL task is split into multiple SQL subtasks.

[0051] S200, each SQL subtask is executed in a distributed manner through each cluster node to obtain the corresponding subtask data results, wherein the cluster node is a node of the computer cluster.

[0052] S300: Aggregate the results of each subtask to obtain the final query result.

[0053] In the embodiments of this application, the SQL query task submitted by the submitting end is intelligently split. Complex SQL query tasks are decomposed into multiple subtasks that can be processed in parallel, significantly improving query speed by leveraging the parallel computing capabilities of the computer cluster. When executing subtasks, the split SQL subtasks are distributed across multiple computing nodes. The parallel execution mechanisms of some cluster frameworks can handle multiple tasks simultaneously, thereby shortening the overall processing time. Simultaneously, utilizing distributed processing with the computer cluster allows for dynamic allocation of computing resources and flexible adjustment of resource allocation, improving resource utilization. Therefore, the technical solution of this application solves the problem of low processing efficiency in existing centralized processing systems.

[0054] It should be noted that the computer clusters used in the embodiments of this application are distributed clusters, such as Ray Cluster, Hadoop YARN, Apache Spark, etc., and this application is not limited thereto. The following embodiments will all be described using Ray Cluster as an example.

[0055] In step S100, the SQL query task submitted by the submitting end is intelligently split. The complex SQL query task is decomposed into multiple subtasks that can be processed in parallel, so as to significantly improve the query speed by utilizing the parallel computing capabilities of the computer cluster.

[0056] Specifically, in some embodiments, the specific implementation of step S100 can be found in the following embodiments. This embodiment is based on... Figure 2 According to the detailed description of step S100 in the distributed query method shown in the corresponding embodiment, step S100 in the distributed query method may include the following steps:

[0057] The SQL task is preprocessed into SQL statement text.

[0058] The SQL statement text input task segmentation model is used to obtain multiple SQL subtasks.

[0059] In this embodiment, the SQL task is preprocessed into SQL statement text, and then input into a neural network model (i.e., a task segmentation model) to segment the SQL task.

[0060] Specifically, in some embodiments, the task segmentation model may include a recurrent attention network and a language segmentation network. When performing task segmentation, the SQL statement text can be input into the recurrent attention network to obtain multiple window vectors and sentence compression vectors. Then, all the window vectors are concatenated to form a concatenated vector. The concatenated vector and the sentence compression vector are input together into the language segmentation network to obtain multiple SQL subtasks.

[0061] Since the task segmentation model is trained on a large number of samples, it can be maintained simply by adjusting the samples. Compared with other prediction methods, the maintenance cost is lower. Basically, it only needs to maintain the SQL statement text, which reduces maintenance costs and improves code stability.

[0062] Specifically, the training methods for the above-mentioned task segmentation model include:

[0063] Obtain the SQL statement text sample set, which contains multiple SQL statement text samples, each of which is labeled with a corresponding SQL subtask tag.

[0064] The SQL statement text samples are input one by one into the task segmentation model to obtain the SQL subtask results.

[0065] Based on the output SQL subtask results and the SQL subtask labels, the parameters of the task segmentation model are updated until the predetermined termination condition is met, thus ending the training and obtaining the trained task segmentation model.

[0066] In the embodiments of this application, during training, a set of SQL statement text samples containing multiple SQL statement text samples can be obtained first, and each SQL statement text sample is labeled with a corresponding SQL subtask label; then, the multiple SQL statement text samples are divided into a training set, a validation set, and a test set according to a predetermined ratio; then, the parameters of the encoder and decoder in the task segmentation model are adjusted and determined according to the SQL statement text samples included in the training set, validation set, and test set, and the trained task segmentation model is obtained.

[0067] When training the model, the SQL statement text sample set can be divided into a training set, a validation set, and a test set. Then, the model is trained based on the training set, validated based on the validation set, and tested based on the test set to obtain a trained task segmentation model.

[0068] Before training on the training set, the SQL statement text samples in the training set can be preprocessed. Preprocessing includes vector adjustment, normalization, data augmentation, and class encoding.

[0069] After obtaining the enhanced training set, the task segmentation model can be trained based on the enhanced training set, and the parameters and weights in the network can be updated.

[0070] Specifically, the SQL statement text samples from the training set are input into the task segmentation model to obtain the SQL subtask results output by the model. The SQL subtask results are compared with their labels, and the loss function is calculated. Then, the stochastic gradient descent method is used to minimize the loss function. The parameters and weights in the task segmentation model are updated via backpropagation until the loss function meets predetermined conditions, such as convergence or falling below a predetermined threshold.

[0071] In some embodiments, the SQL subtask result includes a target contour and target contour parameters, and the SQL subtask label may include at least one of a contour label and a contour parameter label. The loss function is calculated by comparing the target contour with the contour label and / or comparing the target contour parameters with the contour parameter label.

[0072] After training, the image SQL subtask network with updated parameters from the training set can be validated using a validation set. Specifically, the image SQL subtask network is debugged based on the validation set data. When the loss function meets predetermined conditions, the model parameters for that stage are output. If the loss function does not meet the predetermined conditions, hyperparameters such as the learning rate are automatically adjusted, and the next round of network model training is performed.

[0073] When the loss function computed on the validation set meets predetermined conditions, the parameters and weights can be retained, and then the retained parameters are tested based on the test set. Specifically, the input test set data enables the image SQL subtask network with retained parameters and weights to output the SQL subtask results and model weights. By comparing the model loss and corresponding weights across multiple rounds, the model weights with the minimum loss are output, thus determining the trained task segmentation model.

[0074] After obtaining the trained task segmentation model, SQL subtasks can be completed based on the task segmentation model.

[0075] Furthermore, data augmentation is not required for the input data in the validation and test sets.

[0076] Specifically, in other embodiments, the specific implementation of step S100 can be found in [reference needed]. Figure 3 . Figure 3 It is based on Figure 2 According to the detailed description of step S100 in the distributed query method shown in the corresponding embodiment, step S100 in the distributed query method may include the following steps:

[0077] S110, Obtain the task information of the SQL task.

[0078] S120, if the task information meets the first splitting condition, the SQL task is split in the first splitting mode to obtain multiple SQL subtasks.

[0079] S130, if the task information meets the second splitting condition, the SQL task is split in the second splitting mode to obtain multiple SQL subtasks.

[0080] In this embodiment, when splitting SQL tasks, the splitting conditions are first determined. For SQL tasks that meet different conditions, different splitting modes are used to achieve intelligent task splitting. On the one hand, complex SQL query tasks are decomposed into multiple parallel processing subtasks, which can significantly improve query speed by leveraging the parallel computing capabilities of computer clusters such as Ray Cluster. On the other hand, splitting complex SQL tasks using targeted splitting modes according to their own characteristics can also better balance the load among nodes, thereby improving task execution efficiency and achieving efficient, flexible, and secure data processing.

[0081] In S110, the task information for an SQL task may include the time range of the requested query, the database range to be queried, the length of the filter conditions, and the number of filter conditions. Based on this task information, condition judgments are made, and then, according to the conditions satisfied by the SQL task, the corresponding splitting mode is selected to split the task.

[0082] In S120, the first condition can be met by the following: the time range of the requested query exceeds the predetermined time range threshold, the database range of the requested query exceeds the predetermined database range threshold, the length of the filter condition exceeds the predetermined length threshold, or the number of filter conditions exceeds the predetermined number threshold. The following explanation will take the example of the time range of the requested query exceeding the predetermined time range threshold.

[0083] Specifically, in some embodiments, the specific implementation of step S120 can be found in the following embodiments. This embodiment is based on... Figure 3 In the detailed description of step S120 in the distributed query method shown in the corresponding embodiment, the task information in the distributed query method includes the time range of the requested query, and step S120 may include the following steps:

[0084] If the time range requested for the query exceeds the predetermined time range threshold, then the first request condition is met.

[0085] Based on the time range of the requested query, the SQL task is split into multiple SQL subtasks, using time periods as units.

[0086] In this embodiment, the predetermined time range threshold needs to be determined based on the actual situation such as the amount of data in the database, the number of nodes, and the processing capacity of each node. It can also be set by the user, and can be a quarter, half a year, or a year, etc. This application limits this step.

[0087] When the time range of a requested query exceeds a predetermined time range threshold, the corresponding splitting condition is met. At this point, the SQL task can be split into multiple subtasks based on the requested time range, each subtask corresponding to a specific time segment of the query. Specifically, for example, an SQL task requesting a year can be split by month or quarter; an SQL task requesting half a year can be split by month or ten-day period; and an SQL task requesting a month can be split by week or by time segment such as 5 or 6 days.

[0088] Generally, data in a database is evenly distributed over time. When the requested time range is too large, the large time range is broken down into multiple smaller time segments, forming multiple small subtasks. Each subtask is distributed across multiple computing nodes and executed in parallel, which speeds up the analysis of long-term and complex security data and shortens the response time.

[0089] In other embodiments, the SQL task can also be split based on the number of cluster nodes. That is, based on the number of cluster nodes and the time range of the query request, the time range corresponding to each subtask is obtained, and the SQL task is split so that the split subtasks can be more easily distributed to each computing node. This means that computing resources are allocated more intelligently, computing resources are dynamically allocated according to the actual needs of the task, the load of each node is balanced, efficiency and resource utilization are balanced, resource fragmentation and waste are avoided, and the overall processing time is shortened.

[0090] In other embodiments, the load of each cluster node can be further considered when splitting the SQL task, so that the split subtasks can be more evenly distributed to each computing node, thereby further shortening the overall processing time.

[0091] In S130, the second condition can also be satisfied if the time range of the requested query exceeds a predetermined time range threshold, the database range of the requested query exceeds a predetermined database range threshold, the length of the filter condition exceeds a predetermined length threshold, or the number of filter conditions exceeds a predetermined number threshold, etc. However, the second condition and the first condition cannot be the same. The following explanation uses the example of the length of the filter condition exceeding the predetermined length threshold.

[0092] Specifically, in some embodiments, the specific implementation of step S130 can be found in the following embodiments. This embodiment is based on... Figure 3In the detailed description of step S130 in the distributed query method shown in the corresponding embodiment, the task information includes the length of the filtering conditions, and step S130 may include the following steps:

[0093] If the length of the filtering criteria exceeds the predetermined length threshold, then the second request condition is satisfied.

[0094] The SQL task is split into multiple SQL subtasks based on the length of the filtering criteria and the number of cluster nodes.

[0095] In this embodiment, the predetermined length threshold needs to be determined based on the actual situation such as the number of nodes and the processing capacity of each node. It can also be set by the user and can be 2T, 3T, 5T, 8T, etc. This application limits it in this step.

[0096] When the length of the filter condition exceeds the predetermined length threshold, the corresponding splitting condition is met. At this time, the SQL task can be split according to the length of the filter condition to obtain multiple subtasks, each of which corresponds to a WHERE clause.

[0097] Generally speaking, a WHERE clause can correspond to a small query task. When the filtering conditions are too long, the complex filtering conditions are broken down into multiple smaller filtering conditions, forming multiple small subtasks. Each subtask is distributed and executed in parallel on multiple computing nodes, which speeds up the analysis of long-term, complex security data and shortens the response time.

[0098] In other embodiments, the SQL task can also be split based on the number of cluster nodes. That is, the corresponding WHERE clause is obtained according to the number of cluster nodes and the length of the filtering conditions, and then the SQL task is split so that the split subtasks can be more easily distributed to each computing node. That is, computing resources are allocated more intelligently, the load of each node is balanced, efficiency and resource utilization are balanced, thereby shortening the overall processing time.

[0099] In other embodiments, the load of each cluster node can be further considered when splitting the SQL task, so that the split subtasks can be more evenly distributed to each computing node, thereby further shortening the overall processing time.

[0100] It should be noted that the splitting conditions in step S100 are not limited to two, but can include multiple conditions, and each splitting condition must be different.

[0101] For example, if the range of databases requested for querying exceeds a predetermined database range threshold, the SQL task for different databases can be split into corresponding SQL subtasks based on the databases being queried. Alternatively, it can be further split by WHERE clause to make the split subtasks easier to distribute evenly among the computing nodes, i.e., to allocate computing resources more intelligently, balance the load of each node, balance efficiency and resource utilization, and thus shorten the overall processing time.

[0102] In step S200, the split SQL subtasks are distributed to multiple computing nodes for execution. Ray's parallel execution mechanism can handle multiple tasks simultaneously, thus shortening the overall processing time. Furthermore, Ray Cluster has a built-in intelligent scheduling mechanism; the Ray framework can flexibly adjust resource allocation based on task priority and resource requirements to ensure efficient task execution and avoid resource fragmentation and waste.

[0103] Specifically, in some embodiments, the specific implementation of step S200 can be found in [reference needed]. Figure 4 . Figure 4 It is based on Figure 2 According to the detailed description of step S200 in the distributed query method shown in the corresponding embodiment, step S200 in the distributed query method may include the following steps:

[0104] S210, Use the SQL parser to analyze the SQL subtask and obtain the parsing information.

[0105] S220, the parsed information is converted into an executable API call, and the API interface is called to directly obtain the data and get the corresponding subtask data results.

[0106] In this embodiment, for a cluster node, an SQL parser can be used to convert SQL query tasks into executable API calls. By calling the API interface, data can be obtained directly, bypassing the database layer, reducing the burden on traditional databases, thereby improving efficiency. At the same time, it can also ensure the security and reliability of data transmission.

[0107] For example, data can be retrieved via a RESTful API, using HTTP / HTTPS protocols for data transmission. This API should be designed to support high-concurrency data requests.

[0108] In S210, the key information for executing the SQL query task is first extracted from the SQL subtask to construct the API call.

[0109] Specifically, in some embodiments, the specific implementation of step S210 can be found in the following embodiments. This embodiment is based on... Figure 4According to the detailed description of step S210 in the distributed query method shown in the corresponding embodiment, step S210 in the distributed query method may include the following steps:

[0110] The SQL subtask is analyzed using an SQL parser to generate an abstract syntax tree;

[0111] The query information is extracted from the abstract syntax tree to form the parsing information.

[0112] In this embodiment, the SQL query task is transformed into a structured tree representation, which enables subsequent processing to understand the logical structure of the query task more efficiently. It also facilitates the extraction of key information from the abstract syntax tree, such as table names, field names, filtering conditions, sorting requirements, and other parsing information, for constructing API call requests.

[0113] In S220, direct access to the data source via API reduces the burden of real-time database queries. Simultaneously, retrieving data from the API interface ensures data integrity and reliability, guaranteeing that data is not lost or corrupted during transmission.

[0114] Specifically, in some embodiments, the specific implementation of step S220 can be found in the following embodiments. This embodiment is based on... Figure 4 According to the detailed description of step S220 in the distributed query method shown in the corresponding embodiment, step S220 in the distributed query method may include the following steps:

[0115] An API call request is constructed based on the parsed information, and the API call request includes query parameters.

[0116] The API call request is sent to the target API interface to retrieve data, and the corresponding subtask data result is obtained.

[0117] In this embodiment, based on the parsing information extracted from the SQL query task, an API call request containing necessary query parameters is constructed to ensure that the call request is accurately mapped to the requirements of the SQL query, thereby bypassing the database layer to directly call the API interface and obtain the corresponding subtask data results.

[0118] In some embodiments of this application, after step S200, the method further includes:

[0119] The results of the subtasks are cached.

[0120] In this embodiment, the data retrieval results (i.e., the subtask data results) are cached. When the cache is hit, it can be used directly to avoid repeated queries and reduce the real-time query burden on the database.

[0121] In some embodiments, distributed caching systems such as Redis and Memcached can be used to store the pre-aggregated results. Specific caching strategies can include TTL (Time To Live) and LRU (Least Recently Used) caching strategies.

[0122] Specifically, after caching the data results of the subtask, when executing subsequent SQL subtasks, the system first checks whether the cache contains the required results. If the cache is hit, the cached result is returned directly; otherwise, the SQL subtask continues to be executed. This embodiment can reduce system load and latency, significantly improve query speed, and reduce the number of database or API calls.

[0123] It should be noted that the aforementioned computer cluster can also be a blockchain cluster, with each cluster node acting as a blockchain node. Each blockchain node processes different SQL subtasks, obtains the corresponding subtask results, and then aggregates these results into the final result. The immutability of the blockchain guarantees the authenticity and integrity of all processing steps and results. The execution and recording of all tasks are public and transparent, enhancing system trust, reducing reliance on centralized servers, and improving system robustness and fault tolerance. Simultaneously, the encryption properties of the blockchain enhance data security, preventing malicious tampering and unauthorized access.

[0124] In step S300, after obtaining the results of each SQL subtask after execution, the results of each subtask need to be aggregated to obtain the final query result.

[0125] Specifically, in some embodiments, the specific implementation of step S300 can be found in the following embodiments. This embodiment is based on... Figure 2 According to the detailed description of step S300 in the distributed query method shown in the corresponding embodiment, step S300 in the distributed query method may include the following steps:

[0126] The results of each subtask are pre-aggregated to obtain the pre-aggregated results for each subtask.

[0127] The pre-aggregated results corresponding to each subtask are aggregated to obtain the final query result.

[0128] In this embodiment, the results of each SQL subtask are first pre-aggregated to form corresponding pre-aggregated results. Then, all pre-aggregated results are aggregated to obtain the final query result, ensuring the accuracy and consistency of data processing results even under high concurrency. Simultaneously, the staged aggregation method effectively integrates the results of parallel processing, avoiding data conflicts and inconsistencies.

[0129] Specifically, the pre-polymerization steps may include:

[0130] The SQL subtasks corresponding to the data obtained from the API are sharded to enable more efficient merging during the parallel processing phase. Preliminary statistical and merging operations are performed on each data shard, such as calculating intermediate results (e.g., intermediate sums, averages, etc.), awaiting further processing in the final aggregation phase. This reduces the data volume to a smaller range, alleviating the burden of subsequent computations and accelerating data processing and query response.

[0131] Specifically, the specific steps of aggregation may include:

[0132] After obtaining the pre-aggregation results corresponding to each subtask, all pre-aggregation results are first summarized, and the final statistical value or summary information is calculated. Then, complex aggregation operations are processed, such as calculating the sum, average, maximum, and minimum values, to ensure the completeness and accuracy of the results. Finally, the completeness of the aggregation results is checked to ensure that all data has been correctly merged.

[0133] In some other embodiments, Ray's distributed computing capabilities can be leveraged to improve the efficiency of large-scale data processing.

[0134] In some embodiments of this application, after step S300, the method further includes:

[0135] Save the final query results.

[0136] In this embodiment, the aggregated result (i.e. the final query result) is saved for subsequent access and use, ensuring data consistency during the processing and avoiding data loss or duplicate calculations.

[0137] Specifically, the final query results can be stored in a distributed storage system, such as a relational database, NoSQL database, or data warehouse (e.g., Amazon S3, Google BigQuery), and indexes can be created for the stored results to support efficient querying and retrieval. When creating indexes, the database's indexing functionality or a distributed indexing system can be used. Using a distributed storage system ensures the long-term preservation and availability of query results, improving the speed and efficiency of data retrieval.

[0138] The following describes an apparatus embodiment of this application, which can be used to execute the distributed query method in the above embodiments of this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the distributed query method described above.

[0139] Figure 5 A block diagram of a distributed query apparatus according to an embodiment of this application is shown.

[0140] Reference Figure 5As shown, a distributed query device 500 according to an embodiment of this application includes: a task splitting module 510, a task execution module 520, and a result aggregation module 530.

[0141] The task splitting module 510 is used to split the SQL task submitted by the submitting end into multiple SQL subtasks in response to the SQL task submitted by the submitting end; the task execution module 520 is used to execute each of the SQL subtasks in a distributed manner through each cluster node to obtain the corresponding subtask data results, wherein the cluster node is the node of the computer cluster; and the result aggregation module 530 is used to aggregate the results of each of the subtasks to obtain the final query result.

[0142] In some embodiments of this application, the task splitting module specifically includes: a text preprocessing submodule, used to preprocess the SQL task into SQL statement text; and a model segmentation submodule, used to input the SQL statement text into a task segmentation model to obtain multiple SQL subtasks.

[0143] In some embodiments of this application, the task splitting module specifically includes: an information acquisition submodule, used to acquire task information of the SQL task; a first splitting submodule, used to split the SQL task into multiple SQL subtasks in a first splitting mode if the task information meets a first splitting condition; and a second splitting submodule, used to split the SQL task into multiple SQL subtasks in a second splitting mode if the task information meets a second splitting condition.

[0144] In some embodiments of this application, the task information includes the time range of the requested query, and the first splitting submodule specifically includes: a first condition unit, used to satisfy a first request condition if the time range of the requested query exceeds a predetermined time range threshold; and a first splitting unit, used to split the SQL task into multiple SQL subtasks based on the time range of the requested query, using time periods as units.

[0145] In some embodiments of this application, the task information includes the length of the filtering conditions, and the second splitting submodule specifically includes: a second condition unit, used to satisfy the second request condition if the length of the filtering conditions exceeds a predetermined length threshold; and a second splitting unit, used to split the SQL task into multiple SQL subtasks according to the length of the filtering conditions and the number of cluster nodes.

[0146] In some embodiments of this application, the task execution module specifically includes: a parsing and analysis submodule, used to analyze the SQL subtask using an SQL parser to obtain parsing information; and an interface call submodule, used to convert the parsing information into an executable API call, call the API interface to directly obtain data, and obtain the corresponding subtask data result.

[0147] In some embodiments of this application, the parsing and analysis submodule specifically includes: a task parsing unit, used to analyze the SQL subtask using an SQL parser to generate an abstract syntax tree; and an information extraction unit, used to extract query information from the abstract syntax tree to form parsing information.

[0148] In some embodiments of this application, the interface call submodule specifically includes: a request construction unit, used to construct an API call request based on the parsed information, wherein the API call request includes query parameters; and a request sending unit, used to send the API call request to the target API interface to obtain data and obtain the corresponding subtask data result.

[0149] In some embodiments of this application, the result aggregation module specifically includes: a result pre-aggregation module, used to pre-aggregate the results of each subtask to obtain the pre-aggregated result corresponding to each subtask; and a result aggregation module, used to aggregate the pre-aggregated results corresponding to each subtask to obtain the final query result.

[0150] In some embodiments of this application, the distributed query device further includes a result caching unit for caching the subtask data results.

[0151] In some embodiments of this application, the distributed query device further includes a result storage unit for storing the final query result.

[0152] In the embodiments of this application, the SQL query task submitted by the submitting end is intelligently split. Complex SQL query tasks are decomposed into multiple subtasks that can be processed in parallel, significantly improving query speed by leveraging the parallel computing capabilities of the computer cluster. When executing subtasks, the split SQL subtasks are distributed across multiple computing nodes. The parallel execution mechanisms of some cluster frameworks can handle multiple tasks simultaneously, thereby shortening the overall processing time. Simultaneously, utilizing distributed processing with the computer cluster allows for dynamic allocation of computing resources and flexible adjustment of resource allocation, improving resource utilization. Therefore, the technical solution of this application solves the problem of low processing efficiency in existing centralized processing systems.

[0153] Figure 6 A schematic diagram of the structure of a computer system suitable for implementing the electronic device of the present application is shown.

[0154] It should be noted that, Figure 6 The computer system of the electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0155] like Figure 6 As shown, the computer system includes a Central Processing Unit (CPU) 1801, which can perform various appropriate actions and processes based on programs stored in Read-Only Memory (ROM) 1802 or programs loaded from storage portion 1808 into Random Access Memory (RAM) 1803, such as performing the methods described in the above embodiments. The RAM 1803 also stores various programs and data required for system operation. The CPU 1801, ROM 1802, and RAM 1803 are interconnected via a bus 1804. An Input / Output (I / O) interface 1805 is also connected to the bus 1804.

[0156] The following components are connected to I / O interface 1805: an input section 1806 including a keyboard, mouse, etc.; an output section 1807 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 1808 including a hard disk, etc.; and a communication section 1809 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 1809 performs communication processing via a network such as the Internet. A drive 1810 is also connected to I / O interface 1805 as needed. Removable media 1811, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 1810 as needed so that computer programs read from them can be installed into storage section 1808 as needed.

[0157] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program including a computer program for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 1809, and / or installed from removable medium 1811. When the computer program is executed by central processing unit (CPU) 1801, it performs various functions defined in the system of this application.

[0158] It should be noted that the computer-readable medium shown in the embodiments of this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying a computer-readable computer program. The transmitted data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0159] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0160] The units described in the embodiments of this application can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.

[0161] In another aspect, this application also provides a computer-readable medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the methods described in the above embodiments.

[0162] This specification also provides a computer program product that stores at least one instruction, said at least one instruction being loaded and executed by the processor as described above. Figures 1-4 The method described in the illustrated embodiment can be found in the following document for a detailed execution process. Figures 1-4 The specific details of the illustrated embodiments will not be elaborated here.

[0163] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0164] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, touch terminal, or network device, etc.) to execute the method according to the embodiments of this application.

[0165] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the embodiments disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein.

[0166] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A distributed query method, characterized in that, When applied in a computer cluster, the distributed query method includes: In response to the SQL task submitted by the submitting end, the SQL task is split into multiple SQL subtasks; Each SQL subtask is executed in a distributed manner on each cluster node to obtain the corresponding subtask data results, wherein the cluster node is a node of the computer cluster; The results of each subtask are aggregated to obtain the final query result.

2. The distributed query method as described in claim 1, characterized in that, The process of splitting the SQL task into multiple SQL subtasks specifically includes: The SQL task is preprocessed into SQL statement text; The SQL statement text input task segmentation model is used to obtain multiple SQL subtasks.

3. The distributed query method as described in claim 1, characterized in that, The process of splitting the SQL task into multiple SQL subtasks specifically includes: Obtain the task information of the SQL task; If the task information meets the first splitting condition, the SQL task is split in the first splitting mode to obtain multiple SQL subtasks; If the task information meets the second splitting condition, the SQL task is split into multiple SQL subtasks using the second splitting mode.

4. The distributed query method as described in claim 3, characterized in that, The task information includes the time range of the requested query. If the task information meets the first splitting condition, the SQL task is split according to the first splitting mode to obtain multiple SQL subtasks, specifically including: If the time range requested for the query exceeds the predetermined time range threshold, then the first request condition is met; Based on the time range of the requested query, the SQL task is split into multiple SQL subtasks, using time periods as units.

5. The distributed query method as described in claim 3, characterized in that, The task information includes the length of the filtering conditions. If the task information meets the second splitting condition, the SQL task is split according to the second splitting mode to obtain multiple SQL subtasks, specifically including: If the length of the filtering criteria exceeds the predetermined length threshold, then the second request condition is met; The SQL task is split into multiple SQL subtasks based on the length of the filtering criteria and the number of cluster nodes.

6. The distributed query method as described in claim 1, characterized in that, The step of distributing the execution of each SQL subtask across cluster nodes to obtain the corresponding subtask data results specifically includes: The SQL subtask is analyzed using an SQL parser to obtain parsing information; The parsed information is converted into an executable API call, which directly retrieves the data and obtains the corresponding subtask data results.

7. The distributed query method as described in claim 6, characterized in that, The process of using an SQL parser to analyze the SQL subtask and obtain parsing information specifically includes: The SQL subtask is analyzed using an SQL parser to generate an abstract syntax tree; The query information is extracted from the abstract syntax tree to form the parsing information.

8. A distributed query device, characterized in that, The distributed query device includes: The task splitting module is used to split the SQL task submitted by the submitting end into multiple SQL subtasks in response to the SQL task submitted by the submitting end. The task execution module is used to execute each of the SQL subtasks in a distributed manner through each cluster node to obtain the corresponding subtask data results, wherein the cluster node is a node of the computer cluster; The result aggregation module is used to aggregate the results of each subtask to obtain the final query result.

9. A computer-readable medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the distributed query method as described in any one of claims 1 to 7.

10. An electronic device, characterized in that, include: One or more processors; A storage device for storing one or more programs, which, when executed by one or more processors, cause the one or more processors to implement the distributed query method as described in any one of claims 1 to 7.

Images