Data processing method and system, electronic device, and storage medium

Abstract

The present disclosure provides a data processing method and system, an electronic device, and a storage medium. The method comprises: monitoring for a data query request, the data query request being used to request querying of data to be executed; in response to the data query request, allocating a target scheduling time to each queue on the basis of resource configuration information of the queue, the resource configuration information being used to characterize processing resources allowed to be scheduled for the queue during operation; determining time difference information between the target scheduling time of the queue and an already scheduled time of the queue, the time difference information being used to indicate a difference between the target scheduling time and the already scheduled time; on the basis of the time difference information of each queue, determining a target queue among a plurality of queues, wherein the time difference information corresponding to the target queue is greater than a difference information threshold; and querying the data to be executed from the target queue.

Description

Data processing method, system, electronic device and storage medium

[0001] Cross-reference to related applications

[0002] The present disclosure claims priority from a Chinese patent application No. 202411231285.X filed on September 3, 2024, and entitled "Data processing method, system, electronic device and storage medium", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0003] The present disclosure relates to the field of computers, and in particular, to a data processing method, system, electronic device and storage medium. BACKGROUND

[0004] At present, with the continuous development of the data era, the processing of data tends to integrate various different database servers into a unified database system (shared database system). In the shared database system, the simultaneously running data query requests often share the central processing unit (CPU), memory, disk input / output (I / O) and other resources in the database system, which means that multiple data query requests will compete for system resources during execution. If some data query requests consume huge resources without being limited, other concurrently running data query requests will have to compete for resources with the above data query requests, thereby affecting the overall performance of the database system.

[0005] In related technologies, in the data warehouse to which the database system belongs, the Small Query Acceleration (SQA) function can be used to split the queue of small queries (data query requests with short execution time and less resource consumption) to improve the speed of data processing. However, small queries can still be affected by resource contention and competition from large queries (data query requests with long execution time and more resource consumption), resulting in the execution of small queries being affected. Therefore, there is still the technical problem of low efficiency of data processing.

[0006] At present, there is no effective solution to the above problems. SUMMARY

[0007] The embodiments of the present disclosure provide a data processing method, system, electronic device and storage medium to at least solve the technical problem of low efficiency of data processing.

[0008] According to an aspect of embodiments of the present disclosure, a data processing method is provided. The method can include: monitoring a data query request, wherein the data query request is used to request to query to-be-executed data; in response to the data query request, allocating a target scheduling time to a queue based on resource configuration information of the queue, wherein the resource configuration information is used to represent processing resources allowed to be scheduled to the queue in a running process, and the target scheduling time is used to represent a time allocated to the queue; determining time difference information between the target scheduling time of the queue and a scheduled time of the queue, wherein the time difference information is used to represent a difference between the target scheduling time and the scheduled time; determining a target queue from a plurality of queues based on the time difference information of the queue, wherein the target queue corresponds to time difference information greater than a difference information threshold; and querying the to-be-executed data from the target queue.

[0009] According to another aspect of embodiments of the present disclosure, a data processing method is provided. The method applied to an execution end can include: monitoring a data query request from a thread, wherein the data query request is used to request to query to-be-executed data; in response to the data query request, allocating a target scheduling time to a queue based on resource configuration information of the queue in a data warehouse, wherein the resource configuration information is used to represent processing resources allowed to be scheduled to the queue in a running process, the target scheduling time is used to represent a time allocated to the queue, and the data warehouse includes a plurality of queues; determining time difference information between the target scheduling time of the queue and a scheduled time of the queue, wherein the time difference information is used to represent a difference between the target scheduling time and the scheduled time; determining a target queue from the plurality of queues based on the time difference information of the queue, wherein the target queue corresponds to time difference information greater than a difference information threshold; querying the to-be-executed data from the target queue; and returning the queried to-be-executed data to the thread execution.

[0010] According to a further aspect of the embodiments of the present disclosure, a method for processing data is provided. The method can include: monitoring a data query request by invoking a first interface, wherein the data query request is used to request to query to-be-executed data, and the first interface includes a first parameter, and a parameter value of the first parameter is the data query request; in response to the data query request, allocating a target scheduling time to a queue based on resource configuration information of the queue, wherein the resource configuration information is used to represent processing resources allowed to be scheduled to the queue in a running process, and the target scheduling time is used to represent a time allocated to the queue; determining time difference information between the target scheduling time of the queue and a scheduled time of the queue, wherein the time difference information is used to represent a difference between the target scheduling time and the scheduled time; determining a target queue from a plurality of queues based on the time difference information of the queue, wherein the target queue corresponds to time difference information greater than a difference information threshold; querying the to-be-executed data from the target queue; and outputting the to-be-executed data by invoking a second interface, wherein the second interface includes a second parameter, and a parameter value of the second parameter is the to-be-executed data.

[0011] According to a further aspect of the embodiments of the present disclosure, a system for processing data is provided. The system for processing data can include: a client configured to send a data query request, wherein the data query request is used to request to query to-be-executed data; and a server configured to, in response to the data query request, allocate a target scheduling time to a queue based on resource configuration information of the queue, wherein the resource configuration information is used to represent processing resources allowed to be scheduled to the queue in a running process, and the target scheduling time is used to represent a time allocated to the queue; determine time difference information between the target scheduling time of the queue and a scheduled time of the queue, wherein the time difference information is used to represent a difference between the target scheduling time and the scheduled time; determine a target queue from a plurality of queues based on the time difference information of the queue, wherein the target queue corresponds to time difference information greater than a difference information threshold; query the to-be-executed data from the target queue; and return the to-be-executed data to the client.

[0012] According to a further aspect of the embodiments of the present disclosure, an electronic device is also provided. The electronic device can include a memory and a processor: the memory is configured to store computer executable instructions, and the processor is configured to execute the computer executable instructions. When the computer executable instructions are executed by the processor, the data processing method of the embodiments of the present disclosure is implemented.

[0013] According to a further aspect of the embodiments of the present disclosure, a processor is also provided. The processor is configured to run a program. When the program is running, the data processing method of the embodiments of the present disclosure is executed.

[0014] According to another aspect of the embodiments of the present disclosure, a computer readable storage medium is further provided. The computer readable storage medium includes a stored program, wherein the program controls the device where the storage medium is located to perform the data processing method of the embodiments of the present disclosure when the program is running.

[0015] According to another aspect of the embodiments of the present disclosure, a computer program product is further provided. The computer program product includes a computer program, which, when executed by a processor, implements the data processing method of the embodiments of the present disclosure.

[0016] In the embodiments of the present disclosure, if the data to be queried by the data query request needs to be queried, the received data query request from the client can be monitored to analyze the to-be-executed data that needs to be queried from the server. According to the resource configuration information of each queue, the processing resources that the data query request in the queue is allowed to be scheduled to in the running process can be analyzed, and the target scheduling time when the current data query request is allocated to the processing resources in the queue for execution can be determined. The time difference information between the target scheduling time and the scheduled time of the corresponding queue can also be determined, and the target queue can be selected according to the size relationship between the time difference information of each queue and the difference information threshold. The queue with the time difference information greater than the difference information threshold can be determined as the target queue, so that the current data query request can use the target queue to schedule the corresponding processing resources to query the corresponding to-be-executed data. In the embodiments of the present disclosure, by setting the corresponding resource configuration information for each queue in the database system, the soft isolation between tasks of different priorities, that is, data query requests of different degrees, is realized. In the process of executing the data query request, the target scheduling time of different queues can be automatically calculated according to the resource configuration information, and a more suitable queue can be selected based on the target scheduling time to query the to-be-executed data, thereby avoiding the resource contention and competition problems in the execution of large queries and small queries, achieving the technical effect of improving the efficiency of data processing, and solving the technical problem of low efficiency of data processing.

[0017] It is easy to note that the general description above and the detailed description below are merely for exemplifying and explaining the present disclosure, and do not constitute a limitation on the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0018] The drawings described herein are used to provide further understanding of the present disclosure, and form a part of the present disclosure. The illustrative embodiments of the present disclosure and their descriptions serve to explain the present disclosure, and do not constitute an improper limitation on the present disclosure. In the drawings:

[0019] FIG. 1 is a schematic diagram of an application scenario of a data processing method according to an embodiment of the present disclosure;

[0020] Figure 2 is a flow chart of a data processing method according to an embodiment of the present disclosure;

[0021] Figure 3 is a flow chart of a data processing method according to an embodiment of the present disclosure;

[0022] Figure 4 is a flow chart of a data processing method according to an embodiment of the present disclosure;

[0023] Figure 5 is a schematic diagram of a data processing system according to an embodiment of the present disclosure;

[0024] Figure 6 is a schematic diagram of a preliminary version of a processing queue processing process;

[0025] Figure 7 is a schematic diagram of an improved multi-level priority queue and scheduling time according to an embodiment of the present disclosure;

[0026] Figure 8 is a schematic diagram of a multi-level priority queue according to an embodiment of the present disclosure;

[0027] Figure 9 is a schematic diagram of an overall architecture of a priority queue-based data warehouse hybrid load isolation according to an embodiment of the present disclosure;

[0028] Figure 10(a) is a flow chart of a process of obtaining a task from a queue for processing according to an embodiment of the present disclosure;

[0029] Figure 10(b) is a flow chart of a process of placing a task into a queue according to an embodiment of the present disclosure;

[0030] Figure 11 is a schematic diagram of a scheduling example according to an embodiment of the present disclosure;

[0031] Figure 12 is a schematic diagram of CPU usage of a 16-thread full system according to an embodiment of the present disclosure;

[0032] Figure 13 is a schematic diagram of a data processing apparatus according to an embodiment of the present disclosure;

[0033] Figure 14 is a schematic diagram of another data processing apparatus according to an embodiment of the present disclosure;

[0034] Figure 15 is a schematic diagram of another data processing apparatus according to an embodiment of the present disclosure;

[0035] Figure 16 is a structural block diagram of a computer terminal according to an embodiment of the present disclosure;

[0036] Figure 17 is a block diagram of an electronic device for implementing a data processing method according to an embodiment of the present disclosure;

[0037] Figure 18 is a hardware structural block diagram of a computer terminal (or mobile device) for implementing a data processing method according to an embodiment of the present disclosure;

[0038] FIG. 19 is a structural block diagram of a computing environment of a data processing method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0039] In order to enable persons skilled in the art to better understand the present disclosure scheme, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, not all the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by persons skilled in the art without creative labor should be within the scope of protection of the present disclosure.

[0040] It should be noted that the terms "first", "second" and the like in the specification and claims of the present disclosure and the above-described drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or a chronological sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present disclosure described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or components does not have to be limited to those steps or components clearly listed, but can include other steps or components not clearly listed or inherent to the above-mentioned process, method, product or device.

[0041] First, some nouns or terms that appear in the description of the embodiments of the present disclosure are applicable to the following explanations:

[0042] Workload Group is an important concept in open source distributed Structured Query Language (SQL) query engine, which can be used to organize and manage different query workloads, so as to allocate different Central Processing Unit (CPU) resources and memory limits for different queries;

[0043] The concurrent upper limit refers to the maximum number of queries that can be simultaneously queued for execution within a specific time period. In a database system or data warehouse system, there will be an execution queue to store the query requests waiting for execution. Setting the upper limit of the queued query can control and manage the number of queries waiting for execution, prevent the queue from overloading and cause the performance of the database system to decline and the response to delay;

[0044] CPU Share, also known as cpu_share, refers to the proportion of CPU resources allocated to a query, which can be expressed in percentage. In a database system or data warehouse system, CPU resources are one of the key resources for executing queries and processing data. By setting CPU Share, the database system or data warehouse system can determine the proportion of CPU resources that each query can use, thereby controlling the execution priority and resource occupation of each query;

[0045] Elastic Space refers to the ability of the open-source distributed SQL query engine to dynamically adjust the memory size allocated to a query according to needs. This mechanism allows the database system or data warehouse system to dynamically adjust the memory space required by the query during query execution according to actual needs, in order to improve query performance and resource utilization efficiency. The characteristic of Elastic Space is that it can flexibly allocate memory resources according to the requirements of different queries and the load of the database system or data warehouse system, so as to better meet the execution needs of the query;

[0046] Queue is an important concept in data warehouse for managing query execution order and priority, which can be sorted and scheduled according to the priority and execution time of the query, ensuring that different queries can be executed in order according to the set rules and strategies;

[0047] Large Query refers to a query that consumes a large amount of CPU resources and memory, involving the processing and calculation of a large amount of data, which may cause increased load, decreased performance, and resource contention. Since large queries may have a significant impact on the database system or data warehouse system, they need to be optimized and limited to ensure the stability and performance of the database system or data warehouse system;

[0048] CPU Time refers to the time consumed by the CPU during query execution. In a database system or data warehouse system, CPU Time can be used to represent the actual execution time of a query or task on the CPU. CPU Time is one of the important indicators for measuring the execution efficiency and performance of a query or task;

[0049] Memory Upper Limit refers to the maximum memory size allocated to a query for storing data and intermediate results required during query execution. In a database system or data warehouse system, Memory Upper Limit is an important configuration parameter that can help control and manage memory resource occupation during query execution;

[0050] Max scan rows, refers to the maximum number of rows that can be scanned during query execution. In a database system or data warehouse system, max scan rows is an important configuration parameter that limits the maximum number of rows that a query can handle when scanning data. Setting max scan rows can help control the efficiency of query execution and resource consumption, avoiding excessive scanning of data by queries, which can lead to performance degradation or excessive load on the database system or data warehouse system;

[0051] Short query, refers to a query with a short execution time, which does not need to be optimized and limited. In a database system or data warehouse system, short query can refer to a query with a short execution time and small data processing volume, which can be completed quickly in a short time. The above type of query can not need to be optimized or limited, because the execution time is short and the resource consumption is low, which will not have a significant impact on the performance of the database system or data warehouse system;

[0052] Resource group, is an open-source distributed SQL query engine that manages query resources. Resource group allows users to allocate different query workloads to different groups according to task requirements and query characteristics, so as to independently configure and optimize each Resource Group. Through Resource Group, users can allocate different CPU resources and memory limits to different queries to meet the resource requirements of different queries and optimize query execution efficiency;

[0053] Max concurrency, refers to the maximum number of queries that can be executed simultaneously within a certain period of time. In a database system or data warehouse system, max concurrency is an important performance indicator that reflects the ability of the database system or data warehouse system to handle queries simultaneously. By setting the maximum concurrency, the number of queries that can be handled simultaneously by the database system or data warehouse system can be controlled, avoiding excessive load on the database system or data warehouse system, resource contention and performance degradation;

[0054] Max queue size, refers to the maximum number of queries that can be queued and waiting for execution within a certain period of time. In a database system or data warehouse system, when the database system or data warehouse system is overloaded or lacks resources, queries may enter the queue to wait for execution. Setting max queue size can help control the number of queries in the queue, avoiding excessive accumulation of the queue, which can lead to performance degradation or response delay of the database system or data warehouse system;

[0055] Workload Management (WLM) rules are rules used in data warehouses to manage queries. Through WLM rules, users can configure the priority of different queries so that the database system or data warehouse system can sort and schedule queries according to these rules. Through WLM rules, users can prioritize queries according to task requirements and query importance to ensure that critical queries can be processed in a timely manner;

[0056] IO skew refers to the imbalance of I / O operations during query execution due to uneven data distribution or other reasons. This situation can cause the data processing speed of some nodes or partitions to slow down significantly, affecting the performance of the entire query;

[0057] Workload Manager Queue is used to manage and control the execution order and priority of queries. Through Workload Manager Queue, users can assign different priorities to different queries and sort and schedule queries according to these priorities and execution times;

[0058] Automatic adjustment refers to a mechanism in which the database system or data warehouse automatically adjusts query resource allocation based on query execution and resource usage. This mechanism can help the database system or data warehouse system dynamically optimize query execution performance and efficiency in real time to adapt to changing workloads and resource requirements;

[0059] Manual configuration refers to a mechanism in which users manually adjust query resource allocation based on query execution and resource usage. Compared with automatic adjustment, manual configuration requires users to directly intervene and adjust according to actual conditions to meet specific requirements and optimize query execution performance;

[0060] Priority queue is a data structure used to sort and schedule elements according to their priority. In a database system, priority queue can be used to manage the execution order of queries or tasks, and determine which queries should be executed first according to their priority to optimize database system resource utilization and improve overall performance.

[0061] According to the embodiments of the present disclosure, a data processing method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0062] The data processing method provided by the embodiments of the present disclosure can be applied to the application scenario as shown in FIG. 1, but is not limited thereto. In the application scenario as shown in FIG. 1, the server 10 can be a cloud server. The server 10 can be connected to one or more client devices 20 through a local area network connection, a wide area network connection, an Internet connection, or other types of data networks. The client devices 20 can include, but are not limited to, smart phones, tablet computers, notebook computers, palm computers, personal computers, smart home devices, vehicle-mounted devices, and the like. The client devices collectively constitute a client opposite to the server. On a graphical user interface on the client device, an operation interface for obtaining a memory access request for accessing data in a memory address of a device can be deployed. The operation interface can be an operation interface for detecting a relevant operation performed by a user to generate a memory access request. The client device 20 can interact with the user through the graphical user interface to implement the data processing method provided by the embodiments of the present disclosure.

[0063] In the embodiments of the present disclosure, the data processing system constituted by the client device 20 and the server 10 can perform the following steps: if a user has a demand for querying data in a database system on the server, the user can perform a corresponding operation on an operation interface on the client device 20 to upload a data query request corresponding to the data to be queried. After receiving the data query request, the data query request can be sent to the server 10 through a network. A database system, such as a data warehouse, can be deployed in the server 10. Specifically, the following steps can be performed in the server 10:

[0064] Step S102, monitoring a data query request, wherein the data query request is used to request to query to-be-executed data.

[0065] Step S104, in response to the data query request, allocating a target scheduling time to a queue based on resource configuration information of the queue, wherein the resource configuration information is used to represent processing resources allowed to be scheduled to the queue during running, and the target scheduling time is used to represent a time allocated to the queue.

[0066] Step S106, determining time difference information between the target scheduling time of the queue and a scheduled time of the queue, wherein the time difference information is used to represent a difference between the target scheduling time and the scheduled time.

[0067] Step S108, determining a target queue from a plurality of queues based on the time difference information of the queue, wherein the target queue corresponds to time difference information greater than a difference information threshold.

[0068] Step S110, querying the to-be-executed data from the target queue.

[0069] In the above process, the data query request in the server 10 can be monitored through the network, and the target queue can be screened out from a large number of queues in the data warehouse, so as to call the processing resource to query the to-be-executed data to be queried by the data query request, and the obtained to-be-executed data can be sent to the client device 20. The to-be-executed data can be displayed on the operation interface of the client device 20 for the user to view.

[0070] In the embodiments of the present disclosure, by setting the corresponding resource configuration information for each queue in the database system, the soft isolation between tasks of different priorities, that is, data query requests of different degrees, is realized. In the process of executing the data query request, the target scheduling time of different queues can be automatically calculated according to the resource configuration information, and a more suitable queue can be selected to query the to-be-executed data based on the target scheduling time, thereby avoiding the resource contention and competition problem of large queries and small queries and the like during execution, and realizing the technical effect of improving the efficiency of data processing, and solving the technical problem of low efficiency of data processing.

[0071] The embodiments of the present disclosure provide a data processing method as shown in FIG. 2 in the above application scenario. FIG. 2 is a flowchart of a data processing method according to an embodiment of the present disclosure. As shown in FIG. 2, the method can include the following steps:

[0072] Step S202, monitoring a data query request.

[0073] In the technical solution provided in the foregoing step S202 of the present disclosure, the data query request can be used to request to query the to-be-executed data, which can be referred to as a query, and can include small queries and large queries, and can be a request for performing a query or a task (split) operation, that is, when the data query request is received, the data query request can be divided into different queues to call resource processing. The to-be-executed data can be data in a database system. The database system is a system for storing, managing, and retrieving data, and can be composed of a database and a database management system. The database management system is a core component of the database system, and can be responsible for functions such as creation, maintenance, query, backup, and the like of the database. The database management system can be used in combination with a data warehouse. The database management system provides basic management functions of data, and the data warehouse provides better levels of data analysis and query functions on the basis of the database management system, and can store and manage data on multiple computers or servers. The computers are connected to each other through a network and work cooperatively to provide data storage and access services. The data warehouse is a database system for storing and managing large amounts of data, and can support an organization to collect, integrate, and analyze data from different sources and different time points. For example, the data warehouse can be an online analytical processing data warehouse (OLAP) type data warehouse.

[0074] Optionally, the queue is an important management concept in the data warehouse, and is used to manage the execution order and priority of the query. By reasonably setting the queue rules and scheduling strategies, the execution order and time of the query can be effectively controlled, the performance and stability of the database system can be improved, and it is ensured that the query can be executed in order according to the set priority, thereby improving the overall efficiency and reliability of the database system.

[0075] Optionally, the data warehouse is an application of the database management system, which is established on the basis of the database management system and is used to support data analysis and decision support requirements. The data warehouse is a data storage system for centralized, comprehensive, storage, and management of large amounts of structured and unstructured data. The data warehouse aims to support data analysis, decision support, and task intelligence requirements, and provides users with convenient and efficient data access and analysis capabilities by integrating data from different data sources into a unified data storage.

[0076] In this embodiment, whether the client device corresponding to the user has a data query request can be monitored.

[0077] Optionally, after the data query request is monitored, the data query request can be analyzed to determine to-be-executed data in the database system requested by the data query request.

[0078] For example, after obtaining the data query request, the query statement in the data query request can be parsed to understand the specific content and requirements of the query request. According to the table name and field name involved in the query statement, the table and field where the to-be-executed data to be queried from the database system can be determined. By analyzing the condition statement in the query statement, the filtering condition in the query request can be determined. According to the keyword in the query statement, it is determined whether the query request is a single-table query or a multi-table association query, whether an index needs to be used, and the like. According to the characteristics of the query request and the performance characteristics of the database system, it is considered whether performance optimization needs to be performed, such as using an index, adjusting a query order, optimizing a query statement, and the like. Through the above method, the to-be-executed data to be queried from the database system can be accurately analyzed from the data query request.

[0079] It should be noted that the above process and method of analyzing query to-be-executed data from a data query request are only for illustration, and are not specifically limited herein.

[0080] For example, if a user needs to view a certain data in the database system of the server, the user can perform corresponding operations on the client device corresponding to the user to input the requirement of viewing the data, thereby forming a data query request. After detecting the data query request on the client device, the data query request can be transmitted to the server through the network, so as to analyze the data query request by using the server.

[0081] In step S204, in response to the data query request, a target scheduling time is allocated to the queue based on the queue-based resource configuration information.

[0082] In the technical solution provided in step S204 of the present disclosure, the queue can also be referred to as a priority queue, and can be a pending split queue, a priority schedule split queue, or a level waiting split queue. The queue can include a plurality of Simple Multi Level Split Queue (SMLSQ) queues, which can be used to store data query requests and group the data query requests according to their sizes and resource requirements, so as to more effectively manage and schedule the data query requests. The resource configuration information can be used to represent the processing resources that the corresponding queue is allowed to schedule to during operation, which can help the database system to more flexibly manage and allocate various processing resources, achieve soft isolation of various resources of the priority queue, and thus optimize the performance and resource utilization of the database system. The resource configuration information can also be referred to as a configuration, such as cpu_share, which can be used to set the proportion of CPU time obtained by the priority queue, and achieve soft isolation of CPU resources. The target scheduling time can be used to represent the time at which the database system calls the corresponding queue, such as the scheduling time between SMLSQ queues, the expected scheduling time, or the target schedule time. The processing resources can include CPU resources, and can also include other resources in the database system, such as memory, I / O, etc. The above is only an example and is not limited in particular.

[0083] It should be noted that the above resource configuration information and processing resources are only examples and are not limited in particular.

[0084] In this embodiment, after monitoring the data query request, the target scheduling time can be allocated to the corresponding queue based on the resource configuration information of each queue.

[0085] Optionally, each queue can be configured with corresponding resource configuration information, which can represent the weight of CPU time obtained by the queue. The database system can be configured to set the proportion of CPU resources obtained by each queue. The database system determines the scheduling time of the queue according to the resource configuration information of the queue to which the query belongs.

[0086] Optionally, the target scheduling time is determined through the resource configuration information and the scheduled time of each queue. For example, the resource configuration information and the scheduled time of each queue are obtained, the total scheduling time of each queue is calculated through the two parameters, a total scheduling time meeting the condition is selected as a baseline value from the total scheduling times of each queue, and the target scheduling time of each queue is obtained according to the CPU share ratio distribution of the baseline value, so as to ensure that each queue is reasonably allocated with the scheduling time according to the configuration and scheduling conditions, and the task scheduling efficiency of the database system is improved.

[0087] In the related art, the small query is not blocked by the large query and queued after the SQA function, that is, the problem of reducing the performance and response speed of the entire data warehouse caused by the small query waiting for the large query to complete before obtaining the database system resource for execution is avoided. However, even through the queue splitting, the small query can be executed independently in the queue, but in the actual database system execution process, the small query and the large query still need to share the database system resource, and the resource contention problem still exists between the two. Therefore, the technical problem of low data processing efficiency still exists.

[0088] However, in the embodiment of the present disclosure, by configuring the resource configuration information of each queue, the weight of the CPU time obtained by each queue can be controlled, so as to realize the reasonable allocation and optimization of resources. The high-priority queue (which is given a higher execution priority in the database system and will be given priority for execution in the database system scheduling to ensure that the high-priority queue is completed in time) can obtain more CPU resources to ensure that important tasks or urgent tasks can be processed in time, while the low-priority queue (which is given a lower execution priority in the database system and will be given lower consideration for execution in the database system scheduling and can be processed after the high-priority task is completed) can still obtain a certain CPU time under the condition of limited resources to ensure the overall operation of the database system. The scheduling time of each queue is determined according to the resource configuration information and the scheduled time of each queue, so as to realize that the scheduling time of each queue basically meets the CPU share ratio, thereby avoiding that some queues occupy CPU resources for a long time, causing other queues to be unable to obtain enough resources for execution, so as to improve the resource utilization and performance of the database system, and thus the technical effect of improving the efficiency of data processing is realized.

[0089] In addition, by configuring the resource configuration information of each queue, the processing of tasks with different priorities can be realized. The queue with high resource configuration information is preferentially scheduled to ensure the timely execution of high-priority tasks; and the queue with low resource configuration information can be processed only after the high-priority tasks are satisfied, thereby ensuring the rationality of the resource distribution of the database system. The resource configuration information and the scheduling time of each queue can be flexibly adjusted according to the requirements and performance requirements of the database system. Different task scenarios and application requirements can be configured according to the actual situation to configure the resource configuration information and the scheduling time of the queue, so as to realize a better resource allocation and scheduling strategy.

[0090] In summary, the determination of the scheduling time of the queue based on the resource configuration information of each queue can help the database system to realize the rational allocation of resources, the priority processing of tasks, the control of scheduling time, improve the performance and stability of the database system, and meet the flexibility and customizability of different task scenarios and requirements.

[0091] Step S206, determining time difference information between the target scheduling time of the queue and the scheduled time of the corresponding queue.

[0092] In the technical solution provided by the above step S206 of the present disclosure, the scheduled time can be the time that has been scheduled, which can be referred to as the level scheduled time. The time difference information can be used to represent the difference between the target scheduling time and the scheduled time, for example, the difference can be the difference ratio between the target scheduling time and the scheduled time. It should be noted that the above difference ratio is only an example, and this disclosure does not make specific limitations here, as long as the time difference information can reflect the difference between the target scheduling time and the scheduled time, which is within the protection scope of the embodiments of the present disclosure.

[0093] Optionally, the scheduled time can refer to the length of time that a queue has been scheduled and started to execute. During the queue scheduling process, the length of time from the start of execution to the current time of the queue is recorded as the scheduled time of the queue. The calculation of the scheduled time can be used to measure the length of time of the queue scheduling, evaluate the data processing efficiency of the queue scheduling, and perform the priority scheduling and resource allocation of the queue. By monitoring and recording the scheduled time of the queue, the execution order of the queue in the database system can be better managed, and the queue can be scheduled according to certain rules and priorities, thereby improving the efficiency and performance of the database system.

[0094] In this embodiment, after the target scheduling time of the corresponding queue is determined based on the resource configuration information of each queue, the time difference information between the target scheduling time and the scheduled time can be determined.

[0095] Optionally, a time difference information is determined by determining the difference between the target scheduling time and the scheduled time. For example, the time difference information can be calculated by subtracting the scheduled time from the target scheduling time. The time difference information can also be converted into a ratio form, i.e., the ratio (difference ratio) of the target scheduling time to the scheduled time, which can be calculated by dividing the time difference information by the scheduled time.

[0096] It should be noted that the above process of determining the time difference information and the form of the determined time difference information are only examples and are not limited in this regard. In the embodiments of the present disclosure, as long as the process and method can understand the relationship between the target scheduling time and the scheduled time, and the time difference information between the two can be used to measure how to allocate the queue for the current data query, they are within the protection scope of the embodiments of the present disclosure.

[0097] In the embodiments of the present disclosure, the time difference information can be used to measure the difference between different queues, which helps to determine the priority and urgency of each queue. Through the time difference information, the scheduling of queues with different time spans can be uniformly analyzed and compared, which improves the standardization level of scheduling. According to the size of the time difference information, the database system can automatically adjust the execution order, priority or resource allocation of the task to optimize the performance and efficiency of the database system. By calculating the time difference information between the target scheduling time and the scheduled time, intelligent queue scheduling and resource allocation can be achieved, and the priority of queue execution can be adjusted according to the time difference information to ensure the rational use of resources. The time difference information can be used as a real-time monitoring index to help the database system to monitor the queue execution in real time and adjust the scheduling strategy in a timely manner to adapt to the real-time changes of the database system load and task demand. Through the time difference information, resources can be allocated more effectively, and different tasks can be reasonably scheduled according to their priority and urgency, thereby improving the overall resource utilization and performance of the database system.

[0098] In summary, determining the time difference information between the target scheduling time and the scheduled time can provide a more accurate reference for the queue scheduling and resource allocation of the database system, which helps to improve the intelligence and efficiency of the database system.

[0099] In step S208, a target queue is determined from the multiple queues based on the time difference information of the queues.

[0100] In the technical solution provided in step S208 of the present disclosure, the difference information threshold value can be a pre-set numerical value or a numerical value set according to actual conditions, which is not specifically limited here. The target queue corresponds to time difference information greater than the difference information threshold value, for example, a queue with a large difference.

[0101] Optionally, a queue with large time difference information can fail to obtain sufficient CPU time due to uneven resource allocation, resulting in a large difference between the target scheduling time and the scheduled time. A queue with large time difference information can include tasks with low priority, which are less urgent or important than tasks in other queues, and thus have a short scheduling time. A queue with large time difference information can require more scheduling time, which can indicate that the tasks in the queue require more resources or time to complete during execution, and thus the performance of the database system can need to be optimized or the resource allocation can need to be adjusted.

[0102] In this embodiment, after determining the time difference information of the queue by using the target scheduling time and the corresponding scheduled time, the time difference information and the difference information threshold value can be compared, and a queue with time difference information greater than the difference information threshold value can be determined as the target queue.

[0103] In the embodiments of the present disclosure, by determining the target queue with time difference information greater than the difference information threshold value, an uneven resource allocation in the database system can be found, which helps to optimize the resource allocation and ensure that each queue can obtain reasonable scheduling time. The target queue with time difference information greater than the difference information threshold value can include tasks with low priority, and adjusting the priority of these tasks can better meet the actual needs of the database system and improve the overall operation efficiency of the database system. By analyzing the target queue with time difference information greater than the difference information threshold value, performance bottlenecks or resource allocation problems in the database system can be found, which helps to optimize the performance of the database system and improve the execution efficiency and stability of the database system.

[0104] In summary, by determining the target queue with time difference information greater than the difference information threshold value, resource allocation imbalance, unreasonable task priority, or performance optimization problems can be found, thereby optimizing task scheduling and resource allocation and improving the performance and efficiency of the database system.

[0105] Optionally, the database system can select a suitable SMLSQ queue as the target queue according to the resource configuration information and the scheduled time.

[0106] Optionally, the queue can help the database system manage the execution order of queries, and the queries are queued and scheduled according to the order and priority of the queries entering the queue. Through the queue, the queries can be ensured to be executed in a certain order, avoiding confusion and conflict between queries. According to the priority of the queries, the queries can be sorted and scheduled to ensure that high-priority queries can be executed first. According to the importance and urgency of different queries, different priority rules can be set to ensure that critical queries can be processed in time. According to the execution time of the queries, the queries can be sorted and scheduled to ensure that long-running queries do not block the execution of other queries. By setting the execution time limit and scheduling strategy, the execution time of the queries can be effectively controlled to avoid long-running queries affecting the performance of the database system. By managing the execution order and priority of the queries through the queue, the allocation and utilization of resources can be better balanced, and the stability and reliability of the database system can be improved. Through the queue, the execution order of the queries can be reasonably scheduled to avoid excessive consumption of resources and excessive load of the database system, ensuring that the database system can run smoothly.

[0107] In step S210, the to-be-executed data of the request query corresponding to the data query request is queried from the target queue.

[0108] In the technical solution provided by the above step S210 of the present disclosure, after determining the target queue from the plurality of queues based on the time difference information, the to-be-executed data of the request query corresponding to the data query request can be queried from the target queue.

[0109] Optionally, querying the to-be-executed data from the target queue can include the following steps: determining the target queue, obtaining the data query request from the target queue, including the data content, query conditions and other information required for the query. According to the obtained data query request, the corresponding query operation is performed in the database to obtain the required to-be-executed data. The obtained to-be-executed data can be returned to the external thread or the database system for further processing or display of the to-be-executed data.

[0110] In the embodiment of the present disclosure, the target queue can indicate that the queue needs more scheduling time to reach the target scheduling time, and can contain some important or urgent data query requests. Therefore, processing these requests first can improve the performance and efficiency of the database system. The target queue may need more scheduling time due to unbalanced resource allocation, so timely processing of the requests in the queue can optimize the allocation and utilization of resources. If the target queue contains some important data query requests, timely processing of these requests can avoid delays and affect the response speed of the database system. By processing the data query requests in the target queue, the task scheduling and resource allocation strategy of the database system can be optimized, and the overall performance and efficiency of the database system can be improved.

[0111] In summary, the data query request can be processed preferentially from the target queue whose time difference information is greater than the difference information threshold, so as to make full use of resources, improve the performance of the database system, avoid delay, and realize efficient operation of the database system.

[0112] According to the steps S202 to S210 of the present disclosure, if the data to be queried by the data query request needs to be queried, the data query request received from the client can be monitored to analyze the to-be-executed data that needs to be queried from the server. According to the resource configuration information of each queue, the processing resources allowed to be scheduled in the process of the data query request in the queue can be analyzed, and the target scheduling time when the current data query request is allocated to the processing resources in the queue for execution can be determined. The time difference information between the target scheduling time and the scheduled time of the corresponding queue can also be determined. The queue whose time difference information is greater than the difference information threshold is determined as the target queue, so that the current data query request can use the target queue to schedule the corresponding processing resources to query the corresponding to-be-executed data. In the embodiment of the present disclosure, by setting the corresponding resource configuration information for each queue in the database system, the soft isolation between tasks of different priorities, that is, data query requests of different degrees, is realized. In the process of executing the data query request, the target scheduling time of different queues can be automatically calculated according to the resource configuration information, and a more suitable queue can be selected to query the to-be-executed data based on the target scheduling time, thereby avoiding the resource contention and competition problem in the execution of large queries and small queries, realizing the technical effect of improving the efficiency of data processing, and solving the technical problem of low efficiency of data processing.

[0113] The above method of the embodiment will be further introduced below.

[0114] As an optional implementation, in step S204, the target scheduling time is allocated to the queue in response to the data query request and based on the resource configuration information of the queue, including: in response to the data query request, the total scheduling time allowed to be scheduled by the queue is determined based on the resource configuration information of the queue and the scheduled time of the queue, where the total scheduling time is used to represent the total scheduling time allowed to be scheduled by the queue; and the target scheduling time is allocated to the corresponding queue based on the total scheduling time of the queue.

[0115] In this embodiment, in the process of determining the target scheduling time of each queue based on the resource configuration information of the multiple queues, the total scheduling time allowed to be scheduled by the queue can be determined based on the resource configuration information and the scheduled time of the queue, and the target scheduling time allocated to each queue can be determined based on the total scheduling time.

[0116] Optionally, the total scheduling time refers to a theoretical total scheduling time that the queue should obtain according to the resource configuration information and the scheduled time of the queue, which can be the total scheduling time allowed for the queue. Specifically, the total scheduling time is determined by adjusting the scheduled time according to the proportion of the resource configuration information of the queue to determine the length of the scheduling time that each queue should obtain.

[0117] In task scheduling and resource allocation, the concept of total scheduling time can balance the allocation of scheduling time to some extent according to the priority and resource demand of the queue, ensuring that each queue can obtain corresponding scheduling time. By calculating the total scheduling time, the execution order of tasks and the allocation of resources can be better managed, improving the performance and efficiency of the database system.

[0118] Optionally, the resource configuration information and the scheduled time of the queue are obtained. According to the resource configuration information, the total scheduling time of the queue can be calculated. The total scheduling time is determined according to the resource configuration information and the scheduled time of the queue to determine the length of the total scheduling time that the queue should obtain. For example, the scheduled time can be divided by the proportion of the resource configuration information to calculate the total scheduling time. According to the calculated total scheduling time, the total scheduling time may need to be adjusted to ensure that each queue can obtain reasonable scheduling time. This may involve modifying or optimizing the total scheduling time according to the actual situation of the database system. The final total scheduling time will be used in the task scheduling and resource allocation process to determine the length of the scheduling time that each queue should obtain. The execution order of tasks and the allocation of resources will be arranged according to the total scheduling time to improve the performance and efficiency of the database system.

[0119] Through the above method, the total scheduling time that each queue should obtain can be calculated according to the resource configuration information and the scheduled time, to ensure the reasonable scheduling of tasks and the effective allocation of resources, thereby improving the overall running efficiency of the database system.

[0120] As an optional implementation, based on the total scheduling time of the queue, the target scheduling time is allocated to the queue, including: determining a first total scheduling time from the total scheduling times of a plurality of queues, wherein the first total scheduling time is greater than a first time threshold; based on the first total scheduling time and the resource configuration information of the queue, the target scheduling time is allocated to the queue.

[0121] In this embodiment, in the process of determining the target scheduling time required to be allocated to each queue based on the total scheduling time, a first total scheduling time greater than the first time threshold can be determined from the total scheduling times of the multiple queues, and the target scheduling time can be allocated to the corresponding queue based on the first total scheduling time and the resource configuration information. The first total scheduling time can be used as a baseline, that is, a reference standard, to help the database system determine the current scheduling situation and the scheduling plan of future tasks.

[0122] Optionally, a baseline value can be selected from the total scheduling times of the queues as a reference standard, and the maximum value can be selected as the baseline. The baseline value is selected by comparing the total scheduling times of the queues and selecting one that meets the condition as the baseline, for example, the total scheduling time that is long or meets the first time threshold. After the baseline value is determined, the baseline value can be used as a reference standard for subsequent task scheduling and resource allocation. The baseline value can help the database system determine the current scheduling situation and the scheduling plan of future tasks, and guide the execution order of tasks and the allocation of resources. It should be noted that the above process and method of determining the baseline from the total scheduling times of the queues are only for illustration and are not limited specifically herein.

[0123] Through the above method, the baseline value can be determined by calculating the total scheduling time, thereby playing a guiding role in task scheduling and resource allocation, better managing the execution order of tasks and the allocation of resources, and improving the performance and efficiency of the database system.

[0124] In the embodiments of the present disclosure, in order to select a reference standard to determine the current scheduling situation and the scheduling plan of future tasks, the baseline can be determined by the total scheduling time. Determining the baseline can better understand the scheduling time situation of each queue, thereby optimizing the allocation of resources, ensuring that each queue can obtain reasonable scheduling time, and improving the performance and efficiency of the database system. The baseline value as a reference standard can guide the execution order of tasks and the allocation of resources, ensure that tasks are scheduled according to certain rules and priorities, and improve the overall running efficiency of the database system. By determining the baseline, problems of uneven scheduling time and uneven resource allocation can be found in time, providing a basis for performance optimization of the database system, and better managing and scheduling tasks.

[0125] Through the above method, the database system can be more reasonable and efficient in task scheduling and resource allocation, improve the performance and stability of the database system, optimize the execution order of tasks and the utilization efficiency of resources, be conducive to improving the overall running efficiency of the database system, and ensure that tasks can be reasonably executed according to the predetermined scheduling time.

[0126] As an optional implementation, the resource configuration information of the queue includes a resource proportion of the processing resource allowed to be scheduled by the queue in a running process to the total processing resource, and the target scheduling time of the queue is allocated based on the first total scheduling time and the resource configuration information of the queue, including: determining the target scheduling time corresponding to the resource configuration information of the queue from the first total scheduling time, wherein the target scheduling time of the queue occupies the same time proportion of the first total scheduling time as the resource proportion indicated by the resource configuration information of the queue.

[0127] In this embodiment, in the process of allocating the target scheduling time of the corresponding queue based on the first total scheduling time and the resource configuration information, the target scheduling time corresponding to the resource configuration information can be determined from the first total scheduling time, wherein the resource configuration information can include the processing resource allowed to be scheduled by the corresponding queue in a running process, which can be a cpu_share value. The target scheduling time occupies the same time proportion of the first total scheduling time as the resource proportion indicated by the resource configuration information of the queue, that is, the proportion between the scheduling time and the cpu_share is consistent.

[0128] Optionally, the proportion between the scheduling time and the cpu_share is maintained to be substantially consistent through scheduling. This means that when the database system performs task scheduling, it will determine the scheduling time that each queue should obtain according to the cpu_share value of the queue, so as to maintain the proportion relationship between the cpu_share and the scheduling time. The maintenance of the above consistency can ensure that the scheduling time is reasonably allocated according to the priority and resource configuration of each queue, thereby guaranteeing the performance and efficiency of the database system. If the proportion relationship between the cpu_share value and the scheduling time can be substantially consistent, that is, the scheduling time obtained by each queue is proportional to its cpu_share value, the resources can be more accurately allocated, the execution order of the tasks can be optimized, and the overall operation efficiency of the database system can be improved. The maintenance of the above consistency helps the database system to maintain balanced allocation of resources, and avoids that some queues occupy resources for a long time, affecting the execution efficiency of other queues. By continuously maintaining the proportion consistency between the cpu_share and the scheduling time, the execution order of the tasks and the allocation of resources can be better managed, and the performance and stability of the database system can be improved.

[0129] Optionally, in addition to obtaining the resource configuration information of each queue, including the CPU share value, the scheduled time of each queue can also be obtained. The CPU share value represents the weight of the queue in obtaining CPU time, and the scheduled time represents the length of time that the queue has been scheduled to execute. According to the baseline value and the CPU share value and the scheduled time of the queue, the target scheduling time of each queue is calculated. The target scheduling time is obtained by allocating the baseline value to each queue in proportion to the CPU share value of the queue, so as to ensure that each queue can obtain reasonable scheduling time.

[0130] Optionally, when determining the target scheduling time of each queue, the maximum value can be selected from the calculated total scheduling time of each queue as the current baseline value. The baseline value can be used as a reference standard to determine the longest scheduling time in the current database system as the basis for subsequent calculation of the target scheduling time.

[0131] For example, the total scheduling time of each queue is calculated, and the total scheduling time of each queue is calculated according to the CPU share value and the scheduled time of the queue. From these calculated total scheduling times, the one that meets the condition is selected as the baseline value. Based on the baseline value, the target scheduling time of each queue is further calculated to ensure that the scheduling time of each queue meets the proportional relationship of the corresponding CPU share value. The purpose of selecting the total scheduling time that meets the condition as the baseline value is to ensure that the longest scheduling time in the database system is not ignored, so as to ensure that the resource allocation and task scheduling of the database system are more reasonable and effective. The baseline value can be used as a reference standard for subsequent scheduling and resource allocation decisions.

[0132] As an optional implementation, the resource configuration information of the queue is the resource proportion of the processing resource allowed to be scheduled by the queue in the running process to the total processing resource, and in response to the data query request, the total scheduling time allowed to be scheduled by the queue is determined based on the resource configuration information of the queue and the scheduled time of the queue, including: in response to the data query request, obtaining the quotient between the scheduled time of the queue and the resource proportion of the queue; determining the quotient as the total scheduling time of the queue.

[0133] In this embodiment, in the process of determining the total scheduling time based on the resource configuration information of the queue and the corresponding scheduled time, the quotient between the scheduled time and the resource proportion can be obtained, and the quotient can be determined as the total scheduling time, wherein the resource configuration information can be the resource proportion of the processing resource allowed to be scheduled by the corresponding queue in the running process to the total processing resource, that is, the cpu_share value proportion, which can also be referred to as the proportion of cpu_share.

[0134] Optionally, each queue calculates its total scheduling time according to its own scheduled time and the proportion of the CPU share value. Specifically, the total scheduling time is determined according to the CPU share value and the scheduled time of the queue, and the length of the total scheduling time that the queue should obtain. The CPU share value represents the weight of the queue to obtain the CPU time, and the higher the proportion represents that the queue should obtain more CPU time. The scheduled time represents the length of time that the queue has been scheduled to execute. By proportionally calculating the CPU share value and the scheduled time, the total scheduling time that each queue should obtain can be obtained. Through the above method, the scheduling time of each queue can be determined according to the configuration and actual scheduling situation of each queue, so as to reasonably allocate resources and the execution order of tasks.

[0135] Optionally, the total scheduling time can be calculated by the scheduled time and the proportion of the cpu_share value of each queue. For example, the total scheduling time can be calculated by dividing the scheduled time by the proportion of the cpu_share value.

[0136] For example, if the database system includes five queues, such as queue 0, queue 1, queue 2, queue 3 and queue 4, and the cpu_share values of each queue are 15, 8, 4, 2 and 1 respectively, and each queue has been scheduled for 140, 90, 40, 30 and 10 respectively. According to the proportion of the cpu_share value and the scheduled time of each queue, the total scheduling time of each queue is determined. The cpu_share value of the total processing resource is the sum of the cpu_share values of the above five queues, that is, 30. The total scheduling time of queue 0 is 140 / (15 / 30) = 280, the total scheduling time of queue 1 is 90 / (8 / 30) = 337.5, the total scheduling time of queue 2 is 40 / (4 / 30) = 300, the total scheduling time of queue 3 is 30 / (2 / 30) = 450, and the total scheduling time of queue 4 is 10 / (1 / 30) = 300.

[0137] For another example, from the total scheduling time corresponding to the above five queues respectively, the larger one can be selected as the baseline, that is, 450 can be selected as the baseline. According to the proportion of the cpu_share value, the total scheduling time baseline is allocated as the target scheduling time, that is, the target scheduling time of queue 0 is 450*(15 / 30) = 225, the target scheduling time of queue 1 is 450*(8 / 30) = 120, the target scheduling time of queue 2 is 450*(4 / 30) = 60, the target scheduling time of queue 3 is 450*(2 / 30) = 30, and the target scheduling time of queue 4 is 450*(1 / 30) = 15.

[0138] It should be noted that the number of queues, cpu_share value and scheduled time in the above database system are only for illustration and are not specifically limited here.

[0139] Alternatively, cpu_share can be a mandatory configuration, taking an integer value between 1 and 1000, used to set the proportion of CPU time obtained by the priority queue, to achieve soft isolation of CPU resources. cpu_share is a relative value, indicating the weight of the running priority queue that can obtain CPU resources.

[0140] In the embodiments of the present disclosure, the cpu_share parameter is used to set the proportion of CPU time obtained by the priority queue, to achieve soft isolation of CPU resources. The cpu_share parameter is a relative value, indicating the weight of the running priority queue that can obtain CPU resources. By setting the cpu_share parameter, the proportion of CPU resources obtained by different priority queues can be controlled, thereby achieving allocation control of CPU resources. Soft isolation of CPU resources can also be achieved, that is, CPU resources are allocated to different priority queues according to the set proportion, ensuring that each queue can obtain the CPU time it deserves, avoiding a queue from occupying CPU resources for a long time and affecting the execution of other queues. By reasonably setting the cpu_share parameter, the performance of the database system can be optimized, and the response speed and processing efficiency of the database system to different priority tasks can be improved. According to actual task requirements and database system load conditions, the cpu_share parameter can be flexibly adjusted, so that the database system can reasonably allocate CPU resources according to different priority requirements, improving the resource utilization of the database system. By setting the cpu_share parameter, reasonable allocation and management of CPU resources can be achieved, ensuring that the database system can efficiently process various query tasks of different priorities, improving the performance and stability of the entire database system.

[0141] Optionally, the cpu_share parameter can be exposed to the user, and the purpose of exposure is to enable the user to set the proportion of CPU time obtained by the priority queue according to the task demand and priority requirement of the user, and to realize soft isolation of CPU resources. The user can operate the cpu_share parameter, and by setting the cpu_share value of different priority queues, the user can affect the CPU resource allocation of the database system to different priority tasks. The cpu_share parameter can be used to realize the following functions: the user can set the cpu_share value of different queues according to the task demand and task priority of the user, so as to realize customized allocation of CPU resources and ensure that different priority tasks can obtain appropriate CPU resource support. By adjusting the cpu_share parameter, the user can optimize the performance of the database system, improve the response speed and processing efficiency of the database system to different priority tasks, and ensure that important tasks can be processed in priority. Soft isolation of CPU resources can also be realized, avoiding that a queue occupies CPU resources for a long time to affect the execution of other queues, and ensuring the reasonable allocation and utilization of database system resources. The user can flexibly adjust the cpu_share parameter according to the actual task demand and database system load, so that the database system can reasonably allocate CPU resources according to different priority requirements, and improve the resource utilization and performance of the database system.

[0142] In general, the cpu_share parameter is exposed to the customer (user), and the customer can operate the parameter to optimize the performance of the database system, realize resource isolation and control, and customize the allocation of CPU resources according to different task requirements, thereby improving the stability and efficiency of the database system.

[0143] It should be noted that whether the above resource configuration information is exposed to the user is optional in the embodiments of the present disclosure, and is not specifically limited here, and can be set according to the actual data processing requirement. Here, the exposure of the resource configuration information to the user is not specifically limited.

[0144] Optionally, the cpu_share value of the SQA queue is equal to the sum of all other queues, which means that the SQA queue can obtain the sum of CPU resources not used by other queues.

[0145] As an optional implementation, the method can include: determining the priority of the queue; and configuring the resource configuration information to the queue according to the priority.

[0146] In this embodiment, the priority of each queue can be determined, and the resource configuration information can be configured to different queues according to the priority, wherein the priority can also be referred to as the minimum priority level (level min priority) or priority level.

[0147] Optionally, the priority of the queue is obtained by configuring different priority queues in the queue, which can have three, for example, a low priority queue, a normal priority queue, and a high priority queue, wherein the low priority queue can be used to store some tasks or data that are not urgent and do not need to be processed immediately, which can be processed when the database system is idle, and will not affect the normal operation of the database system. The normal priority queue can be used to store general tasks or data, which need to be processed in a certain order, and these tasks or data have a certain impact on the overall performance of the database system, but do not have to be processed immediately. The high priority queue can be used to store tasks or data that are urgent and need to be processed immediately, which have an important impact on the normal operation of the database system and need to be processed in priority to ensure the stability and performance of the database system.

[0148] For example, if 3 priority queues are configured, low, normal and high, and their cpu_share are 10, 30 and 40 respectively. Assuming that at a certain time, the low and normal queues are running tasks, and the high queue has no tasks, the low queue can obtain 25% of the CPU resources (10 / (10+30)), and the normal queue can obtain 75% of the CPU resources. If the database system has only one priority queue with queries running, no matter what the value of its cpu_share is, the queue can obtain the entire CPU resources of the database system.

[0149] In the embodiments of the present disclosure, by configuring different priority queues for the queue, the tasks or data can be processed according to different priorities. By setting different priority queues, it can be ensured that high-priority tasks or data can be processed in time, improving the response speed and efficiency of the database system. By setting different priority queues, important tasks or data can be processed in priority, improving the overall performance of the database system. High-priority tasks or data can be processed faster, improving user experience and satisfaction. Different priority queues can be set according to specific needs, and various situations can be flexibly handled. High-priority tasks or data can be processed in priority, speeding up the response speed of the database system. By setting different priority queues, it can be ensured that the database system can process important tasks or data in time, improving the stability and reliability of the database system. By reasonably setting different priority queues, the resources of the database system can be better utilized, improving the utilization efficiency of resources. By dividing tasks or data into different priority levels, they can be processed according to specific conditions, improving the efficiency and performance of the database system.

[0150] As an optional implementation, in step S206, the time difference information between the target scheduling time of the queue and the scheduled time of the queue is determined, including: obtaining a target time ratio of the target scheduling time of the queue to the scheduled time of the queue; and determining the target time ratio as the time difference information of the queue.

[0151] In this embodiment, in the process of determining the time difference information between the target scheduling time and the scheduled time of the corresponding queue, the target time ratio of the target scheduling time to the scheduled time can be obtained, and the target time ratio can be determined as the time difference information, wherein the target time ratio can be the ratio of the difference between the two scheduling times.

[0152] Alternatively, the time difference information is determined by the ratio of the difference between the target scheduling time and the actual scheduling time.

[0153] For example, in five queues, the target scheduling time of queue 0 is 225, the target scheduling time of queue 1 is 120, the target scheduling time of queue 2 is 60, the target scheduling time of queue 3 is 30, and the target scheduling time of queue 4 is 15. The scheduled time of each of the above queues is 140, 90, 40, 30 and 10, respectively. By the ratio of the target scheduling time to the scheduled time, the target time ratio of the corresponding queue can be determined. The difference ratio of queue 0: 225 / 140=1.607, the difference ratio of queue 1: 120 / 90=1.333, the difference ratio of queue 2: 60 / 40=1.5, the difference ratio of queue 3: 30 / 30=1, and the difference ratio of queue 4: 15 / 10=1.5.

[0154] It should be noted that the above process and method of determining time difference information are only examples, and the form of time difference information and the process of determining time difference information are not limited in the embodiments of the present disclosure.

[0155] As an optional implementation, in step S208, the target queue is determined from the plurality of queues based on the time difference information of the queues, including: determining the maximum time difference information from the time difference information of the plurality of queues; and determining the queue corresponding to the maximum time difference information from the plurality of queues as the target queue.

[0156] In this embodiment, in the process of determining the target queue from the plurality of queues based on the time difference information, the maximum time difference information can be determined from the plurality of time difference information corresponding to the plurality of queues, and the queue corresponding to the maximum time difference information can be determined as the target queue, where the maximum time difference information can be the target gap ratio.

[0157] Optionally, the queue corresponding to the maximum time difference information can have a long time unprocessed task or data, which needs to be processed as soon as possible to avoid affecting the normal operation of the database system. By processing the data of the queue, the overall performance and response speed of the database system can be improved faster. Prior processing of the data of the queue can improve the stability and reliability of the database system, and ensure that the urgent task is processed in time.

[0158] In the embodiments of the present disclosure, processing the data of the maximum queue can improve the response speed of the database system, so that the user can obtain the required result faster. By preferentially processing the data of the maximum queue, the database system resources can be better utilized, and the efficiency and performance of the database system can be improved. Processing the data of the maximum queue can ensure that the database system can process important tasks in time, and improve the stability and reliability of the database system. Preferential processing of the data of the maximum queue can improve the user experience, so that the user can obtain the required information or result faster, and improve the user satisfaction. In summary, taking the queue with the largest gap as the target queue for data query can improve the performance, response speed and stability of the database system, optimize resource utilization, improve user experience, and bring many benefits in many aspects.

[0159] For example, if the gap ratio of queue 0 is 1.607, the gap ratio of queue 1 is 1.333, the gap ratio of queue 2 is 1.5, the gap ratio of queue 3 is 1, and the gap ratio of queue 4 is 1.5, among the gap ratios of the above queues, the target gap ratio is 1.607, and at this time, the queue 0 of the target gap ratio is the target queue.

[0160] As an optional implementation, the target queue is determined from the plurality of queues based on the time difference information of the queues, including: obtaining thread configuration information of the queues, wherein the thread configuration information is used to at least represent a maximum thread number allowed to be used by a corresponding queue; in response to the used thread number of the queue satisfying the thread configuration information of the queue, determining, from the time difference information of the plurality of queues, second largest time difference information corresponding to the largest time difference information; and determining, as the target queue, a queue corresponding to the second largest time difference information in the plurality of queues.

[0161] In this embodiment, the thread configuration information of the queue can be obtained, and when the used thread number of the queue satisfies the thread configuration information, the second largest time difference information corresponding to the largest time difference information can be determined from the plurality of time difference information corresponding to the plurality of queues. The queue corresponding to the second largest time difference information in the queue can be determined as the target queue. The thread configuration information can be used to at least represent a maximum thread number (max thread number, max thread for short) allowed to be used by a corresponding queue, which can be max_thread. The queue corresponding to the second largest time difference information can also be the second largest queue in the queue.

[0162] Optionally, the queue corresponding to the largest time difference information is selected, and a split is taken out to return to an external thread for execution. If the queue is empty or has reached the upper limit of the number of threads that can be used, the second largest queue is selected.

[0163] Optionally, when a certain queue has reached the upper limit of the number of threads that can be used, that is, the number of threads currently used by the queue has reached the maximum thread number, if the second largest queue is not selected, resource waste may occur, causing the performance of the database system to decrease. Selecting the second largest queue can ensure that the resources of the database system are more fully utilized, and the overall performance of the database system is improved. If the queue corresponding to the current largest time difference information is empty or full, the second largest queue can contain more urgent tasks that need to be processed first.

[0164] Optionally, in the process of determining whether the second largest queue needs to be determined, it can be checked whether the largest queue is empty or has reached the upper limit of the number of threads. If the largest queue is empty or has reached the upper limit of the number of threads, the second largest queue is selected. A split is taken out from the second largest queue to return to an external thread for execution.

[0165] In the embodiments of the present disclosure, when the number of threads available in the queue reaches the upper limit, selecting the second largest queue can ensure that the database system resources are more fully utilized, and improve the resource utilization rate of the database system. Prior processing of the tasks of the second largest queue can improve the response speed of the database system and ensure that urgent tasks can be processed in time. Selecting the second largest queue can avoid wasting idle resources and improve the efficiency and performance of the database system. Dynamically selecting the second largest queue to process tasks according to the state of the queue improves the flexibility and adaptability of the database system. In summary, the reason for selecting the second largest queue to process tasks is to avoid resource waste, improve the utilization rate and response speed of the database system.

[0166] As an optional implementation, determining the target queue from the plurality of queues based on the time difference information of the queues includes: in response to the current queue being empty, determining, from the time difference information of the plurality of queues, second largest time difference information corresponding to the largest time difference information; and determining, as the target queue, a queue corresponding to the second largest time difference information in the plurality of queues.

[0167] In this embodiment, if the current queue is empty, the second largest time difference information corresponding to the largest time difference information can be determined from the plurality of time difference information corresponding to the plurality of queues, and a queue corresponding to the second largest time difference information in the plurality of queues can be determined as the target queue.

[0168] Optionally, if the current queue is empty, not selecting the second largest queue will result in idle resources, affecting the efficiency and performance of the database system. Selecting the second largest queue can ensure that the database system resources are more fully utilized, and improve the overall performance of the database system. The second largest queue can contain more urgent tasks that need to be processed first to ensure the stability and performance of the database system.

[0169] Optionally, it is checked whether the current queue is empty. If the queue is empty, the second largest queue is selected. A split is taken out of the second largest queue and returned to an external thread for execution.

[0170] In the embodiments of the present disclosure, when the current queue is empty, selecting the second largest queue can ensure that the database system resources are more fully utilized, and improve the resource utilization rate of the database system. Prior processing of the tasks of the second largest queue can improve the response speed of the database system and ensure that urgent tasks can be processed in time. Selecting the second largest queue can avoid wasting idle resources and improve the efficiency and performance of the database system. Processing the tasks of the second largest queue can ensure that the database system can process urgent tasks in time and improve the stability and reliability of the database system. In summary, the reason for selecting the second largest queue to process tasks is to avoid resource idle, improve the utilization rate and response speed of the database system.

[0171] As an optional implementation, before querying the to-be-executed data from the target queue, the method further comprises: in response to the target queue being non-empty, storing the to-be-executed data into the target queue.

[0172] In this embodiment, before querying the to-be-executed data from the target queue, if the target queue is non-empty, the to-be-executed data can be stored into the target queue.

[0173] Optionally, when the database system needs to put a split into the queue, if the queue is non-empty, the split can be directly put into the queue, in which case, the queue is non-empty, and if there is already a task or split in the queue, the new split can be directly put into the queue without waiting. When the queue is non-empty, directly putting the split can avoid waiting time, improve the performance and efficiency of the database system. Without waiting for the queue to be empty before putting the split, the task can be directly processed, improving the response speed of the database system. Directly putting the split can ensure that the task is processed in a timely manner, avoiding task delay or waiting.

[0174] In summary, when the database system needs to put a split into the queue, if the queue is non-empty, directly putting the split can improve the performance and response speed of the database system and ensure that the task is processed in a timely manner.

[0175] As an optional implementation, before querying the to-be-executed data from the target queue, the method further comprises: determining a total scheduling time allowed to be scheduled by the target queue based on resource configuration information of the target queue and a scheduled time of the target queue; in response to the target queue being empty, determining a second total scheduling time from the total scheduling times of the plurality of queues, wherein the second total scheduling time is less than a second time threshold; determining a target time threshold corresponding to the target queue based on the second total scheduling time and the resource configuration information of the target queue; in response to the scheduled time of the target queue being less than the target time threshold, adjusting the scheduled time of the target queue to the target time threshold, and storing the to-be-executed data into the adjusted target queue; and in response to the scheduled time of the target queue being greater than or equal to the target time threshold, maintaining the scheduled time of the target queue, and storing the to-be-executed data into the maintained target queue.

[0176] In this embodiment, before querying the to-be-executed data from the target queue, a total scheduling time allowed to be scheduled by the target queue can be determined based on resource configuration information of the target queue and a scheduled time of the target queue. When the target queue is empty, a second total scheduling time can be determined from the total scheduling times of the plurality of queues. The target time threshold corresponding to the target queue can be determined based on the second total scheduling time and the resource configuration information of the target queue, and the size relationship between the scheduled time of the target queue and the target time threshold can be determined.

[0177] Optionally, if the scheduled time is less than the target time threshold, the scheduled time of the target queue can be adjusted to the target time threshold, and the to-be-executed data can be stored in the adjusted target queue. If the scheduled time is greater than or equal to the target time threshold, the scheduled time of the target queue can be maintained, and the to-be-executed data is stored in the maintained target queue. Through the above steps, the scheduling time of each queue in the database system can be ensured to be within a reasonable range, and the total scheduling time is too small to cause the database system to tend to schedule a certain queue, thereby maintaining the balance and fairness of the database system. The second total scheduling time can be less than the second time threshold, and the second total scheduling time can be the smaller total scheduling time of the total scheduling time of each queue, and the smaller total scheduling time of the total scheduling time is taken as the current baseline. The second time threshold can be a value set according to actual conditions, or can be a pre-set value. Here, only an example is given, and no specific limitation is made. The target time threshold can be the scheduling time that should be allocated to each queue.

[0178] Optionally, when the target queue is empty, if the scheduled time of the target queue is not adjusted, the target queue can be excessively inclined to be scheduled, which can cause other queues to be unable to be fully processed, thereby affecting the overall performance of the database system. The scheduling time is too small, which can cause the queue to be excessively scheduled, and can occupy the resources of other queues, thereby affecting the fairness and efficiency of the database system. Therefore, in the embodiment of the present disclosure, when the target queue is empty, the scheduled time of the target queue needs to be pulled into a reasonable range to avoid the database system excessively inclined to schedule the target queue, and to cause the other queues to be occupied.

[0179] In the embodiment of the present disclosure, by pulling the scheduled time of the target queue into a reasonable range, the fair scheduling of each queue by the database system can be maintained, and the situation that a certain queue is excessively scheduled can be avoided. Adjusting the scheduling time of the queue can optimize the allocation of resources of the database system, so that each queue can be reasonably processed, and the overall efficiency and performance of the database system can be improved. Reasonably adjusting the scheduled time of the target queue can avoid a certain queue occupying the resources of other queues, and can ensure the balance and stability of each part of the database system. In summary, pulling the scheduled time of the target queue into a reasonable range can avoid the database system excessively scheduling a certain queue, avoid queue occupation, and improve the fairness, resource utilization efficiency, and stability of the database system.

[0180] Optionally, according to the time already scheduled by each queue and the proportion of the cpu_share value, the total scheduling time of each queue can be calculated to ensure that the queue is not empty to avoid too small total scheduling time. The minimum value of the total scheduling time of each queue is selected as a baseline, which is used as a reference value of a reasonable scheduling time range. According to the cpu_share proportion of each queue, the total scheduling time baseline is allocated. For an empty queue, according to the cpu_share proportion of the empty queue, the total scheduling time that should be allocated to the empty queue is calculated, and the total scheduling time baseline is determined. If the scheduling time of the queue is less than the calculated value, the scheduling time of the queue needs to be adjusted to the value, that is, the scheduling time of the queue is adjusted to the target time threshold. If the scheduling time of the queue is greater than the calculated value, it remains unchanged and does not need to be adjusted. After the above adjustment, the split is put into the corresponding queue. Through the above process, it can be ensured that when the queue is empty, the scheduling time of the queue is adjusted to a reasonable range, avoiding over-scheduling of a queue by the database system, and maintaining the fairness and efficiency of the database system.

[0181] As an optional implementation, the resource configuration information of the target queue is a resource proportion of the processing resource allowed to be scheduled by the target queue in the running process to the total processing resource. The target time threshold corresponding to the target queue is determined based on the second total scheduling time and the resource configuration information of the target queue, including: determining the target time threshold corresponding to the resource configuration information of the target queue from the second total scheduling time, wherein the target time threshold of the target queue occupies the same time proportion of the second total scheduling time as the resource proportion indicated by the resource configuration information of the target queue.

[0182] In this embodiment, in the process of determining the target time threshold corresponding to the queue based on the second total scheduling time and the resource configuration information, the target time threshold corresponding to the resource configuration information can be determined from the second total scheduling time, wherein the time proportion of the target time threshold to the second total scheduling time is the same as the resource proportion.

[0183] Optionally, the total scheduling time is calculated according to the time already scheduled by each queue and the proportion of the cpu_share value to ensure that the queue is not empty. The minimum total scheduling time in all queues is selected as a baseline to ensure that each queue has a reasonable scheduling time. According to the cpu_share value proportion of each queue and the above baseline, the scheduling time that should be allocated to each queue is calculated. The scheduling time of each queue is checked, and if it is less than the calculated corresponding scheduling time that should be allocated, the scheduling time of the queue needs to be adjusted to the value, and if it is greater than the value, no change is needed. The split is put into the queue.

[0184] For example, if the total scheduling time of queue 0 is 280, the total scheduling time of queue 1 is 337.5, the total scheduling time of queue 2 is 300, the total scheduling time of queue 3 is 450, and the total scheduling time of queue 4 is 300, the smaller one of the total scheduling times of the five queues can be selected as the baseline, i.e., 280 can be selected as the baseline. According to the cpu_share value 2 of queue 3 in the above queues and the baseline, the scheduling time that should be allocated to queue 3 can be determined: 280*2 / 30=18.6.

[0185] As an optional implementation, in step S202, the data query request is monitored, including: monitoring the data query request from the thread, wherein the data query request is used to request to query the target task to be executed by the thread; and in step S210, the to-be-executed data is queried from the target queue, including: determining the subtask data of the target task in the target queue as the to-be-executed data; and returning the to-be-executed data to the thread for execution.

[0186] In this embodiment, in the process of monitoring the data query request, the data query request from the thread can be monitored. The subtask data of the target task in the target queue can be determined as the to-be-executed data, and the to-be-executed data can be returned to the thread for execution, wherein the data query request can be used to request to query the target task to be executed by the thread. The target task can be the task to be executed by the thread.

[0187] Optionally, there is a thread pool of 256 threads on the execution node, each thread takes out a split from the pending Spilts queue and executes a time slice in an infinite loop. In this case, the pending Spilts queue is a Priority Blocking Queue, in which the queries basically follow the priority of First In, First Out (FIFO). Specifically, the Priority Blocking Queue is a priority-supported unbounded blocking queue, and elements are processed in order of their respective priorities. Here, the priority of execution is determined according to the order in which the elements enter the pending Spilts queue, i.e., the principle of first in, first out. Each thread obtains a split from the pending Spilts queue and executes a time slice, which can effectively utilize the thread pool resources of 256 threads while maintaining the order of the queries. Since the Priority Blocking Queue sorts elements according to their priorities, queries are processed according to the priority of FIFO, ensuring the order of query execution. The above setting can ensure that queries are processed in a timely manner and maintain the order of queries, avoiding possible priority inversion or starvation. At the same time, since each thread in the thread pool is in an infinite loop to obtain and execute queries, multi-thread parallel processing of queries can be more fully utilized, improving the concurrency performance and response speed of the database system.

[0188] Optionally, the above node can be an execution node, which refers to a computing node responsible for executing tasks or processing data. The execution node has a thread pool containing 256 threads, each thread pool can cyclically remove a split from the pending Spilts queue and execute a time slice. The thread pool is used to execute tasks concurrently, improving the processing capacity and efficiency of the database system.

[0189] In the embodiments of the present disclosure, through the above settings, the execution node can efficiently process tasks, the threads in the thread pool continuously take out tasks from the queue for execution, and the FIFO priority rule ensures that tasks are executed in the correct order, improving the efficiency and performance of the database system. At the same time, using the Priority Blocking Queue can process tasks according to their priorities, ensuring that important tasks can be executed first, improving the response speed and stability of the database system.

[0190] Optionally, the pending Spilts queue is improved so that it is no longer a single queue, but a multi-level feedback queue composed of multiple priority queues. The scheduling time allocated to different priority queues decreases by a factor of 2 layer by layer, that is, the scheduling time of the queue at each level is twice that of the previous level. When a query enters, it will be allocated to the first level priority queue, and as the execution time increases, the query will gradually descend to lower level queues to avoid large queries (large queries) from excessively affecting the execution of small queries (small queries).

[0191] In the embodiments of the present disclosure, through the setting of the multi-level feedback queue, the queries can be more effectively managed and scheduled, and the response speed and performance of the database system to the queries can be improved. The queues at different levels gradually decrease according to the execution time, which can ensure that large queries do not occupy database system resources for a long time, thereby maintaining the overall stability and performance of the database system. Small queries will enter the first level queue first and be executed preferentially, thereby avoiding large queries from excessively affecting the execution of small queries. Through the above setting of the multi-level priority queue, different priority queries can be more efficiently processed, and it can be ensured that large queries and small queries can be reasonably processed and allocated resources, thereby improving the overall performance and stability of the database system.

[0192] Optionally, the node can include a resource control (RC) node. The RC node has 6 priority queues, which can be a root (ROOT) queue, a SQL queue, a high priority queue, a normal priority queue, a low priority queue, and a lowest (LOWEST / lowest) priority queue. The first two queues are not exposed to users, and the last four queues are exposed to users. Under the above setting, users can set the priority of the query to make the query enter the corresponding queue, and also can modify the maximum concurrency and maximum queue number of the queue. Although there is no difference between the queues on the execution side, by setting different priority queues, reasonable scheduling and processing can be performed according to the importance and urgency of the task.

[0193] It should be noted that whether the above different priority queues are exposed to users is a setting of the embodiments of the present disclosure, which is not specifically limited here, and can be set according to actual data processing needs. Here, the priority queue is not specifically limited whether it is exposed to users.

[0194] In the embodiments of the present disclosure, by setting the priority of the query, the user can place the query into different priority queues according to the task requirements, thereby controlling the execution order and priority of the query. The user can modify the maximum concurrency and the maximum queuing number of the queue according to the own requirements, so as to adapt to different workloads and resource requirements. Although different queues behave differently on the client side, there is no difference on the execution side, that is, the query will be executed in the same way in all queues. This can simplify the implementation and maintenance of the database system and reduce the logical complexity between different queues. Since different queues have no difference on the execution side, different priority query requests can be processed more evenly, avoiding performance imbalance problems caused by queue differences. In general, the above setting allows the client to set and manage the priority and queue properties of the query according to the task requirements, while keeping the logic simple and unified on the execution side to improve the stability, maintainability and performance of the database system.

[0195] Optionally, the database system can monitor the data query requests from the threads, that is, the tasks or queries submitted by different threads. According to the priority of the task, the sub-task data of the target task in the target queue is determined as the to-be-executed data. This means that the database system will analyze and process the tasks in the target queue, filter out the sub-task data that needs to be executed, and prepare for execution. The sub-task data determined as the to-be-executed data is returned to the thread for execution. This means that the database system will pass the filtered and processed data to the corresponding thread for the thread to perform the actual data query operation.

[0196] In the embodiments of the present disclosure, by monitoring the data query requests, determining the to-be-executed data and returning it to the thread for execution, the query processing efficiency of the database system can be improved, and the tasks can be executed in a timely manner. According to the priority or other standards of the task, the to-be-executed data can be determined to flexibly schedule the tasks and ensure that important or urgent tasks are processed in priority. The to-be-executed data is returned to the thread for execution, which realizes the allocation and execution of the tasks and ensures that the tasks are properly processed in the database system. In summary, by monitoring the data query requests and determining the to-be-executed data, the tasks can be effectively processed and allocated to the threads for execution, so as to improve the efficiency and performance of the database system and ensure that the tasks are processed in a timely manner.

[0197] As an optional implementation, the method further includes: determining a minimum thread number allowed to be used by the queue based on the number of cores of the processor; determining the minimum thread number allowed to be used by the queue based on the resource configuration information of the queue; and adding the minimum thread number to the thread configuration information of the queue.

[0198] In this embodiment, the minimum number of threads allowed to be used in the queue can be determined based on the number of cores of the processor, and the minimum number of threads allowed to be used in the queue can be determined based on the resource configuration information, wherein the thread configuration information is also used to represent the minimum number of threads allowed to be used in the corresponding queue. The number of cores of the processor can be the number of CPU cores.

[0199] Optionally, max_thread can be an optional configuration. It can be used to set the upper limit of the number of threads that the queue can use, with a value range of (0, 1). The maximum number of threads that the queue can use will be the total number of threads multiplied by this proportion and then rounded down, but in order to ensure that the query of a queue can also make full use of the database system resources, the lower limit of the restriction is CPU core number multiplied by 2, that is, the minimum number of threads is CPU core number * 2. When not manually set, the default is to determine according to the proportion of cpu_share.

[0200] Optionally, by setting max_thread, the number of threads that the queue can use can be limited to avoid consuming too many database system resources. Limiting the number of threads can maintain the stability of the database system and avoid the situation that the performance decreases or crashes due to too many threads and high load of the database system. Adjusting the number of threads of the queue according to the actual situation can better allocate database system resources and improve the efficiency of the database system. Setting max_thread can limit the number of threads that the queue can use to avoid wasting and competing for database system resources due to too many threads. By reasonably setting max_thread, the load of the database system can be adjusted to ensure that the database system runs within a reasonable range. Excessive thread competition can be avoided to optimize the performance of the database system. Limiting the number of threads can avoid wasting resources and ensure that the database system resources are effectively utilized. Controlling the number of threads can avoid high load of the database system and improve the stability and reliability of the database system. According to the actual demand and the load of the database system, the value of max_thread can be flexibly adjusted to adapt to different scenes and demands.

[0201] In summary, the purpose of setting max_thread is to control database system resources, ensure database system stability, adjust database system load, improve database system performance, avoid resource waste, improve database system stability, and flexibly adjust the number of threads, etc.

[0202] In the embodiments of the present disclosure, the main purpose of limiting the maximum number of threads of the queue is to avoid small queries being crushed by large queries in terms of CPU resources, and to ensure that small queries can obtain sufficient execution opportunities and resource support. Limiting the maximum number of threads of the queue can help the database system to allocate thread resources more reasonably, and avoid some queues occupying too many thread resources and affecting the execution of other queues. If the number of threads that a queue can use is not limited under a 256-thread pool, the threads may have to select a split of a large query because they cannot obtain a split of a small query. This may cause CPU competition and lead to small queries being crushed by large queries in terms of CPU resources. Small queries can have a small number of splits and a short execution time, and limiting the maximum number of threads of the queue can ensure that small queries obtain sufficient CPU resources and execution opportunities, and avoid being preempted by large queries to affect execution efficiency. By limiting the maximum number of threads of the queue, resource contention and excessive consumption can be avoided, and the performance and response speed of the database system can be optimized, and different types of query tasks can be reasonably processed.

[0203] In summary, limiting the maximum number of threads of the queue can help the database system to better manage and schedule different types of query tasks, avoid unreasonable allocation of resources and competition, and improve the performance and stability of the database system. The above method helps to protect small queries from being disturbed by large queries, and ensures that the database system can efficiently process query tasks of different priorities.

[0204] Optionally, the max_thread value is -1, indicating that the SQA queue can use all the thread numbers, i.e., there is no limit on the number of threads, which can ensure that the SQA queue has priority and higher resource utilization in terms of CPU resources and thread numbers.

[0205] The embodiments of the present disclosure also provide a data processing method. FIG. 3 is a flowchart of a data processing method according to an embodiment of the present disclosure. As shown in FIG. 3, the method can be applied to an execution end and can include the following steps:

[0206] In step S302, a data query request from a thread is monitored, wherein the data query request is used to request to query to-be-executed data.

[0207] In the technical solution provided in the above step S302 of the present disclosure, whether a user has a data query request can be monitored. After the data query request is monitored, the data query request can be analyzed to determine to-be-executed data in a database system requested by the data query request.

[0208] For example, if a user needs to view some data in a database system in a server, the user can perform corresponding operations on a client device corresponding to the user to input the requirement of viewing the data, forming a data query request. After detecting the data query request on the client device, the data query request can be transmitted to the server through the network to analyze the data query request by using the server.

[0209] In step S304, in response to the data query request, a target scheduling time is allocated to the queue based on resource configuration information of the queue in the data warehouse, wherein the resource configuration information is used to represent the processing resources allowed to be scheduled by the queue during operation, the target scheduling time is used to represent the time allocated to the queue, and the data warehouse includes a plurality of queues.

[0210] In the technical solution provided by step S304 of the present disclosure, after detecting the data query request, the target scheduling time of the corresponding queue can be allocated based on the resource configuration information of each queue.

[0211] Optionally, the target scheduling time is determined by the resource configuration information and the scheduled time of each queue. Each queue has a corresponding resource configuration information configured, which can represent the weight of the CPU time obtained by the queue. The database system can be configured to set the proportion of CPU resources obtained by each queue. The database system determines the scheduling time of the queue according to the resource configuration information of the queue to which the query belongs.

[0212] In step S306, time difference information between the target scheduling time of the queue and the scheduled time of the corresponding queue is determined, wherein the time difference information is used to represent the difference between the target scheduling time and the scheduled time.

[0213] In the technical solution provided by step S306 of the present disclosure, after determining the target scheduling time of the corresponding queue based on the resource configuration information of each queue, the time difference information between the target scheduling time and the scheduled time can be determined.

[0214] In the embodiments of the present disclosure, the time difference information can measure the difference between different queues, which helps to determine the priority and urgency of each queue. Through the time difference information, the scheduling of queues with different time spans can be uniformly analyzed and compared, improving the standardization level of scheduling. According to the size of the time difference information, the database system can automatically adjust the execution order, priority or resource allocation of the task to optimize the performance and efficiency of the database system. By calculating the time difference information between the target scheduling time and the scheduled time, intelligent queue scheduling and resource allocation can be realized, and the priority of the queue execution is adjusted according to the time difference information to ensure the rational use of resources.

[0215] In step S308, a target queue is determined from the plurality of queues based on the time difference information of the queues, wherein the target queue corresponds to time difference information greater than the difference information threshold.

[0216] In the technical solution provided by step S308 of the present disclosure, the time difference information and the difference information threshold can be compared, and the queue with time difference information greater than the difference information threshold can be determined as the target queue.

[0217] Optionally, by determining the target queue with time difference information greater than the difference information threshold, it can be found that there is an imbalance in the resource allocation of the database system, which helps to optimize the allocation of resources and ensure that each queue can obtain reasonable scheduling time. The target queue with time difference information greater than the difference information threshold may contain some tasks with low priority, and adjusting the priority of these tasks can better meet the actual needs of the database system and improve the overall operation efficiency of the database system.

[0218] In step S310, the to-be-executed data is queried from the target queue.

[0219] In the technical solution provided by step S310 of the present disclosure, the to-be-executed data corresponding to the request query of the data query request can be queried from the target queue.

[0220] Optionally, by determining the target queue, the data query request can be obtained from the target queue, including the data content, query condition and other information to be queried.

[0221] In step S312, the queried to-be-executed data is returned to the thread execution.

[0222] In the technical solution provided by step S312 of the present disclosure, the queried to-be-executed data can be returned to the thread for execution.

[0223] Optionally, according to the acquired data query request, a corresponding query operation is performed in the database to acquire the required to-be-executed data. The acquired to-be-executed data can be returned to the external thread or the database system for further processing or display.

[0224] Through the steps S302 to S312 of the present disclosure, the data query request from the thread is monitored; in response to the data query request, the target scheduling time is allocated to the corresponding queue based on the resource configuration information of the plurality of queues in the data warehouse; the time difference information between the target scheduling time and the scheduled time of the corresponding queue is determined; the target queue is determined from the plurality of queues based on the time difference information; the to-be-executed data is queried from the target queue; and the queried to-be-executed data is returned to the thread execution. In the embodiment of the present disclosure, by setting the corresponding resource configuration information for each queue in the database system, the soft isolation between tasks of different priorities, i.e., data query requests of different degrees, is realized. In the process of executing the data query request, the target scheduling time of different queues can be automatically calculated according to the resource configuration information, and a more suitable queue is selected to query the to-be-executed data based on the target scheduling time, thereby avoiding the resource contention and competition problem in the execution of large queries and small queries, and realizing the technical effect of improving the efficiency of data processing, and solving the technical problem of low efficiency of data processing.

[0225] The embodiment of the present disclosure also provides a data processing method, and FIG. 4 is a flowchart of a data processing method according to an embodiment of the present disclosure. As shown in FIG. 4, the method can include the following steps:

[0226] In step S402, a data query request is monitored by calling a first interface, wherein the data query request is used to request to query to-be-executed data, and the first interface includes a first parameter, and a parameter value of the first parameter is the data query request.

[0227] In the technical solution provided in the step S402 of the present disclosure, the first interface can be called to monitor the data query request, wherein the first interface can include the first parameter, and the parameter value of the first parameter can be the data query request.

[0228] Optionally, whether a user has a data query request can be monitored through the first interface. After the data query request is monitored, the data query request can be analyzed to determine the to-be-executed data in the database system requested by the data query request.

[0229] In step S404, in response to the data query request, a target scheduling time is allocated to a queue based on resource configuration information of the queue, wherein the resource configuration information is used to represent the processing resources allowed to be scheduled by the queue in the running process, and the target scheduling time is used to represent the time allocated by the queue for scheduling.

[0230] In the technical solution provided in the step S404 of the present disclosure, the target scheduling time of each queue can be allocated based on the resource configuration information of each queue.

[0231] Optionally, the target scheduling time is determined based on the resource configuration information and the scheduled time of each queue. For example, the resource configuration information and the scheduled time of each queue are obtained, the total scheduling time of each queue is calculated based on the two parameters, a baseline value is selected from the total scheduling time of each queue, and the target scheduling time of each queue is obtained based on the CPU share ratio and the baseline value.

[0232] In the step S406, time difference information between the target scheduling time of each queue and the scheduled time of the corresponding queue is determined, wherein the time difference information is used to represent the difference between the target scheduling time and the scheduled time.

[0233] In the technical solution provided in the step S406 of the present disclosure, the time difference information between the target scheduling time and the scheduled time can be determined.

[0234] Optionally, the difference between the target scheduling time and the scheduled time is determined to obtain the time difference information. For example, the time difference is converted into a ratio, i.e., the ratio between the target scheduling time and the scheduled time, which can be obtained by dividing the time difference by the scheduled time.

[0235] In the step S408, a target queue is determined from the plurality of queues based on the time difference information of each queue, wherein the time difference information of the target queue is greater than a difference information threshold.

[0236] In the technical solution provided in the step S408 of the present disclosure, the time difference information and the difference information threshold can be compared, and the queue with the time difference information greater than the difference information threshold can be determined as the target queue.

[0237] Optionally, the database system can select a suitable SMLSQ queue as the target queue based on the resource configuration information and the scheduled time.

[0238] Optionally, the queue can manage the execution order of the queries, and the queries are queued and scheduled according to the order and priority of the queries entering the queue. Through the queue, it can be ensured that the queries are executed in a certain order, avoiding confusion and conflict between the queries. The queue can sort and schedule the queries according to the priority of the queries, ensuring that the queries with high priority can be executed preferentially. Different priority rules can be set according to the importance and urgency of different queries to ensure that critical queries can be processed in time. The queue can also sort and schedule the queries according to the execution time of the queries, ensuring that the queries running for a long time do not block the execution of other queries.

[0239] In step S410, the to-be-executed data of the query is queried from the target queue.

[0240] In the technical solution provided by the above step S410 of the present disclosure, the to-be-executed data of the query corresponding to the data query request can be queried from the target queue.

[0241] Optionally, the database system needs to determine the target queue, and obtain the data query request from the target queue, which includes the data content, query conditions and other information that needs to be queried.

[0242] In step S412, the to-be-executed data is output by calling the second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter is the to-be-executed data.

[0243] In the technical solution provided by the above step S412 of the present disclosure, the to-be-executed data can be output through the second interface, and the to-be-executed data queried can be returned to the thread for execution, wherein the second interface can include a second parameter, and the parameter value of the second parameter can be the to-be-executed data.

[0244] Optionally, according to the obtained data query request, a corresponding query operation is performed in the database to obtain the required to-be-executed data. The obtained to-be-executed data can be returned to the external thread or the database system for further processing or display.

[0245] By the steps S402 to S412 of the present disclosure, the data query request is monitored by calling the first interface; in response to the data query request, the target scheduling time is allocated to the corresponding queue based on the resource configuration information of the multiple queues respectively; the time difference information between the target scheduling time and the scheduled time of the corresponding queue is determined, wherein the time difference information is used to represent the difference between the target scheduling time and the scheduled time; the target queue is determined from the multiple queues based on the time difference information; and the to-be-executed data is output by calling the second interface. In the embodiment of the present disclosure, by setting the corresponding resource configuration information for each queue in the database system, the tasks of different priorities, i.e., the soft isolation between data query requests of different degrees, are realized. In the process of executing the data query request, the target scheduling time of different queues can be automatically calculated according to the resource configuration information, and a more suitable queue is selected to query the to-be-executed data based on the target scheduling time, thereby avoiding the resource contention and competition problems in the execution of large queries and small queries, and realizing the technical effect of improving the efficiency of data processing, and solving the technical problem of low efficiency of data processing.

[0246] According to the embodiment of the present disclosure, an embodiment of a data processing system is also provided. FIG. 5 is a schematic diagram of a data processing system according to an embodiment of the present disclosure. As shown in FIG. 5, the data processing system 500 can include a client 501 and a server 502.

[0247] The client 501 is configured to send a data query request, wherein the data query request is used to request to query to-be-executed data.

[0248] In this embodiment, the data query request corresponding to the to-be-executed data required by the user to query can be obtained by the client 501. After obtaining the data query request, the data query request can be transmitted to the server 502 through the network.

[0249] The server 502 is configured to, in response to the data query request, allocate a target scheduling time to a queue based on resource configuration information of the queue, wherein the resource configuration information is used to represent the processing resources allowed to be scheduled by the queue during the running process, and the target scheduling time is used to represent the time allocated to the queue; determine time difference information between the target scheduling time of the queue and a scheduled time of the corresponding queue, wherein the time difference information is used to represent the difference between the target scheduling time and the scheduled time; determine a target queue from multiple queues based on the time difference information of the queue, wherein the time difference information corresponding to the target queue is greater than a difference information threshold; query the to-be-executed data from the target queue; and return the to-be-executed data to the client.

[0250] In this embodiment, after the server 502 receives the data query request from the client 501, the resource configuration information of each queue can be analyzed to determine the processing resources that the data query request in the queue is allowed to schedule during the running process, and the target scheduling time when the current data query request is allocated to the queue to schedule the processing resources for execution can be determined. The time difference information between the target scheduling time and the corresponding queue scheduled time can also be determined. The queue with time difference information greater than the difference information threshold is determined as the target queue, so that the current data query request can use the target queue to schedule the corresponding processing resources to query the corresponding to-be-executed data.

[0251] Optionally, the resource configuration information and the scheduled time of the queue are obtained, the total scheduling time of each queue is calculated through the above two parameters, the total scheduling time that meets the condition is selected as the baseline value from the total scheduling time of each queue, and the target scheduling time of each queue is obtained according to the CPU share ratio allocation baseline value, so that the scheduling time of each queue can be reasonably allocated according to the configuration and scheduling conditions, and the task scheduling efficiency of the data processing system is improved.

[0252] Optionally, the gap between the target scheduling time and the scheduled time is determined to obtain the time difference information. For example, the time difference is converted into a proportional form, that is, the ratio (gap ratio) between the target scheduling time and the scheduled time, which can be obtained by dividing the time difference by the scheduled time.

[0253] Optionally, the suitable SMLSQ queue is selected as the target queue according to the resource configuration information and the scheduled time.

[0254] Optionally, the data query request can be obtained from the target queue by determining the target queue, including the data content to be queried, the query condition and other information. According to the obtained data query request, the corresponding query operation is performed in the database to obtain the required to-be-executed data. The obtained to-be-executed data can be returned to the external thread or the data processing system for further processing or display.

[0255] In the embodiment, a data processing system is provided. A data query request is sent by a client 501, wherein the data query request is used to request to query to-be-executed data; a target scheduling time is allocated to a corresponding queue based on resource configuration information of a plurality of queues by a server 502 in response to the data query request, wherein the resource configuration information is used to represent processing resources allowed to be scheduled by the corresponding queue in a running process; time difference information between the target scheduling time and a scheduled time of the corresponding queue is determined, wherein the time difference information is used to represent a difference between the target scheduling time and the scheduled time; a target queue is determined from the plurality of queues based on the time difference information, wherein the target queue corresponds to time difference information greater than a difference information threshold; the to-be-executed data is queried from the target queue; and the to-be-executed data is returned to the client, thereby achieving a technical effect that can effectively improve the efficiency of data processing, and solving the technical problem of low efficiency of data processing.

[0256] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present disclosure, such as data for inspection, are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose authorization or refusal.

[0257] With the development of the data era, companies are increasingly valuing data and its management. Integrating multiple different database servers under a unified database has many advantages. A unified database can better integrate and manage data, making it easier to maintain and use. Data sharing and exchange between different database systems become more convenient, reducing the risk of data redundancy and inconsistency. At the same time, a unified database system can provide more efficient and accurate data query and analysis functions, better supporting decision-making and task development. A unified database system can achieve multi-source big data analysis, helping to discover more value in data. By integrating multiple data sources, we can better understand customer needs, market trends, and competitor dynamics, and better develop relevant strategies, product plans, and task development directions. Such comprehensive data analysis can help discover new business opportunities and growth points, improving competitiveness and innovation capabilities. From a cost perspective, a shared database system often means lower costs. Maintaining a unified database system is more efficient and economical than maintaining multiple database systems, reducing waste of manpower, time, and resources. In addition, a unified database system can better utilize hardware and software resources, reducing investment and operating costs of information technology (IT) infrastructure.

[0258] In summary, integrating multiple different database servers under a unified database is a wise choice that can improve data management and analysis efficiency and quality, reduce costs, and discover more value, thereby promoting sustainable development and innovation.

[0259] Under the above trend, multiple different types of workloads run in the shared database, including core task queries and temporary analysis non-core queries. These workloads have different characteristics, some requiring high overall response time and low resource consumption, while others are complex and resource-intensive. Users have different performance requirements for different types of workloads, so the database system needs to effectively manage and schedule these diverse workloads to meet user performance demands and improve the overall efficiency and responsiveness of the database system.

[0260] Simultaneously running queries in the data warehouse share the database system's CPU, memory, disk I / O, and other resources. If some queries consume a lot of database system resources without restrictions, other queries will have to compete for resources, affecting overall performance. Therefore, to ensure the stability and performance of the database system, queries that consume a lot of resources must be limited, and database system resources must be allocated and managed reasonably to ensure that each query receives adequate resource support, improving the overall efficiency and responsiveness of the database system.

[0261] In the current data warehouse, four different priority queues have been provided for customers, and different queries will enter different queues according to their own priority, but there is only a difference in the maximum concurrent limit on the front-end node between these queues, and the priority of the query is not reflected on the execution side. At the same time, the isolation between different queues is very weak and cannot meet the customer's task demand well. Due to the problems existing in the priority queue setting of the current data warehouse, including the fact that the query priority is not reflected on the execution side and the insufficient isolation between different queues. In order to better meet the customer's task demand, it is necessary to further optimize and improve the database system, strengthen the management of query priority and resource isolation, and improve the performance and flexibility of the database system, so as to better meet the different task demands of customers and improve the overall performance and user experience of the database system.

[0262] In addition, the SQA of the current data warehouse splits the queue to make small queries not blocked by large queries and queued, but there is still resource contention and competition on the execution end. The previously developed Multi level split queue implements a mechanism for gradually differentiating the execution of large and small queries over time, gradually reducing the time slice resources allocated to large queries. The above mechanism can more effectively manage the resource allocation of different size queries and improve the execution efficiency and query experience optimization of the database system.

[0263] In actual testing, it is found that since all queries start from the first layer of the Multi level split queue and gradually descend, small queries in the first layer are often affected by newly entered queries. At the same time, due to the long execution time of large queries and the large number of splits, under the current 256 threads, it is not possible to well limit the scheduling time of large queries and the resource contention for small queries.

[0264] Under the current Multi level split queue execution model, although small queries have an advantage to some extent, the overall isolation effect still has a lot of room for improvement. That is, the above method still has the technical problem of low data processing efficiency.

[0265] Further, the present disclosure provides a priority queue-based OLAP data warehouse hybrid load isolation scheme, which creates a new pending split queue on the execution side, including five SMLSQ queues. New incoming queries are executed in different Simple Multi Level Split Queues according to the queue they belong to on the RC. The SMLSQ here basically realizes the function of the original Multi Level Split Queue, but simplifies the scheduling process and reduces the complexity of scheduling logic and lock overhead. Each SMLSQ also has a certain resource tilt for small queries, avoiding the impact of high resource consumption of large queries on the performance of other queries. The pending split queue is divided into five SMLSQ queues, and queries are allocated to different queues for execution according to the different queues they belong to on the RC, realizing the function of multi-level splitting. By simplifying the scheduling process and reducing the lock overhead, the execution efficiency and performance of the database system are improved. Each SMLSQ has a certain resource tilt for small queries, avoiding the impact of high resource consumption of large queries on the performance of other queries, optimizing the allocation and utilization of database system resources. Through this setting, different types of queries can be better managed, the execution efficiency and performance of the database system are improved, and the query experience and overall performance of the database system are optimized. Further, the resource contention and competition problems during the execution of large queries and small queries are avoided, the technical effect of improving the efficiency of data processing is realized, and the technical problem of low efficiency of data processing is solved.

[0266] The above method of the embodiment will be further described below.

[0267] In the embodiments of the present disclosure, a new pending split queue is proposed, including five Simple Multi Level Split Queue (SMLSQ) queues. The design of this new queue aims to allocate new incoming queries to different Simple Multi Level Split Queues for execution according to the queue they belong to on the RC. The purpose of this is to better utilize the previously developed large query differentiation mechanism, while simplifying the scheduling process and reducing the complexity of scheduling logic and lock overhead.

[0268] Optionally, in the simplified scheduling process, the following methods can be adopted: Simplifying the scheduling process can reduce the locking operation on shared resources, thereby reducing the overhead and competition of locks, and improving the concurrency performance of the database system. Simplifying the scheduling process can reduce the complex scheduling logic, simplify the task allocation process, and improve the execution efficiency of the database system. Simplifying the scheduling process can adopt a simpler task scheduling strategy, reduce the complex judgment of query priority and resource demand, and improve the response speed and stability of the database system. Simplifying the scheduling process can pay more attention to the utilization efficiency of resources, avoid waste and excessive consumption of resources, and thus improve the overall performance of the database system. In summary, simplifying the scheduling process can help reduce the complexity and overhead of the database system, improve the execution efficiency and performance of the database system, while maintaining the stability and reliability of the database system.

[0269] Optionally, in this new design, Simple Multi Level Split Queue basically realizes the function of the original Multi Level Split Queue, but simplifies the scheduling process. Each Simple Multi Level Split Queue can also have a certain resource tilt for small queries to avoid excessive resource consumption of large queries affecting the performance of other queries. Through this setting, the following advantages can be achieved: By allocating queries to different Simple Multi Level Split Queues for execution, the resources of the database system can be more effectively utilized, and the overall performance of the database system can be improved. Simplifying the scheduling process can reduce the complexity of the database system, reduce the overhead of scheduling, and improve the execution efficiency of the database system. Each Simple Multi Level Split Queue has a certain resource tilt for small queries, which can avoid excessive resource consumption of large queries, thereby improving the stability of the database system and the performance of queries.

[0270] In summary, the above-mentioned new pending split queue setting can effectively improve the performance and stability of the database system, simplify the scheduling logic and fully utilize the previous query differentiation mechanism, thereby improving the overall performance of the database system.

[0271] Optionally, in the above-mentioned queue, resource inclination to small queries means allocating more resources to small queries inside the Simple Multi Level Split Queue to ensure that the queue can execute faster and obtain more database system resources. The implementation of this resource inclination can be realized in the following ways: adjusting the priority of small queries in the queue, so that the queue has a higher execution priority and can be executed faster. Allocate more computing resources such as CPU, memory, etc. to small queries, and prioritize the resource needs of small queries when executing. Limit the number of large queries executed in the queue at the same time to provide more execution opportunities for small queries.

[0272] Optionally, the effect of resource inclination includes: by allocating more resources to small queries, the execution speed of small queries can be accelerated, and the response speed and performance of the database system can be improved. Through resource inclination, the excessive resource consumption of large queries can be avoided, thereby avoiding affecting the execution performance of other queries. Resource inclination can help balance the utilization of database system resources and avoid the situation that uneven allocation of resources leads to performance degradation or crash of the database system. Through resource inclination to small queries in the Simple Multi Level Split Queue, the overall performance and stability of the database system can be effectively improved, while ensuring that various types of queries can be reasonably allocated resources and executed.

[0273] In this embodiment, Figure 6 is a schematic diagram of a preliminary version of a pending split queue processing process. As shown in Figure 6, there is a thread pool of 256 threads on the execution node, and each thread is in an infinite loop to take out a split from the pending Spilts queue and execute a time slice. In this case, the pending Spilts queue is described as a Priority Blocking Queue, in which queries basically follow the FIFO priority. Specifically, the Priority Blocking Queue is a priority-supported unbounded blocking queue, and elements are processed in order of their priority. Here, queries determine the priority of execution according to the order in which they enter the pending Spilts queue, i.e. the first-in-first-out principle. Each thread gets a split from the pending Spilts queue and executes a time slice, and the above mechanism can effectively utilize the thread pool resources of 256 threads while maintaining the order of queries. Since the Priority Blocking Queue will sort elements according to their priority, queries are processed according to the FIFO priority, ensuring the execution order of queries.

[0274] The above setting mode can ensure that the query is processed in time and the order of the query is maintained, avoiding the possible priority inversion or starvation situation. At the same time, since each thread in the thread pool is in an infinite loop to obtain and execute the query, the multi-thread parallel processing of the query can be more fully utilized, and the concurrent performance and response speed of the database system are improved.

[0275] Optionally, there is a thread pool with 256 threads on the execution node. Each thread will be in an infinite loop to obtain a split from the pending Spilts queue and execute a time slice. The pending Spilts queue here is a Priority Blocking Queue, which is a priority-supported unbounded blocking queue, and the query basically follows the FIFO priority. Specifically, each thread obtains a split from the queue and executes according to the FIFO priority, which means that the split that enters the queue first will be processed first. The Priority Blocking Queue will sort the elements according to their priority to ensure that splits with higher priority are processed first.

[0276] Through the above setting, the thread pool resources of 256 threads can be fully utilized, and the order of the query can be maintained. Each thread obtains and executes a split from the queue, which can effectively process the query task in the database system and maintain the efficiency and responsiveness of the database system. The FIFO priority mechanism based on the Priority Blocking Queue can ensure that the query is processed according to the order of entering the queue, avoiding the potential priority inversion or starvation situation.

[0277] Optionally, there is a thread pool with 256 threads on the execution node. Each thread will be in an infinite loop to obtain a split from the pending Spilts queue and execute a time slice, and the pending Spilts queue at this time is a Priority Blocking Queue, and the query basically follows the FIFO priority.

[0278] In the embodiments of the present disclosure, FIG. 7 is a diagram of an improved multi-level priority queue and scheduling time according to an embodiment of the present disclosure. As shown in FIG. 7, the pending Spilts queue is improved so that it is no longer a single queue, but a multi-level feedback queue (Multi level feedback queue) composed of multiple priority queues, including Level 0, Level 1, Level 2, Level 3, and Level 4 priority queues. The scheduling time allocated to different priority queues decreases by a factor of 2 layer by layer, that is, the expected scheduling time can decrease by a factor of 2 layer by layer according to different priority queues. When a query enters, it will be in the first level priority queue, and then it will gradually decrease the level as the execution time increases, thereby avoiding the excessive impact of large queries on the execution of small queries.

[0279] Optionally, FIG. 8 is a diagram of a multi-level priority queue according to an embodiment of the present disclosure. As shown in FIG. 8, there are six priority queues on the RC node, namely, the ROOT, SQA, high, normal, low, and LOWEST queues.

[0280] As shown in FIG. 8, in the LOWEST queue, the initial priority of data extraction, transformation, and loading (ETL) or query is set to the lowest (lowest), that is, query_priority = lowest, and then adjusted to 5, which is in the medium priority range between 0 and 9 (query_priority = 5 (query_priority = 0-9)). This process can mean that initially the task is considered to be the least urgent or the least important, but later it is promoted to a medium priority in the database system so as to be executed faster. In the low queue, the initial priority of batch tasks or queries is set to low (low), that is, query_priority = low, and then adjusted to 15, which is in the higher priority range between 10 and 19 (query_priority = 15 (query_priority = 10-19)). This process can mean that initially the task is considered to be less urgent or important, but later it is promoted to a higher priority in the database system so as to be executed faster.

[0281] As shown in FIG. 8, in the normal queue, the initial priority of the task is set to the default value or normal, i.e., query_priority = normal, and then adjusted to 25, which is in the higher priority range between 20 and 29 (query_priority = 25 (query_priority = 20-29)). This process indicates that the task is initially considered to have a general priority, but later promoted to a higher priority in the database system in order to be processed faster. In the high queue, the initial priority of the task is set to high, i.e., query_priority = high, and then adjusted to 35, which is in the higher priority range between 30 and 39 (query_priority = 35 (query_priority = 30-39)). This indicates that the task is considered to have a high urgency and importance, and is set to a higher priority in the database system to ensure its fast processing and execution.

[0282] As shown in FIG. 8, in the ROOT queue, the query type of the task is set to system, i.e., query_type = system, and then the priority of the task is adjusted to 45, which is in the priority range between 40 and 49 (query_priority = 45 (query_priority = 40-49)). This indicates that the task is considered to be a system-level task with a high priority to ensure the normal operation of the database system and the timely processing of critical tasks. In the SQA queue, the query priority of the task is set to SQA, i.e., query_priority = sqa, and the priority of the task is adjusted to 255 (query_priority = 255). This value can represent the highest priority to ensure that the SQA task is processed quickly and efficiently. This setting indicates that the SQA task has the highest urgency and importance in the database system to ensure effective management and maintenance of the database system quality and performance.

[0283] Optionally, two configurations, such as cpu_share and max_thread, can be added to the four priority queues of high, normal, low, and LOWEST. The introduction of these two configurations can help the database system to more flexibly manage and allocate CPU resources and thread quantities, achieve CPU resource soft isolation and thread quantity limitation for priority queues, and thus optimize the performance and resource utilization of the database system. By reasonably configuring cpu_share and max_thread, the CPU resource allocation and thread utilization of different priority queues can be better controlled, ensuring that the database system can efficiently process various priority queries.

[0284] For example, cpu_share is a mandatory configuration, an integer between 1 and 1000, used to set how much cpu time the priority queue can get, which can achieve cpu resource soft isolation. cpu_share is a relative value, indicating the weight of the running priority queue that can get cpu resources. For example, 3 priority queues are configured, low, normal and high, cpu_share is 10, 30 and 40 respectively, at a certain moment, low and normal are running tasks, and high has no task, at this moment, low can get 25% (10 / (10+30)) of cpu resources, and normal can get 75% of cpu resources. If the database system has only one priority queue with queries running, regardless of the value of its cpu_share, it can get all the cpu resources.

[0285] For example, max_thread is an optional configuration, used to set the upper limit of the number of threads that the queue can use, ranging from (0, 1), the maximum number of threads that the queue can use will be the total number of threads multiplied by this proportion and rounded down, but in order to ensure that the queries of a queue can also use up the resources of the database system, the lower limit of the restriction is CPU core number * 2. The configuration proportion defaults to the proportion of cpu_share when not manually set.

[0286] Optionally, SQA queue is treated specially, the cpu_share value of SQA queue is equal to the sum of other queues, and max_thread is -1, which can use all the thread quantity. The queries of the ROOT queue temporarily enter the SQA queue on the execution side, and in the later stage, a separate thread pool may be considered to execute the database system queries. The queries of the ROOT queue temporarily enter the SQA queue on the execution side, which means that the queries of the ROOT queue will use the resources of the SQA queue first when executing. This temporary entry into the SQA queue can improve the execution efficiency and priority of the queries of the ROOT queue. In the later stage, a separate thread pool may be considered to execute the database system queries. This approach can ensure that the database system query tasks can be executed independently and are not affected by other queues, ensuring the stability and reliability of the database system queries. Through the special setting and processing of SQA queue and ROOT queue, the database system can better manage and schedule different types of query tasks, improving the performance and efficiency of the database system. At the same time, a separate thread pool is isolated to execute the database system queries, which can further optimize the running and resource utilization of the database system, ensuring that the core functions of the database system can be fully supported and guaranteed.

[0287] In this embodiment, a new pending split queue is created on the execution side, including five SMLSQ queues. New incoming queries are executed in different Simple Multi Level Split Queues according to the queue they belong to on the RC. These SMLSQ queues basically implement the functions of the original Multi Level Split Queue, but simplify the scheduling process and reduce the complexity of scheduling logic and lock overhead. At the same time, each SMLSQ queue can also tilt resources to small queries to avoid excessive resource consumption of large queries affecting the performance of other queries.

[0288] Optionally, when the thread requests execution of the split to the priority schedule split queue, the appropriate SMLSQ queue can be selected according to the CPU share value and the time already scheduled. This ensures that the scheduling time between different SMLSQ queues basically conforms to the proportion of CPU share, and thus realizes reasonable scheduling and execution of different priority tasks. This setting fully utilizes the resource allocation and task scheduling of different SMLSQ queues, while simplifying the scheduling logic and resource management, improving the performance and efficiency of the database system. At the same time, resource tilting for small queries can ensure that the database system can better handle different types of query tasks, improving the overall performance and stability of the database system. By selecting the appropriate SMLSQ queue according to the CPU share value and the time already scheduled, query tasks can be better managed and scheduled, improving the resource utilization and performance of the database system.

[0289] Optionally, Figure 9 is a schematic diagram of an overall architecture of a priority queue-based data warehouse hybrid load isolation according to an embodiment of the present disclosure, as shown in Figure 9, a new pending split queue is created on the execution side, including five SMLSQ queues, which can include simple multi-level split queue 0, simple multi-level split queue 1, simple multi-level split queue 2, simple multi-level split queue 3, and simple multi-level split queue 4. New incoming queries enter different Simple Multi Level Split Queue for execution according to the queue they belong to on the RC. In order to fully utilize the size query differentiation mechanism developed earlier, the SMLSQ here basically realizes the function of the original Multi Level Split Queue, but simplifies the scheduling process to reduce the complexity of scheduling logic and lock overhead, so that each SMLSQ can also have a certain resource tilt for small queries, avoiding the high resource consumption of large queries affecting the performance of other queries. When a thread requests to take a split execution from the priority schedule split queue, it can select the appropriate SMLSQ queue according to the CPU share value and the time already scheduled, so that the scheduling time between different SMLSQ queues basically conforms to the proportion of CPU share.

[0290] Optionally, as shown in Figure 9, due to the small number of splits and short execution time of small queries, under a 256-thread pool, if the number of threads available to the queue is not limited, it is often found that threads have to choose large query splits because they cannot get small query splits, and eventually completely crush small queries in CPU competition. Therefore, the maximum number of threads is set for different queues to avoid small queries being crushed by large queries in the number of threads and being preempted in CPU resources.

[0291] Optionally, in the internal scheduling logic, the scheduling can continuously maintain the proportion of the scheduling time between queues consistent with the proportion of CPU share. This means that the database system will select the appropriate queue according to the CPU share value and the time already scheduled to balance the scheduling time proportion between different queues and maintain consistency with the CPU share value.

[0292] Figure 10(a) is a flowchart of a method of taking a task from a queue for processing according to an embodiment of the present disclosure, as shown in Figure 10(a), the method of taking a split can include the following steps:

[0293] Step S1001, calculate max target total scheduling time, index = 0.

[0294] In this embodiment, the maximum target total scheduling time is calculated, and index is set to 0, ready to start the loop processing queue.

[0295] Optionally, the maximum target total scheduling time is calculated first, and index is set to 0, ready to start the loop processing queue. The purpose of this step is to determine the overall scheduling time, and initialize the index value.

[0296] Step S1002, Index < 5.

[0297] In this embodiment, it is determined whether the index is less than 5. If yes, step S1003 is executed, otherwise step S1005 is executed. This step is to ensure that the index value is within the valid range. If the index is less than 5, the queue is continued to be processed, otherwise the processing is ended.

[0298] Step S1003, it is determined whether the queue with index number index is not empty and the number of threads does not exceed the maximum limit.

[0299] In this embodiment, it is determined whether the queue with index number index is not empty and the number of threads does not exceed the maximum limit. If the above conditions are met, step S1004 can be executed, otherwise step S1002 is returned to execute.

[0300] Step S1004, the ratio of target scheduling time / actual scheduling time is calculated and saved, index=index+1.

[0301] In this embodiment, if the queue with index number index is not empty and the number of threads does not exceed the maximum limit. The ratio of target scheduling time to actual scheduling time is calculated and saved in the array, and then index is added by 1. This step is to calculate the scheduling time ratio of the current queue and save it in the array, and then continue to process the next queue.

[0302] Step S1005, whether the array storing the ratio is empty.

[0303] In this embodiment, it is checked whether the array storing the ratio is empty. If empty, step S1006 is executed, otherwise step S1007 is executed.

[0304] Optionally, it is checked whether the array storing the ratio is empty. If empty, return null, indicating that there is no queue that meets the conditions. This step is to check whether the ratio array is empty after all queues are processed in the loop. If empty, it indicates that there is no queue that meets the conditions.

[0305] Step S1006, return null.

[0306] In this embodiment, null is returned, indicating that there is no queue that meets the condition.

[0307] Step S1007, a split is taken out from the index queue corresponding to max(ratio) and returned.

[0308] In this embodiment, the index corresponding to the maximum ratio in the storage ratio array is found, and then a split is taken out from the queue and returned.

[0309] Figure 10(b) is a flowchart of a method of putting a task into a queue according to an embodiment of the present disclosure. As shown in Figure 10(b), the above-mentioned method of putting a split can include the following steps:

[0310] Step S1008, the queue corresponding to the priority of the split is obtained.

[0311] In this embodiment, the queue corresponding to the priority of the split is obtained. The purpose of this step is to determine the queue that needs to be put in according to the priority of the split.

[0312] Step S1009, it is determined whether the queue is empty.

[0313] In this embodiment, it is determined whether the queue is empty. If yes, step S1010 can be performed, otherwise, step S1011 can be performed. This step is to check whether the target queue is empty, so as to determine whether the scheduling time needs to be padded or the split is directly put in.

[0314] Step S1010, the scheduling time of the corresponding SMLSQ is padded to the minimum value by calculating the minimum target total time.

[0315] In this embodiment, the scheduling time of the corresponding SMLSQ is padded to the minimum value by calculating the minimum target total time. This step is to pad the scheduling time of the corresponding queue to ensure that the minimum target total time is reached.

[0316] Step S1011, the split is put into the corresponding SMLSQ.

[0317] In this embodiment, the split is put into the corresponding SMLSQ. This step is to put the split into the corresponding queue, completing the operation of integrating the queue.

[0318] Optionally, when executing the take split process, i.e., the process of obtaining and executing the query task, the following steps can be followed: the put split into the corresponding queue process, first, the database system can put the query task into the corresponding queue according to the priority of the query and other related parameters. According to the description, it may be according to the CPU share value and the scheduled time to select the appropriate queue. Once the query task enters the queue, according to the scheduling logic and strategy, it can be determined when to take out the query task from the queue and execute it. According to the scheduling time ratio between the queues and the CPU share value, the appropriate opportunity to execute the query task can be selected. According to the priority of the queue, the CPU share value and other parameters, the database system resources can be reasonably allocated to the query tasks in different queues, so as to ensure that the database system can efficiently process various types of query tasks. Through such internal scheduling logic and process, the database system resources and the scheduling of query tasks can be reasonably allocated according to the priority and CPU share value of different queues, so as to realize the effective processing and execution of different types of query tasks. This scheduling method can help the database system to balance the use of resources and improve the performance and stability of the database system.

[0319] For example, FIG. 11 is a schematic diagram of a scheduling example according to an embodiment of the present disclosure. As shown in FIG. 11, it is assumed that the customer has set the cpu_share values of the high, normal, low, and lowest queues to be 8, 4, 2, and 1 respectively, and the cpu_share value of the SQA queue is the sum of them, which is 15. The maximum number of threads that can be used is 256, 128, 64, 32, and 16 respectively. The currently scheduled time of each queue is 140, 90, 40, 30, and 10 respectively. When a thread in the database system needs to take a split to execute, each queue first calculates the total scheduling time according to the scheduling time and the cpu_share value ratio of itself. From the total scheduling time, the current baseline is selected, and the total scheduling time baseline is further allocated as the target scheduling time according to the cpu_share ratio. The gap ratio is obtained by the target scheduling time / scheduling time, the target gap ratio is selected from the gap ratio, and the corresponding queue is determined to take out a split and return it to the external thread for execution. If the queue is empty or the queue has reached the upper limit of the number of threads that can be used, the second largest queue is selected.

[0320] Optionally, in the database system, when a split needs to be put into a queue, the scheduling time of the queue and the influence of the cpu_share value need to be considered. If the queue is not empty, the split can be directly put into the queue. However, if the queue is empty, it may mean that the queue has been empty for a long time, and its scheduling time is less. At this time, directly putting the split may cause the database system to tend to schedule this queue, causing preemption to other queues. In order to avoid this situation, when putting the split, the scheduling time of the queue needs to be pulled to a reasonable range.

[0321] Figure 12 is a schematic diagram of CPU usage of a 16-thread full system according to an embodiment of the present disclosure. As shown in Figure 12, the horizontal axis of the image is time, and the vertical axis is CPU usage, wherein the average CPU usage of the control node, the maximum CPU usage of the control node, the storage computing node CPU usage, and the computing node CPU usage are described. 16 threads are sufficient to fully utilize the CPU resources of the database system, which means that the database system can achieve maximum CPU utilization when using 16 threads. Letting the database system reach maximum CPU utilization can be referred to as full system. When the CPU utilization of the database system reaches 100%, it means that the CPU resources of the database system have been fully utilized, and there is no idle processing capacity. This may mean that the database system is processing a large number of tasks or calculations, or running an application that requires a large amount of CPU resources. In this case, the database system may exhibit high response speed and processing capacity, but it may also cause the database system to be overloaded, affecting the execution of other tasks. By setting the database system to use 16 threads to fully utilize CPU resources, it can ensure that the database system can maximize the processing capacity of the CPU when processing tasks, and improve the overall performance and efficiency of the database system. By reasonably allocating threads and resources, the load of the database system can be balanced, avoiding resource waste or excessive occupation, thereby improving the stability and performance of the database system.

[0322] In summary, by using 16 threads to fully utilize the CPU resources of the database system, the database system can achieve high CPU utilization, improve the processing capacity and performance of the database system. This approach can help the database system process tasks more efficiently and improve the response speed and efficiency of the database system.

[0323] In the embodiments of the present disclosure, if the data to be queried needs to be queried according to the data query request, the received data query request from the client can be monitored to analyze the to-be-executed data that needs to be queried from the server. According to the resource configuration information of each queue, the processing resources that the data query request in the queue is allowed to schedule in the running process can be analyzed, and the target scheduling time when the current data query request is allocated to the processing resources in the queue for execution can be determined. The time difference information between the target scheduling time and the scheduled time of the corresponding queue can also be determined. The queue with time difference information greater than the difference information threshold is determined as the target queue, so that the current data query request can use the target queue to schedule the corresponding processing resources to query the corresponding to-be-executed data. In the embodiments of the present disclosure, by setting the corresponding resource configuration information for each queue in the database system, the soft isolation between tasks of different priorities, that is, data query requests of different degrees, is realized. In the process of executing the data query request, the target scheduling time of different queues can be automatically calculated according to the resource configuration information, and a more suitable queue can be selected based on the target scheduling time to query the to-be-executed data, thereby avoiding the resource contention and competition problems in the execution of large queries and small queries, realizing the technical effect of improving the efficiency of data processing, and solving the technical problem of low efficiency of data processing.

[0324] According to the embodiments of the present disclosure, a data processing apparatus for implementing the data processing method shown in FIG. 2 is also provided.

[0325] FIG. 13 is a schematic diagram of a data processing apparatus according to an embodiment of the present disclosure. As shown in FIG. 13, the data processing apparatus 1300 can include a first monitoring component 1302, a first allocation component 1304, a first determination component 1306, a second determination component 1308, and a first query component 1310.

[0326] The first monitoring component 1302 is configured to monitor a data query request, wherein the data query request is used to request to query to-be-executed data.

[0327] The first allocation component 1304 is configured to allocate a target scheduling time to a queue based on resource configuration information of the queue in response to the data query request, wherein the resource configuration information is used to represent the processing resources that the queue is allowed to schedule in the running process, and the target scheduling time is used to represent the time allocated to the queue.

[0328] The first determination component 1306 is configured to determine time difference information between the target scheduling time of the queue and the scheduled time of the queue, wherein the time difference information is used to represent the difference between the target scheduling time and the scheduled time.

[0329] The second determining component 1308 is configured to determine a target queue from the plurality of queues based on time difference information of the queues, where the target queue corresponds to time difference information greater than a difference information threshold.

[0330] The first querying component 1310 is configured to query the to-be-executed data from the target queue.

[0331] It should be noted that the first monitoring component 1302, the first allocating component 1304, the first determining component 1306, the second determining component 1308, and the first querying component 1310 correspond to steps S202 to S210 in the above embodiment, and the five components have the same instances and application scenarios as the corresponding steps, but are not limited to the content disclosed in the above embodiment. It should be noted that the above components can be hardware components or software components stored in a memory (for example, the memory 1604) and processed by one or more processors (for example, the processors 1602a, 1602b,..., 1602n), or the above components can be a part of the apparatus and can run in the computer terminal A provided in the following embodiment.

[0332] According to the embodiments of the present disclosure, a data processing apparatus for implementing the data processing method of the device shown in FIG. 3 is also provided.

[0333] FIG. 14 is a schematic diagram of another data processing apparatus according to an embodiment of the present disclosure. As shown in FIG. 14, the data processing apparatus 1400 can include a second monitoring component 1402, a second allocating component 1404, a third determining component 1406, a fourth determining component 1408, a second querying component 1410, and a first executing component 1414.

[0334] The second monitoring component 1402 is configured to monitor a data query request from a thread, where the data query request is used to request to query to-be-executed data.

[0335] The second allocating component 1404 is configured to allocate a target scheduling time to a queue based on resource configuration information of the queue in a data warehouse in response to the data query request, where the resource configuration information is used to represent processing resources allowed to be scheduled by the queue during running, the target scheduling time is used to represent a time allocated to the queue, and the data warehouse includes a plurality of queues.

[0336] The third determining component 1406 is configured to determine time difference information between the target scheduling time of the queue and a scheduled time of the corresponding queue, where the time difference information is used to represent a difference between the target scheduling time and the scheduled time.

[0337] The fourth determining component 1408 is configured to determine a target queue from the plurality of queues based on the time difference information of the queues, where the target queue corresponds to time difference information greater than the difference information threshold.

[0338] The second querying component 1410 is configured to query the to-be-executed data from the target queue.

[0339] The first executing component 1414 is configured to return the queried to-be-executed data to the thread execution.

[0340] It should be noted that the second monitoring component 1402, the second allocating component 1404, the third determining component 1406, the fourth determining component 1408, the second querying component 1410, and the first executing component 1414 correspond to steps S302 to S312 in the above embodiment, and the six components have the same instances and application scenarios as the corresponding steps, but are not limited to the content disclosed in the above embodiment. It should be noted that the above components can be hardware components or software components stored in a memory (for example, the memory 1604) and processed by one or more processors (for example, the processors 1602a, 1602b,..., 1602n), or the above components can be a part of the apparatus and can run in the computer terminal A provided in the following embodiment.

[0341] According to the embodiments of the present disclosure, a data processing apparatus for implementing the data processing method of the device shown in FIG. 4 is also provided.

[0342] FIG. 15 is a schematic diagram of another data processing apparatus according to an embodiment of the present disclosure. As shown in FIG. 15, the data processing apparatus 1500 can include a third monitoring component 1502, a third allocating component 1504, a fifth determining component 1506, a sixth determining component 1508, a third querying component 1510, and a second executing component 1512.

[0343] The third monitoring component 1502 is configured to monitor a data query request by calling a first interface, where the data query request is used to request to query to-be-executed data, and the first interface includes a first parameter, and a parameter value of the first parameter is the data query request.

[0344] The third allocating component 1504 is configured to allocate a target scheduling time to a corresponding queue based on resource configuration information of the queue in response to the data query request, where the resource configuration information is used to represent processing resources allowed to be scheduled to the queue during running, and the target scheduling time is used to schedule a time allocated to the queue.

[0345] The fifth determining component 1506 is configured to determine time difference information between the target scheduling time of the queue and the scheduled time corresponding to the queue, where the time difference information is used to represent the difference between the target scheduling time and the scheduled time.

[0346] The sixth determining component 1508 is configured to determine the target queue from the plurality of queues based on the time difference information of the queue, where the target queue corresponds to time difference information greater than the difference information threshold.

[0347] The third querying component 1510 is configured to query the to-be-executed data from the target queue.

[0348] The second executing component 1512 is configured to output the to-be-executed data by calling the second interface, where the second interface includes a second parameter, and the parameter value of the second parameter is the to-be-executed data.

[0349] It should be noted that the third monitoring component 1502, the third allocating component 1504, the fifth determining component 1506, the sixth determining component 1508, the third querying component 1510 and the second executing component 1512 correspond to steps S302 to S312 in the above embodiment, and the six components have the same instances and application scenarios as the corresponding steps, but are not limited to the content disclosed in the above embodiment. It should be noted that the above components can be hardware components or software components stored in the memory (for example, the memory 1604) and processed by one or more processors (for example, the processors 1602a, 1602b, …, 1602n), and the above components can also be a part of the device and can run in the computer terminal A provided in the following embodiment.

[0350] In the data processing apparatus, if the data to be queried by the data query request needs to be queried, the received data query request from the client can be monitored to analyze the to-be-executed data to be queried from the server. According to the resource configuration information of each queue, the processing resource to which the data query request in the queue is allowed to be scheduled during running can be analyzed, and the target scheduling time when the current data query request is allocated to the processing resource in the queue for execution can be determined. The time difference information between the target scheduling time and the scheduled time of the corresponding queue can also be determined. The queue with the time difference information greater than the difference information threshold is determined as the target queue, so that the current data query request can use the target queue to schedule the corresponding processing resource to query the corresponding to-be-executed data. In the embodiment of the present disclosure, by setting the corresponding resource configuration information for each queue in the database system, the soft isolation between tasks of different priorities, that is, data query requests of different degrees, is realized. During the execution of the data query request, the target scheduling time of different queues can be automatically calculated according to the resource configuration information, and a more appropriate queue can be selected based on the target scheduling time to query the to-be-executed data, thereby avoiding the resource contention and competition problem during the execution of large queries and small queries, and realizing the technical effect of improving the efficiency of data processing, and solving the technical problem of low efficiency of data processing.

[0351] The embodiment of the present disclosure can provide a computer terminal, which can be any one of the computer terminal devices in the computer terminal group. Alternatively, in the embodiment, the computer terminal can be replaced by a mobile terminal or other terminal device.

[0352] Alternatively, in the embodiment, the computer terminal can be located in at least one of the network devices in the computer network.

[0353] In the embodiment, the computer terminal can execute the program code of the following steps in the data processing method: monitoring a data query request, wherein the data query request is used to request to query to-be-executed data; in response to the data query request, based on the resource configuration information of a queue, allocating a target scheduling time to the queue, wherein the resource configuration information is used to represent the processing resource to which the queue is allowed to be scheduled during running, and the target scheduling time is used to represent the time to which the queue is allocated; determining the time difference information between the target scheduling time of the queue and the scheduled time of the corresponding queue, wherein the time difference information is used to represent the difference between the target scheduling time and the scheduled time; based on the time difference information of the queue, determining a target queue from a plurality of queues, wherein the time difference information corresponding to the target queue is greater than a difference information threshold; and querying the to-be-executed data from the target queue.

[0354] Optionally, FIG. 16 is a structural block diagram of a computer terminal according to an embodiment of the present disclosure. As shown in FIG. 16, the computer terminal A can include one or more (only one is shown in the figure) processors 1602, a memory 1604, and a transmission device 1606.

[0355] The memory can be configured to store software programs and components, such as program instructions / components corresponding to the data processing method and device in the embodiments of the present disclosure. The processor can execute various function applications and data processing by running the software programs and components stored in the memory, that is, implement the above-mentioned data processing method. The memory can include a high-speed random access memory, and can further include a non-volatile memory, such as one or more magnetic storage devices, flash memories, or other non-volatile solid-state memories. In some examples, the memory can further include a memory remotely arranged with respect to the processor, and the above-mentioned remote memory can be connected to the computer terminal A through a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0356] The processor can call information and application programs stored in the memory through the transmission device to execute the following steps: in response to a data query request, determining a total scheduling time allowed to be scheduled by a queue based on resource configuration information of the queue and scheduled time of the queue; and allocating a target scheduling time to the queue based on the total scheduling time of the queue.

[0357] Optionally, the above-mentioned processor can further execute program codes of the following steps: determining a first total scheduling time from total scheduling times of a plurality of queues; and allocating a target scheduling time to a corresponding queue based on the first total scheduling time and resource configuration information of the queue.

[0358] Optionally, the above-mentioned processor can further execute program codes of the following steps: determining a target scheduling time corresponding to the resource configuration information of the queue from the first total scheduling time.

[0359] Optionally, the above-mentioned processor can further execute program codes of the following steps: in response to a data query request, obtaining a quotient between the scheduled time of the queue and the resource proportion of the queue; and determining the quotient as the total scheduling time of the queue.

[0360] Optionally, the above-mentioned processor can further execute program codes of the following steps: determining a priority of the queue; and configuring resource configuration information to the queue according to the priority.

[0361] Optionally, the above-mentioned processor can further execute program codes of the following steps: obtaining a target time proportion of the target scheduling time of the queue in the scheduled time of the queue; and determining the target time proportion as time difference information of the queue.

[0362] Optionally, the processor can further execute program codes of the following steps: determining maximum time difference information from the time difference information of the plurality of queues; and determining a queue corresponding to the maximum time difference information as the target queue.

[0363] Optionally, the processor can further execute program codes of the following steps: obtaining thread configuration information of the queue; in response to the number of used threads of the queue satisfying the thread configuration information of the queue, determining second maximum time difference information from the time difference information of the plurality of queues; and determining a queue corresponding to the second maximum time difference information as the target queue.

[0364] Optionally, the processor can further execute program codes of the following steps: in response to the current queue being empty, determining second maximum time difference information from the time difference information of the plurality of queues; and determining a queue corresponding to the second maximum time difference information as the target queue.

[0365] Optionally, the processor can further execute program codes of the following steps: in response to the target queue being non-empty, storing the to-be-executed data into the target queue.

[0366] Optionally, the processor can further execute program codes of the following steps: determining a total scheduling time allowed to be scheduled by the target queue based on resource configuration information of the target queue and a scheduled time of the target queue; in response to the target queue being empty, determining second total scheduling time from the total scheduling time of the plurality of queues; determining a target time threshold corresponding to the target queue based on the second total scheduling time and the resource configuration information of the target queue; in response to the scheduled time of the target queue being less than the target time threshold, adjusting the scheduled time of the target queue to the target time threshold and storing the to-be-executed data into the adjusted target queue; and in response to the scheduled time of the target queue being greater than or equal to the target time threshold, keeping the scheduled time of the target queue and storing the to-be-executed data into the kept target queue.

[0367] Optionally, the processor can further execute program codes of the following steps: determining the target time threshold corresponding to the resource configuration information of the target queue from the second total scheduling time.

[0368] Optionally, the processor can further execute program codes of the following steps: monitoring a data query request from a thread; determining sub-task data of the target task in the target queue as the to-be-executed data; and returning the to-be-executed data to the thread for execution.

[0369] Optionally, the processor can further execute program codes of the following steps: determining the minimum thread number allowed to be used by the queue based on the number of cores of the processor; determining the minimum thread number allowed to be used by the queue based on the resource configuration information of the queue; and adding the minimum thread number to the thread configuration information of the queue.

[0370] Optionally, the processor can call the information and application stored in the memory through the transmission device to execute the following steps: monitoring the data query request from the thread; in response to the data query request, allocating the target scheduling time to the queue based on the resource configuration information of the queue in the data warehouse; determining the time difference information between the target scheduling time of the queue and the scheduled time of the queue; determining the target queue from the plurality of queues based on the time difference information of the queue; querying the to-be-executed data from the target queue; and returning the queried to-be-executed data to the thread for execution.

[0371] Optionally, the processor can call the information and application stored in the memory through the transmission device to execute the following steps: monitoring the data query request by calling the first interface; in response to the data query request, allocating the target scheduling time to the queue based on the resource configuration information of the queue; determining the time difference information between the target scheduling time of the queue and the scheduled time of the queue; determining the target queue from the plurality of queues based on the time difference information of the queue; querying the to-be-executed data from the target queue; and outputting the to-be-executed data by calling the second interface.

[0372] The embodiment of the present disclosure provides a data processing method. In the embodiment of the present disclosure, if the data to be queried by a data query request needs to be queried, the data query request received from the client can be monitored to analyze the to-be-executed data to be requested and queried from the server. The processing resource to which the data query request in each queue is allowed to be dispatched during running can be analyzed according to the resource configuration information of the queue, and the target dispatch time when the current data query request is allocated to the processing resource in the queue for execution can be determined. The time difference information between the target dispatch time and the scheduled time of the corresponding queue can also be determined. The queue with time difference information greater than the difference information threshold is determined as the target queue, so that the current data query request can use the target queue to dispatch the corresponding processing resource to query the corresponding to-be-executed data. In the embodiment of the present disclosure, by setting the corresponding resource configuration information for each queue in the database system, the soft isolation between tasks of different priorities, that is, data query requests of different degrees, is realized. During the execution of the data query request, the target dispatch time of different queues can be automatically calculated according to the resource configuration information, and a more suitable queue is selected to query the to-be-executed data based on the target dispatch time, thereby avoiding the resource contention and competition problem during the execution of large queries and small queries, and realizing the technical effect of improving the efficiency of data processing, and solving the technical problem of low efficiency of data processing.

[0373] Those skilled in the art can understand that the structure shown in FIG. 16 is only schematic, and the computer terminal A can also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a palm computer, a mobile Internet device (Mobile Internet Devices, referred to as MID), a PAD, and the like. FIG. 16 does not limit the structure of the above computer terminal A. For example, the computer terminal A can further include more or fewer components (such as a network interface, a display device, etc.) than those shown in FIG. 16, or have a different configuration from that shown in FIG. 16.

[0374] Those skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer readable storage medium, which can include a flash disk, a read-only memory (Read-Only Memory, referred to as ROM), a random access memory (Random Access Memory, referred to as RAM), a magnetic disk or an optical disk, etc.

[0375] The embodiment of the present disclosure further provides a computer readable storage medium. Optionally, in the embodiment, the computer readable storage medium can be used to save the program code executed by the data processing method provided in the first embodiment.

[0376] Optionally, in the embodiment, the computer readable storage medium can be located in any one of the computer terminals in the computer terminal group in the computer network, or in any one of the mobile terminals in the mobile terminal group.

[0377] Optionally, in the embodiment, the computer readable storage medium is configured to store program code for performing the following steps: monitoring a data query request, wherein the data query request is used to request to query to-be-executed data; in response to the data query request, based on resource configuration information of a queue, allocating a target scheduling time to the queue, wherein the resource configuration information is used to represent processing resources allowed to be scheduled to the queue in a running process; determining time difference information between the target scheduling time of the queue and a scheduled time corresponding to the queue, wherein the time difference information is used to represent a difference between the target scheduling time and the scheduled time; based on the time difference information of the queue, determining a target queue from a plurality of queues, wherein the target queue corresponds to time difference information greater than a difference information threshold; and querying the to-be-executed data from the target queue.

[0378] The processor can call information and application programs stored in the memory through the transmission device to perform the following steps: in response to the data query request, based on the resource configuration information of the queue and the scheduled time of the queue, determining a total scheduling time allowed to be scheduled to the queue; and based on the total scheduling time of the queue, allocating the target scheduling time to the queue.

[0379] The processor can call information and application programs stored in the memory through the transmission device to perform the following steps: determining a first total scheduling time from total scheduling times corresponding to a plurality of queues; and based on the first total scheduling time and the resource configuration information of the queue, allocating the target scheduling time to the corresponding queue.

[0380] The processor can call information and application programs stored in the memory through the transmission device to perform the following steps: determining the target scheduling time corresponding to the resource configuration information of the queue from the first total scheduling time.

[0381] The processor can call information and application programs stored in the memory through the transmission device to perform the following steps: in response to the data query request, obtaining a quotient between the scheduled time of the queue and the resource proportion of the queue; and determining the quotient as the total scheduling time of the queue.

[0382] The processor can invoke information and application programs stored in the memory through the transmission device to perform the following steps: determining the priority of the queue; and configuring resource configuration information to the queue according to the priority.

[0383] The processor can invoke information and application programs stored in the memory through the transmission device to perform the following steps: obtaining a target time ratio of a target scheduling time of the queue to a scheduled time of the queue; and determining the target time ratio as time difference information of the queue.

[0384] The processor can invoke information and application programs stored in the memory through the transmission device to perform the following steps: determining maximum time difference information from time difference information corresponding to a plurality of queues; and determining a queue corresponding to the maximum time difference information as a target queue from the plurality of queues.

[0385] The processor can invoke information and application programs stored in the memory through the transmission device to perform the following steps: obtaining thread configuration information of the queue; in response to a number of used threads of the queue satisfying the thread configuration information of the queue, determining secondary maximum time difference information from maximum time difference information from time difference information of a plurality of queues; and determining a queue corresponding to the secondary maximum time difference information as a target queue from the plurality of queues.

[0386] The processor can invoke information and application programs stored in the memory through the transmission device to perform the following steps: in response to the current queue being empty, determining secondary maximum time difference information from maximum time difference information from time difference information corresponding to a plurality of queues; and determining a queue corresponding to the secondary maximum time difference information as a target queue from the plurality of queues.

[0387] The processor can invoke information and application programs stored in the memory through the transmission device to perform the following steps: in response to the target queue being non-empty, storing to-be-executed data to the target queue.

[0388] The processor can invoke information and application programs stored in the memory through the transmission device to perform the following steps: determining a total scheduling time allowed to be scheduled by the target queue based on resource configuration information of the target queue and a scheduled time of the target queue; in response to the target queue being empty, determining a second total scheduling time from total scheduling times corresponding to a plurality of queues; determining a target time threshold corresponding to the target queue based on the second total scheduling time and the resource configuration information of the target queue; in response to the scheduled time of the target queue being less than the target time threshold, adjusting the scheduled time of the target queue to the target time threshold and storing to-be-executed data to the adjusted target queue; and in response to the scheduled time of the target queue being greater than or equal to the target time threshold, maintaining the scheduled time of the target queue and storing to-be-executed data to the maintained target queue.

[0389] The processor can call information and application programs stored in the memory through the transmission device to perform the following steps: determining a target time threshold corresponding to the resource configuration information of the target queue from the second total scheduling time.

[0390] The processor can call information and application programs stored in the memory through the transmission device to perform the following steps: monitoring a data query request from a thread; determining the subtask data of the target task in the target queue as to-be-executed data; and returning the to-be-executed data to the thread for execution.

[0391] The processor can call information and application programs stored in the memory through the transmission device to perform the following steps: determining a minimum thread number allowed to be used by the queue based on the number of cores of the processor; determining the minimum thread number allowed to be used by the queue based on the resource configuration information of the queue; and adding the minimum thread number to the thread configuration information of the queue.

[0392] Optionally, the computer readable storage medium can further execute program codes of the following steps: monitoring a data query request from a thread; in response to the data query request, allocating a target scheduling time to the queue based on the resource configuration information of the queue in the data warehouse; determining time difference information between the target scheduling time of the queue and the scheduled time of the corresponding queue; determining a target queue from a plurality of queues based on the time difference information of the queue; querying to-be-executed data from the target queue; and returning the queried to-be-executed data to the thread for execution.

[0393] Optionally, the computer readable storage medium can further execute program codes of the following steps: monitoring a data query request through the first interface; in response to the data query request, allocating a target scheduling time to the queue based on the resource configuration information of the queue; determining time difference information between the target scheduling time of the queue and the scheduled time of the queue; determining a target queue from a plurality of queues based on the time difference information of the queue; querying to-be-executed data from the target queue; and outputting the to-be-executed data through the second interface.

[0394] Embodiments of the present disclosure can provide an electronic device, which can include a memory and a processor.

[0395] FIG. 17 is a block diagram of an electronic device for implementing a method of processing data according to an embodiment of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not intended to limit the implementations of the present disclosure described and / or claimed in this document.

[0396] As shown in FIG. 17, the device 1700 includes a computing component 1701 that can perform various suitable actions and processes according to computer programs stored in a read-only memory (ROM) 1702 or computer programs loaded into a random access memory (RAM) 1703 from a storage component 1708. Various programs and data used by the device 1700, in addition to the computing component 1701, the ROM 1702, and the RAM 1703, are connected to each other by a bus 1704. An input / output (I / O) interface 1705 is also connected to the bus 1704.

[0397] Various components in the device 1700 are connected to the I / O interface 1705, including an input component 1706, such as a keyboard, a mouse, etc., an output component 1704, such as various types of displays, speakers, etc., a storage component 1708, such as a magnetic disk, an optical disk, etc., and a communication component 1709, such as a network card, a modem, a wireless communication transceiver, etc. The communication component 1709 allows the device 1700 to exchange information / data with other devices through computer networks, such as the Internet, and / or various telecommunication networks.

[0398] The computing component 1701 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing component 1701 include, but are not limited to, a central processing component (CPU), a graphics processing component (GPU), various specialized artificial intelligence (AI) computing chips, various computing components running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processors, controllers, microcontrollers, and the like. The computing component 1701 performs various methods and processes described above, such as the processing method of data. For example, in some embodiments, the processing method of data can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage component 1708. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 1700 via the ROM 1702 and / or the communication component 1709. When the computer program is loaded into the RAM 1703 and executed by the computing component 1701, one or more steps of the processing method of data described above can be performed. Alternatively, in other embodiments, the computing component 1701 can be configured to perform the processing method of data by any other appropriate means, such as by means of firmware.

[0399] Embodiments of the present disclosure further provide a computer program product. Optionally, in the embodiments, the computer program product can include a computer program, and the computer program, when executed by a processor, implements the processing method of data of the above-mentioned embodiments of the present disclosure.

[0400] According to the embodiments of the present disclosure, a processing method of data is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that here.

[0401] The method provided in the embodiments of the present disclosure can be executed in a mobile terminal, a computer terminal or a similar computing device. FIG. 18 is a hardware structure block diagram of a computer terminal (or a mobile device) for implementing a data processing method according to an embodiment of the present disclosure. As shown in FIG. 18, the computer terminal 180 (or a mobile device) can include one or more processors 1802 (processors 1802 can include, but are not limited to, a microcontroller unit (MCU) or a field programmable gate array (FPGA) and the like processing device), a memory 1804 for storing data, and a transmission device 1806 for communication function. In addition, it can also include a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which can be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. Those skilled in the art can understand that the structure shown in FIG. 18 is only schematic, which does not limit the structure of the above-mentioned electronic device. For example, the computer terminal 180 can include more or less components than those shown in FIG. 18, or have a different configuration from that shown in FIG. 18.

[0402] The hardware structure block diagram shown in FIG. 18 can not only be used as an exemplary block diagram of the above-mentioned computer terminal 180 (or a mobile device), but also as an exemplary block diagram of the above-mentioned server. In an optional embodiment, FIG. 18 shows an embodiment using the above-mentioned computer terminal 180 (or a mobile device) shown in FIG. 18 as a computing node in a computing environment 1901.

[0403] FIG. 19 is a structure block diagram of a computing environment for a data processing method according to an embodiment of the present disclosure. As shown in FIG. 19, the computing environment 1901 includes a plurality of computing nodes (such as servers) running on a distributed network. The computing nodes all contain local processing and memory resources, and an end user 1902 can remotely run an application or store data in the computing environment 1901. The application can be provided as a plurality of services 1920-1, 1920-2, 1920-3 and 1920-4 in the computing environment 1901, representing services “F”, “G”, “I” and “H” respectively.

[0404] End users 1902 can provide and access services through a web browser or other software application on a client, in some embodiments, provisioning and / or requests of end users 1902 can be provided to ingress gateway 1930. Ingress gateway 1930 can include a corresponding proxy to handle provisioning and / or requests for services (one or more services provided in computing environment 1901).

[0405] Services are provided or deployed according to various virtualization technologies supported by computing environment 1901. In some embodiments, services can be provided according to virtual machine (VM) based virtualization, container-based virtualization, and / or the like. Virtual machine based virtualization can be emulating a real computer by initializing a virtual machine to execute programs and applications without directly accessing any actual hardware resources. While a virtual machine virtualization machine, according to container-based virtualization, a container can be launched to virtualize an entire operating system so that multiple workloads can run on a single operating system instance.

[0406] In one embodiment of container-based virtualization, several containers of a service can be assembled into a Pod (e.g., a Kubernetes Pod). For example, as shown in FIG. 19, service 1920-2 can be equipped with one or more Pods 1940-1, 1940-2,..., 1940-N (collectively, Pods). A Pod can include a proxy 1945 and one or more containers 1942-1, 1942-2,..., 1942-M (collectively, containers). The one or more containers in a Pod handle requests related to one or more corresponding functions of the service, and the proxy 1945 can control network functions related to the service, such as routing, load balancing, and the like. Other services can also be equipped with Pods similar to the Pods.

[0407] In operation, executing a user request from an end user 1902 can require invoking one or more services in computing environment 1901, and executing one or more functions of a service can require invoking one or more functions of another service. As shown in FIG. 19, service “F” 1920-1 receives a user request from an end user 1902 from ingress gateway 1930, service “F” 1920-1 can invoke service “G” 1920-2, and service “G” 1920-2 can request service “I” 1920-3 to execute one or more functions.

[0408] The computing environment described above can be a cloud computing environment, with the allocation of resources managed by a cloud service provider, allowing the development of functionality without the need to consider implementing, tuning or scaling servers. The computing environment allows developers to execute code in response to events without building or maintaining complex infrastructure. Services can be split into a set of functions that can automatically scale independently, rather than scaling a single hardware device to handle potential loads.

[0409] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an Application-Specific Integrated Circuit (ASIC), a System-on-a-Chip (SOC), a complex programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0410] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces the functions / operations specified in the flowcharts and / or the block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine as a standalone software package, partially on a machine and partially on a remote machine or entirely on a remote machine or server.

[0411] In the context of this disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fiber, compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0412] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a Cathode Ray Tube (CRT) or a Liquid Crystal Display (LCD) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0413] The systems and techniques described herein can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described herein, or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0414] The computer system can include clients and servers. A client and server are generally remote from each other and can interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0415] It should be noted that the above-mentioned sequence numbers of the embodiments of the present disclosure are only for description, and do not represent the advantages and disadvantages of the embodiments.

[0416] In several embodiments provided by the present disclosure, it should be understood that the disclosed technology can be implemented in other ways. For example, the above-described device embodiments are only schematic, and the division of components is only a logical function division, and actual implementation can have another division manner, for example, a plurality of components or components can be combined or integrated into another system, or some features can be omitted or not executed. In addition, the coupling or direct coupling or communication connection between the shown or discussed components can be indirect coupling or communication connection through some interfaces, components, or in electrical or other forms.

[0417] The components illustrated as separate components can or can not be physical separate, and the components illustrated as components can or can not be physical components, i.e., can be located in one place, or can be distributed to multiple network components. Part or all of the components can be selected according to actual needs to achieve the purpose of the embodiments.

[0418] In addition, each functional component in the present disclosure can be integrated in one processing component, or each component can exist physically separately, or two or more components can be integrated in one component. The integrated component can be realized in the form of hardware or in the form of a software functional component.

[0419] When the integrated component is realized in the form of a software functional component and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present disclosure, essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods in the present disclosure. The aforementioned storage medium includes various media that can store program codes, such as a U disk, a read-only memory, a random access memory, a mobile hard disk, a magnetic disk or an optical disk, etc.

[0420] The above is only the preferred embodiment of the present disclosure, and it should be pointed out that, for those skilled in the art, without departing from the principles of the present disclosure, several improvements and refinements can be made, and the above improvements and refinements should be regarded as the protection scope of the present disclosure. Industrial applicability

[0421] The scheme provided by the embodiments of the present disclosure can be applied in the process of a data query request. If the data to be queried by the data query request needs to be queried, the received data query request from the client can be monitored to analyze the to-be-executed data that needs to be requested and queried from the server. According to the resource configuration information of each queue, the processing resources that the data query request in the queue is allowed to be scheduled to in the running process can be analyzed, and the target scheduling time when the current data query request is allocated to the queue to schedule the processing resources for execution can be determined. The time difference information between the target scheduling time and the scheduled time of the corresponding queue can also be determined, and the target queue can be selected according to the size relationship between the time difference information of each queue and the difference information threshold. The queue with the time difference information greater than the difference information threshold can be determined as the target queue, so that the current data query request can use the target queue to schedule the corresponding processing resources to query the corresponding to-be-executed data. In the embodiments of the present disclosure, by setting the corresponding resource configuration information for each queue in the database system, the soft isolation between tasks of different priorities, that is, data query requests of different degrees, is realized. In the process of executing the data query request, the target scheduling time of different queues can be automatically calculated according to the resource configuration information, and a more suitable queue can be selected to query the to-be-executed data based on the target scheduling time, thereby avoiding the resource contention and competition problems in the execution of large queries and small queries, and realizing the technical effect of improving the efficiency of data processing, and solving the technical problem of low efficiency of data processing.

Claims

1. A method for processing data, comprising: monitoring a data query request, wherein the data query request is used to request querying to-be-executed data; allocating a target scheduling time to a queue based on resource configuration information of the queue in response to the data query request, wherein the resource configuration information is used to represent processing resources allowed to be scheduled by the queue during running, and the target scheduling time is used to represent a time at which the queue is allocated to be scheduled; determining time difference information between the target scheduling time of the queue and a scheduled time of the queue, wherein the time difference information is used to represent a difference between the target scheduling time and the scheduled time; determining a target queue from a plurality of queues based on the time difference information of the queue, wherein the target queue corresponds to the time difference information greater than a difference information threshold value; querying the to-be-executed data from the target queue.

2. The method of claim 1, wherein, The step of allocating the target scheduling time to the queue based on the resource configuration information of the queue in response to the data query request comprises: determining total scheduling time allowed to be scheduled by the queue based on the resource configuration information of the queue and the scheduled time of the queue in response to the data query request, wherein the total scheduling time is used to represent all scheduling time allowed to be scheduled by the queue; allocating the target scheduling time to the queue based on the total scheduling time of the queue.

3. The method of claim 2, wherein, The step of allocating the target scheduling time to the queue based on the total scheduling time of the queue comprises: determining a first total scheduling time from total scheduling time of a plurality of queues, wherein the first total scheduling time is greater than a first time threshold value; allocating the target scheduling time to the queue based on the first total scheduling time and the resource configuration information of the queue.

4. The method of claim 3, wherein, The resource configuration information of the queue comprises a resource proportion of the processing resources allowed to be scheduled by the queue during running in total processing resources, and the step of allocating the target scheduling time to the queue based on the first total scheduling time and the resource configuration information of the queue comprises: determining the target scheduling time corresponding to the resource configuration information of the queue from the first total scheduling time, wherein the target scheduling time of the queue occupies a same time proportion of the first total scheduling time as the resource proportion indicated by the resource configuration information of the queue.

5. The method of claim 2, wherein, The resource configuration information of the queue is a resource proportion of the processing resources allowed to be scheduled by the queue during running in total processing resources, and the step of determining the total scheduling time allowed to be scheduled by the queue based on the resource configuration information of the queue and the scheduled time of the queue in response to the data query request comprises: obtaining a quotient between the scheduled time of the queue and the resource proportion of the queue in response to the data query request; determining the quotient as the total scheduling time of the queue.

6. The method of claim 1, wherein, The method further comprises: determining a priority of the queue; configuring the resource configuration information to the queue according to the priority.

7. The method of claim 1, wherein, The step of determining the time difference information between the target scheduling time of the queue and the scheduled time of the queue comprises: obtaining a target time proportion of a target scheduling time of the queue in a scheduled time of the queue; determining the target time proportion as time difference information of the queue.

8. The method of claim 1, wherein, determining a target queue from a plurality of the queues based on the time difference information of the queue, including: determining maximum time difference information from the time difference information of the plurality of the queues; determining a queue corresponding to the maximum time difference information from the plurality of the queues as the target queue.

9. The method of claim 1, wherein, determining a target queue from a plurality of the queues based on the time difference information of the queue, including: obtaining thread configuration information of the queue, wherein the thread configuration information is used to represent at least a maximum number of threads allowed to be used by the queue; in response to a number of threads used by the queue satisfying the thread configuration information of the queue, determining secondary maximum time difference information corresponding to the maximum time difference information from the time difference information of the plurality of the queues; determining a queue corresponding to the secondary maximum time difference information from the plurality of the queues as the target queue.

10. The method of claim 1, wherein, determining a target queue from a plurality of the queues based on the time difference information of the queue, including: in response to the current queue being empty, determining secondary maximum time difference information corresponding to the maximum time difference information from the time difference information of the plurality of the queues; determining a queue corresponding to the secondary maximum time difference information from the plurality of the queues as the target queue.

11. The method of claim 1, wherein, before querying the to-be-executed data from the target queue, the method further includes: in response to the target queue being non-empty, storing the to-be-executed data in the target queue.

12. The method of claim 1, wherein, before querying the to-be-executed data from the target queue, the method further includes: determining a total scheduling time allowed to be scheduled by the target queue based on resource configuration information of the target queue and a scheduled time of the target queue; in response to the target queue being empty, determining a second total scheduling time from total scheduling times of the plurality of the queues, wherein the second total scheduling time is less than a second time threshold; determining a target time threshold corresponding to the target queue based on the second total scheduling time and the resource configuration information of the target queue; in response to the scheduled time of the target queue being less than the target time threshold, adjusting the scheduled time of the target queue to the target time threshold, and storing the to-be-executed data in the adjusted target queue; in response to the scheduled time of the target queue being greater than or equal to the target time threshold, maintaining the scheduled time of the target queue, and storing the to-be-executed data in the maintained target queue.

13. The method of claim 12, wherein, the resource configuration information of the target queue is a resource proportion of processing resources allowed to be scheduled by the target queue in a running process in total processing resources, and determining the target time threshold of the target queue based on the second total scheduling time and the resource configuration information of the target queue includes: From the second total scheduling time, a target time threshold corresponding to resource configuration information of the target queue is determined, wherein the target time threshold of the target queue is the same as a time proportion of the second total scheduling time and a resource proportion indicated by the resource configuration information of the target queue.

14. The method of any one of claims 1 to 13, wherein, The data query request is monitored, including: The data query request from the thread is monitored, wherein the data query request is used to request to query a target task to be executed by the thread; From the target queue, the to-be-executed data is queried, including: determining sub-task data of the target task in the target queue as the to-be-executed data; and returning the to-be-executed data to the thread for execution.

15. The method of any one of claims 1 to 13, wherein, The method further includes: Based on the number of cores of the processor, a minimum thread number allowed to be used by the queue is determined; Based on resource configuration information of the queue, a minimum thread number allowed to be used by the queue is determined; The minimum thread number is added to thread configuration information of the queue.

16. A data processing method applied to an execution end, including: Monitoring a data query request from a thread, wherein the data query request is used to request to query to-be-executed data; In response to the data query request, based on resource configuration information of a queue in a data warehouse, a target scheduling time is allocated to the queue, wherein the resource configuration information is used to represent processing resources allowed to be scheduled by the queue during running, and the target scheduling time is used to represent time allocated to schedule the queue; the data warehouse includes a plurality of the queues; Determining time difference information between the target scheduling time of the queue and scheduled time of the queue, wherein the time difference information is used to represent a difference between the target scheduling time and the scheduled time; Based on the time difference information of the queue, a target queue is determined from the plurality of queues, wherein the target queue corresponds to time difference information greater than a difference information threshold value; From the target queue, the to-be-executed data is queried; The queried to-be-executed data is returned to the thread for execution.

17. A data processing method, including: Monitoring a data query request by calling a first interface, wherein the data query request is used to request to query to-be-executed data, and the first interface includes a first parameter, and a parameter value of the first parameter is the data query request; In response to the data query request, based on resource configuration information of a queue, a target scheduling time is allocated to the queue, wherein the resource configuration information is used to represent processing resources allowed to be scheduled by the queue during running, and the target scheduling time is used to represent time allocated to schedule the queue; Determining time difference information between the target scheduling time of the queue and scheduled time of the queue, wherein the time difference information is used to represent a difference between the target scheduling time and the scheduled time; Based on the time difference information of the queue, a target queue is determined from the plurality of queues, wherein the target queue corresponds to time difference information greater than a difference information threshold value; query the to-be-executed data from the target queue; output the to-be-executed data by calling a second interface, wherein the second interface comprises a second parameter, and a parameter value of the second parameter is the to-be-executed data.

18. A data processing system, comprising: a client configured to send a data query request, wherein the data query request is used to request to query to-be-executed data; a server configured to, in response to the data query request, allocate a target scheduling time to the queue based on resource configuration information of the queue, wherein the resource configuration information is used to represent processing resources allowed to be scheduled to the queue during running of the queue, the target scheduling time is used to represent a time at which the queue is scheduled to be allocated to, determine time difference information between the target scheduling time of the queue and a scheduled time of the queue, wherein the time difference information is used to represent a difference between the target scheduling time and the scheduled time, determine a target queue from a plurality of queues based on the time difference information of the queue, wherein the target queue corresponds to the time difference information greater than a difference information threshold, query the to-be-executed data from the target queue, and return the to-be-executed data to the client.

19. An electronic device, comprising: a memory storing an executable program; a processor configured to run the program, wherein the program performs the method of any one of claims 1 to 17 when running.

20. A computer readable storage medium comprising a stored executable program, wherein, controlling a device in which the storage medium is located to perform the method of any one of claims 1 to 17 when the executable program is running.

21. A computer program product, comprising a computer program which, when executed by a processor, implements the method of any one of claims 1 to 17.

22. A computer program product, comprising: a non-volatile computer-readable storage medium storing a computer program which, when executed by a processor, implements the method of any one of claims 1 to 17.

23. A computer program which, when executed by a processor, implements the method of any one of claims 1 to 17.

Images