Data processing method, system, electronic device and storage medium

Abstract

The present application discloses a data processing method, system, electronic device and storage medium. The method may include: monitoring data query requests, wherein the data query request is used to request query of data to be executed; in response to the data query request, allocating a target scheduling time to the queue based on the resource configuration information of the queue, wherein the resource configuration information is used to characterize the processing resources that the queue is allowed to schedule during operation; determining the time difference information between the target scheduling time of the queue and the scheduled time of the queue, wherein the time difference information is used to represent the difference between the target scheduling time and the scheduled time; determining a target queue from multiple queues based on the time difference information of the queue, wherein the time difference information corresponding to the target queue is greater than the difference information threshold; querying the data to be executed from the target queue. The present application solves the technical problem of low efficiency of data processing.

Description

Technical Field

[0001] The present application relates to the field of computers, and in particular, to a data processing method, system, electronic device and storage medium. Background Art

[0002] At present, with the continuous development of the data era, it is more inclined to integrate various database servers into a unified database system (shared database system). In a shared database system, data query requests running simultaneously often share resources such as the central processing unit (CPU), memory, and disk input / output (I / O) in the database system. This means that multiple data query requests will compete for system resources during execution. If some data query requests consume huge resources and are not restricted, other concurrently running data query requests will have to compete for resources with the above data query requests, thereby affecting the overall performance of the database system.

[0003] In the related art, in the data warehouse of the database system, the Small Query Acceleration (SQA) function can be used to split the queues of small queries (data query requests with short execution time and less resource consumption) to improve the speed of data processing. However, small queries may still be affected by the resource contention and competition of large queries (data query requests with long execution time and more resource consumption), resulting in the execution of small queries being affected. Therefore, there is still a technical problem of low data processing efficiency.

[0004] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0005] The embodiments of the present application provide a data processing method, system, electronic device and storage medium to at least solve the technical problem of low efficiency of data processing.

[0006] According to one aspect of an embodiment of the present application, a method for processing data is provided. The method may include: monitoring a data query request, wherein the data query request is used to request a query for data to be executed; in response to the data query request, allocating a target scheduling time to the queue based on the resource configuration information of the queue, wherein the resource configuration information is used to characterize the processing resources that the queue is allowed to schedule during operation, and the target scheduling time is used to indicate the time allocated to the scheduling queue; determining the time difference information between the target scheduling time of the queue and the scheduled time of the queue, wherein the time difference information is used to indicate the difference between the target scheduling time and the scheduled time; determining a target queue from multiple queues based on the time difference information of the queue, wherein the time difference information corresponding to the target queue is greater than the difference information threshold; and querying the data to be executed from the target queue.

[0007] According to another aspect of the embodiment of the present application, a data processing method is provided. The method is applied to the execution end and may include: monitoring a data query request from a thread, wherein the data query request is used to request a query for data to be executed; in response to the data query request, based on the resource configuration information of the queue in the data warehouse, assigning a target scheduling time to the queue, wherein the resource configuration information is used to characterize the processing resources that the queue is allowed to schedule during operation, the target scheduling time is used to indicate the time allocated to the scheduling queue, and the data warehouse includes multiple queues; determining the time difference information between the target scheduling time of the queue and the scheduled time of the queue, wherein the time difference information is used to indicate the difference between the target scheduling time and the scheduled time; based on the time difference information of the queue, determining the target queue from multiple queues, wherein the time difference information corresponding to the target queue is greater than the difference information threshold; querying the data to be executed from the target queue; returning the queried data to be executed to the thread for execution.

[0008] According to another aspect of the embodiment of the present application, a method for processing data is provided. The method may include: monitoring a data query request by calling a first interface, wherein the data query request is used to request querying data to be executed, the first interface includes a first parameter, and the parameter value of the first parameter is the data query request; in response to the data query request, based on the resource configuration information of the queue, assigning a target scheduling time to the queue, wherein the resource configuration information is used to characterize the processing resources allowed to be scheduled by the queue during operation, and the target scheduling time is used to indicate the time allocated to the scheduling queue; determining the time difference information between the target scheduling time of the queue and the scheduled time of the queue, wherein the time difference information is used to indicate the difference between the target scheduling time and the scheduled time; based on the time difference information of the queue, determining a target queue from multiple queues, wherein the time difference information corresponding to the target queue is greater than the difference information threshold; querying the data to be executed from the target queue; outputting the data to be executed by calling a second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter is the data to be executed.

[0009] According to another aspect of the embodiment of the present application, a data processing system is provided. The system may include: a client, for sending a data query request, wherein the data query request is used to request to query the data to be executed; a server, for responding to the data query request, and allocating a target scheduling time to the queue based on the resource configuration information of the queue, wherein the resource configuration information is used to characterize the processing resources that the queue is allowed to schedule during operation, and the target scheduling time is used to indicate the time allocated to the scheduling queue; determining the time difference information between the target scheduling time of the queue and the scheduled time of the queue, wherein the time difference information is used to indicate the difference between the target scheduling time and the scheduled time; determining a target queue from multiple queues based on the time difference information of the queue, wherein the time difference information corresponding to the target queue is greater than the difference information threshold; querying the data to be executed from the target queue; and returning the data to be executed to the client.

[0010] According to another aspect of the embodiment of the present application, an electronic device is also provided. The electronic device may include a memory and a processor: the memory is used to store computer executable instructions, and the processor is used to execute the computer executable instructions. When the computer executable instructions are executed by the processor, the data processing method of the embodiment of the present application is implemented.

[0011] According to another aspect of an embodiment of the present application, a processor is further provided. The processor is used to run a program, wherein the data processing method of the embodiment of the present application is executed when the program is running.

[0012] According to another aspect of the embodiment of the present application, a computer-readable storage medium is provided, which includes a stored program, wherein when the program is executed, the device where the storage medium is located is controlled to execute the data processing method of the embodiment of the present application.

[0013] According to another aspect of the embodiment of the present application, a computer program product is also provided. The computer program product includes a computer program, and when the computer program is executed by a processor, the data processing method of the embodiment of the present application is implemented.

[0014] In an embodiment of the present application, if it is necessary to query the data to be queried by the data query request, the received data query request from the client can be monitored to analyze the pending data to be requested from the server. According to the resource configuration information of each queue, the processing resources that the data query request in the queue is allowed to be scheduled during operation can be analyzed, and the target scheduling time when the current data query request is assigned to the scheduling processing resources in the queue for execution can be determined. It is also possible to determine the time difference information between the target scheduling time and the corresponding queue scheduled time, and filter out the target queue according to the size relationship between the time difference information of each queue and the difference information threshold. The queue whose time difference information is greater than the difference information threshold can be determined as the target queue, so that the current data query request can use the target queue to schedule the corresponding processing resources to query the corresponding pending data. In an embodiment of the present application, by setting the corresponding resource configuration information for each queue in the database system, tasks of different priorities, that is, soft isolation between data query requests of different degrees, are realized. In the process of executing data query requests, the target scheduling time of different queues can be automatically calculated according to the resource configuration information, and a more appropriate queue can be selected to query the data to be executed based on the target scheduling time, thereby avoiding resource scramble and competition problems during the execution of large queries and small queries, achieving the technical effect of improving the efficiency of data processing and solving the technical problem of low data processing efficiency.

[0015] It is easy to notice that the above general description and the following detailed description are only for the purpose of exemplifying and explaining the present application, and do not constitute a limitation of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0017] Figure 1 is a schematic diagram of an application scenario of a data processing method according to an embodiment of the present application;

[0018] Figure 2 is a flow chart of a data processing method according to an embodiment of the present application;

[0019] Figure 3 is a flow chart of a data processing method according to an embodiment of the present application;

[0020] Figure 4 is a flow chart of a data processing method according to an embodiment of the present application;

[0021] Figure 5 is a flow chart of a data processing system according to an embodiment of the present application;

[0022] Figure 6 It is a schematic diagram of the initial version of the pending segmentation queue processing process;

[0023] Figure 7 is a schematic diagram of an improved multi-level priority queue and scheduling time according to an embodiment of the present application;

[0024] Figure 8 is a schematic diagram of a multi-level priority queue according to an embodiment of the present application;

[0025] Fig. 9 It is a schematic diagram of an overall architecture of data warehouse hybrid load isolation based on priority queues according to an embodiment of the present application;

[0026] FIG10( a ) is a flowchart of obtaining a task from a queue for processing according to an embodiment of the present application;

[0027] FIG10( b ) is a flow chart of placing a task into a queue according to an embodiment of the present application;

[0028] Fig.11 is a schematic diagram of a scheduling example according to an embodiment of the present application;

[0029] Fig.12 is a schematic diagram of a CPU usage status of a system with 16 threads fully occupied according to an embodiment of the present application;

[0030] Fig.13 is a schematic diagram of a data processing device according to an embodiment of the present application;

[0031] Fig.14 is a schematic diagram of another data processing device according to an embodiment of the present application;

[0032] Fig.15 is a schematic diagram of another data processing device according to an embodiment of the present application;

[0033] Fig.16 is a structural block diagram of a computer terminal according to an embodiment of the present application;

[0034] Fig.17 is a block diagram of an electronic device according to a data processing method of an embodiment of the present application;

[0035] Fig.18 It is a hardware structure block diagram of a computer terminal (or mobile device) for implementing a data processing method according to an embodiment of the present application;

[0036] Fig.19 It is a structural block diagram of a computing environment of a data processing method according to an embodiment of the present application. DETAILED DESCRIPTION

[0037] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0038] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to the above-mentioned process, method, product or device.

[0039] First, some nouns or terms that appear in the description of the embodiments of the present application are subject to the following explanations:

[0040] Workload Group is an important concept in the open source distributed Structured Query Language (SQL) query engine. It can be used to organize and manage different query workloads, so that different CPU resources and memory limits can be allocated to different queries.

[0041] The concurrency limit refers to the maximum number of queries that can be queued for execution at the same time within a specific time period. In a database system or data warehouse system, there is usually an execution queue to store query requests waiting to be executed. Setting a queue query limit can help the system control and manage the number of queries waiting to be executed, preventing excessive queue accumulation from causing system performance degradation and response delays;

[0042] CPU Share or cpu_share refers to the proportion of CPU resources allocated to a query, usually expressed as a percentage. In a database system or data warehouse system, CPU resources are one of the key resources for executing queries and processing data. By setting CPU Share, the system can determine the proportion of CPU resources that can be used by each query, thereby controlling the execution priority and resource usage of each query;

[0043] Elastic space refers to the ability to dynamically adjust the memory size allocated for queries in an open source distributed SQL query engine as needed. This mechanism allows the system to dynamically adjust the memory space required for queries according to actual needs during query execution to improve query performance and resource utilization efficiency. The characteristic of elastic space is that it can flexibly allocate memory resources according to the requirements of different queries and system load conditions, so as to better meet the execution needs of queries;

[0044] Queues are an important concept used in data warehouses to manage query execution order and priority. Queues can help the system sort and schedule queries based on their priority and execution time, ensuring that different queries can be executed in an orderly manner according to the set rules and strategies.

[0045] Large queries are queries that consume a lot of CPU resources and memory, usually involving the processing and calculation of a large amount of data, which may lead to problems such as increased system load, performance degradation, and resource competition. Since large queries may have a significant impact on the system, they usually need to be optimized and restricted to ensure system stability and performance;

[0046] CPU time refers to the time consumed by the CPU during query execution. In database systems or data warehouse systems, CPU time is usually used to indicate the time a query or task actually takes to execute on the CPU. CPU time is one of the important indicators for measuring the efficiency and performance of query or task execution.

[0047] The memory limit refers to the maximum memory size allocated for a query, which is used to store the data and intermediate results required during the query execution. In a database system or data warehouse system, the memory limit is an important configuration parameter that helps control the usage and management of memory resources during query execution;

[0048] The maximum number of scanned rows refers to the maximum number of rows that can be scanned during query execution. In a database system or data warehouse system, the maximum number of scanned rows is an important configuration parameter used to limit the maximum number of rows that a query can process when scanning data. Setting the maximum number of scanned rows can help control query execution efficiency and resource consumption, and avoid excessive query scanning of data, which may lead to performance degradation or excessive system load.

[0049] Small query / short query refers to a query with a short execution time, which usually does not need to be optimized or restricted. In a database system or data warehouse system, short query usually refers to a query with a short execution time and a small amount of data to process, which can be completed quickly in a short time. Such queries usually do not require complex optimization or restriction because of their short execution time and low resource consumption, which will not have a significant impact on system performance;

[0050] Resource Group is an organizational unit used to manage query resources in the open source distributed SQL query engine. Resource Group allows users to allocate different query workloads to different groups for management based on task requirements and query characteristics, so that each Resource Group can be independently configured and tuned. Through Resource Group, users can allocate different CPU resources and memory limits to different queries to meet the resource requirements of different queries and optimize query execution efficiency;

[0051] Maximum concurrency (max_concurrency) refers to the maximum number of queries that the system can execute simultaneously within a specific period of time. In a database system or data warehouse system, maximum concurrency is an important performance indicator that reflects the system's ability to process queries simultaneously. By setting the maximum concurrency, you can control the number of queries that the system processes simultaneously, avoiding system overload, resource contention, and performance degradation.

[0052] The maximum queue size (max_queue_size) refers to the maximum number of queries that can be queued for execution in a specific time period. In a database system or data warehouse system, when the system load is too high or resources are insufficient, queries may enter the queue for execution. Setting the maximum queue size can help control the number of queries in the queue and avoid excessive queue accumulation that may cause system performance degradation or response delays.

[0053] Workload Management (WLM) rules are rules used to manage queries in the data warehouse. Through WLM rules, users can configure the priorities of different queries so that the system can sort and schedule queries according to these rules. Through WLM rules, users can prioritize queries based on task requirements and query importance to ensure that key queries can be processed in a timely manner;

[0054] IO skew refers to the imbalance of I / O operations during query execution due to uneven data distribution or other reasons. This situation will cause the data processing speed of some nodes or partitions to be significantly slowed down, thus affecting the performance of the entire query;

[0055] Workload Manager Queue: used to manage and control the execution order and priority of queries. Through Workload Manager Queue, users can assign different priorities to different queries and sort and schedule queries based on these priorities and execution times;

[0056] Automatic adjustment refers to a mechanism by which the system automatically adjusts query resource allocation based on query execution and resource usage. This mechanism can help the system dynamically optimize query execution performance and efficiency at runtime to adapt to changing workloads and resource requirements;

[0057] Manual configuration refers to a mechanism whereby users manually adjust query resource allocation based on query execution and resource usage. Compared with automatic adjustment, manual configuration requires users to directly intervene and make adjustments based on actual conditions to meet specific needs and optimize query execution performance;

[0058] A priority queue is a data structure used to sort and schedule elements according to their priority. In database systems, priority queues are often used to manage the execution order of queries or tasks, determining which queries should be executed first based on their priority, in order to optimize the utilization of system resources and improve overall performance.

[0059] According to an embodiment of the present application, a method for processing data is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0060] The data processing method provided in the embodiment of the present application can be applied to Figure 1The application scenarios shown are not limited to these. Figure 1 In the application scenario shown, the server 10 can be a cloud. The server 10 can be connected to one or more client devices 20 via a local area network connection, a wide area network connection, an Internet connection, or other types of data networks. The client devices 20 here may include but are not limited to: smart phones, tablet computers, laptops, PDAs, personal computers, smart home devices, vehicle-mounted devices, etc. The client devices together constitute the client relative to the server. On the graphical user interface of the client device, an operation interface for obtaining a memory access request for accessing data in the memory address of the device can be deployed. The operation interface can be an operation interface for detecting related operations performed by the user to generate a memory access request. The client device 20 can interact with the user through a graphical user interface to implement the data processing method provided in the embodiment of the present application.

[0061] In the embodiment of the present application, the system composed of the client device 20 and the server 10 can perform the following steps: if the user needs to query the data in the database system on the server, the user can perform corresponding operations in the operation interface on the client device 20 to upload the data query request corresponding to the data to be queried. After receiving the above data query request, the data query request can be sent to the server 10 through the network. The database system, such as a data warehouse, can be deployed in the server 10. Specifically, the following steps can be performed in the server 10:

[0062] Step S102: monitoring data query requests, wherein the data query request is used to request query of to-be-executed data.

[0063] Step S104, in response to the data query request, allocating a target scheduling time to the queue based on the resource configuration information of the queue, wherein the resource configuration information is used to characterize the processing resources that the queue is allowed to schedule during operation, and the target scheduling time is used to represent the time allocated to the scheduling queue.

[0064] Step S106, determining time difference information between the target scheduling time of the queue and the scheduled time of the queue, wherein the time difference information is used to represent the difference between the target scheduling time and the scheduled time.

[0065] Step S108: determining a target queue from multiple queues based on the time difference information of the queues, wherein the time difference information corresponding to the target queue is greater than a difference information threshold.

[0066] Step S110, querying the data to be executed from the target queue.

[0067] In the above process, the data query request in the server 10 can be monitored through the network, and the target queue can be screened out from a large number of queues in the data warehouse to call processing resources to query the to-be-executed data to be queried by the above data query request, and the obtained to-be-executed data can be sent to the client device 20. The to-be-executed data can be displayed on the operation interface of the client device 20 for the user to view.

[0068] In the embodiment of the present application, by setting corresponding resource configuration information for each queue in the database system, soft isolation between tasks of different priorities, that is, data query requests of different degrees, is achieved. In the process of executing data query requests, the target scheduling time of different queues can be automatically calculated according to the resource configuration information, and a more suitable queue can be selected to query the data to be executed based on the target scheduling time, thereby avoiding resource scramble and competition problems when executing large queries and small queries, achieving the technical effect of improving the efficiency of data processing, and solving the technical problem of low efficiency of data processing.

[0069] The present application embodiment proposes the following method. In the above application scenario, the present application provides the following Figure 2 The data processing method shown. Figure 2 is a flow chart of a data processing method according to an embodiment of the present application, such as Figure 2 As shown, the following steps may be included:

[0070] Step S202: monitoring data query request.

[0071] In the technical solution provided in the above step S202 of the present application, the data query request can be used to request to query the data to be executed, which can be referred to as query, which can include small query and large query, etc., and can be a request for executing a query or task (split) operation, that is, when the data query request is received, the data query request can be divided into different queues to call system resources for processing. The data to be executed can be data in a database system. The database system is a system for storing, managing and retrieving data, which can be composed of a database and a database management system. The database management system is the core component of the database system, which is responsible for managing the creation, maintenance, query, backup and other functions of the database. Among them, the database management system can be used in combination with the data warehouse. The database management system provides basic data management functions, and the data warehouse provides a better level of data analysis and query functions on this basis, and can distribute and store and manage data on multiple computers or servers. These computers are connected to each other through a network and work together to provide data storage and access services. The data warehouse is a database system for storing and managing large amounts of data, which can support organizations to collect, integrate and analyze data from different sources and at different time points. For example, the data warehouse may be an Online Analytical Processing Data Warehouse (OLAP for short) type data warehouse.

[0072] Optionally, the above queue is an important management concept in the data warehouse, which is used to manage the execution order and priority of queries. By setting queue rules and scheduling strategies reasonably, the system can effectively control the execution order and time of queries, improve the performance and stability of the system, and ensure that queries can be executed in order according to the set priority, thereby improving the overall efficiency and reliability of the system.

[0073] Optionally, the above-mentioned data warehouse is an application of a database management system, which is built on the basis of a database management system to support data analysis and decision support needs. A data warehouse is a data storage system used to centralize, integrate, store and manage large amounts of structured and unstructured data. The data warehouse is designed to support data analysis, decision support and task intelligence needs by integrating data from different data sources into a unified data storage to provide users with convenient and fast data access and analysis capabilities.

[0074] In this embodiment, it may be monitored whether the client device corresponding to the user has a data query request.

[0075] Optionally, after the data query request is monitored, the data query request may be analyzed to determine the to-be-executed data in the database system requested to be queried by the data query request.

[0076] For example, after obtaining a data query request, the query statement in the data query request can be parsed to understand the specific content and requirements of the query request. Based on the table name and field name involved in the query statement, the table and field where the pending data to be queried from the database system are located can be determined. By analyzing the conditional statement in the query statement, the screening conditions in the query request can be determined. Based on the keywords in the query statement, the query request method is determined, whether it is a single-table query or a multi-table associated query, whether an index is required, etc. Based on the characteristics of the query request and the performance characteristics of the database system, consider whether performance optimization is required, such as using indexes, adjusting the query order, optimizing query statements, etc. Through the above steps, the pending data to be queried from the database system can be accurately analyzed from the data query request.

[0077] It should be noted that the above process and method of analyzing and querying the to-be-executed data from the data query request are only for illustration and are not specifically limited here.

[0078] For example, if a user needs to view certain data in a database system in a server, the user can perform corresponding operations on the client device corresponding to the user to input the need to view the data, thereby forming a data query request. After the data query request is detected on the client device, the data query request can be transmitted to the server through the network so that the server can analyze the data query request.

[0079] Step S204: In response to the data query request, a target scheduling time is allocated to the queue based on the resource configuration information of the queue.

[0080] In the technical solution provided in the above step S204 of the present application, the queue can also be called a priority queue, which can be a pending split queue, a priority schedule split queue, or a level waiting split queue, which can include multiple simple multi-level split queues (Simple Multi Level Split Queue, referred to as SMLSQ), SMLSQ can be used to store data query requests, and group them according to the size and resource requirements of the data query requests, so as to more effectively manage and schedule these data query requests. Resource configuration information can be used to characterize the processing resources that the corresponding queue is allowed to schedule during operation, which can help the database system to manage and allocate various processing resources more flexibly, and realize soft isolation of various resources of the priority queue, thereby optimizing the performance and resource utilization of the system, and can also be referred to as configuration, for example, it can include cpu_share, which can be used to set the proportion of CPU time obtained by the priority queue to realize soft isolation of CPU resources. The target scheduling time can be used to indicate the time when the database system calls the corresponding queue, for example, it can be the scheduling time between SMLSQ queues, it can be the expected scheduling time, and it can also be called the target scheduled time. Processing resources may include CPU resources, and may also include other resources in the database system, such as memory, I / O, etc. This is only an example and is not specifically limited.

[0081] It should be noted that the above resource configuration information and processing resources are only for illustration and no specific limitation is imposed here.

[0082] In this embodiment, after the data query request is monitored, the target scheduling time may be allocated to the corresponding queue based on the resource configuration information of each queue.

[0083] Optionally, each queue may be configured with corresponding resource configuration information, which may indicate the weight of the queue to obtain CPU time. This may be set through the system configuration to determine the proportion of CPU resources obtained by each queue. The database system determines the scheduling time of the queue based on the resource configuration information of the queue to which it is queried.

[0084] Optionally, the target scheduling time is determined by the resource configuration information and scheduled time of each queue. For example, the resource configuration information and scheduled time of the queue are obtained, the total scheduling time of each queue is calculated by the above two parameters, and the total scheduling time that meets the conditions is selected from the total scheduling time of each queue as the baseline value, and the baseline value is allocated according to the CPU share ratio to obtain the target scheduling time of each queue. In this way, it can ensure that each queue reasonably allocates the scheduling time according to the configuration and scheduling situation, and improve the task scheduling efficiency of the system.

[0085] In the related art, the SQA function is used to make small queries not have to be blocked by large queries and queued, that is, to avoid the problem of reduced performance and response speed of the entire data warehouse due to the need for small queries to wait for large queries to complete before obtaining system resources for execution. However, even if the small query can be executed independently in the queue through queue splitting, in the actual system execution process, small queries and large queries still need to share system resources, and there will still be a problem of resource competition between the two. Therefore, there is still a technical problem of low data processing efficiency.

[0086] However, in an embodiment of the present application, by configuring the resource configuration information of each queue, it is possible to adjust the weight of different queues to obtain CPU time, thereby realizing reasonable allocation and optimization of resources. A high-priority queue (a queue that is assigned a higher execution priority in the system and will be given priority for execution during system scheduling to ensure its timely completion) can obtain more CPU resources to ensure that important tasks or urgent tasks can be processed in a timely manner, while a low-priority queue (a queue that is assigned a lower execution priority in the system and will be considered for execution at a lower level during system scheduling, and will usually be processed only after high-priority tasks are completed) can still obtain a certain amount of CPU time under limited resources to ensure the overall operation of the system. By determining the scheduling time of the queue based on the resource configuration information of each queue and the time that has been scheduled, it is possible to achieve that the scheduling time between different queues basically conforms to the ratio of CPU share. This can prevent some queues from occupying CPU resources for a long time, resulting in other queues being unable to obtain sufficient resources for execution, thereby improving the resource utilization and performance of the system, and thus achieving the technical effect of improving the efficiency of data processing.

[0087] In addition, by configuring the resource configuration information of each queue, you can process tasks of different priorities. Queues with high resource configuration information will be scheduled first to ensure the timely execution of high-priority tasks; queues with low resource configuration information will be processed only after high-priority tasks are satisfied, ensuring the rational allocation of system resources. By configuring the resource configuration information and scheduling time of each queue, you can flexibly adjust it according to the needs and performance requirements of the system. Different task scenarios and application requirements can configure the resource configuration information and scheduling time of the queue according to actual conditions to achieve better resource allocation and scheduling strategies.

[0088] In summary, determining the corresponding scheduling time of queues based on the resource configuration information of each queue can help the system to achieve reasonable allocation of resources, priority processing of tasks, and control of scheduling time, improve the performance and stability of the system, while meeting the flexibility and customizability of different task scenarios and requirements.

[0089] Step S206, determining the time difference information between the target scheduling time of the queue and the scheduled time of the corresponding queue.

[0090] In the technical solution provided in the above step S206 of the present application, the scheduled time can be the time that has been scheduled, can be the actual scheduled time, and can also be called the level scheduled time. The time difference information can be used to indicate the difference between the target scheduling time and the scheduled time. For example, the difference can be the difference ratio between the target scheduling time and the scheduled time. It should be noted that the above difference ratio is only for example purposes and is not specifically limited here. As long as the time difference information can reflect the difference between the target scheduling time and the scheduled time, it is within the protection scope of the embodiments of the present application.

[0091] Optionally, the scheduled time usually refers to the length of time that a queue has been scheduled and started to execute. During the queue scheduling process, the length of time from the start of the queue execution to the current time is recorded as the scheduled time of the queue. The calculation of the scheduled time can be used to measure the duration of queue scheduling, evaluate the efficiency of queue scheduling for data processing, and perform priority scheduling and resource allocation of queues. By monitoring and recording the scheduled time of queues, the execution order of queues in the database system can be better managed to ensure that queues are scheduled according to certain rules and priorities, thereby improving the efficiency and performance of the system.

[0092] In this embodiment, after the target scheduling time for scheduling the corresponding queue is determined based on the resource configuration information of each queue, the time difference information between the target scheduling time and the scheduled time can be determined.

[0093] Optionally, the difference between the target scheduling time and the scheduled time is determined to obtain time difference information. For example, the difference between the target scheduling time and the scheduled time is calculated, which can be calculated by subtracting the scheduled time from the target scheduling time to obtain the time difference information between the two. The time difference information can also be converted into a ratio form, that is, a ratio (difference ratio) between the target scheduling time and the scheduled time, which can be obtained by dividing the time difference by the scheduled time.

[0094] It should be noted that the above-mentioned process of determining the time difference information and the form of the determined time difference information are for illustrative purposes only and are not specifically limited here. In the embodiments of the present application, as long as the process and method can understand the relationship between the target scheduling time and the scheduled time, and can use the time difference information between the two to measure how to allocate queues for the current data query data, they are all within the protection scope of the embodiments of the present application.

[0095] In the embodiment of the present application, the time difference information can be used to measure the difference between different queues, which helps to determine the priority and urgency of each queue. Through the time difference information, the scheduling conditions between queues with different time spans can be uniformly analyzed and compared, which improves the standardization level of scheduling. And according to the size of the time difference information, the database system can automatically adjust the execution order, priority or resource allocation of the task to optimize the performance and efficiency of the system. By calculating the time difference information between the target scheduling time and the scheduled time, the system can realize intelligent queue scheduling and resource allocation, adjust the priority of queue execution according to the time difference information, and ensure the rational use of resources. The time difference information can be used as an indicator for real-time monitoring to help the system grasp the execution of the queue in real time and adjust the scheduling strategy in time to adapt to the real-time changing system load and task requirements. Through the time difference information, the system can allocate resources more effectively, so that different tasks can be reasonably scheduled according to their priority and urgency, thereby improving the overall resource utilization and performance of the system.

[0096] In summary, determining the time difference information between the target scheduling time and the scheduled time can provide a more accurate reference for the system's queue scheduling and resource allocation, and help improve the intelligence and efficiency of the database system.

[0097] Step S208: determining a target queue from multiple queues based on the time difference information of the queues.

[0098] In the technical solution provided in step S208 of the present application, the difference information threshold can be a preset value or a value set according to actual conditions, and no specific limitation is made here. The time difference information corresponding to the target queue is greater than the difference information threshold, for example, it can be a queue with a relatively large difference.

[0099] Alternatively, a queue with large time difference information may not obtain enough CPU time due to uneven resource allocation, resulting in a large gap between the target scheduling time and the scheduled time. A queue with large time difference information may contain some lower priority tasks, which are less urgent or important than tasks in other queues, so the scheduling time is shorter. A queue with large time difference information may require more scheduling time, which may indicate that the tasks in the queue require more resources or time to complete during execution, and the system performance may need to be tuned or resource allocation may need to be adjusted.

[0100] In this embodiment, after determining the time difference information of the queue through the target scheduling time and the corresponding scheduled time, the time difference information and the difference information threshold can be compared, and the queue whose time difference information is greater than the difference information threshold can be determined as the target queue.

[0101] In an embodiment of the present application, by determining the target queue whose time difference information is greater than the difference information threshold, it can be found that there is an imbalance in resource allocation in the system, which helps to optimize the allocation of resources and ensure that each queue can obtain a reasonable scheduling time. The target queue whose time difference information is greater than the difference information threshold may contain some tasks with lower priority. Adjusting the priority of these tasks can better meet the actual needs of the system and improve the overall operating efficiency of the system. By analyzing the target queue whose time difference information is greater than the difference information threshold, it is possible to find performance bottlenecks or resource allocation problems in the system, which helps to optimize the performance of the system and improve the execution efficiency and stability of the system.

[0102] In summary, determining the target queue whose time difference information is greater than the difference information threshold can help the system discover problems such as uneven resource allocation, unreasonable task priority, or performance tuning, thereby optimizing task scheduling and resource allocation and improving system performance and efficiency.

[0103] Optionally, the database system selects a suitable SMLSQ queue as the target queue based on resource configuration information and the scheduled time.

[0104] Optionally, queues can help the system manage the execution order of queries, queuing and scheduling them according to the order and priority of queries entering the queue. Through queues, the system can ensure that queries are executed in a certain order to avoid confusion and conflicts between queries. Queues can sort and schedule queries according to the priority of queries to ensure that high-priority queries are executed first. The system can set different priority rules according to the importance and urgency of different queries to ensure that key queries can be processed in a timely manner. Queues can also sort and schedule queries according to the execution time of queries to ensure that long-running queries do not block the execution of other queries. By setting execution time limits and scheduling strategies, the system can effectively control the execution time of queries to avoid long-running queries affecting system performance. By managing the execution order and priority of queries through queues, the system can better balance the allocation and utilization of resources and improve the stability and reliability of the system. Queues can help the system reasonably schedule the execution order of queries, avoid excessive resource consumption and system overload, and ensure that the system can run smoothly.

[0105] Step S210, querying the data to be executed from the target queue.

[0106] In the technical solution provided in the above step S210 of the present application, after determining the target queue from multiple queues based on the time difference information, the to-be-executed data of the request query corresponding to the data query request can be queried from the target queue.

[0107] Optionally, querying the to-be-executed data from the target queue generally includes the following steps: the system needs to determine the target queue, and obtain a data query request from the target queue, including information such as the data content to be queried and query conditions. According to the obtained data query request, a corresponding query operation is performed in the database to obtain the required to-be-executed data. The obtained to-be-executed data can be returned to an external thread or system for further processing or display of the above-mentioned to-be-executed data by the external thread or system.

[0108] In an embodiment of the present application, the target queue may indicate that the queue needs more scheduling time to reach the target scheduling time, and may contain some important or urgent data query requests. Therefore, giving priority to these requests can improve the performance and efficiency of the system. The target queue may require more scheduling time due to uneven resource allocation, so timely processing of the requests in the queue can optimize the allocation and utilization of resources. If the target queue contains some important data query requests, timely processing of these requests can avoid delays and affect the response speed of the system. By processing the data query requests in the target queue, the task scheduling and resource allocation strategies of the system can be optimized, and the overall performance and efficiency of the system can be improved.

[0109] In summary, querying data query requests from the target queue whose time difference information is greater than the difference information threshold can give priority to those requests that require more scheduling time to fully utilize resources, improve system performance, avoid delays, and thus achieve efficient operation of the system.

[0110] Through the above steps S202 to S210 of the present application, if it is necessary to query the data to be queried by the data query request, the data query request received from the client can be monitored, and the pending data that needs to be requested to be queried from the server can be analyzed. According to the resource configuration information of each queue, the processing resources that the data query request in the queue is allowed to be scheduled during operation can be analyzed, and the target scheduling time when the current data query request is assigned to the scheduling processing resources in the queue for execution can be determined. The time difference information between the target scheduling time and the corresponding queue scheduled time can also be determined. The queue whose time difference information is greater than the difference information threshold is determined as the target queue, so that the current data query request can use the target queue to schedule the corresponding processing resources to query the corresponding pending data. In an embodiment of the present application, by setting corresponding resource configuration information for each queue in the database system, tasks of different priorities, that is, soft isolation between data query requests of different degrees is achieved. In the process of executing data query requests, the target scheduling time of different queues can be automatically calculated according to the resource configuration information, and a more appropriate queue can be selected to query the data to be executed based on the target scheduling time, thereby avoiding resource scramble and competition problems during the execution of large queries and small queries, achieving the technical effect of improving the efficiency of data processing and solving the technical problem of low data processing efficiency.

[0111] The above method of this embodiment is further introduced below.

[0112] As an optional implementation, step S204, in response to a data query request, allocates a target scheduling time to a queue based on the resource configuration information of the queue, including: in response to the data query request, determining the total scheduling time allowed for the queue based on the resource configuration information of the queue and the scheduled time of the queue, wherein the total scheduling time is used to characterize the total scheduling time that the queue is allowed to be scheduled; and allocating a target scheduling time to the corresponding queue based on the total scheduling time of the queue.

[0113] In this embodiment, in the process of determining the target scheduling time of each queue based on the resource configuration information of multiple queues, the total scheduling time allowed for the queue can be determined based on the resource configuration information and the scheduled time of the queue, and the target scheduling time allocated to each queue can be determined based on the total scheduling time.

[0114] Optionally, the total scheduling time refers to the total scheduling time that a queue should theoretically obtain, calculated based on the resource configuration information of the queue and the scheduled time, and can be the total scheduling time that the queue is allowed to be scheduled. Specifically, the total scheduling time is determined by adjusting the scheduled time according to the proportion of the resource configuration information of the queue to determine the length of scheduling time that each queue should obtain.

[0115] In task scheduling and resource allocation, the concept of total scheduling time can help the system to distribute scheduling time evenly to a certain extent according to the priority and resource requirements of the queues, ensuring that each queue can get the corresponding scheduling time. By calculating the total scheduling time, the system can better manage the execution order of tasks and the allocation of resources, and improve the performance and efficiency of the system.

[0116] Optionally, obtain the resource configuration information and scheduled time of the queue. According to the resource configuration information, calculate the total scheduling time of the queue. The total scheduling time is the total scheduling time length that the queue should obtain based on the resource configuration information and the scheduled time of the queue. For example, the total scheduling time can be calculated by dividing the scheduled time by the ratio of the resource configuration information. According to the calculated total scheduling time, the total scheduling time may need to be adjusted to ensure that each queue can obtain a reasonable scheduling time. This may involve correcting or optimizing the total scheduling time according to the actual situation of the system. The final total scheduling time will be used in the task scheduling and resource allocation process to determine the scheduling time length that each queue should obtain. The system will arrange the execution order of tasks and the allocation of resources according to the total scheduling time to improve the performance and efficiency of the system.

[0117] Through the above steps, the total scheduling time that each queue should obtain can be calculated based on the resource configuration information and the scheduled time to ensure the reasonable scheduling of tasks and the effective allocation of resources, thereby improving the overall operating efficiency of the system.

[0118] As an optional implementation, allocating a target scheduling time to a queue based on the total scheduling time of the queue, including: determining a first total scheduling time from the total scheduling times of multiple queues, wherein the first total scheduling time is greater than a first time threshold; allocating a target scheduling time to the queue based on the first total scheduling time and resource configuration information of the queue.

[0119] In this embodiment, in the process of determining the target scheduling time that needs to be allocated to each queue based on the total scheduling time, a first total scheduling time can be determined from the total scheduling times corresponding to multiple queues, and based on the first total scheduling time and resource configuration information, the target scheduling time can be allocated to the corresponding queue, wherein the first total scheduling time is a total scheduling time greater than the first time threshold, and the first total scheduling time can be used as a baseline, that is, a reference standard, which can help the system determine the current scheduling situation and the scheduling plan for future tasks.

[0120] Optionally, a baseline value is selected from the total scheduling time of each queue as a reference standard, and the maximum value is usually selected as the baseline. The baseline value is obtained by comparing the total scheduling time of each queue, and selecting the one that meets the conditions as the baseline. For example, the one with a long total scheduling time or that meets the first time threshold is used as the baseline. After determining the baseline value, it can be used as a reference standard for subsequent task scheduling and resource allocation. The baseline value can help the system determine the current scheduling situation and the scheduling plan for future tasks, and guide the execution order of tasks and the allocation of resources. It should be noted that the above process and method of how to determine the baseline from the total scheduling time of each queue is only for illustration and is not specifically limited here.

[0121] Through the above steps, the baseline value can be determined by calculating the total scheduling time, which can play a guiding role in the task scheduling and resource allocation process, help the system better manage the execution order of tasks and the allocation of resources, and improve the performance and efficiency of the system.

[0122] In an embodiment of the present application, the reason and purpose for determining the baseline through the total scheduling time is to select a reference standard to help the system determine the current scheduling situation and the scheduling plan for future tasks. Determining the baseline can help the system better understand the scheduling time situation of each queue, thereby optimizing the allocation of resources, ensuring that each queue can obtain a reasonable scheduling time, and improving the performance and efficiency of the system. As a reference standard, the baseline value can guide the execution order of tasks and the allocation of resources, ensure that tasks are scheduled according to certain rules and priorities, and improve the overall operating efficiency of the system. By determining the baseline, problems such as uneven scheduling time and uneven resource allocation in the system can be discovered in a timely manner, providing a basis for system performance optimization, and helping the system to better manage and schedule tasks.

[0123] Through the above measures, the system can be made more reasonable and efficient in task scheduling and resource allocation, improve the system's performance and stability, optimize the execution order of tasks and resource utilization efficiency. Doing so is conducive to improving the overall operating efficiency of the system and ensuring that tasks can be reasonably executed according to the scheduled scheduling time.

[0124] As an optional implementation, the resource configuration information of the queue includes the ratio of processing resources allowed to be scheduled by the queue during operation to the total processing resources, and based on the first total scheduling time and the resource configuration information of the queue, a target scheduling time is allocated to the queue, including: determining the target scheduling time corresponding to the resource configuration information of the queue from the first total scheduling time, wherein the time ratio of the target scheduling time of the queue to the first total scheduling time and the resource ratio indicated by the resource configuration information of the queue are the same.

[0125] In this embodiment, in the process of allocating the target scheduling time to the corresponding queue based on the first total scheduling time and the resource configuration information, the target scheduling time corresponding to the resource configuration information can be determined from the first total scheduling time, wherein the resource configuration information can include the processing resources that the corresponding queue is allowed to schedule during operation, which can be a cpu_share (CPU share) value. The time ratio of the target scheduling time to the first total scheduling time is the same as the resource ratio indicated by the resource configuration information of the queue, that is, the ratio of the scheduling time is consistent with the ratio between cpu_share.

[0126] Optionally, the ratio of the scheduling time between each queue and the ratio of the CPU share are basically consistent through scheduling. This means that when the system performs task scheduling, it will determine the scheduling time that each queue should obtain based on the CPU share value of the queue to keep the proportional relationship between the CPU share and the scheduling time roughly consistent. This consistency maintenance is to ensure that the system can reasonably allocate scheduling time according to the priority and resource allocation of each queue, thereby ensuring the performance and efficiency of the system. If the proportional relationship between the CPU share value and the scheduling time can be basically consistent, that is, the scheduling time obtained by each queue is proportional to its CPU share value, the system can allocate resources more accurately, optimize the execution order of tasks, and improve the overall operation efficiency of the system. This consistency maintenance helps the system maintain a balanced allocation of resources and avoid some queues occupying resources for a long time, affecting the execution efficiency of other queues. By continuously maintaining the consistency of the ratio of CPU share and scheduling time, the system can better manage the execution order of tasks and the allocation of resources, and improve the performance and stability of the system.

[0127] Optionally, obtain resource configuration information for each queue, including the CPU share value, and obtain the scheduled time of each queue. The CPU share value indicates the weight of the queue to obtain CPU time, and the scheduled time indicates the length of time the queue has been scheduled for execution. Calculate the target scheduling time for each queue based on the baseline value, the CPU share value of the queue, and the scheduled time. The target scheduling time is calculated by allocating the baseline value to each queue in proportion to the CPU share value of the queue to ensure that each queue can obtain a reasonable scheduling time.

[0128] Optionally, when determining the target scheduling time for each queue, the maximum value is selected from the calculated total scheduling time for each queue as the current baseline value. The baseline value can be used as a reference standard to determine the longest scheduling time in the current system as a basis for subsequent calculation of the target scheduling time.

[0129] For example, the total scheduling time of each queue is calculated based on the CPU share value of the queue and the scheduled time. From these calculated total scheduling times, the one that meets the conditions is selected as the baseline value. Based on this baseline value, the target scheduling time of each queue is further calculated to ensure that the scheduling time of each queue meets the proportional relationship of its CPU share value. The purpose of selecting the one that meets the conditions in the total scheduling time as the baseline value is to ensure that the longest scheduling time in the system is not ignored, so as to ensure that the resource allocation and task scheduling of the system can be more reasonable and effective. The baseline value can be used as a reference standard for subsequent scheduling and resource allocation decisions.

[0130] As an optional implementation, the resource configuration information of a queue is the ratio of processing resources allowed to be scheduled during operation of the queue to the total processing resources. In response to a data query request, the total scheduling time allowed for the queue is determined based on the resource configuration information of the queue and the scheduled time of the queue, including: in response to the data query request, obtaining the quotient between the scheduled time of the queue and the resource ratio of the queue; and determining the quotient as the total scheduling time of the queue.

[0131] In this embodiment, in the process of determining the total scheduling time based on the resource configuration information of the queue and the corresponding scheduled time, the quotient between the scheduled time and the resource ratio can be obtained, and the quotient can be determined as the total scheduling time, wherein the resource configuration information can be the resource ratio of the processing resources scheduled for the corresponding queue during operation to the total processing resources, that is, it can be the cpu_share value ratio, which can also be called the cpu_share ratio.

[0132] Optionally, each queue calculates its total scheduling time based on the ratio of its own scheduled time and CPU share value. Specifically, the total scheduling time is determined based on the CPU share value of the queue and the time that has been scheduled. The CPU share value indicates the weight of the queue in obtaining CPU time. The higher the ratio, the more CPU time the queue should get. The scheduled time indicates the length of time that the queue has been scheduled for execution. By calculating the ratio of the CPU share value and the scheduled time, the total scheduling time that each queue should obtain can be obtained. In this way, the scheduling time of each queue can be determined according to the configuration of each queue and the actual scheduling situation, so as to reasonably allocate resources and the execution order of tasks.

[0133] Optionally, the total scheduling time may be calculated by the scheduled time of each queue and the cpu_share value ratio. For example, the total scheduling time may be calculated by dividing the scheduled time by the cpu_share value ratio.

[0134] For example, if the database system includes five queues, such as queue 0, queue 1, queue 2, queue 3 and queue 4, and if the cpu_share values ​​of each queue are 15, 8, 4, 2 and 1 respectively, the time that each queue has been scheduled is 140, 90, 40, 30 and 10. According to the cpu_share value ratio of each queue and the time that has been scheduled, the total scheduling time of each queue is determined. The cpu_share value of the total processing resource is the sum of the cpu_share values ​​of the above five queues, that is, 30. The total scheduling time of queue 0: 140 / (15 / 30)=280, the total scheduling time of queue 1: 90 / (8 / 30)=337.5, the total scheduling time of queue 2: 40 / (4 / 30)=300, the total scheduling time of queue 3: 30 / (2 / 30)=450, and the total scheduling time of queue 4: 10 / (1 / 30)=300.

[0135] For another example, from the total scheduling times corresponding to the above five queues, the largest one can be selected as the baseline, that is, 450 can be selected as the baseline. The total scheduling time baseline is allocated as the target scheduling time according to the cpu_share value ratio, that is, the target scheduling time of queue 0: 450*(15 / 30)=225, the target scheduling time of queue 1: 450*(8 / 30)=120, the target scheduling time of queue 2: 450*(4 / 30)=60, the target scheduling time of queue 3: 450*(2 / 30)=30, and the target scheduling time of queue 4: 450*(1 / 30)=15.

[0136] It should be noted that the number of queues, cpu_share value and scheduled time in the above database system are only examples and are not specifically limited here.

[0137] Optionally, cpu_share is a mandatory configuration whose value range is an integer between 1 and 1000. It is used to set the proportion of CPU time that the priority queue obtains to achieve soft isolation of CPU resources. cpu_share is a relative value, indicating the weight of the CPU resources that the running priority queue can obtain.

[0138] In the embodiment of the present application, the cpu_share parameter is used to set the ratio of the priority queue to obtain CPU time, and realize the soft isolation of CPU resources. It is a relative value, indicating the weight of the running priority queue that can obtain CPU resources. By setting the cpu_share parameter, the ratio of different priority queues to obtain CPU resources can be controlled, thereby realizing the allocation control of CPU resources. It is also possible to realize the soft isolation of CPU resources, that is, to allocate CPU resources to different priority queues according to the set ratio, to ensure that each queue can obtain its due CPU time, and to avoid a queue occupying CPU resources for a long time to affect the execution of other queues. By reasonably setting the cpu_share parameter, the performance of the system can be optimized, and the response speed and processing efficiency of the system to tasks of different priorities can be improved. According to the actual task requirements and the system load, the cpu_share parameter can be flexibly adjusted so that the system can reasonably allocate CPU resources according to different priority requirements and improve the resource utilization of the system. By setting the cpu_share parameter, the reasonable allocation and management of CPU resources can be realized, ensuring that the system can efficiently process query tasks of various priorities and improve the performance and stability of the entire system.

[0139] Optionally, the cpu_share parameter can be exposed to the user. Its purpose is to allow users to set the proportion of CPU time obtained by priority queues according to their own task requirements and priority requirements, so as to achieve soft isolation of CPU resources. Customers can operate the cpu_share parameter to affect the system's CPU resource allocation to tasks of different priorities by setting the cpu_share value of different priority queues. The cpu_share parameter can be used to achieve the following functions: Users can set the cpu_share value of different queues according to their own task requirements and task priorities to achieve customized allocation of CPU resources and ensure that tasks of different priorities can obtain appropriate CPU resource support. By adjusting the cpu_share parameter, users can optimize the performance of the system, improve the system's response speed and processing efficiency for tasks of different priorities, and ensure that important tasks can be processed first. It can also achieve soft isolation of CPU resources to prevent a queue from occupying CPU resources for a long time and affecting the execution of other queues, so as to ensure the reasonable allocation and utilization of system resources. Users can flexibly adjust the cpu_share parameter according to actual task requirements and system load conditions, so that the system can reasonably allocate CPU resources according to different priority requirements and improve the resource utilization and performance of the system.

[0140] In general, the cpu_share parameter is exposed to customers, who can manipulate it to optimize system performance, implement resource isolation and control, and customize CPU resource allocation according to different task requirements, thereby improving system stability and efficiency.

[0141] It should be noted that whether the above resource configuration information is exposed to the user is a choice in the embodiments of the present invention, and no specific restriction is made here. It can be set according to the actual data processing needs. No specific restriction is made here on whether the resource configuration information is exposed to the user.

[0142] Optionally, the cpu_share value of the SQA queue is equal to the sum of all other queues, which means that the SQA queue can obtain the sum of CPU resources not used by other queues.

[0143] As an optional implementation, the method may include: determining a priority of a queue; and configuring resource configuration information for the queue according to the priority.

[0144] In this embodiment, the priority of each queue may be determined, and resource configuration information may be configured for different queues according to the priority, wherein the priority may also be referred to as level min priority or priority level.

[0145] Optionally, the priority of a queue is obtained by configuring different priority queues in the queue. There can be three priority queues, such as a low priority queue, a normal priority queue, and a high priority queue. The low priority queue can be used to store tasks or data that are not urgent and do not need to be processed immediately. These tasks or data can be processed when the system is idle and will not affect the normal operation of the system. The normal priority queue can be used to store general tasks or data that need to be processed in a certain order. These tasks or data have a certain impact on the overall performance of the system, but do not have to be processed immediately. The high priority queue can be used to store urgent tasks or data that need to be processed immediately. These tasks or data have an important impact on the normal operation of the system and need to be processed with priority to ensure the stability and performance of the system.

[0146] For example, if three priority queues are configured, namely low, normal, and high, and their cpu_shares are 10, 30, and 40 respectively. Assuming that at a certain moment, both the low and normal queues are running tasks, and the high queue has no tasks, then the low queue can obtain 25% of the CPU resources (10 / (10+30)), and the normal queue can obtain 75% of the CPU resources. If the system has only one priority queue with queries running, then regardless of its cpu_share value, this queue can obtain the CPU resources of the entire database system.

[0147] In an embodiment of the present application, by configuring priority queues of different priorities for queues, tasks or data can be processed according to different priorities. By setting different priority queues, it is possible to ensure that high-priority tasks or data can be processed in a timely manner, thereby improving the response speed and efficiency of the system. By setting different priority queues, important tasks or data can be processed with priority, thereby improving the overall performance of the system. High-priority tasks or data can be processed faster, thereby improving user experience and satisfaction. Different priority queues can be set according to specific needs, thereby flexibly responding to various situations. High-priority tasks or data can be processed with priority, thereby speeding up the response speed of the system. By setting different priority queues, it is possible to ensure that the system can process important tasks or data in a timely manner, thereby improving the stability and reliability of the system. By reasonably setting different priority queues, system resources can be better utilized, thereby improving the utilization efficiency of resources. By dividing tasks or data into different priority levels, the system can process them according to specific circumstances, thereby improving the efficiency and performance of the system.

[0148] As an optional implementation, step S206, determining the time difference information between the target scheduling time of the queue and the scheduled time of the queue, includes: obtaining the target time ratio of the target scheduling time of the queue to the scheduled time of the queue; and determining the target time ratio as the time difference information of the queue.

[0149] In this embodiment, in the process of determining the time difference information between the target scheduling time and the scheduled time of the corresponding queue, the target time ratio of the target scheduling time to the scheduled time can be obtained, and the target time ratio can be determined as the time difference information, wherein the target time ratio can be the difference ratio between the above two scheduling times.

[0150] Optionally, the time difference information is determined by the difference ratio between the target scheduling time and the actual scheduling time.

[0151] For example, among the five queues, the target scheduling time of queue 0 is 225, the target scheduling time of queue 1 is 120, the target scheduling time of queue 2 is 60, the target scheduling time of queue 3 is 30, and the target scheduling time of queue 4 is 15. The scheduled times corresponding to the above queues are 140, 90, 40, 30 and 10 respectively. The target time ratio of the corresponding queue can be determined by the ratio between the target scheduling time and the scheduled time. The gap ratio of queue 0: 225 / 140=1.607, the gap ratio of queue 1: 120 / 90=1.333, the gap ratio of queue 2: 60 / 40=1.5, the gap ratio of queue 3: 30 / 30=1, and the gap ratio of queue 4: 15 / 10=1.5.

[0152] It should be noted that the above process and method for determining the time difference information are only for illustrative purposes. This is only a specific embodiment for determining the time difference information by proportional means. In the embodiments of the present application, the form of the time difference information and the determination process are not limited.

[0153] As an optional implementation, step S208, based on the time difference information of the queues, determines the target queue from multiple queues, including: determining the maximum time difference information from the time difference information of the multiple queues; and determining the queue corresponding to the maximum time difference information among the multiple queues as the target queue.

[0154] In this embodiment, in the process of determining the target queue from multiple queues based on time difference information, the maximum time difference information can be determined from multiple time difference information corresponding to the multiple queues, and the queue corresponding to the maximum time difference information can be determined as the target queue, wherein the maximum time difference information can be the target gap ratio.

[0155] Optionally, the queue corresponding to the maximum time difference information may contain tasks or data that have not been processed for a long time, and need to be processed as soon as possible to avoid affecting the normal operation of the system. By processing the data in this queue, the performance and response speed of the overall system can be improved more quickly. Prioritizing the processing of the data in this queue can improve the stability and reliability of the system and ensure that urgent tasks are processed in a timely manner.

[0156] In an embodiment of the present application, processing the data of the largest queue can improve the response speed of the system and allow users to obtain the required results more quickly. By giving priority to processing the data of the largest queue, system resources can be better utilized and the efficiency and performance of the system can be improved. Processing the data of the largest queue can ensure that the system can handle important tasks in a timely manner and improve the stability and reliability of the system. Giving priority to processing the data of the largest queue can improve the user experience, allow users to obtain the required information or results more quickly, and improve user satisfaction. In summary, using the queue with the largest gap as the target queue for data query can improve the performance, response speed and stability of the system, optimize resource utilization, and improve user experience. It brings many benefits.

[0157] For example, if among the five queues, the gap ratio of queue 0 is 1.607, the gap ratio of queue 1 is 1.333, the gap ratio of queue 2 is 1.5, the gap ratio of queue 3 is 1, and the gap ratio of queue 4 is 1.5, among the gap ratios of the above queues, the target gap ratio is 1.607, and queue 0 of the target gap ratio is the target queue.

[0158] As an optional implementation, based on the time difference information of the queues, a target queue is determined from multiple queues, including: obtaining thread configuration information of the queue, wherein the thread configuration information is used to at least characterize the maximum number of threads allowed to be used by the corresponding queue; in response to the number of used threads of the queue satisfying the thread configuration information of the queue, determining the second largest time difference information corresponding to the maximum time difference information from the time difference information of multiple queues; and determining the queue corresponding to the second largest time difference information among the multiple queues as the target queue.

[0159] In this embodiment, the thread configuration information of the queue can be obtained. When the number of used threads in the queue meets the thread configuration information, the second largest time difference information corresponding to the maximum time difference information can be determined from the multiple time difference information corresponding to the multiple queues. The queue corresponding to the second largest time difference information in the queue can be determined as the target queue, wherein the thread configuration information can be used to at least indicate the maximum number of threads (max threadnumber, referred to as max thread) allowed to be used by the corresponding queue, for example, max_thread. The queue corresponding to the second largest time difference information can also be the second largest queue in the queue.

[0160] Optionally, the queue corresponding to the maximum time difference information is selected, and a split is taken out and returned to the external thread for execution. If the queue is empty or the upper limit of the number of available threads has been reached, the next largest queue is selected.

[0161] Optionally, when a queue has reached the upper limit of the number of threads that can be used, that is, the number of threads currently used by the queue has reached the maximum number of threads, if the second largest queue is not selected, it may cause a waste of resources and cause system performance to degrade. Selecting the second largest queue can ensure that system resources are more fully utilized and improve the overall performance of the system. If the queue corresponding to the current maximum time difference information is empty or full, the second largest queue may contain more urgent tasks and need to be processed first.

[0162] Optionally, in the process of selecting whether to determine the second largest queue, it is possible to check whether the current largest queue is empty or has reached the upper limit of the number of threads. If the largest queue is empty or has reached the upper limit of the number of threads, the second largest queue is selected. A split is taken from the second largest queue and returned to the external thread for execution.

[0163] In an embodiment of the present application, when the number of threads available in a queue reaches an upper limit, selecting the second largest queue can ensure that system resources are more fully utilized and improve the resource utilization of the system. Prioritizing the tasks of the second largest queue can improve the response speed of the system and ensure that urgent tasks can be processed in a timely manner. Selecting the second largest queue can avoid the waste of idle resources and improve the efficiency and performance of the system. Dynamically selecting the second largest queue to process tasks according to the state of the queue improves the flexibility and adaptability of the system. In summary, the reason for selecting the second largest queue to process tasks is to avoid waste of resources and improve the utilization and response speed of the system.

[0164] As an optional implementation, based on the time difference information of the queues, a target queue is determined from multiple queues, including: in response to the current queue being empty, determining the second largest time difference information corresponding to the maximum time difference information from the time difference information of the multiple queues; and determining the queue corresponding to the second largest time difference information among the multiple queues as the target queue.

[0165] In this embodiment, if the current queue is empty, the second largest time difference information corresponding to the maximum time difference information can be determined from the multiple time difference information corresponding to the multiple queues, and the queue corresponding to the second largest time difference information in the multiple queues can be used as the target queue.

[0166] Optionally, if the current queue is empty, if you do not select the next largest queue, resources will be idle, affecting system efficiency and performance. Selecting the next largest queue can ensure that system resources are more fully utilized and improve the overall performance of the system. The next largest queue may contain more urgent tasks that need to be processed first to ensure system stability and performance.

[0167] Optionally, check if the current queue is empty. If the queue is empty, select the next largest queue. Take a split from the next largest queue and return it to the external thread for execution.

[0168] In an embodiment of the present application, when the current queue is empty, selecting the second largest queue can ensure that system resources are more fully utilized and improve the resource utilization of the system. Prioritizing the tasks of the second largest queue can improve the response speed of the system and ensure that urgent tasks can be processed in a timely manner. Selecting the second largest queue can avoid the waste of idle resources and improve the efficiency and performance of the system. Processing the tasks of the second largest queue can ensure that the system can process urgent tasks in a timely manner and improve the stability and reliability of the system. In summary, the reason for selecting the second largest queue to process tasks is to avoid idle resources and improve the utilization and response speed of the system.

[0169] As an optional implementation, before querying the target queue for the data to be executed, the method further includes: in response to the target queue being non-empty, storing the data to be executed in the target queue.

[0170] In this embodiment, before querying the to-be-executed data from the target queue, if the target queue is not empty, the to-be-executed data may be stored in the target queue.

[0171] Optionally, when the system needs to put a split into the queue, if the queue is not empty, the split can be put directly into the queue. In this case, if the queue is not empty and there are already tasks or splits in the queue, the system can directly put the new split into the queue without waiting. When the queue is not empty, putting the split directly into the queue can avoid waiting time and improve the performance and efficiency of the system. There is no need to wait for the queue to be empty before putting the split in, and the task can be processed directly, improving the responsiveness of the system. Putting the split directly into the queue can ensure that the task can be processed in time, avoiding task delays or waiting.

[0172] To sum up, when the system needs to put a split into the queue, if the queue is not empty, directly putting the split in can improve system performance, response speed, and ensure that the task is processed in a timely manner.

[0173] As an optional implementation, before querying the data to be executed from the target queue, the method also includes: determining the total scheduling time allowed for the target queue to be scheduled based on the resource configuration information of the target queue and the scheduled time of the target queue; in response to the target queue being empty, determining a second total scheduling time from the total scheduling time of multiple queues, wherein the second total scheduling time is less than a second time threshold; determining a target time threshold corresponding to the target queue based on the second total scheduling time and the resource configuration information of the target queue; in response to the scheduled time of the target queue being less than the target time threshold, adjusting the scheduled time of the target queue to the target time threshold, and storing the data to be executed in the adjusted target queue; in response to the scheduled time of the target queue being greater than or equal to the target time threshold, maintaining the scheduled time of the target queue, and storing the data to be executed in the maintained target queue.

[0174] In this embodiment, before querying the to-be-executed data from the target queue, the total scheduling time allowed for scheduling of the target queue can be determined based on the resource configuration information of the target queue and the scheduled time of the target queue. When the target queue is empty, a second total scheduling time can be determined from the total scheduling time of multiple queues. The target time threshold corresponding to the target queue can be determined based on the second total scheduling time and the resource configuration information of the target queue, and the size relationship between the scheduled time of the target queue and the target time threshold can be determined.

[0175] Optionally, if the scheduled time is less than the target time threshold, the scheduled time of the target queue can be adjusted to the target time threshold, and the data to be executed can be stored in the adjusted target queue. If the scheduled time is greater than or equal to the target time threshold, the scheduled time of the target queue can be maintained, and the data to be executed can be stored in the maintained target queue. Through the above steps, it can be ensured that the scheduling time of each queue in the system is within a reasonable range, avoiding the occurrence of a too small total scheduling time that causes the system to tend to schedule a certain queue, thereby maintaining the balance and fairness of the system. Among them, the second total scheduling time can be less than the second time threshold, and the second total scheduling time can be the smaller total scheduling time of the total scheduling time of each queue, and the smaller total scheduling time of the total scheduling time is used as the current baseline. The second time threshold can be a value set by itself according to actual conditions, or it can be a pre-set value. This is only for example and is not specifically limited. The target time threshold can be the scheduling time that each queue should be allocated.

[0176] When the target queue is empty, if its scheduled time is not adjusted, the system may tend to schedule the queue too much, resulting in other queues not being fully processed, thereby affecting the overall performance of the system. Too small a scheduling time will cause the queue to be overscheduled, which may preempt the resources of other queues, affecting the fairness and efficiency of the system. Therefore, in an embodiment of the present application, when the target queue is empty, it is necessary to bring the scheduled time of the target queue into a reasonable range to avoid the system tending to schedule this queue too much, causing preemption of other queues.

[0177] In an embodiment of the present application, by aligning the scheduled time of the target queue to a reasonable range, the system can maintain fair scheduling of each queue and avoid the situation where a queue is overscheduled. Adjusting the scheduling time of the queue can optimize the allocation of system resources, so that each queue is reasonably processed and the overall efficiency and performance of the system are improved. Reasonable adjustment of the scheduled time of the target queue can avoid a queue from preempting the resources of other queues and ensure the balance and stability of various parts of the system. In summary, aligning the scheduled time of the target queue to a reasonable range can avoid the system from over-scheduling a queue, avoid queue preemption, and improve the fairness, resource utilization efficiency and stability of the system.

[0178] Optionally, the total scheduling time of each queue is calculated based on the time that each queue has been scheduled and the proportion of the cpu_share value. Ensure that the queue is not empty to avoid a total scheduling time that is too small. Select the minimum value from the total scheduling time of each queue as the baseline, as a reference value for a reasonable scheduling time range. Allocate the total scheduling time baseline according to the cpu_share ratio of the queue. For an empty queue, calculate the total scheduling time that should be allocated to the queue according to its cpu_share ratio, and determine the baseline of the total scheduling time. If the scheduling time of the queue is less than the calculated value, the scheduling time of the queue needs to be adjusted to this value, that is, the scheduling time of the queue is adjusted to the target time threshold. If the scheduling time of the queue is greater than the calculated value, it remains unchanged and no adjustment is required. After completing the above adjustment, put the split into the corresponding queue. Through the above process, it can be ensured that when the queue is empty, the scheduling time of the queue is adjusted to a reasonable range, avoiding the system from over-scheduling a queue and maintaining the fairness and efficiency of the system.

[0179] As an optional implementation, the resource configuration information of the target queue is the ratio of processing resources allowed to be scheduled by the target queue during operation to the total processing resources. Based on the second total scheduling time and the resource configuration information of the target queue, the target time threshold corresponding to the target queue is determined, including: from the second total scheduling time, determining the target time threshold corresponding to the resource configuration information of the target queue, wherein the target time threshold of the target queue accounts for the time ratio of the second total scheduling time and the resource ratio indicated by the resource configuration information of the target queue, which are the same.

[0180] In this embodiment, in the process of determining the target time threshold corresponding to the queue based on the second total scheduling time and resource configuration information, the target time threshold corresponding to the resource configuration information can be determined from the second total scheduling time, wherein the time ratio of the target time threshold to the second total scheduling time is the same as the resource ratio.

[0181] Optionally, calculate the total scheduling time based on the ratio of the time each queue has already scheduled and the cpu_share value to ensure that the queue is not empty. Select the minimum total scheduling time among all queues as the baseline to ensure that each queue has a reasonable scheduling time. Based on the cpu_share value ratio of each queue and the above baseline, the scheduling time that should be allocated to each queue can be calculated. Check the scheduling time of each queue. If it is less than the calculated corresponding scheduling time that should be allocated, the scheduling time of the queue needs to be adjusted to this value. If it is greater than this value, no change is required. Put the split into the queue.

[0182] For example, if among the five queues, the total scheduling time of queue 0 is 280, the total scheduling time of queue 1 is 337.5, the total scheduling time of queue 2 is 300, the total scheduling time of queue 3 is 450, and the total scheduling time of queue 4 is 300. From the total scheduling times corresponding to the above five queues, the smaller one can be selected as the baseline, that is, 280 can be selected as the baseline. According to the cpu_share value 2 of queue 3 in the above queues and the above baseline, the scheduling time that queue 3 should be allocated can be determined: 280*2 / 30=18.6.

[0183] As an optional implementation, step S202, monitoring data query requests, includes: monitoring data query requests from threads, wherein the data query requests are used to request querying the target tasks to be executed by the threads; step S210, querying the data to be executed from the target queue, includes: determining the subtask data of the target task in the target queue as the data to be executed; and returning the data to be executed to the thread for execution.

[0184] In this embodiment, in the process of monitoring data query requests, data query requests from threads can be monitored. The subtask data of the target task in the target queue can be determined as data to be executed, and the data to be executed can be returned to the thread for execution, wherein the data query request can be used to request the target task to be executed by the query thread. The target task can be a task to be executed by the thread.

[0185] Optionally, there is a thread pool of 256 threads on the execution node, and each thread takes a split from the pendingSpilts queue and executes a time slice in an infinite loop. In this case, the pendingSpilts queue is a priority blocking queue (PriorityBlockingQueue), in which queries basically follow the first-in, first-out (FIFO) priority. Specifically, PriorityBlockingQueue is an unbounded blocking queue that supports priorities, and elements are processed in the order of their priorities. Here, queries are prioritized according to the order in which they enter the pendingSpilts queue, that is, the first-in, first-out principle. Each thread takes a split from the pendingSpilts queue and executes a time slice. This mechanism can effectively utilize the thread pool resources of 256 threads while maintaining the order of queries. Since PriorityBlockingQueue sorts elements according to their priorities, queries are processed according to FIFO priority, ensuring the execution order of queries. This setting ensures that queries are processed in a timely manner and maintains the order of queries, avoiding possible priority inversion or starvation. At the same time, since each thread in the thread pool is acquiring and executing queries in an infinite loop, the system can make better use of multi-threaded parallel processing of queries, improving the system's concurrency performance and response speed.

[0186] Optionally, the above node can be an execution node, which refers to a computing node responsible for executing tasks or processing data. On the execution node, there is a thread pool containing 256 threads, each of which can cyclically remove splits from the pendingSpilts queue and execute a time slice. The thread pool is used to execute tasks concurrently to improve the processing capacity and efficiency of the system.

[0187] In the embodiment of the present application, through the above settings, the execution node can efficiently process tasks, the threads in the thread pool continuously take tasks from the queue for execution, follow the FIFO priority rule to ensure that tasks are executed in the correct order, and improve the efficiency and performance of the system. At the same time, the use of PriorityBlockingQueue can be processed according to the priority of the task, ensuring that important tasks can be executed first, and improving the response speed and stability of the system.

[0188] Optionally, the pendingSpilts queue is improved so that it is no longer a single queue, but a multi-level feedback queue composed of multiple priority queues. The scheduling time assigned to different priority queues is reduced by a multiple of 2 layer by layer, that is, the scheduling time of each level of queue is twice that of the previous level. When a query first enters, it will be assigned to the first level of priority queue. As the execution time increases, the query will gradually drop to a lower level queue to avoid large queries affecting the execution of small queries too much.

[0189] In an embodiment of the present application, through the setting of a multi-level feedback queue, it is possible to more effectively manage and schedule queries, and improve the response speed and performance of the system to queries. Queues of different levels gradually decrease according to the execution time, which can ensure that large queries will not occupy system resources for a long time, thereby maintaining the overall stability and performance of the system. Small queries will first enter the first-level queue and be executed first, thereby avoiding large queries from affecting the execution of small queries too much. Through the setting of this multi-level priority queue, the system can more efficiently process queries of different priorities, and ensure that both large and small queries can be reasonably processed and resource allocated, thereby improving the overall performance and stability of the system.

[0190] Optionally, the node may include a Resource Control (RC) node. The RC node has six priority queues, which may be the root queue, SQL queue, high priority queue, normal priority queue, low priority queue, and lowest priority queue. The first two queues mentioned above are not exposed to users, while the last four queues are exposed to users. Under this setting, users can set the priority of queries to enter the corresponding queues, and can also modify the maximum concurrency and maximum queue number of the queues. Although the queues are the same on the execution side, by setting different priority queues, reasonable scheduling and processing can be performed according to the importance and urgency of the tasks.

[0191] It should be noted that whether the above-mentioned different priority queues are exposed to users is a setting of an embodiment of the present invention, and is not specifically restricted here. It can be set by yourself according to actual data processing requirements. No specific restriction is made here on whether the priority queues are exposed to users.

[0192] In an embodiment of the present application, by setting the priority of the query, the user can place the query into different priority queues according to the task requirements, thereby controlling the execution order and priority of the query. The user can modify the maximum concurrency and maximum queue number of the queue according to their own needs to adapt to different workloads and resource requirements. Although different queues behave differently on the client side, there is no difference on the execution side, that is, the query will be executed in all queues in the same way. This can simplify the implementation and maintenance of the system and reduce the logical complexity between different queues. Since different queues are no different on the execution side, the system can process query requests of different priorities more evenly, avoiding the problem of uneven performance caused by queue differences. In general, this setting allows customers to set and manage the priority and queue properties of queries according to task requirements, while keeping the logic simple and unified on the execution side to improve the stability, maintainability and performance of the system.

[0193] Optionally, the system can monitor data query requests from threads, i.e., tasks or queries submitted by different threads. According to the priority of the task, the subtask data of the target task in the target queue is determined as data to be executed. This means that the system will analyze and process the tasks in the target queue, filter out the subtask data to be executed, and prepare for execution. The subtask data determined as data to be executed is returned to the thread for execution. This means that the system will pass the filtered and processed data to the corresponding thread so that the thread can perform the actual data query operation.

[0194] In an embodiment of the present application, by monitoring data query requests, determining the data to be executed and returning it to the thread for execution, the query processing efficiency of the system can be improved to ensure that the task is executed in a timely manner. By determining the data to be executed according to the priority of the task or other standards, the tasks can be flexibly scheduled to ensure that important tasks or urgent tasks are given priority. Returning the data to be executed to the thread for execution realizes the allocation and execution of tasks and ensures that the tasks are properly processed in the system. In summary, by monitoring data query requests and determining the data to be executed, the system can effectively process tasks and assign tasks to threads for execution to improve the efficiency and performance of the system and ensure that tasks are processed in a timely manner.

[0195] As an optional implementation, the method also includes: determining the minimum number of threads allowed for the queue based on the number of processor cores; determining the minimum number of threads allowed for the queue based on the resource configuration information of the queue; and adding the minimum number of threads to the thread configuration information of the queue.

[0196] In this embodiment, the minimum number of threads allowed to be used in the queue can be determined based on the number of cores of the processor, and the minimum number of threads allowed to be used in the queue can be determined based on the resource configuration information, wherein the thread configuration information is also used to characterize the minimum number of threads allowed to be used in the corresponding queue. The number of cores of the processor can be the number of CPU cores.

[0197] Optionally, max_thread can be configured as an optional parameter. It can be used to set the upper limit ratio of the number of threads that a queue can use, and the value range is between (0, 1). The maximum number of threads that a queue can use will be the total number of threads multiplied by this ratio and rounded down. However, in order to ensure that a queue's queries can also fully utilize system resources, the lower limit is the number of CPU cores multiplied by 2, that is, the minimum number of threads is the number of CPU cores * 2. If not set manually, it is determined by the ratio of cpu_share by default.

[0198] Optionally, by setting max_thread, you can limit the number of threads that the queue can use to avoid consuming too many system resources. Limiting the number of threads can maintain system stability and avoid performance degradation or crashes due to excessive system load caused by too many threads. Adjusting the number of threads in the queue according to actual conditions can better allocate system resources and improve system efficiency. Setting max_thread can limit the number of threads that the queue can use to avoid waste and competition of system resources caused by too many threads. By setting max_thread reasonably, you can adjust the system load and ensure that the system runs within a reasonable range. You can avoid excessive thread competition and optimize the performance of the system. Limiting the number of threads can avoid waste of resources and ensure that system resources are effectively used. Controlling the number of threads can avoid excessive system load and improve system stability and reliability. According to actual needs and system load, you can flexibly adjust the value of max_thread to adapt to different scenarios and needs.

[0199] To sum up, the purpose of setting max_thread is to control system resources, ensure system stability, and adjust system load. Its technical effects include improving system performance, avoiding resource waste, improving system stability, and flexibly adjusting the number of threads.

[0200] In the embodiment of the present application, the main purpose of limiting the maximum number of threads of the queue is to avoid small queries being crushed by large queries in terms of CPU resources, and to ensure that small queries can obtain sufficient execution opportunities and resource support. Limiting the maximum number of threads of the queue can help the system allocate thread resources more reasonably, and avoid some queues occupying too many thread resources and affecting the execution of other queues. Under the 256 thread pool, if the number of threads that the queue can use is not limited, it may cause the thread to have to choose the split of the large query because it cannot obtain the split execution of the small query. This may cause CPU competition, causing small queries to be crushed by large queries in terms of CPU resources. Small queries usually have a smaller number of splits and a shorter execution time. Limiting the maximum number of threads of the queue can ensure that small queries obtain sufficient CPU resources and execution opportunities, and avoid being preempted by large queries to affect execution efficiency. By limiting the maximum number of threads of the queue, resource contention and excessive consumption can be avoided, the performance and response speed of the system can be optimized, and different types of query tasks can be ensured to be reasonably processed.

[0201] In general, limiting the maximum number of threads in a queue can help the system better manage and schedule different types of query tasks, avoid unreasonable resource allocation and competition, and improve system performance and stability. This approach helps protect small queries from being interfered with by large queries and ensures that the system can efficiently handle query tasks of different priorities.

[0202] Optionally, the max_thread value is -1, which means that the SQA queue can use all threads, that is, there is no limit on the number of threads. This setting ensures that the SQA queue has priority and higher resource utilization in obtaining CPU resources and thread numbers.

[0203] The present application also provides a data processing method. Figure 3 is a flow chart of a data processing method according to an embodiment of the present application. Figure 3 As shown, the method can be applied to the execution end and can include the following steps:

[0204] Step S302 , monitoring a data query request from a thread, wherein the data query request is used to request a query of data to be executed.

[0205] In the technical solution provided in the above step S302 of the present application, it is possible to monitor whether the user has a data query request. After the data query request is monitored, the data query request can be analyzed to determine the data to be executed in the database system requested by the data query request.

[0206] For example, if a user needs to view certain data in a database system in a server, the user can perform corresponding operations on the client device corresponding to the user to input the need to view the data, thereby forming a data query request. After the data query request is detected on the client device, the data query request can be transmitted to the server through the network so that the server can analyze the data query request.

[0207] Step S304, in response to the data query request, assign a target scheduling time to the queue based on the resource configuration information of the queue in the data warehouse, wherein the resource configuration information is used to characterize the processing resources that the queue is allowed to be scheduled to during operation, and the target scheduling time is used to represent the time allocated to the scheduling queue, and the data warehouse includes multiple queues.

[0208] In the technical solution provided in the above step S304 of the present application, after the data query request is monitored, the target scheduling time can be allocated to the corresponding queue based on the resource configuration information of each of the multiple queues.

[0209] Optionally, the target scheduling time is determined by the resource configuration information and scheduled time of the multiple queues. Each queue has a corresponding resource configuration information configured, and the resource configuration information can represent the weight of the queue to obtain CPU time. It can be set through the system configuration to determine the proportion of CPU resources obtained by each queue. The database system determines the scheduling time of the queue based on the resource configuration information of the queue to which it belongs.

[0210] Step S306, determining the time difference information between the target scheduling time of the queue and the scheduled time of the corresponding queue, wherein the time difference information is used to represent the difference between the target scheduling time and the scheduled time.

[0211] In the technical solution provided in the above step S306 of the present application, after determining the target scheduling time for scheduling the corresponding queue based on the resource configuration information of the multiple queues, the time difference information between the target scheduling time and the scheduled time can be determined.

[0212] In the embodiment of the present application, the time difference information can be used to measure the differences between different queues, which helps to determine the priority and urgency of each queue. Through the time difference information, the scheduling conditions between queues with different time spans can be uniformly analyzed and compared, which improves the standardization level of scheduling. And according to the size of the time difference information, the database system can automatically adjust the execution order, priority or resource allocation of the task to optimize the performance and efficiency of the system. By calculating the time difference information between the target scheduling time and the scheduled time, the system can realize intelligent queue scheduling and resource allocation, adjust the priority of queue execution according to the time difference information, and ensure the rational use of resources.

[0213] Step S308: determining a target queue from multiple queues based on the time difference information of the queues, wherein the time difference information corresponding to the target queue is greater than a difference information threshold.

[0214] In the technical solution provided in the above step S308 of the present application, the time difference information and the difference information threshold can be compared, and the queue whose time difference information is greater than the difference information threshold can be determined as the target queue.

[0215] Optionally, by determining the target queue whose time difference information is greater than the difference information threshold, it can be found that there is an imbalance in resource allocation in the system, which helps to optimize resource allocation and ensure that each queue can obtain a reasonable scheduling time. The target queue whose time difference information is greater than the difference information threshold may contain some tasks with lower priorities. Adjusting the priorities of these tasks can better meet the actual needs of the system and improve the overall operating efficiency of the system.

[0216] Step S310, querying the data to be executed from the target queue.

[0217] In the technical solution provided in the above step S310 of the present application, the to-be-executed data of the request query corresponding to the data query request can be queried from the target queue.

[0218] Optionally, the system needs to determine a target queue and obtain a data query request from the target queue, including information such as the data content to be queried and query conditions.

[0219] Step S312: Return the retrieved data to be executed to the thread for execution.

[0220] In the technical solution provided in the above step S312 of the present application, the queried data to be executed can be returned to the thread for execution.

[0221] Optionally, according to the acquired data query request, a corresponding query operation is performed in the database to acquire the required data to be executed. The acquired data to be executed can be returned to the external thread or system for further processing or display.

[0222] Through the above steps S302 to S312 of the present application, the data query request from the thread is monitored; in response to the data query request, the target scheduling time is allocated to the corresponding queue based on the resource configuration information of multiple queues in the data warehouse; the time difference information between the target scheduling time and the scheduled time of the corresponding queue is determined; based on the time difference information, the target queue is determined from multiple queues; from the target queue, the data to be executed is queried; the data to be executed is returned to the thread for execution. In an embodiment of the present application, by setting corresponding resource configuration information for each queue in the database system, soft isolation between tasks of different priorities, that is, data query requests of different degrees is achieved. In the process of executing data query requests, the target scheduling time of different queues can be automatically calculated according to the resource configuration information, and a more suitable queue can be selected based on the target scheduling time to query the data to be executed, thereby avoiding resource scramble and competition problems when executing large queries and small queries, etc., achieving the technical effect of improving the efficiency of data processing, and solving the technical problem of low efficiency of data processing.

[0223] The present application also provides a data processing method. Figure 4 is a flow chart of a data processing method according to an embodiment of the present application. Figure 4 As shown, the method may include the following steps:

[0224] Step S402: monitoring a data query request by calling a first interface, wherein the data query request is used to request a query of to-be-executed data, and the first interface includes a first parameter, and a parameter value of the first parameter is the data query request.

[0225] In the technical solution provided in the above step S402 of the present application, the first interface may be called to monitor the data query request, wherein the first interface may include a first parameter, and the parameter value of the first parameter may be the data query request.

[0226] Optionally, the first interface may be used to monitor whether the user has a data query request. After the data query request is monitored, the data query request may be analyzed to determine the to-be-executed data in the database system requested to be queried by the data query request.

[0227] Step S404, in response to the data query request, allocating a target scheduling time to the queue based on the resource configuration information of the queue, wherein the resource configuration information is used to characterize the processing resources that the queue is allowed to schedule during operation, and the target scheduling time is used to represent the time allocated for scheduling the queue.

[0228] In the technical solution provided in the above step S404 of the present application, the target scheduling time can be allocated to the corresponding queues based on the resource configuration information of each of the multiple queues.

[0229] Optionally, the target scheduling time is determined by the resource configuration information and scheduled time of each of the multiple queues. For example, the resource configuration information and scheduled time of the queues are obtained, the total scheduling time of each queue is calculated by the above two parameters, the total scheduling time that meets the conditions is selected from the total scheduling time of each queue as the baseline value, and the baseline value is allocated according to the CPU share ratio to obtain the target scheduling time of each queue.

[0230] Step S406, determining the time difference information between the target scheduling time of the queue and the scheduled time of the corresponding queue, wherein the time difference information is used to represent the difference between the target scheduling time and the scheduled time.

[0231] In the technical solution provided in the above step S406 of the present application, the time difference information between the target scheduling time and the scheduled time can be determined.

[0232] Optionally, the difference between the target scheduling time and the scheduled time is determined to obtain time difference information. For example, the time difference is converted into a ratio form, that is, the ratio between the target scheduling time and the scheduled time, which can be obtained by dividing the time difference by the scheduled time.

[0233] Step S408: determining a target queue from multiple queues based on the time difference information of the queues, wherein the time difference information corresponding to the target queue is greater than a difference information threshold.

[0234] In the technical solution provided in the above step S408 of the present application, the time difference information and the difference information threshold can be compared, and the queue whose time difference information is greater than the difference information threshold can be determined as the target queue.

[0235] Optionally, the database system selects a suitable SMLSQ queue as the target queue based on resource configuration information and the scheduled time.

[0236] Optionally, queues can help the system manage the order in which queries are executed, queuing and scheduling them according to the order and priority of the queries entering the queue. Through queues, the system can ensure that queries are executed in a certain order to avoid confusion and conflicts between queries. Queues can be sorted and scheduled according to the priority of the queries, ensuring that high-priority queries are executed first. The system can set different priority rules based on the importance and urgency of different queries to ensure that critical queries are processed in a timely manner. Queues can also be sorted and scheduled according to the execution time of queries, ensuring that long-running queries do not block the execution of other queries.

[0237] Step S410, querying the to-be-executed data from the target queue.

[0238] In the technical solution provided in the above step S410 of the present application, the to-be-executed data of the request query corresponding to the data query request can be queried from the target queue.

[0239] Optionally, the system needs to determine a target queue and obtain a data query request from the target queue, including information such as the data content to be queried and query conditions.

[0240] Step S412: outputting the data to be executed by calling the second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter is the data to be executed.

[0241] In the technical solution provided in the above step S412 of the present application, the data to be executed can be output through the second interface, and the queried data to be executed can be returned to the thread for execution, wherein the second interface can include a second parameter, and the parameter value of the second parameter can be the data to be executed.

[0242] Optionally, according to the acquired data query request, a corresponding query operation is performed in the database to acquire the required data to be executed. The acquired data to be executed can be returned to the external thread or system for further processing or display.

[0243] Through the above steps S402 to S412 of the present application, the data query request is monitored by calling the first interface; in response to the data query request, the target scheduling time is allocated to the corresponding queue based on the resource configuration information of multiple queues respectively; the time difference information between the target scheduling time and the scheduled time of the corresponding queue is determined, wherein the time difference information is used to represent the difference between the target scheduling time and the scheduled time; based on the time difference information, the target queue is determined from multiple queues; and the data to be executed is output by calling the second interface. In an embodiment of the present application, by setting corresponding resource configuration information for each queue in the database system, soft isolation between tasks of different priorities, that is, data query requests of different degrees is achieved. In the process of executing data query requests, the target scheduling time of different queues can be automatically calculated according to the resource configuration information, and a more suitable queue can be selected based on the target scheduling time to query the data to be executed, thereby avoiding resource scramble and competition problems when executing large queries and small queries, etc., achieving the technical effect of improving the efficiency of data processing, and solving the technical problem of low efficiency of data processing.

[0244] According to an embodiment of the present application, an embodiment of a data processing system is also provided. Figure 5 is a schematic diagram of a data processing system according to an embodiment of the present application, such as Figure 5 As shown, the data processing system 500 may include: a client 501 and a server 502 .

[0245] The client 501 is used to send a data query request, wherein the data query request is used to request to query the data to be executed.

[0246] In this embodiment, the data query request corresponding to the to-be-executed data that the user needs to query can be obtained through the client 501. After the data query request is obtained, the data query request can be transmitted to the server 502 through the network.

[0247] Server 502 is used to respond to a data query request and allocate a target scheduling time to a queue based on resource configuration information of the queue, wherein the resource configuration information is used to characterize the processing resources that the queue is allowed to schedule during operation, and the target scheduling time is used to represent the time allocated to the scheduling queue; determine the time difference information between the target scheduling time of the queue and the scheduled time of the corresponding queue, wherein the time difference information is used to represent the difference between the target scheduling time and the scheduled time; determine a target queue from multiple queues based on the time difference information of the queue, wherein the time difference information corresponding to the target queue is greater than a difference information threshold; query the data to be executed from the target queue; and return the data to be executed to the client.

[0248] In this embodiment, after the server 502 receives the data query request from the client 501, the processing resources that the data query request in the queue is allowed to be scheduled to during operation can be analyzed according to the resource configuration information of each queue, and the target scheduling time when the current data query request is assigned to the scheduling processing resources in the queue for execution can be determined. The time difference information between the target scheduling time and the corresponding queue scheduled time can also be determined. The queue whose time difference information is greater than the difference information threshold is determined as the target queue, so that the current data query request can use the target queue to schedule the corresponding processing resources to query the corresponding data to be executed.

[0249] Optionally, obtain the resource configuration information and scheduled time of the queue, calculate the total scheduling time of each queue by using the above two parameters, select the total scheduling time that meets the conditions from the total scheduling time of each queue as the baseline value, and allocate the baseline value according to the CPU share ratio to obtain the target scheduling time of each queue. This ensures that each queue reasonably allocates the scheduling time according to the configuration and scheduling situation, and improves the task scheduling efficiency of the system.

[0250] Optionally, the difference between the target scheduling time and the scheduled time is determined to obtain time difference information. For example, the time difference is converted into a ratio form, that is, a ratio between the target scheduling time and the scheduled time (difference ratio), which can be obtained by dividing the time difference by the scheduled time.

[0251] Optionally, a suitable SMLSQ queue is selected as the target queue according to resource configuration information and the scheduled time.

[0252] Optionally, the system needs to determine the target queue and obtain the data query request from the target queue, including the data content to be queried, query conditions and other information. According to the obtained data query request, the corresponding query operation is performed in the database to obtain the required data to be executed. The obtained data to be executed can be returned to the external thread or system for further processing or display.

[0253] In this embodiment, a data processing system is provided. A data query request is sent through a client 501, wherein the data query request is used to request to query the data to be executed; a server 502 responds to the data query request, and respectively allocates a target scheduling time to the corresponding queue based on the resource configuration information of multiple queues, wherein the resource configuration information is used to characterize the processing resources that the corresponding queue is allowed to schedule during operation; determines the time difference information between the target scheduling time and the scheduled time of the corresponding queue, wherein the time difference information is used to represent the difference between the target scheduling time and the scheduled time; determines a target queue from multiple queues based on the time difference information, wherein the time difference information corresponding to the target queue is greater than the difference information threshold; queries the data to be executed from the target queue; and returns the data to be executed to the client, thereby achieving a technical effect of effectively improving the efficiency of data processing and solving the technical problem of low efficiency of data processing.

[0254] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application, such as the data for inspection, are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0255] At present, with the development of the data era, companies are paying more and more attention to the value and management of data. Integrating multiple different database servers into a unified database has many advantages. A unified database can better integrate and manage data, making it easier to maintain and use. Data sharing and exchange between different database systems become more convenient, reducing the risk of data redundancy and inconsistency. At the same time, a unified database system can also provide more efficient and accurate data query and analysis functions, thereby better supporting decision-making and task development. A unified database system can realize multi-source big data analysis and help discover more value in data. By integrating multiple data sources, you can have a more comprehensive understanding of customer needs, market trends and competitor dynamics, so as to better formulate relevant strategies, product planning and task development directions. This comprehensive data analysis also helps to discover new business opportunities and growth points, and enhance competitiveness and innovation capabilities. From a cost perspective, a shared database system often means less cost. Maintaining a unified database system is more efficient and economical than maintaining multiple database systems, reducing the waste of manpower, time and resources. In addition, a unified database system can also make better use of hardware and software resources, reducing the investment and operating costs of IT infrastructure.

[0256] In summary, it is a wise choice to integrate various database servers into a unified database. This will not only help improve the efficiency and quality of data management and analysis, but also reduce costs and discover more value, thus promoting continuous development and innovation.

[0257] Under the above trend, many different types of workloads are running in shared databases, including core task queries and non-core queries for temporary analysis. These workloads have different characteristics, some have high requirements for overall response time and low resource overhead, while others are complex and consume high resources. Users have different performance requirements for different types of workloads, so the database system needs to be able to effectively manage and schedule these diverse workloads to meet the different performance requirements of users and improve the overall efficiency and responsiveness of the system.

[0258] In the data warehouse, queries running simultaneously share the database system's CPU, memory, disk IO and other resources. If some queries that consume huge system resources are not restricted, other queries will have to compete with them for resources, thus affecting the overall performance. Therefore, in order to ensure the stability and performance of the system, it is necessary to restrict queries that consume large resources, reasonably allocate and manage system resources, ensure that each query can get enough resource support, and improve the overall efficiency and responsiveness of the system.

[0259] In the current data warehouse, four different priority queues have been provided to customers. Different queries will enter different queues according to their own priorities. However, the only difference between these queues is the maximum concurrency limit on the front-end node, and the query priority is not reflected on the execution side. At the same time, the isolation between different queues is very weak and cannot meet the customer's task requirements well. There are certain problems with the current data warehouse priority queue setting, including the query priority is not reflected on the execution side and the isolation between different queues is insufficient. In order to better meet the customer's task requirements, the system needs to be further optimized and improved, strengthen the management of query priority and resource isolation, and improve the performance and flexibility of the system. This can better meet the different task requirements of customers and improve the overall performance and user experience of the system.

[0260] In addition, the current SQA of the data warehouse splits the queue so that small queries will not be blocked by large queries and queue up, but there is still resource contention and competition on the execution side. The previously developed Multi level split queue implements a mechanism for the gradual differentiation of large and small queries over time, with large queries gradually reducing the allocated time slice resources. This mechanism can more effectively manage the resource allocation of queries of different sizes, improve the execution efficiency of the system and optimize the query experience.

[0261] In actual tests, we found that since all queries must gradually descend from the first level of the Multi-level split queue, small queries in the first level are often affected by new queries. At the same time, since large queries take a long time to execute and have a large number of splits, the current 256 threads cannot effectively limit the scheduling time of large queries and the resource competition for small queries.

[0262] Under the current Multi level split queue execution model, although small queries have advantages to a certain extent, the overall isolation effect still has a lot of room for improvement. In other words, the above method still has the technical problem of low data processing efficiency.

[0263] Furthermore, the present application provides a hybrid load isolation solution for OLAP data warehouse based on priority queues, and creates a new pending split queue on the execution side, including five SMLSQ queues. Newly entered queries enter different Simple Multi Level Split Queues for execution according to the queues to which they belong on RC. The SMLSQ here basically implements the functions of the original Multi Level Split Queue, but simplifies the scheduling process and reduces the complex scheduling logic and lock overhead. Each SMLSQ also has a certain resource inclination towards small queries to avoid the excessive resource consumption of large queries affecting the performance of other queries. The pending split queue is divided into five SMLSQ queues, and the queries are assigned to different queues for execution according to the different queues to which the queries belong on RC, realizing the function of multi-level segmentation. By simplifying the scheduling process and reducing the lock overhead, the execution efficiency and performance of the system are improved. Each SMLSQ has a certain resource inclination towards small queries to avoid the excessive resource consumption of large queries affecting the performance of other queries, and optimizes the allocation and utilization of system resources. Through this setting, different types of queries can be better managed, the execution efficiency and performance of the system can be improved, and the query experience and the overall performance of the system can be optimized. This avoids the resource scramble and competition problems when executing large queries and small queries, achieves the technical effect of improving data processing efficiency, and solves the technical problem of low data processing efficiency.

[0264] The above method of this embodiment is further introduced below.

[0265] In the embodiment of the present application, a new pending split queue is proposed, including five Simple Multi Level Split Queue (SMLSQ) queues. The design of this new queue is to allocate new incoming queries to different Simple Multi Level Split Queues for execution according to the queue to which the query belongs on the RC. The purpose of this is to better utilize the previously developed large and small query differentiation mechanism, while simplifying the scheduling process and reducing the complex scheduling logic and lock overhead.

[0266] Optionally, in the simplified scheduling process, some of the following measures may be taken: Simplifying the scheduling process may reduce the locking operations on shared resources, thereby reducing the lock overhead and contention, and improving the concurrent performance of the system. Simplifying the scheduling process may reduce complex scheduling logic, simplify the task allocation process, and improve the execution efficiency of the system. Simplifying the scheduling process may adopt a simpler task scheduling strategy, reduce the complex judgment of query priority and resource requirements, and improve the response speed and stability of the system. Simplifying the scheduling process may pay more attention to the efficiency of resource utilization, avoid waste and excessive consumption of resources, and thus improve the overall performance of the system. In general, simplifying the scheduling process can help reduce the complexity and overhead of the system, improve the execution efficiency and performance of the system, while maintaining the stability and reliability of the system.

[0267] Optionally, in this new design, Simple Multi Level Split Queue basically implements the functions of the original Multi Level Split Queue, but simplifies the scheduling process. Each Simple Multi Level Split Queue can also have a certain resource bias towards small queries to avoid excessive resource consumption of large queries affecting the performance of other queries. Through this setting, the following advantages can be achieved: By allocating queries to different Simple Multi Level Split Queues for execution, system resources can be used more effectively and the overall performance of the system can be improved. Simplifying the scheduling process can reduce the complexity of the system, reduce scheduling overhead, and improve the execution efficiency of the system. Each Simple Multi Level Split Queue has a certain resource bias towards small queries to avoid excessive resource consumption of large queries, thereby improving the stability of the system and query performance.

[0268] In general, this new pending split queue setting can effectively improve the performance and stability of the system, simplify the scheduling logic and make full use of the previous query differentiation mechanism, thereby improving the overall performance of the system.

[0269] Optionally, in the above queue, resource tilting for small queries means allocating more resources to small queries inside the Simple Multi LevelSplit Queue to ensure that they can execute faster and get more system resources. This resource tilting can be achieved in the following ways: Adjust the priority of small queries in the queue so that they have a higher execution priority so that they can be executed faster. Allocate more computing resources, such as CPU, memory, etc. to small queries, and give priority to meeting the resource requirements of small queries when executing. Limit the number of large queries executed simultaneously in the queue to provide more execution opportunities for small queries.

[0270] Optionally, resource tilting can be used to speed up the execution of small queries and improve the responsiveness and performance of the system by allocating more resources to them. Resource tilting can avoid excessive resource consumption by large queries, thereby avoiding affecting the execution performance of other queries. Resource tilting can help balance the utilization of system resources and avoid situations where uneven resource allocation leads to system performance degradation or system crashes. By tilting resources for small queries in Simple Multi Level SplitQueue, the overall performance and stability of the system can be effectively improved, while ensuring that all types of queries can be reasonably allocated resources and executed.

[0271] In this embodiment, Figure 6 This is a schematic diagram of the initial version of the pending segmentation queue processing process, such as Figure 6 As shown in the figure, there is a thread pool of 256 threads on the execution node, and each thread takes a split from the pendingSpilts queue and executes a time slice in an infinite loop. In this case, the pendingSpilts queue is described as a PriorityBlockingQueue, where queries basically follow the FIFO priority. Specifically, PriorityBlockingQueue is an unbounded blocking queue that supports priorities, and elements are processed in the order of their priorities. Here, queries are prioritized according to the order in which they enter the pending Spilts queue, that is, the first-in-first-out principle. Each thread takes a split from the pending Spilts queue and executes a time slice. This mechanism can effectively utilize the thread pool resources of 256 threads while maintaining the order of queries. Since the PriorityBlockingQueue sorts the elements according to their priorities, queries are processed according to the FIFO priority, ensuring the execution order of queries.

[0272] The above settings can ensure that queries are processed in a timely manner and maintain the order of queries, avoiding possible priority inversion or starvation. At the same time, since each thread in the thread pool is acquiring and executing queries in an infinite loop, the system can make better use of multi-threaded parallel processing of queries, improving the system's concurrency performance and response speed.

[0273] Optionally, there is a thread pool of 256 threads on the execution node. Each thread will take the split from the pending Spilts queue in an infinite loop and execute for a time slice. The pendingSpilts queue here is a Priority Blocking Queue, which is an unbounded blocking queue that supports priorities, in which queries basically follow FIFO priorities. Specifically, each thread takes the split from the queue and executes it according to the FIFO priority, which means that the split that enters the queue first will be processed first. The PriorityBlockingQueue sorts the elements according to their priorities to ensure that splits with higher priorities are processed first.

[0274] With this setup, the system can fully utilize the thread pool resources of 256 threads while maintaining the order of queries. Each thread obtains and executes splits from the queue, which can effectively handle query tasks in the system and maintain the efficiency and responsiveness of the system. This FIFO priority mechanism based on PriorityBlockingQueue can ensure that queries are processed in the order they enter the queue, avoiding potential priority inversion or starvation.

[0275] Optionally, there is a thread pool of 256 threads on the execution node, each thread takes out the split from the pendingSpilts queue in an infinite loop and executes a time slice. At this time, the pendingSpilts queue is a PriorityBlockingQueue, and the query basically follows the FIFO priority.

[0276] In the embodiments of the present application, Figure 7 is a schematic diagram of an improved multi-level priority queue and scheduling time according to an embodiment of the present application, such as Figure 7As shown in the figure, the pending Spilts queue can be improved so that it is no longer a single queue, but a multi-level feedback queue consisting of multiple priority queues, including Level 0, Level 1, Level 2, Level 3, and Level 4. The scheduling time allocated to different priority queues is reduced by a multiple of 2 layer by layer, that is, the expected scheduling time can be reduced by a multiple of 2 layer by layer according to different priority queues. When a query first enters, it will be in the first-level priority queue, and then gradually descend in level as the execution time increases, so as to avoid large queries from affecting the execution of small queries too much.

[0277] Optionally, Figure 8 is a schematic diagram of a multi-level priority queue in an embodiment of the present application, such as Figure 8 As shown in the figure, there are 6 priority queues on the RC node, namely ROOT, SQA, high, normal, low and LOWEST queues.

[0278] like Figure 8 As shown in the figure, in the LOWEST queue, the initial priority of the data extraction, transformation and loading (Extract, Transform, Load, referred to as ETL) or query is set to the lowest (lowest), that is, query_priorty=lowest, and then adjusted to 5, which is in the medium priority range between 0 and 9 (query_priority=5 (query_priority=0~9)). This process may mean that the task was initially considered the least urgent or least important, but was later promoted to a medium priority in the system so that it can be executed faster. In the low queue, the initial priority of the batch task or query is set to low (low), that is, query_priorty=low, and then adjusted to 15, which is in the higher priority range between 10 and 19 (query_priority=15 (query_priority=10~19)). This process may mean that the task was initially considered less urgent or important, but was later promoted to a higher priority in the system so that it can be executed faster.

[0279] like Figure 8As shown in the figure, in the normal queue, the initial priority of the task is set to the default value or normal (normal), that is, query_priorty=normal, and then adjusted to 25, which is within the higher priority range of 20 to 29 (query_priority=25 (query_priority=20~29)). This process shows that the task was initially regarded as having a general priority, but was later promoted to a higher priority in the system to be processed more quickly. In the high queue, the initial priority of the task is set to high (high), that is, query_priorty=high, and then adjusted to 35, which is within the higher priority range of 30 to 39 (query_priority=35 (query_priority=30~39)). This shows that the task is considered to have a high urgency and importance and is set to a high priority in the system to ensure its quick processing and execution.

[0280] like Figure 8 As shown in the figure, in the ROOT queue, the query type of the task is set to system, that is, query_type=system, and then the priority of the task is adjusted to 45, which is in the priority range between 40 and 49 (query_priority=45 (query_priority=40~49)). This means that the task is regarded as a system-level task with a higher priority to ensure the normal operation of the system and the timely processing of critical tasks. In the SQA queue, the query priority of the task is set to SQA, that is, query_priorty=sqa, and the priority of the task is adjusted to 255 (query_priority=255). This value usually indicates the highest priority, which is used to ensure that the SQA task is processed quickly and efficiently. This setting indicates that the SQA task has the highest urgency and importance in the system to ensure that the system quality and performance are effectively managed and maintained.

[0281] Optionally, two configurations can be added to the four priority queues of high, normal, low, and LOWEST, such as cpu_share and max_thread. The introduction of these two configurations can help the system manage and allocate CPU resources and the number of threads more flexibly, and achieve soft isolation of CPU resources and limit the number of threads for priority queues, thereby optimizing system performance and resource utilization. By properly configuring cpu_share and max_thread, you can better control the CPU resource allocation and thread utilization of different priority queues, ensuring that the system can efficiently process query tasks of various priorities.

[0282] For example, cpu_share is a mandatory configuration, an integer between 1 and 1000, which is used to set the amount of CPU time that the priority queue obtains, which can achieve soft isolation of CPU resources. cpu_share is a relative value, indicating the weight of the CPU resources that the running priority queue can obtain. For example, three priority queues are configured, namely low, normal, and high, and cpu_share is 10, 30, and 40 respectively. At a certain moment, low and normal are running tasks, while high has no tasks. At this time, low can obtain 25% (10 / (10+30)) of the CPU resources, and normal can obtain 75% of the CPU resources. If the system has only one priority queue with queries running, it can obtain all the CPU resources regardless of the value of its cpu_share.

[0283] For another example, max_thread is an optional configuration, which is used to set the upper limit ratio of the number of threads that a queue can use. The range is between (0, 1). The maximum number of threads that a queue can use is the total number of threads multiplied by this ratio and rounded down. However, in order to ensure that a query in a queue can also use up all system resources, the lower limit is the number of CPU cores * 2. If this configuration ratio is not manually set, it is determined by the cpu_share ratio by default.

[0284] Optionally, the SQA queue is treated specially, and its cpu_share value is equal to the sum of other queues, and max_thread is -1, which can use all threads. Queries in the ROOT queue temporarily enter the SQA queue on the execution side, and later it may be considered to isolate a separate thread pool to execute system queries. Queries in the ROOT queue temporarily enter the SQA queue on the execution side, which means that queries in the ROOT queue will give priority to using the resources of the SQA queue when executing. This temporary entry into the SQA queue can improve the execution efficiency and priority of queries in the ROOT queue. Later it may be considered to isolate a separate thread pool to execute system queries. This approach can ensure that system query tasks can be executed independently without being affected by other queues, ensuring the stability and reliability of system queries. Through special settings and processing of the SQA queue and the ROOT queue, the system can better manage and schedule different types of query tasks, improving the performance and efficiency of the system. At the same time, isolating a separate thread pool to execute system queries can further optimize the operation and resource utilization of the system, ensuring that the core functions of the system can be fully supported and guaranteed.

[0285] In this embodiment, a new pending split queue is created on the execution side, including five SMLSQ queues. Newly entered queries will be executed in different Simple Multi Level Split Queues according to the queues they belong to on the RC. These SMLSQ queues basically implement the functions of the original Multi Level Split Queue, but simplify the scheduling process, reducing complex scheduling logic and lock overhead. At the same time, each SMLSQ queue can also tilt resources to small queries to avoid excessive resource consumption of large queries affecting the performance of other queries.

[0286] Optionally, in this new system, when a thread makes a request to the Priority Schedule Split Queue to execute a split, the system selects a suitable SMLSQ queue based on the CPU share value and the time already scheduled. This ensures that the scheduling time between different SMLSQ queues basically conforms to the CPU share ratio, thereby achieving reasonable scheduling and execution of tasks of different priorities. This setting makes full use of the resource allocation and task scheduling of different SMLSQ queues, while improving the performance and efficiency of the system by simplifying the scheduling logic and resource management. At the same time, tilting resources towards small queries can ensure that the system can better handle different types of query tasks and improve the overall performance and stability of the system. By selecting the appropriate SMLSQ queue based on the CPU share value and the scheduled time, the system can better manage and schedule query tasks and improve the resource utilization and performance of the system.

[0287] Optionally, Fig. 9 is a schematic diagram of an overall architecture of data warehouse hybrid load isolation based on priority queues according to an embodiment of the present application, such as Fig. 9As shown in the figure, a new pending split queue is created on the execution side, including five SMLSQ queues, which may include Simple Multi-Level Split Queue 0, Simple Multi-Level Split Queue 1, Simple Multi-Level Split Queue 2, Simple Multi-Level Split Queue 3, and Simple Multi-Level Split Queue 4. Newly entered queries enter different Simple Multi Level Split Queues for execution according to the queues they belong to on RC. In order to make full use of the previously developed large and small query differentiation mechanism, the SMLSQ here basically implements the functions of the original Multi Level Split Queue, but simplifies the scheduling process, reduces the complex scheduling logic and lock overhead, and enables each SMLSQ to have a certain resource bias for small queries, avoiding the excessive resource consumption of large queries affecting the performance of other queries. When a thread makes a request to PriorityScheduleSplitQueue to take a split execution, the system will select a suitable SMLSQ queue based on the CPU share value and the time that has been scheduled, so that the scheduling time between different SMLSQ queues basically conforms to the CPU share ratio.

[0288] Alternatively, if Fig. 9 As shown in the figure, since small queries have a smaller number of splits and a shorter execution time, in a 256-thread pool, if the number of threads that a queue can use is not limited, it is often found that threads cannot get the splits of small queries to execute and have to choose the splits of large queries. In the end, the small queries are completely crushed in CPU competition. Therefore, an upper limit on the maximum number of threads is set for different queues to prevent small queries from being crushed by large queries in terms of the number of threads and thus being preempted in terms of CPU resources.

[0289] Optionally, in the internal scheduling logic, the system continuously maintains the scheduling time ratio between queues and the CPU share ratio through scheduling. This means that the system will select the appropriate queue based on the CPU share value and the time already scheduled to achieve a balance between the scheduling time ratios of different queues and maintain consistency with the CPU share value.

[0290] FIG10( a ) is a flowchart of taking a task from a queue for processing according to an embodiment of the present application. As shown in FIG10( a ), the method for taking a task (take split) may include the following steps:

[0291] Step S1001, calculate the max target total scheduling time, index=0.

[0292] In this embodiment, the maximum target total scheduling time is calculated, and the index is set to 0, and a circular processing queue is started.

[0293] Optionally, first calculate the maximum target total scheduling time and set index to 0 to prepare to start looping through the queue. The purpose of this step is to determine the overall scheduling time and initialize the index value.

[0294] Step S1002, Index<5.

[0295] In this embodiment, it is determined whether index is less than 5. If so, step S1003 is executed, otherwise, step S1005 is executed. This step is to ensure that the index value is within the valid range. If index is less than 5, the queue is processed continuously, otherwise the processing is terminated.

[0296] Step S1003, determine whether the queue with sequence number index is empty and the number of threads does not exceed the maximum upper limit.

[0297] In this embodiment, it is determined whether the queue with the serial number index is not empty and the number of threads does not exceed the maximum upper limit. If the above conditions are met, step S1004 can be executed, otherwise, the execution can return to step S1002.

[0298] Step S1004, calculate the ratio of target scheduling time / actual scheduling time and save it, index=index+1.

[0299] In this embodiment, if the queue with index is not empty and the number of threads does not exceed the maximum limit, the ratio of the target scheduling time to the actual scheduling time is calculated and saved in an array, and then the index is increased by 1. This step is to calculate the scheduling time ratio of the current queue, save it in an array, and then continue to process the next queue.

[0300] Step S1005, whether the array storing the ratio is empty.

[0301] In this embodiment, it is checked whether the array storing the ratios is empty, and if so, step S1006 is executed, and if not, step S1007 is executed.

[0302] Optionally, check whether the array storing the ratios is empty. If so, return null, indicating that there are no queues that meet the criteria. This step is to check whether the ratio array is empty after the loop processes all queues. If it is empty, it means that there are no queues that meet the criteria.

[0303] Step S1006, returns null.

[0304] In this embodiment, null is returned, indicating that there is no queue that meets the conditions.

[0305] Step S1007, take out a split from the index queue corresponding to max(ratio) and return it.

[0306] In this embodiment, the index corresponding to the maximum ratio is found from the storage ratio array, and then a split is taken out from the queue and returned.

[0307] FIG10( b ) is a flowchart of placing a task into a queue according to an embodiment of the present application. As shown in FIG10( b ), the method of placing a task (put split) may include the following steps:

[0308] Step S1008, obtaining the queue corresponding to the priority of the split.

[0309] In this embodiment, the queue corresponding to the priority of the split is obtained. The purpose of this step is to determine the queue to be placed according to the priority of the split.

[0310] Step S1009, determine whether the queue is empty.

[0311] In this embodiment, it is determined whether the queue is empty, if so, step S1010 can be executed, otherwise, step S1011 can be executed. This step is to check whether the target queue is empty to determine whether it is necessary to make up the scheduling time or directly put it into split.

[0312] Step S1010, completing the corresponding SMLSQ scheduling time to the minimum value by calculating the minimum target total time.

[0313] In this embodiment, the scheduling time of the corresponding SMLSQ is supplemented to the minimum value by calculating the minimum target total time. This step is to supplement the scheduling time of the corresponding queue to ensure that the minimum target total time is reached.

[0314] Step S1011, put the split into the corresponding SMLSQ.

[0315] In this embodiment, the split is put into the corresponding SMLSQ. This step is to put the split into the corresponding queue and complete the operation of integrating the queues.

[0316] Optionally, when executing the take split process, that is, the process of obtaining and executing query tasks, the following steps are usually followed: Put Split enters the process of the corresponding queue. First, the system puts the query task into the corresponding queue according to the priority of the query and other relevant parameters. According to the description, the appropriate queue may be selected according to the CPU share value and the scheduled time. Once the query task enters the queue, the system will decide when to take the query task out of the queue and execute it according to the scheduling logic and strategy. According to the scheduling time ratio and CPU share value between the queues, the system will choose the appropriate time to execute the query task. The system will reasonably allocate system resources to query tasks in different queues according to the priority, CPU share value and other parameters of the queue, ensuring that the system can efficiently process various types of query tasks. Through such internal scheduling logic and process, the system can reasonably allocate system resources and schedule query tasks according to the priority and CPU share value of different queues, so as to achieve effective processing and execution of different types of query tasks. This scheduling method can help the system maintain balanced resource utilization and improve system performance and stability.

[0317] For example, Fig.11 is a schematic diagram of a scheduling example according to an embodiment of the present application, such as Fig.11 As shown in the figure, it is assumed that the customer currently sets the cpu_share values ​​of the high, normal, low, and lowest queues to 8, 4, 2, and 1 respectively. At this time, the cpu_share value of the SQA queue is their sum of 15, and the maximum number of threads they can use is 256, 128, 64, 32, and 16 respectively. The time that each queue has scheduled is 140, 90, 40, 30, and 10. When a thread in the system needs to take a split out for execution, each queue first calculates the total scheduling time by dividing the scheduling time by the cpu_share value ratio based on its own scheduled time and cpu_share value ratio. The current baseline is selected from the total scheduling time, and the total scheduling time baseline is further allocated as the target scheduling time according to the cpu_share ratio. The gap ratio is obtained by dividing the target scheduling time by the scheduling time, and the target gap ratio is selected from the gap ratio. The corresponding queue is determined, a split is taken out, and it is returned to the external thread for execution. If the queue is empty or the queue has reached the upper limit of the number of threads that can be used, the next largest queue is selected.

[0318] Optionally, in the system, when you need to put a split into a queue, you need to consider the scheduling time of the queue and the impact of the cpu_share value. If the queue is not empty, you can put the split directly into the queue. However, if the queue is empty, it may mean that the queue has been empty for a long time and its scheduling time is low. At this time, directly putting the split into the queue may cause the system to tend to schedule this queue, causing preemption of other queues. To avoid this, you need to bring the scheduling time of the queue into a reasonable range when putting the split.

[0319] Fig.12 is a schematic diagram of a CPU usage status of a system with 16 threads fully utilized according to an embodiment of the present application, such as Fig.12 As shown in the figure, the horizontal axis of the graph is time, and the vertical axis is CPU utilization, which describes the average CPU utilization of the control node, the maximum CPU utilization of the control node, the CPU utilization of the storage computing node, and the CPU utilization of the computing node. 16 threads are enough to fully utilize the CPU resources of the system, which means that the system can achieve the maximum CPU utilization when using 16 threads. Letting the system reach the maximum CPU utilization is usually called full system. When the CPU utilization of the system reaches 100%, it means that the CPU resources of the system have been fully utilized and there is no idle processing power. This may mean that the system is processing a large number of tasks or calculations, or is running an application that requires a lot of CPU resources. In this case, the system may show a high response speed and processing power, but it may also cause the system load to be too high, affecting the execution of other tasks. By setting the system to use 16 threads to fully utilize the CPU resources, it can ensure that the system can maximize the use of the CPU processing power when processing tasks, and improve the overall performance and efficiency of the system. By reasonably allocating threads and resources, the load of the system can be balanced, and resource waste or excessive occupation can be avoided, thereby improving the stability and performance of the system.

[0320] In general, by using 16 threads to fully utilize the system's CPU resources, the system can achieve a higher CPU utilization rate and improve the system's processing power and performance. This approach can help the system process tasks more effectively and improve the system's response speed and efficiency.

[0321] In an embodiment of the present application, if it is necessary to query the data to be queried by the data query request, the data query request received from the client can be monitored to analyze the pending data that needs to be requested to be queried from the server. According to the resource configuration information of each queue, the processing resources that the data query request in the queue is allowed to be scheduled during operation can be analyzed, and the target scheduling time when the current data query request is assigned to the scheduling processing resources in the queue for execution can be determined. The time difference information between the target scheduling time and the corresponding queue scheduled time can also be determined. The queue whose time difference information is greater than the difference information threshold is determined as the target queue, so that the current data query request can use the target queue to schedule the corresponding processing resources to query the corresponding pending data. In an embodiment of the present application, by setting corresponding resource configuration information for each queue in the database system, tasks of different priorities, that is, soft isolation between data query requests of different degrees is achieved. In the process of executing data query requests, the target scheduling time of different queues can be automatically calculated according to the resource configuration information, and a more appropriate queue can be selected to query the data to be executed based on the target scheduling time, thereby avoiding resource scramble and competition problems during the execution of large queries and small queries, achieving the technical effect of improving the efficiency of data processing and solving the technical problem of low data processing efficiency.

[0322] According to an embodiment of the present application, there is also provided a method for implementing the above Figure 2 The data processing method shown is a data processing device.

[0323] Fig.13 is a schematic diagram of a data processing device according to an embodiment of the present application, such as Fig.13 As shown, the data processing device 1300 may include: a first monitoring unit 1302 , a first allocating unit 1304 , a first determining unit 1306 , a second determining unit 1308 and a first querying unit 1310 .

[0324] The first monitoring unit 1302 is used to monitor data query requests, wherein the data query request is used to request to query the to-be-executed data.

[0325] The first allocation unit 1304 is used to allocate a target scheduling time to the queue based on the resource configuration information of the queue in response to the data query request, wherein the resource configuration information is used to characterize the processing resources that the queue is allowed to be scheduled to during operation, and the target scheduling time is used to represent the time allocated to the scheduling queue.

[0326] The first determining unit 1306 is configured to determine time difference information between a target scheduling time of a queue and a scheduled time of the queue, wherein the time difference information is used to indicate a difference between the target scheduling time and the scheduled time.

[0327] The second determining unit 1308 determines a target queue from the multiple queues based on the time difference information of the queues, wherein the time difference information corresponding to the target queue is greater than the difference information threshold.

[0328] The first query unit 1310 queries the to-be-executed data from the target queue.

[0329] Here, the first monitoring unit 1302, the first allocation unit 1304, the first determination unit 1306, the second determination unit 1308 and the first query unit 1310 correspond to steps S202 to S210 in the above embodiment, and the five units are the same as the examples and application scenarios implemented by the corresponding steps, but are not limited to the contents disclosed in the above embodiment. It should be noted that the above units can be hardware components or software components stored in a memory (e.g., memory 1604) and processed by one or more processors (e.g., processors 1602a, 1602b..., 1602n), and the above units can also be run as part of the device in the computer terminal A provided in the following embodiment.

[0330] According to an embodiment of the present application, there is also provided a method for implementing the above Figure 3 The data processing method of the device shown is a data processing device.

[0331] Fig.14 is a schematic diagram of a data processing device according to an embodiment of the present application, such as Fig.14 As shown, the data processing device 1400 may include: a second monitoring unit 1402 , a second allocation unit 1404 , a third determination unit 1406 , a fourth determination unit 1408 , a second query unit 1410 and a first execution unit 1414 .

[0332] The second monitoring unit 1402 is used to monitor a data query request from a thread, wherein the data query request is used to request to query the to-be-executed data.

[0333] The second allocation unit 1404 is used to respond to the data query request and allocate a target scheduling time to the queue based on the resource configuration information of the queue in the data warehouse, wherein the resource configuration information is used to characterize the processing resources that the queue is allowed to be scheduled to during operation, and the target scheduling time is used to represent the time allocated to the scheduling queue, and the data warehouse includes multiple queues.

[0334] The third determining unit 1406 is used to determine the time difference information between the target scheduling time of the queue and the scheduled time of the corresponding queue, wherein the time difference information is used to represent the difference between the target scheduling time and the scheduled time.

[0335] The fourth determining unit 1408 is configured to determine a target queue from the multiple queues based on the time difference information of the queues, wherein the time difference information corresponding to the target queue is greater than the difference information threshold.

[0336] The second query unit 1410 is used to query the to-be-executed data from the target queue.

[0337] The first execution unit 1414 is used to return the queried data to be executed to the thread for execution.

[0338] It should be noted that the second monitoring unit 1402, the second allocation unit 1404, the third determination unit 1406, the fourth determination unit 1408, the second query unit 1410 and the first execution unit 1414 correspond to steps S302 to S312 in the above embodiment, and the six units are the same as the examples and application scenarios implemented by the corresponding steps, but are not limited to the contents disclosed in the above embodiment. It should be noted that the above units can be hardware components or software components stored in a memory (e.g., memory 1604) and processed by one or more processors (e.g., processors 1602a, 1602b..., 1602n), and the above units can also be run as part of the device in the computer terminal A provided in the following embodiment.

[0339] According to an embodiment of the present application, there is also provided a method for implementing the above Figure 4 The data processing method of the device shown is a data processing device.

[0340] Fig.15 is a schematic diagram of a data processing device according to an embodiment of the present application, such as Fig.15 As shown, the data processing device 1500 may include: a third monitoring unit 1502 , a third allocating unit 1504 , a fifth determining unit 1506 , a sixth determining unit 1508 , a third querying unit 1510 and a second executing unit 1512 .

[0341] The third monitoring unit 1502 is used to monitor the data query request by calling the first interface, wherein the data query request is used to request to query the to-be-executed data, and the first interface includes a first parameter, and the parameter value of the first parameter is the data query request.

[0342] The third allocation unit 1504 is used to allocate a target scheduling time to the corresponding queue based on the resource configuration information of the queue in response to the data query request, wherein the resource configuration information is used to characterize the processing resources that the queue is allowed to schedule during operation, and the target scheduling time is used to schedule the time allocated to the queue.

[0343] The fifth determining unit 1506 is used to determine the time difference information between the target scheduling time of the queue and the scheduled time of the corresponding queue, wherein the time difference information is used to represent the difference between the target scheduling time and the scheduled time.

[0344] The sixth determining unit 1508 is configured to determine a target queue from the multiple queues based on the time difference information of the queues, wherein the time difference information corresponding to the target queue is greater than the difference information threshold.

[0345] The third query unit 1510 is used to query the to-be-executed data from the target queue.

[0346] The second execution unit 1512 is used to output the data to be executed by calling the second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter is the data to be executed.

[0347] It should be noted that the third monitoring unit 1502, the third allocation unit 1504, the fifth determination unit 1506, the sixth determination unit 1508, the third query unit 1510 and the second execution unit 1512 correspond to steps S302 to S312 in the above embodiment, and the six units are the same as the examples and application scenarios implemented by the corresponding steps, but are not limited to the contents disclosed in the above embodiment. It should be noted that the above units can be hardware components or software components stored in a memory (e.g., memory 1604) and processed by one or more processors (e.g., processors 1602a, 1602b..., 1602n), and the above units can also be run as part of the device in the computer terminal A provided in the following embodiment.

[0348] In the data processing device, if it is necessary to query the data to be queried by the data query request, the data query request received from the client can be monitored to analyze the pending data that needs to be requested from the server. According to the resource configuration information of each queue, the processing resources that the data query request in the queue is allowed to be scheduled during operation can be analyzed, and the target scheduling time when the current data query request is assigned to the scheduling processing resources in the queue for execution can be determined. The time difference information between the target scheduling time and the corresponding queue scheduled time can also be determined. The queue whose time difference information is greater than the difference information threshold is determined as the target queue, so that the current data query request can use the target queue to schedule the corresponding processing resources to query the corresponding pending data. In an embodiment of the present application, by setting corresponding resource configuration information for each queue in the database system, tasks of different priorities, that is, soft isolation between data query requests of different degrees is achieved. In the process of executing data query requests, the target scheduling time of different queues can be automatically calculated according to the resource configuration information, and a more appropriate queue can be selected to query the data to be executed based on the target scheduling time, thereby avoiding resource scramble and competition problems during the execution of large queries and small queries, achieving the technical effect of improving the efficiency of data processing and solving the technical problem of low data processing efficiency.

[0349] The embodiment of the present application may provide a computer terminal, which may be any computer terminal device in a computer terminal group. Optionally, in this embodiment, the computer terminal may also be replaced by a terminal device such as a mobile terminal.

[0350] Optionally, in this embodiment, the computer terminal may be located in at least one network device among a plurality of network devices of the computer network.

[0351] In this embodiment, the above-mentioned computer terminal can execute the program code of the following steps in the data processing method: monitoring data query requests, wherein the data query requests are used to request query of data to be executed; in response to the data query requests, based on the resource configuration information of the queue, allocating a target scheduling time to the queue, wherein the resource configuration information is used to characterize the processing resources allowed to be scheduled by the queue during operation, and the target scheduling time is used to represent the time allocated to the scheduling queue; determining the time difference information between the target scheduling time of the queue and the scheduled time of the corresponding queue, wherein the time difference information is used to represent the difference between the target scheduling time and the scheduled time; based on the time difference information of the queue, determining the target queue from multiple queues, wherein the time difference information corresponding to the target queue is greater than the difference information threshold; and querying the data to be executed from the target queue.

[0352] Optionally, Fig.16is a structural block diagram of a computer terminal according to an embodiment of the present application. Fig.16 As shown, the computer terminal A may include: one or more (only one is shown in the figure) processors 1602 , a memory 1604 and a transmission device 1606 .

[0353] Among them, the memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the data processing method and device in the embodiment of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, realizing the above-mentioned data processing method. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and the above-mentioned remote memory may be connected to the computer terminal A via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0354] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: in response to a data query request, determine the total scheduling time allowed for the queue based on the resource configuration information of the queue and the scheduled time of the queue; based on the total scheduling time of the queue, allocate a target scheduling time to the queue.

[0355] Optionally, the processor may also execute program code of the following steps: determining a first total scheduling time from the total scheduling times of multiple queues; and allocating a target scheduling time to a corresponding queue based on the first total scheduling time and resource configuration information of the queue.

[0356] Optionally, the processor may further execute program code of the following steps: determining a target scheduling time corresponding to the resource configuration information of the queue from the first total scheduling time.

[0357] Optionally, the processor may also execute program code of the following steps: in response to a data query request, obtaining a quotient between the scheduled time of the queue and the resource ratio of the queue; and determining the quotient as the total scheduling time of the queue.

[0358] Optionally, the processor may also execute program codes of the following steps: determining the priority of the queue; and configuring resource configuration information for the queue according to the priority.

[0359] Optionally, the processor may also execute program codes of the following steps: obtaining a target time ratio of the target scheduling time of the queue to the scheduled time of the queue; and determining the target time ratio as the time difference information of the queue.

[0360] Optionally, the processor may also execute program codes of the following steps: determining maximum time difference information from time difference information of multiple queues; and determining a queue corresponding to the maximum time difference information among the multiple queues as a target queue.

[0361] Optionally, the processor may also execute program code for the following steps: obtaining thread configuration information of a queue; in response to the number of used threads of the queue satisfying the thread configuration information of the queue, determining the second largest time difference information corresponding to the maximum time difference information from the time difference information of multiple queues; and determining the queue corresponding to the second largest time difference information among the multiple queues as the target queue.

[0362] Optionally, the processor may also execute the following program code: in response to the current queue being empty, determining the second largest time difference information corresponding to the maximum time difference information from the time difference information corresponding to multiple queues; and determining the queue corresponding to the second largest time difference information among the multiple queues as the target queue.

[0363] Optionally, the processor may further execute program code of the following steps: in response to the target queue being non-empty, storing the to-be-executed data into the target queue.

[0364] Optionally, the processor may also execute program code for the following steps: determining the total scheduling time allowed for the target queue to be scheduled based on the resource configuration information of the target queue and the scheduled time of the target queue; in response to the target queue being empty, determining a second total scheduling time from the total scheduling times corresponding to multiple queues; determining a target time threshold corresponding to the target queue based on the second total scheduling time and the resource configuration information of the target queue; in response to the scheduled time of the target queue being less than the target time threshold, adjusting the scheduled time of the target queue to the target time threshold, and storing data to be executed in the adjusted target queue; in response to the scheduled time of the target queue being greater than or equal to the target time threshold, maintaining the scheduled time of the target queue, and storing data to be executed in the maintained target queue.

[0365] Optionally, the processor may further execute program code of the following steps: determining a target time threshold corresponding to resource configuration information of the target queue from the second total scheduling time.

[0366] Optionally, the processor may also execute program codes of the following steps: monitoring data query requests from threads; determining subtask data of target tasks in target queues as data to be executed; and returning the data to be executed to threads for execution.

[0367] Optionally, the processor may also execute program code of the following steps: determining the minimum number of threads allowed for the queue based on the number of cores of the processor; determining the minimum number of threads allowed for the queue based on the resource configuration information of the queue; and adding the minimum number of threads to the thread configuration information of the queue.

[0368] Optionally, the processor can call the information and application stored in the memory through the transmission device to perform the following steps: monitor data query requests from the thread; in response to the data query request, assign a target scheduling time to the queue based on the resource configuration information of the queue in the data warehouse; determine the time difference information between the target scheduling time of the queue and the scheduled time of the queue; based on the time difference information of the queue, determine the target queue from multiple queues; query the data to be executed from the target queue; and return the queried data to be executed to the thread for execution.

[0369] Optionally, the processor can call the information and application programs stored in the memory through the transmission device to perform the following steps: monitor data query requests by calling the first interface; in response to the data query request, assign a target scheduling time to the queue based on the resource configuration information of the queue; determine the time difference information between the target scheduling time of the queue and the scheduled time of the queue; based on the time difference information of the queue, determine the target queue from multiple queues; query the data to be executed from the target queue; and output the data to be executed by calling the second interface.

[0370] By adopting the embodiment of the present application, a method for processing data is provided. In the embodiment of the present application, if it is necessary to query the data to be queried by the data query request, the data query request received from the client can be monitored, and the to-be-executed data that needs to be requested to be queried from the server can be analyzed. According to the resource configuration information of each queue, the processing resources that the data query request in the queue is allowed to be scheduled during the operation process can be analyzed, and the target scheduling time when the current data query request is assigned to the scheduling processing resources in the queue for execution can be determined. The time difference information between the target scheduling time and the corresponding queue scheduled time can also be determined. The queue whose time difference information is greater than the difference information threshold is determined as the target queue, so that the current data query request can use the target queue to schedule the corresponding processing resources to query the corresponding to-be-executed data. In the embodiment of the present application, by setting the corresponding resource configuration information for each queue in the database system, tasks of different priorities, that is, soft isolation between data query requests of different degrees is achieved. In the process of executing data query requests, the target scheduling time of different queues can be automatically calculated according to the resource configuration information, and a more appropriate queue can be selected to query the data to be executed based on the target scheduling time, thereby avoiding resource scramble and competition problems during the execution of large queries and small queries, achieving the technical effect of improving the efficiency of data processing and solving the technical problem of low data processing efficiency.

[0371] It can be understood by those skilled in the art that Fig.16 The structure shown is for illustration only, and the computer terminal A may also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile Internet device (MID for short), a PAD, or other terminal devices. Fig.16 It does not limit the structure of the above-mentioned computer terminal A. For example, the computer terminal A may also include Fig.16 More or fewer components (such as network interfaces, display devices, etc.) shown in, or with Fig.16 Different configurations are shown.

[0372] A person of ordinary skill in the art may understand that all or part of the steps in the various methods of the above embodiments may be completed by instructing the hardware related to the terminal device through a program, and the program may be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0373] The embodiment of the present application further provides a computer-readable storage medium. Optionally, in this embodiment, the computer-readable storage medium can be used to store the program code executed by the data processing method provided in the first embodiment.

[0374] Optionally, in this embodiment, the computer-readable storage medium may be located in any one of the computer terminals in a computer terminal group in a computer network, or in any one of the mobile terminals in a mobile terminal group.

[0375] Optionally, in this embodiment, a computer-readable storage medium is configured to store program code for performing the following steps: monitoring data query requests, wherein the data query requests are used to request queries for data to be executed; in response to the data query requests, allocating a target scheduling time to the queue based on the resource configuration information of the queue, wherein the resource configuration information is used to characterize the processing resources that the queue is allowed to be scheduled to during operation; determining time difference information between the target scheduling time of the queue and the scheduled time of the corresponding queue, wherein the time difference information is used to represent the difference between the target scheduling time and the scheduled time; based on the time difference information of the queue, determining a target queue from multiple queues, wherein the time difference information corresponding to the target queue is greater than a difference information threshold; and querying the data to be executed from the target queue.

[0376] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: in response to a data query request, determine the total scheduling time allowed for the queue based on the resource configuration information of the queue and the scheduled time of the queue; based on the total scheduling time of the queue, allocate a target scheduling time to the queue.

[0377] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: determine the first total scheduling time from the total scheduling times corresponding to multiple queues; based on the first total scheduling time and the resource configuration information of the queue, allocate the target scheduling time to the corresponding queue.

[0378] The processor may call the information and application program stored in the memory through the transmission device to perform the following steps: determining a target scheduling time corresponding to the resource configuration information of the queue from the first total scheduling time.

[0379] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: in response to the data query request, obtain the quotient between the scheduled time of the queue and the resource ratio of the queue; determine the quotient as the total scheduling time of the queue.

[0380] The processor can call the information and application programs stored in the memory through the transmission device to perform the following steps: determine the priority of the queue; and configure resource configuration information for the queue according to the priority.

[0381] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: obtain the target time ratio of the queue's target scheduling time to the queue's scheduled time; determine the target time ratio as the time difference information of the queue.

[0382] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: determine the maximum time difference information from the time difference information corresponding to multiple queues; determine the queue corresponding to the maximum time difference information among the multiple queues as the target queue.

[0383] The processor can call the information and application stored in the memory through the transmission device to execute the following steps: obtain the thread configuration information of the queue; in response to the number of used threads of the queue satisfying the thread configuration information of the queue, determine the second largest time difference information corresponding to the maximum time difference information from the time difference information of multiple queues; determine the queue corresponding to the second largest time difference information among the multiple queues as the target queue.

[0384] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: in response to the current queue being empty, determine the second largest time difference information corresponding to the maximum time difference information from the time difference information corresponding to multiple queues; determine the queue corresponding to the second largest time difference information among the multiple queues as the target queue.

[0385] The processor may call the information and application program stored in the memory through the transmission device to execute the following steps: in response to the target queue being non-empty, storing the to-be-executed data in the target queue.

[0386] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: determine the total scheduling time allowed for the target queue to be scheduled based on the resource configuration information of the target queue and the scheduled time of the target queue; in response to the target queue being empty, determine the second total scheduling time from the total scheduling times corresponding to multiple queues; based on the second total scheduling time and the resource configuration information of the target queue, determine the target time threshold corresponding to the target queue; in response to the scheduled time of the target queue being less than the target time threshold, adjust the scheduled time of the target queue to the target time threshold, and store the data to be executed in the adjusted target queue; in response to the scheduled time of the target queue being greater than or equal to the target time threshold, maintain the scheduled time of the target queue, and store the data to be executed in the maintained target queue.

[0387] The processor may call the information and application program stored in the memory through the transmission device to perform the following steps: determining a target time threshold corresponding to the resource configuration information of the target queue from the second total scheduling time.

[0388] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: monitor the data query request from the thread; determine the subtask data of the target task in the target queue as the data to be executed; return the data to be executed to the thread for execution.

[0389] The processor can call the information and application programs stored in the memory through the transmission device to perform the following steps: determine the minimum number of threads allowed to be used by the queue based on the number of cores of the processor; determine the minimum number of threads allowed to be used by the queue based on the resource configuration information of the queue; and add the minimum number of threads to the thread configuration information of the queue.

[0390] Optionally, the computer-readable storage medium may also execute program code for the following steps: monitoring data query requests from threads; in response to data query requests, allocating target scheduling time to queues based on resource configuration information of queues in a data warehouse; determining time difference information between the target scheduling time of a queue and the scheduled time of a corresponding queue; determining a target queue from a plurality of queues based on the time difference information of the queues; querying data to be executed from the target queue; and returning the queried data to be executed to the thread for execution.

[0391] Optionally, the computer-readable storage medium may also execute program codes for the following steps: monitoring data query requests by calling a first interface; allocating a target scheduling time to a queue based on resource configuration information of the queue in response to the data query request; determining time difference information between the target scheduling time of the queue and the scheduled time of the queue; determining a target queue from multiple queues based on the time difference information of the queue; querying data to be executed from the target queue; and outputting data to be executed by calling a second interface.

[0392] An embodiment of the present application may provide an electronic device, which may include a memory and a processor.

[0393] Fig.17It is a block diagram of an electronic device according to a method for processing data in an embodiment of the present application. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or required herein.

[0394] like Fig.17 As shown, the device 1700 includes a computing unit 1701, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1702 or a computer program loaded from a storage unit 1708 into a random access memory (RAM) 1703. In the RAM 1703, various programs and data required for the operation of the device 1700 can also be stored. The computing unit 1701, the ROM 1702, and the RAM 1703 are connected to each other via a bus 1704. An input / output (I / O) interface 1705 is also connected to the bus 1704.

[0395] A number of components in the device 1700 are connected to the I / O interface 1705, including: an input unit 1706, such as a keyboard, a mouse, etc.; an output unit 1704, such as various types of displays, speakers, etc.; a storage unit 1708, such as a disk, an optical disk, etc.; and a communication unit 1709, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1709 allows the device 1700 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0396] The computing unit 1701 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSP), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1701 performs the various methods and processes described above, such as a data processing method. For example, in some embodiments, the data processing method may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 1708. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1700 via the ROM 1702 and / or the communication unit 1709. When the computer program is loaded into the RAM 1703 and executed by the computing unit 1701, one or more steps of the data processing method described above may be performed. Alternatively, in other embodiments, the computing unit 1701 may be configured to execute the data processing method in any other appropriate manner (for example, by means of firmware).

[0397] The embodiment of the present application further provides a computer program product. Optionally, in the present embodiment, the computer program product may include a computer program, and when the computer program is executed by a processor, the data processing method of the embodiment of the present application is implemented.

[0398] According to an embodiment of the present application, a method for processing data is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0399] The method embodiment provided in this embodiment of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Fig.18 is a hardware structure block diagram of a computer terminal (or mobile device) for implementing a data processing method according to an embodiment of the present application, such as Fig.18As shown, the computer terminal 180 (or mobile device) may include one or more (shown in the figure as 1802a, 1802b, ..., 1802n) processors 1802 (the processor 1802 may include but is not limited to a processing device such as a microprocessor (Microcontroller Unit, abbreviated as MCU) or a programmable logic device (Field Programmable Gate Array, abbreviated as FPGA)), a memory 1804 for storing data, and a transmission device 1806 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a Universal Serial Bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. Those skilled in the art will understand that Fig.18 The structure shown is only for illustration and does not limit the structure of the above electronic device. Fig.18 More or fewer components as shown, or with Fig.18 Different configurations are shown.

[0400] Fig.18 The hardware structure block diagram shown can be used not only as an exemplary block diagram of the above-mentioned computer terminal 180 (or mobile device), but also as an exemplary block diagram of the above-mentioned server. In an optional embodiment, Fig.18 The block diagram shows the use of the above Fig.18 The computer terminal 180 (or mobile device) is shown as an embodiment of a computing node in the computing environment 1901 .

[0401] Fig.19 is a structural block diagram of a computing environment of a data processing method according to an embodiment of the present application, such as Fig.19 As shown, computing environment 1901 includes multiple computing nodes (such as servers) running on a distributed network (illustrated by 1910-1, 1910-2, ..., in the figure). The computing nodes all contain local processing and memory resources, and end users 1902 can remotely run applications or store data in computing environment 1901. Applications can be provided as multiple services 1920-1, 1920-2, 1920-3 and 1920-4 in computing environment 1901, representing services "F", "G", "I" and "H" respectively.

[0402] The end user 1902 can provide and access services through a web browser or other software application on the client, and in some embodiments, the end user 1902's provision and / or request can be provided to the entry gateway 1930. The entry gateway 1930 may include a corresponding agent to handle the provision and / or request for the service (one or more services provided in the computing environment 1901).

[0403] Services are provided or deployed based on various virtualization technologies supported by computing environment 1901. In some embodiments, services can be provided based on virtual machine (VM)-based virtualization, container-based virtualization, and / or similar methods. Virtual machine-based virtualization can be to simulate a real computer by initializing a virtual machine, and execute programs and applications without directly contacting any actual hardware resources. While the virtual machine virtualizes the machine, according to container-based virtualization, a container can be started to virtualize the entire operating system so that multiple workloads can run on a single operating system instance.

[0404] In an embodiment based on container virtualization, several containers of a service can be assembled into a Pod (e.g., a Kubernetes Pod). Fig.19 As shown, service 1920-2 can be equipped with one or more Pods 1940-1, 1940-2, ..., 1940-N (collectively referred to as Pods). Pods can include a proxy 1945 and one or more containers 1942-1, 1942-2, ..., 1942-M (collectively referred to as containers). One or more containers in a Pod process requests related to one or more corresponding functions of the service, and the proxy 1945 generally controls network functions related to the service, such as routing, load balancing, etc. Other services can also be equipped with Pods similar to Pods.

[0405] During operation, executing a user request from the end user 1902 may require calling one or more services in the computing environment 1901, and executing one or more functions of a service may require calling one or more functions of another service. Fig.19 As shown, service "F" 1920-1 receives a user request from terminal user 1902 from ingress gateway 1930, service "F" 1920-1 may call service "G" 1920-2, and service "G" 1920-2 may request service "I" 1920-3 to perform one or more functions.

[0406] The computing environment described above can be a cloud computing environment, where the allocation of resources is managed by the cloud service provider, allowing the development of functions without considering the implementation, adjustment or expansion of servers. The computing environment allows developers to execute code in response to events without building or maintaining complex infrastructure. Services can be divided into a set of functions that can be automatically and independently scaled, rather than expanding a single hardware device to handle potential loads.

[0407] Various embodiments of the systems and techniques described above herein may be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard parts (ASSPs), systems-on-a-chip (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. The various embodiments described above may include: being implemented in one or more computer programs, which may be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, which may receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0408] The program code for implementing the method of the present application can be written in any combination of one or more programming languages. The above program code can be provided to a processor or controller of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0409] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. The various embodiments described above may include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0410] The program code for implementing the method of the present application can be written in any combination of one or more programming languages. The above program code can be provided to a processor or controller of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0411] In the context of the present application, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory for short), an optical fiber, a portable compact disk read-only memory (CD-ROM for short), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0412] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a cathode ray tube (CRT) or a liquid crystal display (LCD), a monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a path ball), through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0413] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a Local Area Network (LAN), a Wide Area Network (WAN), and the Internet.

[0414] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0415] It should be noted that the serial numbers of the above-mentioned embodiments of the present application are only for description and do not represent the advantages or disadvantages of the embodiments.

[0416] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0417] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0418] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0419] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0420] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, read-only memory, random access memory, mobile hard disk, disk or optical disk, etc., which can store program code.

[0421] The above are only preferred implementations of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. The above improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A data processing method, characterized in that: include: Monitoring data query requests, wherein the data query request is used to request query of pending data; In response to the data query request, based on the resource configuration information of the queue, a target scheduling time is allocated to the queue, wherein the target scheduling time is used to indicate the time allocated to scheduling the queue, and the resource configuration information is used to indicate the weight of the time consumed by the queue to run the processing resource; Determine time difference information between a target scheduling time of the queue and a scheduled time of the queue, wherein the time difference information is used to represent a difference between the target scheduling time and the scheduled time; Based on the time difference information of the queues, determining a target queue from the multiple queues, wherein the time difference information corresponding to the target queue is greater than a difference information threshold; Query the to-be-executed data from the target queue; Wherein, in response to the data query request, allocating a target scheduling time to the queue based on the resource configuration information of the queue includes: in response to the data query request, determining the target scheduling time based on the resource configuration information and the scheduled time.

2. The method according to claim 1, characterized in that In response to the data query request, allocating a target scheduling time to the queue based on resource configuration information of the queue includes: In response to the data query request, determining a total scheduling time allowed for the queue based on the resource configuration information of the queue and the scheduled time of the queue, wherein the total scheduling time is used to represent the total scheduling time allowed for the queue; The target scheduling time is allocated to the queue based on the total scheduling time of the queue.

3. The method according to claim 2, characterized in that Allocating the target scheduling time to the queue based on the total scheduling time of the queue includes: Determine a first total scheduling time from the total scheduling times of the plurality of queues, wherein the first total scheduling time is greater than a first time threshold; The target scheduling time is allocated to the queue based on the first total scheduling time and the resource configuration information of the queue.

4. The method according to claim 3, characterized in that The resource configuration information of the queue includes a resource ratio of the processing resources that the queue is allowed to schedule during operation to the total processing resources. Based on the first total scheduling time and the resource configuration information of the queue, the target scheduling time is allocated to the queue, including: The target scheduling time corresponding to the resource configuration information of the queue is determined from the first total scheduling time, wherein the time ratio of the target scheduling time of the queue to the first total scheduling time and the resource ratio indicated by the resource configuration information of the queue are the same.

5. The method according to claim 2, characterized in that: The resource configuration information of the queue is the resource ratio of the processing resources allowed to be scheduled by the queue during operation to the total processing resources. In response to the data query request, based on the resource configuration information of the queue and the scheduled time of the queue, determining the total scheduling time allowed to be scheduled by the queue includes: In response to the data query request, obtaining a quotient between the scheduled time of the queue and the resource ratio of the queue; The quotient is determined as the total scheduling time for the queue.

6. The method according to claim 1, characterized in that The method further comprises: determining a priority of the queue; The resource configuration information is configured for the queue according to the priority.

7. The method according to claim 1, characterized in that Determining time difference information between the target scheduling time of the queue and the scheduled time of the queue includes: Obtain a target time ratio of the target scheduling time of the queue to the scheduled time of the queue; The target time ratio is determined as the time difference information of the queue.

8. The method according to claim 1, characterized in that: Determining a target queue from the plurality of queues based on the time difference information of the queues includes: Determining maximum time difference information from the time difference information of the plurality of queues; The queue corresponding to the maximum time difference information among the multiple queues is determined as the target queue.

9. The method according to claim 1, characterized in that: Determining a target queue from the plurality of queues based on the time difference information of the queues includes: Obtaining thread configuration information of the queue, wherein the thread configuration information is used to at least represent a maximum number of threads allowed to be used by the queue; In response to the number of used threads of the queue satisfying the thread configuration information of the queue, determining the second largest time difference information corresponding to the maximum time difference information from the time difference information of the plurality of queues; The queue corresponding to the second largest time difference information among the multiple queues is determined as the target queue.

10. The method according to claim 1, characterized in that Determining a target queue from the plurality of queues based on the time difference information of the queues includes: In response to the current queue being empty, determining the second largest time difference information corresponding to the maximum time difference information from the time difference information of the plurality of queues; The queue corresponding to the second largest time difference information among the multiple queues is determined as the target queue.

11. The method according to claim 1, characterized in that: Before querying the to-be-executed data from the target queue, the method further includes: In response to the target queue being non-empty, storing the to-be-executed data in the target queue.

12. The method according to claim 1, characterized in that Before querying the to-be-executed data from the target queue, the method further includes: Determine the total scheduling time allowed for scheduling of the target queue based on the resource configuration information of the target queue and the scheduled time of the target queue; In response to the target queue being empty, determining a second total scheduling time from total scheduling times of the plurality of queues, wherein the second total scheduling time is less than a second time threshold; Determine a target time threshold corresponding to the target queue based on the second total scheduling time and the resource configuration information of the target queue; In response to the scheduled time of the target queue being less than the target time threshold, adjusting the scheduled time of the target queue to the target time threshold, and storing the to-be-executed data in the adjusted target queue; In response to the scheduled time of the target queue being greater than or equal to the target time threshold, the scheduled time of the target queue is maintained, and the to-be-executed data is stored in the maintained target queue.

13. The method according to claim 12, characterized in that The resource configuration information of the target queue is a resource ratio of processing resources that the target queue is allowed to schedule during operation to total processing resources. Based on the second total scheduling time and the resource configuration information of the target queue, determining a target time threshold of the target queue includes: From the second total scheduling time, determine a target time threshold corresponding to the resource configuration information of the target queue, wherein the target time threshold of the target queue accounts for the same time ratio of the second total scheduling time and the resource ratio indicated by the resource configuration information of the target queue.

14. The method according to any one of claims 1 to 13, characterized in that Monitoring data query request, including: Monitoring the data query request from the thread, wherein the data query request is used to request querying a target task to be executed by the thread; Querying the to-be-executed data from the target queue includes: determining the subtask data of the target task in the target queue as the to-be-executed data; and returning the to-be-executed data to the thread for execution.

15. The method according to any one of claims 1 to 13, characterized in that The method further comprises: Determine the minimum number of threads allowed to be used by the queue based on the number of cores of the processor; Determine the minimum number of threads allowed to be used by the queue based on the resource configuration information of the queue; The minimum number of threads is added to the thread configuration information of the queue.

16. A data processing method, characterized in that: Applied to the execution end, including: Monitoring a data query request from a thread, wherein the data query request is used to request a query of data to be executed; In response to the data query request, based on the resource configuration information of the queue in the data warehouse, a target scheduling time is allocated to the queue, wherein the target scheduling time is used to indicate the time allocated to scheduling the queue, the data warehouse includes a plurality of queues, and the resource configuration information is used to indicate the weight of the time consumed by the queue to be scheduled to run the processing resource; Determine time difference information between a target scheduling time of the queue and a scheduled time of the queue, wherein the time difference information is used to represent a difference between the target scheduling time and the scheduled time; Based on the time difference information of the queues, determining a target queue from the multiple queues, wherein the time difference information corresponding to the target queue is greater than a difference information threshold; Query the to-be-executed data from the target queue; Returning the queried data to be executed to the thread for execution; Wherein, in response to the data query request, based on the resource configuration information of the queue in the data warehouse, a target scheduling time is allocated to the queue, including: in response to the data query request, based on the resource configuration information and the scheduled time, determining the target scheduling time.

17. A data processing method, characterized in that: include: Monitoring a data query request by calling a first interface, wherein the data query request is used to request querying to-be-executed data, the first interface includes a first parameter, and a parameter value of the first parameter is the data query request; In response to the data query request, based on the resource configuration information of the queue, a target scheduling time is allocated to the queue, wherein the target scheduling time is used to indicate the time allocated to scheduling the queue, and the resource configuration information is used to indicate the weight of the time consumed by the queue to run the processing resource; Determine time difference information between a target scheduling time of the queue and a scheduled time of the queue, wherein the time difference information is used to represent a difference between the target scheduling time and the scheduled time; Based on the time difference information of the queues, determining a target queue from the multiple queues, wherein the time difference information corresponding to the target queue is greater than a difference information threshold; Query the to-be-executed data from the target queue; Outputting the data to be executed by calling a second interface, wherein the second interface includes a second parameter, and a parameter value of the second parameter is the data to be executed; Wherein, in response to the data query request, allocating a target scheduling time to the queue based on the resource configuration information of the queue includes: in response to the data query request, determining the target scheduling time based on the resource configuration information and the scheduled time.

18. A data processing system, characterized in that: include: A client is used to send a data query request, wherein the data query request is used to request to query the data to be executed; A server is configured to respond to the data query request and allocate a target scheduling time to the queue based on the resource configuration information of the queue, wherein the resource configuration information is used to represent the weight of the time consumed by the queue to be scheduled to process resource operation, and the target scheduling time is used to represent the time allocated to scheduling the queue, which is determined based on the resource configuration information and the scheduled time; determine the time difference information between the target scheduling time of the queue and the scheduled time of the queue, wherein the time difference information is used to represent the difference between the target scheduling time and the scheduled time; determine a target queue from a plurality of the queues based on the time difference information of the queue, wherein the time difference information corresponding to the target queue is greater than a difference information threshold; query the data to be executed from the target queue; and return the data to be executed to the client.

19. An electronic device, characterized in that: include: A memory storing an executable program; A processor, configured to run the program, wherein the program executes the method according to any one of claims 1 to 17 when running.

20. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored executable program, wherein when the executable program is run, the device where the storage medium is located is controlled to execute the method according to any one of claims 1 to 17.

21. A computer program product, characterized in that It comprises a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 17.

Images