Query engine operation method and device based on virtualization technology

Abstract

The invention provides a query engine running method and device based on the virtualization technology, and the method comprises the steps: executing virtualization operation for a preset query engine to configure the query engine to a container environment, and configuring corresponding containers for a management component and a plurality of execution components in the query engine in the container environment; carrying out resource allocation operation on the query engine so as to run the query engine in the container environment; when a query statement input by a user from a client is received, executing the query statement through a management component and a plurality of execution components in the query engine so as to obtain execution result data for the query statement from a preset data storage system; and returning the execution result data to the client. Thus, the server configures the query engine in the container environment in the container mode to achieve calculation resource isolation, it is guaranteed that the server obtains the needed calculation resources for operation, and the stability of the server for executing data query operation and the efficiency of responding to query statements of the client side are improved.

Description

Technical Field

[0001] The present application belongs to the general data processing technology field of the Internet industry, and specifically relates to a query engine operation method and device based on virtualization technology. Background Art

[0002] At present, with the rapid development of big data technology, big data platform has become an important infrastructure for enterprises to process massive data and mine data value. In the current big data platform architecture, the storage-computing hybrid deployment mode is widely used due to its advantages such as simple deployment and high cost-effectiveness. This mode deploys computing components and storage components on the same physical server, realizing fast data reading, writing and processing, greatly improving data processing efficiency.

[0003] However, the storage-computing hybrid deployment model has gradually exposed its inherent limitations in actual applications. In an environment where multiple tasks are executed concurrently, there are conflicts between the demands for computing resources between different tasks, resulting in reduced performance of query components and longer response times. In addition, the fixed resource allocation threshold of the storage-computing hybrid deployment model is difficult to adapt to the dynamic changes in business needs, resulting in insufficient resource utilization or overload. Summary of the invention

[0004] The embodiment of the present application provides a query engine operation method and device based on virtualization technology. The server uses a container to run the query engine in a container environment to achieve computing resource isolation, so as not to compete for resources with other software running in the container environment, and ensure that it obtains the required computing resources to run. In addition, the server can accurately allocate computing resources to the query engine, limit the lower and upper limits of the computing resources it can use, improve resource utilization, and the efficiency and stability of the server's response to query services.

[0005] In a first aspect, an embodiment of the present application provides a query engine operation method based on virtualization technology, which is applied to a server, and the method includes:

[0006] Performing a virtualization operation on a preset query engine to configure the query engine to a container environment, wherein the query engine is composed of a management component and a plurality of execution components, and the management component and the plurality of execution components are respectively configured with corresponding containers in the container environment;

[0007] Performing a resource allocation operation on the query engine to run the query engine in the container environment, where the resource allocation operation is used to allocate computing resources in the container environment;

[0008] When receiving a query statement input by a user from a client, executing the query statement through a management component and a plurality of execution components in the query engine to obtain execution result data for the query statement from a preset data storage system, where the data storage system is located outside the container environment;

[0009] Returns the execution result data to the client.

[0010] In a second aspect, an embodiment of the present application provides a query engine operation device based on virtualization technology, which is applied to a server, and the device includes:

[0011] A virtualization unit, configured to perform a virtualization operation on a preset query engine to configure the query engine to a container environment, wherein the query engine is composed of a management component and a plurality of execution components, and the management component and the plurality of execution components are respectively configured with corresponding containers in the container environment;

[0012] An operation unit, configured to perform a resource allocation operation on the query engine to run the query engine in the container environment, wherein the resource allocation operation is used to allocate computing resources in the container environment;

[0013] A query execution unit, configured to execute the query statement through a management component and a plurality of execution components in the query engine when receiving a query statement input by a user from a client, so as to obtain execution result data for the query statement from a preset data storage system, where the data storage system is located outside the container environment;

[0014] The data return unit is used to return the execution result data to the client.

[0015] In a third aspect, an embodiment of the present application provides a server, comprising a processor, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the processor, and the program includes instructions for executing the steps in the first aspect of the embodiment of the present application.

[0016] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program / instruction stored thereon, which implements the steps in the first aspect of the embodiment of the present application when the computer program / instruction is executed by a processor.

[0017] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program / instructions, which, when executed by a processor, implement some or all of the steps described in the first aspect of the embodiment of the present application.

[0018] It can be seen that in the embodiment of the present application, the server implements computing resource isolation by running the query engine in a container environment in a container manner, so as to avoid resource competition with other software running in the container environment and ensure that it obtains the required computing resources to run. In addition, the server can accurately allocate computing resources to the query engine, limit the lower and upper limits of the computing resources it can use, improve resource utilization, and the efficiency and stability of the server's response to query services. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1 It is a structural block diagram of a data query system provided by an embodiment of the present application;

[0021] Figure 2 It is a flowchart of a query engine operation method based on virtualization technology provided in an embodiment of the present application;

[0022] Figure 3 It is a structural diagram of a server running a query engine provided by an embodiment of the present application;

[0023] Figure 4 This is a schematic diagram of a scenario of interaction between a client and a query engine provided by an embodiment of the present application;

[0024] Figure 5a It is a scenario diagram of a task dispatching strategy corresponding to a first execution time provided in an embodiment of the present application;

[0025] Figure 5b It is a scenario diagram of a task dispatching strategy corresponding to a second execution time provided in an embodiment of the present application;

[0026] Figure 6 It is a block diagram of the functional units of a query engine running device based on virtualization technology provided by an embodiment of the present application;

[0027] Figure 7 It is a block diagram of the functional units of another query engine running device based on virtualization technology provided by an embodiment of the present application;

[0028] Figure 8 It is a structural block diagram of a server provided in an embodiment of the present application. DETAILED DESCRIPTION

[0029] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0030] The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.

[0031] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0032] See also Figure 1 , Figure 1 is a structural block diagram of a data query system provided by an embodiment of the present application. Figure 1 As shown, the data query system 100 includes a server 110 and a client 120, the server 110 is connected to the client 120 in communication, the server 110 can be a single server, or a server cluster consisting of several servers, or a cloud computing service center, and the client 120 can be an interface provided to the user on any terminal device available on the market that can communicate with the server 110. The user actively initiates requests using the client 120, and the server 110 responds to these requests and provides corresponding services. A data query system 100 can correspond to multiple servers 110 and multiple clients 120 at the same time, or the data query system 100 includes a server 110, and each server 110 corresponds to one or more clients 120.

[0033] Based on this, an embodiment of the present application provides a query engine operation method based on virtualization technology, and the embodiment of the present application is described in detail below in conjunction with the accompanying drawings.

[0034] See also Figure 2 , Figure 2 This is a flowchart of a query engine operation method based on virtualization technology provided by an embodiment of the present application, which is applied to Figure 1 The server 110 in the data query system 100 shown is connected in communication with the client 120, and the method includes:

[0035] Step S201 , performing a virtualization operation on a preset query engine to configure the query engine into a container environment.

[0036] Among them, the query engine is composed of a management component and multiple execution components, and the management component and the multiple execution components are respectively configured with corresponding containers in the container environment. The query engine usually contains multiple components to support the entire process from receiving query requests to returning query results. This application is based on the division of functions and roles, and divides the query engine into management components and execution components. Among them, the management component is mainly responsible for the configuration, monitoring, scheduling, optimization and other management tasks of the execution component. The execution component is the part of the query engine that is responsible for actually executing query tasks.

[0037] Specifically, the management component can include the following functions: 1. Responsible for the management of the configuration information of the execution component, such as database connection information, query optimization parameters, etc.; 2. Monitor the running status of the execution component, record the running log, and help the administrator diagnose problems and optimize performance; 3. Responsible for assigning query tasks to different execution components to optimize resource utilization and query performance; 4. Parse the query statements entered by the user (such as standard programming language (Structured Query Language, SQL) statements, allowing users to define, query, update and manage data in relational databases.) into a structure that the query engine can understand, such as an abstract syntax tree (AST); 5. The generated query structure is combined with statistical information and cost models to optimize the query and generate an efficient execution plan. The execution component can specifically include the following functions: According to the execution plan generated by the management component, call the corresponding storage engine or data access interface, execute the actual query operation, and return the query result.

[0038] Among them, virtualization operation refers to the redivision and dynamic management of computing resources, storage resources, network resources and other resources within the server through virtualization technology. Specifically, virtualization operation can divide the resources on a physical computer (or host) into multiple container environments (or virtual machines, virtual environments), each of which can run its own operating system and application programs, and these container environments are isolated from each other and do not affect each other.

[0039] Specifically, this step achieves the following purposes by configuring the query engine in a container environment: 1. Resource isolation: Through virtualization technology, different applications or services can be deployed in different container environments to achieve resource isolation (such as CPU, (central processing unit) memory, disk, etc.) to avoid conflicts and interference between applications. 2. Improve resource utilization: flexibly allocate and utilize the computing resources of the server, reduce hardware waste, and improve overall resource utilization. Among them, the query engine contains a management component and multiple execution components, and the management component and each execution component are containerized. Containerization is a lightweight virtualization technology that allows applications and their dependencies to be packaged into an independent container and run applications in the container. The container is more lightweight, starts faster, and has higher resource utilization. By containerizing the query engine, it can be deployed on any platform that supports container operation to achieve rapid deployment and expansion. In addition, containerization technology provides good isolation, which can ensure that each component runs independently in the container without interference from other applications or services. This isolation helps to ensure the stability and security of each component's operation.

[0040] Step S202: performing resource allocation operations on the query engine to run the query engine in the container environment.

[0041] Among them, the resource allocation operation is used to allocate computing resources in the container environment. Specifically, in the container environment, the computing resources used to perform the resource allocation operation include at least one of the following: CPU resources, memory resources, storage resources and network resources. Different container technologies (such as Docker, Kubernetes, etc.) may have different configuration and management methods for computing resources. In practical applications, it is necessary to reasonably configure and manage computing resources in the container environment according to specific container technologies and business requirements. Among them, the CPU resources specifically include: 1. Number of cores: The number of CPU cores that the container can be configured to use, which determines the ability of the container to process tasks in parallel; 2. CPU shares (CPU Shares): When multiple containers compete for CPU resources, the CPU shares determine the priority and proportion of the container to obtain CPU resources relative to other containers; 3. CPU limits (CPULimits): The upper limit of the CPU resources that can be used by the container can be set to prevent a container from excessively occupying CPU resources and affecting the operation of other containers. The memory resources specifically include: 1. Memory size: The amount of memory that the container can be configured to use; 2. Memory limits (Memory Limits): The upper limit of the memory resources that the container can use can be set. When the container attempts to use memory exceeding this limit, the system may recycle memory or terminate the container.

[0042] In a possible embodiment, a resource allocation operation is performed on a query engine to run the query engine in a container environment, including: running an initialization startup program outside the container environment to read configuration file information of the query engine; performing a resource application operation on computing resources according to first resource specification information and second resource specification information; after the computing resource allocation is completed, obtaining a first container image and a second container image; sequentially starting a first container according to the first container image, and starting a second container corresponding to the number of components according to the second container image; after the first container and the second container are started, determining that the query engine runs successfully, and maintaining inter-component communication between the started first container and the second container.

[0043] Among them, the configuration file information is used to indicate the first resource specification information, the second resource specification information and the number of components of the execution component to be started, the first resource specification information is used to indicate the computing resource specification required by the management component, the second resource specification information is used to indicate the computing resource specification required by the execution component, the first container image includes the process information required to run when the first container configured by the management component is started, and the second container image includes the process information required to run when the second container configured by the execution component is started. Container image: A container image contains a packaged software, its dependencies, and the process information that it runs when it is started. (A container image is a static representation of a container and is a standard format for packaging container applications. It contains all the files required for the container to run, such as program code, libraries, resources, configurations, etc., as well as some configuration parameters prepared for the runtime (such as environment variables, users, etc.). The image can be regarded as a special file system, which adopts a hierarchical structure to make the construction and distribution of the image efficient and lightweight.) In addition, the content in the first resource specification information and the second resource specification information is also specifically used to indicate the maximum value (i.e., the upper limit of the specification) and the minimum value (i.e., the lower limit of the specification) of the computing resource specifications that can be used by the management component and the corresponding execution component.

[0044] For example, please refer to Figure 3 , Figure 3 Schematic diagram of a server running a query engine provided by an embodiment of the present application. Figure 3As shown, the server reads the configuration file information in the configuration file required to run the query engine by running the initialization startup program outside the container environment. The configuration file information includes the following content: "Management component: first resource specification information (minimum value, maximum value)", "Execution component: second resource specification information (minimum value, maximum value), number of components (i.e., the number of components of the execution component to be started) = 3". After that, the server performs the resource application operation in the container environment based on the first resource specification information and the second resource specification information in the configuration file information to start the first container of the management component and the second containers of the three execution components in sequence. The process of starting the container is to obtain the first container image and the second container image based on the configured computing resources, and start the corresponding first container or second container according to the container image. In addition, the first container of the management component in the query engine communicates with the second containers of the three execution components respectively. The inter-component communication between the first container and the second container is to transmit data. Specifically, the management component monitors the running status of the three execution components, and assigns the parallel tasks to be executed to the execution components. After the execution components complete the corresponding tasks to be executed, the execution components will also transmit the execution result data to the management component to be summarized and returned to the client.

[0045] It can be seen that in this example, the server reads the configuration file information by running the initialization startup program, and performs effective configuration of computing resources for each component in the query engine in the container environment based on the configuration file information, thereby achieving resource isolation and ensuring the use of computing resources, and improving the stability of the query engine operation. While ensuring the effective operation of each component, it can also prevent the query engine from occupying too much computing resources of the server, improving the stability of the server and the compatibility of other application operations.

[0046] In one possible embodiment, a resource application operation for computing resources is performed according to the first resource specification information and the second resource specification information, including: obtaining available resource information; determining the current available resource amount of the container environment according to the available resource information, and determining the current resource application requirement indicated by the resource application operation according to the first resource specification information and the second resource specification information; judging whether the number of processing cores and the remaining memory capacity meet the current resource application requirement: if the number of processing cores and the remaining memory capacity meet the resource application requirement, allocating the current available resource amount to the first container according to the first resource specification information, and allocating the current available resource amount to the second container corresponding to the number of components according to the second resource specification information; and, updating the current available resource amount according to the first resource specification information and the second resource specification information; if the number of processing cores and the remaining memory capacity do not meet the resource application requirement, determining that the query engine has failed to run; and, according to a preset period, re-executing the above resource application operation.

[0047] The available resource information is used to indicate the current available resource amount in the computing resources in the server at the current system time, and the current available resource amount includes the number of processing cores of the central processing unit and the remaining memory capacity.

[0048] Among them, the process of allocating the current available resource amount of the server according to the first resource specification information and the second resource specification information can be specifically: first determine whether the current available resource amount is satisfied according to the large value within the computing resource demand range indicated by the first resource specification information and the second resource specification information, if not satisfied, then gradually adjust the value within the computing resource demand range to determine whether the current available resource amount is satisfied, until it is adjusted to the minimum value within the computing resource demand range. If it is still not satisfied, it is determined that the query engine fails to run. In addition, the first container and the second container can also be directly started according to the minimum value within the computing resource demand range, and in the process of running the query engine after the first container and the second container are started, the dynamic configuration adjustment is started to increase the resource specification guarantee corresponding to the container according to the updated available resource amount. The dynamic configuration adjustment is performed based on the actual resource demand required for the corresponding component to perform the task and the maximum value of the computing resource demand indicated by the resource specification information, that is, if the actual resource demand is greater than the computing resource specification configured at startup, the available resource specification of the corresponding container is gradually increased according to the maximum value of the computing resource demand to ensure the stability of the container in performing the corresponding task.

[0049] It can be seen that in this example, the server ensures that the server can effectively run the query engine, improve the stability of the query engine operation, and ensure the lower limit of computing resources and the flexibility of resource configuration by setting the query engine configuration file information in advance.

[0050] Step S203, when a query statement input by a user from a client is received, the query statement is executed by a management component and a plurality of execution components in the query engine to obtain execution result data for the query statement from a preset data storage system.

[0051] Among them, the data storage system is located outside the container environment.

[0052] In a possible embodiment, executing a query statement through a management component and multiple execution components in a query engine to obtain execution result data for the query statement from a preset data storage system includes: based on the inter-component communication between the first container and the second container, controlling the execution component to execute the query statement on the data storage system through the management component; and when the execution component of the second container executes the query statement, obtaining the first resource consumption information corresponding to the first container and the second resource consumption information of the second container in real time; if it is determined that the first container has resource overlimit according to the first resource consumption information and the first resource specification information, and / or, it is determined that any second container has resource overlimit according to the second resource consumption information and the second resource specification information, then executing a resource adjustment strategy to maintain the continuous operation of the first container and the multiple second containers according to the first container and / or the second container with resource overlimit and the updated current available resource amount, until it is determined that the query statement is executed and the execution result data is obtained; if it is determined that the first container does not have resource overlimit according to the first resource consumption information and the first resource specification information, and, it is determined that the second container does not have resource overlimit according to the second resource consumption information and the second resource specification information, then maintaining the continuous operation of the first container and the second container until it is determined that the query statement is executed and the execution result data is obtained.

[0053] Among them, when the execution component executes the query task assigned by the management component to execute the query statement, the server monitors the resource consumption information of the management component and the execution component during operation in real time, that is, the first resource information and the second resource information. The server will determine whether there is a resource overrun situation based on these resource consumption information and the resource specification information of each container (that is, the maximum amount of resources allocated or allowed to be used by each container). If it is found that the first container or any second container has a resource overrun, the server will execute the resource adjustment strategy to maintain the continuous operation of all containers. This process will continue until the query statement is executed and the execution result data is obtained. If the resource consumption of all containers is within the range allowed by their specification information, that is, there is no resource overrun situation, the server will maintain the continuous operation of these containers until the query statement is executed and the execution result data is obtained. Through the above steps, the server realizes real-time monitoring and adjustment of resource usage to ensure that the query engine can continue to run under high load, so as to successfully complete the query task and obtain the result data.

[0054] Furthermore, when the server determines whether there is a resource overlimit situation in the first container or the second container, the resource amount threshold corresponding to the resource overlimit is determined based on the resource specification information of the corresponding container. The resource amount threshold can be set lower than the maximum amount of resources allocated or allowed to be used by each container, so that the server has time to dynamically increase or decrease resources according to the load situation of the container, so as to ensure the smoothness and stability of the query engine in executing data query operations.

[0055] It can be seen that in this example, the server ensures that the query engine can continue to run under high load by real-time monitoring and adjusting resource usage, which provides stability for the server to run the query engine and improves the scalability of the server. Through reasonable resource allocation and load balancing, it can support higher concurrent query requests, meet the needs of large-scale application scenarios, and ensure that resources are fully utilized to avoid resource waste.

[0056] In a possible embodiment, based on the inter-component communication between the first container and the second container, controlling the execution component to execute a query statement on the data storage system through the management component includes: receiving and parsing the query statement based on the management component in the first container to determine the parsing result; and determining multiple tasks to be executed in parallel according to the type of operation to be executed and the database object involved in the operation type, and obtaining the running status of the execution component in the second container in real time based on the inter-component communication between the first container and the second container; dispatching multiple tasks to be executed to the execution component according to the running status, so as to execute the corresponding tasks to be executed dispatched by the management component on the data storage system through the execution component in the second container; receiving the reference result data returned by the execution component in sequence through the management component, and aggregating the multiple reference result data into execution result data.

[0057] The parsing result indicates the type of operation to be performed and the database object of the data storage system involved in the operation type, and each second container executes a task to be performed. The task to be performed includes any of the following task types: obtaining data from an external storage system, changing (including deleting and modifying) data in an external storage system.

[0058] For example, please refer to Figure 4 , Figure 4 This is a schematic diagram of a scenario of interaction between a client and a query engine provided in an embodiment of the present application. In the first container, the management component is responsible for receiving query statements from the client. Afterwards, the management component further parses the query statement to determine its specific operation content and objectives, that is, the parsing result. This step is the key to understanding and decomposing the query statement, laying the foundation for subsequent task allocation and execution. Based on the parsing results, the management component will identify the types of operations to be performed (such as query, update, delete, etc.) and the database objects involved in these operations (such as tables, indexes, etc.). Based on this information, the management component decomposes complex query statements into multiple tasks to be executed (i.e., Figure 4This is a key step to improve query execution efficiency, because parallel processing can significantly reduce the overall execution time. In addition, the management component also obtains the running status of the execution component in the second container in real time through the inter-component communication mechanism (such as message queues, etc.). This is to ensure that the execution component is in an available state when assigning tasks, so as to avoid tasks being sent to overloaded or unavailable components. The execution component in the second container receives the task (i.e. Figure 4 After the task 1 to be executed, the task 2 to be executed, and the task n to be executed are received, the corresponding operation will be performed on the data storage system, and reference result data will be generated (the corresponding reference result data will be returned according to the corresponding execution task, such as reference result data 1, reference result data 2, and reference result data n). The management component is responsible for receiving these results in turn and aggregating them into the final execution result data. This process ensures the consistency and integrity of the query statement execution.

[0059] It can be seen that in this example, the server improves query efficiency by containerizing multiple components of the query engine and executing query statements based on mutual communication. In addition, by decomposing complex queries into multiple subtasks, the server can handle more complex query logic and meet diverse business needs. The componentized design makes the system easy to expand, and more execution components can be added or the capacity of the data storage system can be expanded according to business needs.

[0060] In a possible embodiment, a plurality of tasks to be executed are dispatched to an execution component according to a running state, so as to realize, through the execution component in the second container, the corresponding tasks to be executed dispatched by the management component for the data storage system, including: obtaining a historical task information set; matching the reference execution time of each task to be executed and the number of tasks to be executed according to the historical task information set; determining the number of dispatchable containers as the number of dispatchable containers; detecting that the number of tasks is greater than the number of dispatchable containers, determining the number of vacant containers according to the number of tasks and the number of dispatchable containers, and determining a first execution time according to the number of dispatchable containers and the reference execution time of each task to be executed; determining a second execution time according to the number of vacant containers, the running state corresponding to the non-dispatched container and the reference execution time of each task to be executed; and implementing a task dispatching strategy corresponding to the first execution time or the second execution time with a smaller execution time, so as to realize the execution component to execute the corresponding task to be executed.

[0061] Among them, the historical task information set is used to store the task attributes and reference execution time of multiple historical execution tasks executed by the execution component in the query engine, and the task attributes are used to indicate the operation type and database object corresponding to the corresponding historical execution task; the dispatchable container is a second container whose running state is in a normal state, and the number of running states of the execution components also includes an abnormal state and a not ready state. The abnormal state is used to characterize that the process of the corresponding second container crashes or the process is unresponsive, and the not ready state is used to characterize that the corresponding second container is not in a state where it can execute and process the tasks to be executed. The first execution time is the time required for the dispatchable container to complete all the tasks to be executed, and the non-dispatcher container is a second container whose running state is in an abnormal state and a not ready state. The restart times corresponding to different running states are different. The second execution time is the time required for the updated dispatchable container to complete all the tasks to be executed after the non-dispatcher container is restarted.

[0062] Among them, the management component obtains a set of historical task information to match the reference execution time and the total number of tasks corresponding to the determined multiple tasks to be executed. Through communication between components, the management component can quickly and accurately determine the operating status of each execution component, thereby determining the dispatchable containers of the dispatchable tasks. When the number of dispatchable containers is greater than or equal to the number of tasks, the management component only needs to determine the tasks to be executed assigned to it based on the second resource specification information corresponding to each second container, so as to ensure that the corresponding second container can effectively execute the corresponding tasks to be executed. When the number of dispatchable containers is less than the number of tasks, the allocation of tasks to be executed is involved. The server can specifically allow the execution component to complete the corresponding tasks to be executed in two ways. One is to let the current dispatchable container complete each task to be executed in turn, but this will involve a single dispatchable container needing to complete multiple tasks to be executed, thereby reducing the efficiency of the final query engine returning the client execution result data, affecting the user experience. The time required for this method is the first execution time; the second is to calculate by determining the number of vacant containers and the running status corresponding to the non-dispatched containers, restart the non-dispatched containers with the corresponding number of vacant containers to make up for the vacancies in the number of containers, and then complete each task to be executed. The time required for this method is the second execution time. However, the restart time of the non-dispatched container in this method will change according to its current running time, and it cannot be ensured that the second execution time is shorter than the first execution time. Therefore, in this embodiment, the first execution time and the second execution time are calculated in advance, and then the task dispatch strategy corresponding to the shorter execution time is determined to execute the task dispatch for the second container.

[0063] For example, please refer to Figure 5a , Figure 5a Schematic diagram of a scenario of a task dispatching strategy corresponding to a first execution time provided by an embodiment of the present application. Figure 5a As shown, execution component 1, execution component 2 and execution component n all send their running status to the first container of the management component through the corresponding second container, wherein the running status of execution component 1 is normal, the running status of execution component 2 is abnormal, and the status of execution component n is not ready. The management component can determine the number of dispatchable containers as 1 and the number of non-dispatched containers as 2 by obtaining their running status. In addition, by obtaining the historical task information set and the previously determined pending tasks 1, 2 and 3, the management component can determine the first execution time required for the dispatchable container (i.e., the second container corresponding to execution component 1) to execute pending tasks 1-3 alone. And the management component determines the second execution time by determining the restart time of the non-dispatched container (i.e., the second container corresponding to execution component 2 and execution component n), and combining the execution time of execution component 1, execution component 2 and execution component n to complete pending tasks 1-3 respectively. If the management component determines that the first execution time is less than the second execution time, then execute as follows Figure 5a The task dispatching strategy shown is to dispatch all pending tasks to execution component 1 (i.e., dispatch pending tasks 1-3). In addition, the management component can also release the computing resources occupied by them by executing container deletion operations (i.e., deleting the second container corresponding to execution component 2 and deleting the second container corresponding to execution component n), thereby increasing the upper limit of the computing resource specification of the second container of execution component 1 and ensuring the smooth execution of pending tasks.

[0064] For further information, please refer to Figure 5b , Figure 5b Schematic diagram of a scenario of a task dispatching strategy corresponding to a second execution time provided in an embodiment of the present application. Figure 5b As shown, if the management component determines that the first execution time is greater than the second execution time, the following is executed: Figure 5b The task dispatching strategies shown are "restart the second container corresponding to execution component 2; dispatch task 2 to be executed", and "restart the second container corresponding to execution component n; dispatch task 3 to be executed", and directly dispatch task 1 to be executed to execution component 1 in a normal state. In addition, dispatching task 1 to be executed to execution component 1, dispatching task 2 to execution component 2, and dispatching task 3 to execution component n is a possible task dispatching selection, and which task to be executed is specifically dispatched to the corresponding execution component is determined based on the second resource specification information of the second container corresponding to each execution component and the reference execution time and estimated resource requirements of the task to be executed, so as to avoid resource overruns in a single container during the execution of the task to be executed, and to avoid too long a time to complete all the tasks to be executed.

[0065] It can be seen that in this example, the server improves the flexibility and efficiency of the server's data query operations when executing the query engine, and improves the stability of the query engine's operation by pre-calculating different execution times to select a shorter execution task dispatching strategy.

[0066] In a possible embodiment, according to the first container and / or the second container with excessive resources and the updated current available resources, a resource adjustment strategy is executed to maintain the continuous operation of the first container and the plurality of second containers until it is determined that the query statement is executed completely to obtain the execution result data, including: determining the first container and / or the second container with excessive resources as the target container, and determining the non-dispatching container and the dispatchable container to which the task to be executed is not dispatched as the reference container; determining the amount of excessive resources additionally required for the target container, and determining the amount of releasable resources according to the second resource specification information corresponding to the reference container; if the amount of releasable resources is greater than or equal to the amount of excessive resources, deleting the reference container to release the amount of releasable resources to maintain the continuous operation of the target container; if the amount of releasable resources is less than the amount of excessive resources, updating the first resource specification information or the second resource specification information corresponding to the target container according to the updated current available resources, and backing up the current container data of the target container; and restarting the target container according to the current container data and the updated first resource specification information or the second resource specification information.

[0067] Among them, in this embodiment, by determining the target container that needs to increase resource configuration and the reference container that can be adjusted, and then calculating the specific resource amount, it is determined whether to delete the reference container to meet the increase of resource configuration of the target container. Compared with updating the first resource specification information or the second resource specification information corresponding to the target container according to the current available resource amount and restarting the target container, this step is executed faster, can effectively improve the efficiency of the query engine in processing query statements, and improve the flexibility and response speed of the system.

[0068] Furthermore, if an event occurs in which the server deletes a reference container to maintain the continuous operation of a target container, the server will record the reference container and its corresponding first resource specification information and second resource specification information, and after the query statement is executed, restart the reference container according to the resource specification information, so that the reference container can be reconfigured to be ready to execute the next query statement received by the management component, thereby ensuring the response speed of the query engine.

[0069] It can be seen that in this example, after the basic resources for running the query engine are allocated, the server implements more fine-grained resource management and adjustment by going deep into the actual resource usage of a single container, and dynamically adjusts resources according to the requirements of specific tasks and the actual load of the container, thereby improving the flexibility and response speed of the system, and reducing the operation and maintenance costs of the server through the intelligent resource adjustment process.

[0070] In a possible embodiment, the method also includes: obtaining execution result data obtained by running the query engine within a first preset time period; generating a unique key based on the query statement, query parameters, and database table structure corresponding to the execution result data according to the execution result data within the preset time period; setting up a data cache system outside the container environment according to the memory cache or distributed cache setting, and storing the correspondence between the unique key and the corresponding execution result data, the unique key and the execution result data in the set data cache system, and the data cache system is also configured with a preset cache elimination strategy and a cache validity period, and the cache validity period is determined by the data update frequency and business needs in the data storage system.

[0071] Among them, the cache elimination strategy refers to a mechanism for selecting which cache data to be removed or replaced according to a certain strategy when the cache space is insufficient. The cache elimination strategy includes any of the following: first-in-first-out strategy, least recently used strategy, and least used strategy. Among them, the first-in-first-out strategy means that the data that enters the cache earliest is eliminated first. By maintaining a queue (or linked list), new data is added to the tail of the queue. When the cache capacity is insufficient, the data at the head of the queue is eliminated; the least recently used strategy is to regularly detect the least used data during this period according to the second preset time period and eliminate it first to retain the most commonly used data. In addition, combined with the setting of the cache validity period, data that exceeds the cache validity period is regularly eliminated to ensure the effectiveness of the data; the least used strategy means that the data with the lowest frequency of use is eliminated first to retain the execution result data with a higher access frequency. Each time the execution result data is accessed, its access count is increased (or the access frequency is updated). When the cache capacity of the data cache system is insufficient, the execution result data with the lowest access frequency is eliminated.

[0072] Among them, the specific application process of this embodiment is: before the query is executed, the server first checks whether there is corresponding execution result data in the data cache system. If it exists, the result is directly read from the data cache system and returned to the client. If it does not exist, the query engine is allowed to execute the query statement, and then the execution result data obtained this time is stored in the data cache system and returned to the client. In addition, the server can also monitor the cache hit rate, cache size and cache elimination status. According to the monitoring results, the cache elimination strategy, cache size and cache validity period are adjusted to regularly clean up expired cache items, prevent cache expansion, and ensure the consistency of cache data with database data. And, when the database table structure changes or the data is updated, the relevant cache items are cleaned up or updated in time.

[0073] It can be seen that in this example, the server sets up a data cache system and configures different strategies for different scenarios, thereby reducing the access pressure on the server based on the historical execution result data obtained by running the query engine, thereby improving the performance and response speed of the system.

[0074] Step S204: Return the execution result data to the client.

[0075] The execution result data varies according to the query statement. If the user changes the data in the data storage system through the query statement, the execution result data is used to indicate whether the query statement is executed successfully. If the user queries the data in the data storage system through the query statement, the execution result data is used to indicate the query data obtained by the query engine when executing the query statement.

[0076] visible, Figure 2 This is a flow chart of a query engine operation method based on virtualization technology provided by an embodiment of the present application. In this embodiment of the present application, the server runs the query engine in a container environment in a container manner to achieve computing resource isolation, so as not to compete for resources with other software running in the container environment, thereby ensuring that it obtains the required computing resources to run. In addition, the server can accurately allocate computing resources to the query engine, limit the lower and upper limits of the computing resources it can use, improve resource utilization, and the efficiency and stability of the server's response to query services.

[0077] The following is an embodiment of the device of the present application. The embodiment of the device of the present application and the embodiment of the method of the present application are of the same concept and are used to execute the method described in the embodiment of the present application. For the convenience of explanation, the embodiment of the device of the present application only shows the part related to the embodiment of the device of the present application. For the specific technical details not disclosed, please refer to the description of the embodiment of the method of the present application, which will not be repeated here one by one.

[0078] The embodiment of the present application provides a query engine running device based on virtualization technology, and the query engine running device based on virtualization technology is applied to Figure 1 The server 110 in the data query system 100 shown is in communication connection with the client 120. Specifically, the query engine operation device based on virtualization technology is used to execute the steps executed by the server 110 in the query engine operation method based on virtualization technology above. The query engine operation device based on virtualization technology provided in the embodiment of the present application may include modules corresponding to the corresponding steps.

[0079] The embodiment of the present application can divide the functional modules of the query engine operation device based on virtualization technology according to the above method example. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing module. The above integrated module can be implemented in the form of hardware or in the form of software functional modules. The division of modules in the embodiment of the present application is schematic and is only a logical functional division. There may be other division methods in actual implementation.

[0080] In the case of dividing each functional module into corresponding functional modules, Figure 6 It is a functional unit composition block diagram of a query engine running device based on virtualization technology provided in an embodiment of the present application; the query engine running device 60 based on virtualization technology includes: a virtualization unit 601, which is used to perform a virtualization operation on a preset query engine to configure the query engine to a container environment, and the query engine is composed of a management component and a plurality of execution components, and the management component and the plurality of execution components are respectively configured with corresponding containers in the container environment; an operation unit 602, which is used to perform a resource allocation operation on the query engine to run the query engine in the container environment, and the resource allocation operation is used to allocate computing resources in the container environment; a query execution unit 603, which is used to execute the query statement through the management component and the plurality of execution components in the query engine when receiving a query statement input by a user from a client, so as to obtain execution result data for the query statement from a preset data storage system, and the data storage system is located outside the container environment; a data return unit 604, which is used to return the execution result data to the client.

[0081] In a possible embodiment, in terms of performing resource allocation operations on a query engine to run the query engine in a container environment, the running unit 602 is specifically used to: run an initialization startup program outside the container environment to read configuration file information of the query engine, the configuration file information is used to indicate first resource specification information, second resource specification information, and the number of components of the execution component to be started, the first resource specification information is used to indicate the computing resource specification required by the management component, and the second resource specification information is used to indicate the computing resource specification required by the execution component; perform a resource application operation for the computing resource according to the first resource specification information and the second resource specification information; after the computing resource allocation is completed, obtain a first container image and a second container image, the first container image includes process information required to run when the first container configured by the management component is started, and the second container image includes process information required to run when the second container configured by the execution component is started; start the first container according to the first container image, and start the second container corresponding to the number of components according to the second container image in sequence; after the first container and the second container are started, determine that the query engine runs successfully, and maintain inter-component communication between the started first container and the second container.

[0082] In one possible embodiment, in terms of executing a resource application operation for computing resources according to the first resource specification information and the second resource specification information, the operation unit 602 is specifically used to: obtain available resource information, the available resource information is used to indicate the current available resource amount in the computing resources in the server under the current system time; determine the current available resource amount of the container environment according to the available resource information, and determine the current resource application requirement indicated by the resource application operation according to the first resource specification information and the second resource specification information, the current available resource amount includes the number of processing cores and the remaining memory capacity of the central processing unit; determine whether the number of processing cores and the remaining memory capacity meet the current resource application requirement: if the number of processing cores and the remaining memory capacity meet the resource application requirement, allocate the current available resource amount to the first container according to the first resource specification information, and allocate the current available resource amount to the second container corresponding to the number of components according to the second resource specification information; and, update the current available resource amount according to the first resource specification information and the second resource specification information; if the number of processing cores and the remaining memory capacity do not meet the resource application requirement, determine that the query engine operation has failed; and, according to a preset period, re-execute the above-mentioned resource application operation.

[0083] In one possible embodiment, in terms of executing a query statement through a management component and multiple execution components in a query engine to obtain execution result data for the query statement from a preset data storage system, the query execution unit 603 is specifically used to: based on the inter-component communication between the first container and the second container, control the execution component through the management component to execute the query statement on the data storage system; and when the execution component of the second container executes the query statement, obtain the first resource consumption information corresponding to the first container and the second resource consumption information of the second container in real time; if it is determined that the first container has resource overlimit according to the first resource consumption information and the first resource specification information, and / or, according to If the second resource consumption information and the second resource specification information determine that any second container has resource overlimit, then, based on the first container and / or the second container with resource overlimit and the updated current available resource amount, a resource adjustment strategy is executed to maintain the continuous operation of the first container and multiple second containers until it is determined that the query statement is executed completely to obtain the execution result data; if it is determined that the first container does not have resource overlimit according to the first resource consumption information and the first resource specification information, and it is determined that the second container does not have resource overlimit according to the second resource consumption information and the second resource specification information, then the continuous operation of the first container and the second container is maintained until it is determined that the query statement is executed completely to obtain the execution result data.

[0084] In a possible embodiment, in terms of controlling the execution component to execute a query statement on the data storage system through the management component based on the inter-component communication between the first container and the second container, the query execution unit 603 is specifically used to: based on the management component in the first container, receive and parse the query statement to determine the parsing result, the parsing result is used to indicate the type of operation to be performed and the database object of the data storage system involved in the operation type; and, determine multiple tasks to be executed in parallel according to the type of operation to be performed and the database object involved in the operation type, and obtain the running status of the execution component in the second container in real time based on the inter-component communication between the first container and the second container; dispatch multiple tasks to be executed to the execution component according to the running status, so as to execute the corresponding tasks to be executed dispatched by the management component for the data storage system through the execution component in the second container, and each second container executes one task to be executed; receive the reference result data returned by the execution component in sequence through the management component, and summarize the multiple reference result data into execution result data.

[0085] In one possible embodiment, in terms of dispatching multiple tasks to be executed to the execution component according to the running state, so as to realize the corresponding tasks to be executed dispatched by the management component of the data storage system through the execution component in the second container, the query execution unit 603 is specifically used to: obtain a historical task information set, the historical task information set is used to store the task attributes and reference execution time of multiple historical execution tasks executed by the execution component in the query engine, and the task attributes are used to indicate the operation type and database object corresponding to the corresponding historical execution task; according to the historical task information set, match the reference execution time of each task to be executed and the number of tasks to be executed; determine the number of dispatchable containers as the number of dispatchable containers, the dispatchable container is the second container whose running state is in a normal state, and the number of execution components. The running state also includes an abnormal state and a not ready state, the abnormal state is used to characterize that the process of the corresponding second container has crashed or the process is unresponsive, and the not ready state is used to characterize The corresponding second container is not in a state where it can execute and process the tasks to be executed; it is detected that the number of tasks is greater than the number of dispatchable containers, the number of vacant containers is determined according to the number of tasks and the number of dispatchable containers, and the first execution time is determined according to the number of dispatchable containers and the reference execution time of each task to be executed, the first execution time being the time required for the dispatchable container to execute and complete all the tasks to be executed; the second execution time is determined according to the number of vacant containers, the running state corresponding to the non-dispatched container and the reference execution time of each task to be executed, the non-dispatched container is a second container whose running state is in an abnormal state and an unready state, and the restart times corresponding to different running states are different, the second execution time is the time required for the updated dispatchable container to execute and complete all the tasks to be executed after the non-dispatched container is restarted; the task dispatching strategy corresponding to the first execution time or the second execution time with smaller execution time is used to implement the execution component to execute the corresponding task to be executed.

[0086] In a possible embodiment, in terms of executing a resource adjustment strategy to maintain the continuous operation of the first container and multiple second containers according to the first container and / or the second container with excessive resources and the updated current available resources until it is determined that the query statement is executed to obtain the execution result data, the query execution unit 603 is specifically used to: determine the first container and / or the second container with excessive resources as the target container, and determine the non-dispatched container and the dispatchable container to which the task to be executed is not dispatched as the reference container; determine the excessive resources additionally required by the target container, and determine the releasable resources according to the second resource specification information corresponding to the reference container; if the releasable resources are greater than or equal to the excessive resources, delete the reference container to release the releasable resources to maintain the continuous operation of the target container; if the releasable resources are less than the excessive resources, update the first resource specification information or the second resource specification information corresponding to the target container according to the updated current available resources, and back up the current container data of the target container; and restart the target container according to the current container data and the updated first resource specification information or the second resource specification information.

[0087] In the case of integrated units, such as Figure 7 As shown, Figure 7 This is a functional unit block diagram of another query engine running device based on virtualization technology provided by an embodiment of the present application. Figure 7 In the embodiment, the query engine operation device 60 based on virtualization technology includes: a processing module 702 and a communication module 701. The processing module 702 is used to control and manage the actions of the query engine operation device based on virtualization technology, for example, the steps of the virtualization unit 601, the operation unit 602, the query execution unit 603 and the data return unit 604, and / or other processes for executing the technology described herein. The communication module 701 is used to support the interaction between the query engine operation device based on virtualization technology and other devices. Figure 7 As shown, the query engine running device based on virtualization technology may include a storage module 703, and the storage module 703 is used to store program codes and data of the query engine running device based on virtualization technology.

[0088] Among them, the processing module 702 can be a processor or a controller, for example, a central processing unit (CPU), a general processor, a digital signal processor (DSP), an ASIC, an FPGA or other programmable logic device, a transistor logic device, a hardware component or any combination thereof. It can implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. The processor can also be a combination that implements a computing function, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like. The communication module 701 can be a transceiver, an RF circuit or a communication interface, and the like. The storage module 703 can be a memory.

[0089] Among them, all relevant contents of each scenario involved in the above method embodiment can be referred to the functional description of the corresponding functional module, and will not be repeated here. The above query engine running device 60 based on virtualization technology can execute the above Figure 2 The query engine running method based on virtualization technology is shown.

[0090] The above embodiments can be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When a computer instruction or computer program is loaded or executed on a computer, a process or function according to an embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. Computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, computer instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired or wireless means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center containing one or more available media sets. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0091] Figure 8 is a structural block diagram of a server provided in an embodiment of the present application. Figure 8As shown, the server 110 may include one or more of the following components: a processor 801, a memory 802 coupled to the processor 801, wherein the memory 802 may store one or more computer programs 803, and the one or more computer programs 803 may be configured to implement the methods described in the above embodiments when executed by one or more processors 801. The server 110 may be the server in the above embodiments.

[0092] The processor 801 may include one or more processing cores. The processor 801 uses various interfaces and lines to connect various parts of the entire server 110, and executes various functions and processes data of the server 110 by running or executing instructions, programs, code sets or instruction sets stored in the memory 802, and calling data stored in the memory 802. Optionally, the processor 801 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 801 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing display content; and the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 801, and it can be implemented separately through a communication chip.

[0093] The memory 802 may include a random access memory (RAM) or a read-only memory (ROM). The memory 802 may be used to store instructions, programs, codes, code sets or instruction sets. The memory 802 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned various method embodiments, etc. The data storage area may also store data created by the server 110 during use, etc.

[0094] It is understandable that the server 110 may include more or fewer structural elements than those in the above structural block diagram, which is not limited here.

[0095] An embodiment of the present application also provides a computer storage medium, on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, part or all of the steps of any method recorded in the above method embodiments are implemented.

[0096] An embodiment of the present application also provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to execute part or all of the steps of any method recorded in the above method embodiments.

[0097] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0098] In the several embodiments provided in the present application, it should be understood that the disclosed methods, devices and systems can be implemented in other ways. For example, the device embodiments described above are only schematic; for example, the division of units is only a logical function division, and there may be other division methods in actual implementation; for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0099] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0100] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may be physically included separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0101] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform some steps of the methods of various embodiments of the present invention. The aforementioned storage medium includes: a USB flash drive, a mobile hard disk, a magnetic disk, an optical disk, a volatile memory or a non-volatile memory. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM) or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (double data rate SDRAM, DDR SDRAM), enhanced synchronous dynamic random access memory (enhanced SDRAM, ESDRAM), synchronous link dynamic random access memory (synchlink DRAM, SLDRAM) and direct memory bus random access memory (directrambus RAM, DR RAM) and other media that can store program code.

[0102] Although the present invention is disclosed as above, the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions without departing from the spirit and scope of the present invention, and can make various changes and modifications, including the combination of the above-mentioned different functions and implementation steps, including software and hardware implementation methods, all of which are within the scope of protection of the present invention.

Claims

1. A query engine operation method based on virtualization technology, characterized in that: Applied to a server, the method comprises: Performing a virtualization operation on a preset query engine to configure the query engine to a container environment, wherein the query engine is composed of a management component and a plurality of execution components, and the management component and the plurality of execution components are respectively configured with corresponding containers in the container environment; Performing a resource allocation operation on the query engine to run the query engine in the container environment, wherein the resource allocation operation is used to allocate computing resources in the container environment; When receiving a query statement input by a user from a client, executing the query statement through the management component and the plurality of execution components in the query engine to obtain execution result data for the query statement from a preset data storage system, wherein the data storage system is located outside the container environment; The execution result data is returned to the client.

2. The method according to claim 1, characterized in that The performing a resource allocation operation on the query engine to run the query engine in the container environment includes: Run the initialization startup program outside the container environment to read configuration file information of the query engine, wherein the configuration file information is used to indicate first resource specification information, second resource specification information, and the number of components of the execution component to be started, the first resource specification information is used to indicate the computing resource specification required by the management component, and the second resource specification information is used to indicate the computing resource specification required by the execution component; Execute a resource application operation for the computing resource according to the first resource specification information and the second resource specification information; After the computing resource allocation is completed, a first container image and a second container image are obtained, wherein the first container image includes process information required to run when the first container configured by the management component is started, and the second container image includes process information required to run when the second container configured by the execution component is started; Start the first container according to the first container image in sequence, and start the second container corresponding to the number of components according to the second container image; After the first container and the second container are started, it is determined that the query engine runs successfully, and the first container and the second container that are started maintain inter-component communication.

3. The method according to claim 2, characterized in that The performing a resource application operation for the computing resource according to the first resource specification information and the second resource specification information includes: Obtaining available resource information, where the available resource information is used to indicate the current available resource amount of the computing resources in the server at the current system time; Determine the current available resource amount of the container environment according to the available resource information, and determine the current resource application requirement of the resource application operation indication according to the first resource specification information and the second resource specification information, wherein the current available resource amount includes the number of processing cores and the remaining memory capacity of the central processing unit; Determine whether the number of processing cores and the remaining memory capacity meet the current resource application requirements: If the number of processing cores and the remaining memory capacity meet the resource application requirements, allocating the currently available resource amount to the first container according to the first resource specification information, and allocating the currently available resource amount to the second container corresponding to the number of components according to the second resource specification information; and, Update the currently available resource amount according to the first resource specification information and the second resource specification information; If the number of processing cores and the remaining memory capacity do not meet the resource application requirements, determining that the query engine fails to run; and, Re-execute the above resource application operation according to the preset cycle.

4. The method according to claim 3, characterized in that: The executing the query statement by the management component and the plurality of execution components in the query engine to obtain execution result data for the query statement from a preset data storage system includes: Based on the inter-component communication between the first container and the second container, controlling the execution component to execute the query statement on the data storage system through the management component; and When the execution component of the second container executes the query statement, first resource consumption information corresponding to the first container and second resource consumption information of the second container are obtained in real time; If it is determined that the first container has resource overlimit according to the first resource consumption information and the first resource specification information, and / or if it is determined that any of the second containers has resource overlimit according to the second resource consumption information and the second resource specification information, then according to the first container and / or the second container with the resource overlimit and the updated current available resource amount, a resource adjustment strategy is executed to maintain the continuous operation of the first container and multiple second containers until it is determined that the query statement is executed completely to obtain the execution result data; If it is determined that the first container does not exceed the resource limit based on the first resource consumption information and the first resource specification information, and if it is determined that the second container does not exceed the resource limit based on the second resource consumption information and the second resource specification information, the first container and the second container are maintained in continuous operation until it is determined that the query statement is executed completely to obtain the execution result data.

5. The method according to claim 4, characterized in that The controlling the execution component to execute the query statement on the data storage system through the management component based on the inter-component communication between the first container and the second container includes: Based on the management component in the first container, receiving and parsing the query statement to determine a parsing result, wherein the parsing result is used to indicate a type of operation to be performed and a database object of the data storage system involved in the type of operation; and Determine a plurality of tasks to be executed in parallel according to the type of operation to be executed and the database object involved in the type of operation, and obtain the running status of the execution component in the second container in real time based on the inter-component communication between the first container and the second container; Dispatching the plurality of tasks to be executed to the execution component according to the running state, so that the execution component in the second container executes the corresponding tasks to be executed assigned by the management component for the data storage system, and each second container executes one task to be executed; The reference result data returned by the execution component are received in sequence by the management component, and a plurality of the reference result data are aggregated into the execution result data.

6. The method according to claim 5, characterized in that The dispatching of the plurality of tasks to be executed to the execution component according to the running state, so as to implement execution of the corresponding tasks to be executed assigned by the management component for the data storage system by the execution component in the second container, includes: Acquire a historical task information set, the historical task information set is used to store task attributes and reference execution times of multiple historical execution tasks executed by the execution component in the query engine, the task attributes are used to indicate the operation type and the database object corresponding to the corresponding historical execution task; According to the historical task information set, matching the reference execution time of each of the tasks to be executed and the task quantity of the tasks to be executed; Determine the number of dispatchable containers as the number of dispatchable containers, the dispatchable container is the second container whose running state is in a normal state, and the number of execution components. The running state also includes an abnormal state and a not ready state, the abnormal state is used to indicate that the process of the corresponding second container crashes or the process is unresponsive, and the not ready state is used to indicate that the corresponding second container is not in a state where it can execute and process the task to be executed; It is detected that the number of tasks is greater than the number of dispatchable containers, the number of vacant containers is determined according to the number of tasks and the number of dispatchable containers, and a first execution time is determined according to the number of dispatchable containers and a reference execution time of each of the tasks to be executed, wherein the first execution time is the time required for the dispatchable containers to complete all the tasks to be executed; Determine a second execution time according to the number of vacant containers, the running state corresponding to the undispatched container, and the reference execution time of each of the tasks to be executed, wherein the undispatched container is a second container whose running state is in the abnormal state and the unready state, and the restart times corresponding to different running states are different from each other, and the second execution time is the time required for the updated dispatchable container to complete all the tasks to be executed after the undispatched container is restarted; The task dispatching strategy corresponding to the first execution time or the second execution time with a smaller execution time is used to enable the execution component to execute the corresponding task to be executed.

7. The method according to claim 6, characterized in that The executing a resource adjustment strategy to maintain the continuous operation of the first container and the plurality of second containers according to the first container and / or the second container having the resource overlimit and the updated current available resource amount, until it is determined that the query statement is executed completely to obtain the execution result data, includes: Determine the first container and / or the second container with the resource overlimit as the target container, and determine the non-dispatcher container and the dispatchable container to which the to-be-executed task is not dispatched as the reference container; Determine the amount of excess resources additionally required by the target container, and determine the amount of releasable resources according to the second resource specification information corresponding to the reference container; If the releasable resource amount is greater than or equal to the excess resource amount, deleting the reference container to release the releasable resource amount to maintain the continuous operation of the target container; If the releasable resource amount is less than the excess resource amount, updating the first resource specification information or the second resource specification information corresponding to the target container according to the updated current available resource amount, and backing up the current container data of the target container; and, Restart the target container according to the current container data and the updated first resource specification information or the second resource specification information.

8. A query engine operation device based on virtualization technology, characterized in that: Applied to a server, the device comprises: A virtualization unit, configured to perform a virtualization operation on a preset query engine to configure the query engine to a container environment, wherein the query engine is composed of a management component and a plurality of execution components, and the management component and the plurality of execution components are respectively configured with corresponding containers in the container environment; an operating unit, configured to perform a resource allocation operation on the query engine to operate the query engine in the container environment, wherein the resource allocation operation is used to allocate computing resources in the container environment; a query execution unit, configured to, when receiving a query statement input by a user from a client, execute the query statement through the management component and the plurality of execution components in the query engine, so as to obtain execution result data for the query statement from a preset data storage system, wherein the data storage system is located outside the container environment; A data returning unit is used to return the execution result data to the client.

9. A server, characterized in that: The method comprises a processor, a memory, a communication interface, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the processor, and the programs include instructions for executing the steps in the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: A computer program for electronic data exchange is stored, wherein the computer program causes a computer to execute the method according to any one of claims 1 to 7.

Images