Virtual warehouse system, and operating method, medium, and electronic device

Abstract

Embodiments of the present disclosure provide a virtual warehouse system. The system comprises a plurality of first virtual warehouses; wherein the first virtual warehouses comprise pre-divided resource groups; the resource groups comprise first distribution nodes and first computing nodes; at least part of the first computing nodes correspond to resource groups comprised by at least two first virtual warehouses; the first distribution nodes are configured to distribute a first operation request to a first target computing node in response to receiving the first operation request; the first target computing node and the first distribution node correspond to the same resource group; and the first target computing node is configured to execute the received first operation request. Thus, the purpose of sharing computing resources in the virtual warehouse can be achieved, and the resource utilization is improved. Furthermore, embodiments of the present disclosure provide an operation method, a medium and an electronic device.

Description

TECHNICAL FIELD

[0001] Embodiments of the present disclosure relate to the field of operations, and more particularly, embodiments of the present disclosure relate to a virtual warehouse system, and an operating method, a medium and an electronic device. BACKGROUND

[0002] This section is intended to provide background information to facilitate a better understanding of embodiments of the present disclosure. Information in this section is not admitted to be prior art.

[0003] At present, a virtual warehouse system can include multiple virtual warehouses implementing different business scenarios. Computing nodes for performing operation tasks are assigned to different virtual warehouses. The computing nodes can only execute operation requests for the corresponding virtual warehouses. The present disclosure does not limit the types of business scenarios. For example, the business scenarios can include Business Intelligence (BI) business, Ad hoc queries business, etc. The operation requests can include data reading requests, data storage requests, etc.

[0004] Assigning computing nodes to different virtual warehouses can ensure that there is no interference during the execution of operation by the computing nodes. However, due to factors such as small amount of operation requests of some virtual warehouses and concentrated time of executing operations, the corresponding computing nodes can often be in an idle state, resulting in low utilization of computing resources. SUMMARY

[0005] In this context, embodiments of the present disclosure aim to provide a virtual warehouse system, and an operating method, a medium and an electronic device.

[0006] In a first aspect of the embodiments of the present disclosure, a virtual warehouse system is provided. The virtual warehouse system includes multiple first virtual warehouses; wherein the first virtual warehouses include pre-divided resource groups; the resource groups include first distribution nodes and first computing nodes; at least part of the first computing nodes correspond to resource groups included in at least two first virtual warehouses; the first distribution nodes are configured to, in response to receiving a first operation request for a first virtual warehouse corresponding to the first distribution nodes, distribute the first operation request to a first target computing node; the first target computing node corresponds to the same resource group as the first distribution nodes; and the first target computing node is configured to execute the received first operation request.

[0007] In a second aspect of the embodiments of the present disclosure, an operation method is provided, applied to a virtual warehouse system, the virtual warehouse system comprising a plurality of first virtual warehouses; wherein the first virtual warehouse comprises a pre-divided resource group; the resource group comprises a first distribution node and a first computing node; at least part of the first computing nodes correspond to resource groups comprised by at least two first virtual warehouses; the method comprises: through the first distribution node, in response to receiving a first operation request for the first virtual warehouse corresponding to the first distribution node, distributing the first operation request to a first target computing node; the first target computing node corresponds to the same resource group as the first distribution node; through the first target computing node, executing the received first operation request.

[0008] In a third aspect of the embodiments of the present disclosure, a medium is provided, the medium storing a computer program, the computer program being used to make a processor execute the operation method shown in any of the preceding embodiments.

[0009] In a fourth aspect of the embodiments of the present disclosure, an electronic device is provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor implements the operation method shown in any of the preceding embodiments by running the executable instructions.

[0010] In the technical solutions described above, at least part of the computing nodes correspond to resource groups comprised by at least two virtual warehouses, so that these computing nodes can execute operation requests for the at least two virtual warehouses, achieving the purpose of sharing computing resources in virtual warehouses. Compared with related technologies, the method disclosed in the present disclosure can share the computing nodes contained in the virtual warehouse with less operation requests and more concentrated operation execution time, thereby avoiding the situation that the computing nodes are often in an idle state, and improving the resource utilization rate. BRIEF DESCRIPTION OF DRAWINGS

[0011] The foregoing and other objects, features and advantages of the exemplary embodiments of the present disclosure will be more clearly understood from the following detailed description taken in conjunction with the accompanying drawings. In the drawings, several embodiments of the present disclosure are shown by way of example and not limitation, in which:

[0012] FIG. 1 An application scenario schematic diagram of a virtual warehouse system according to an embodiment of the present disclosure is shown;

[0013] FIG. 2 A system structure diagram of a virtual warehouse system according to an embodiment of the present disclosure is shown;

[0014] FIG. 3 A flowchart of obtaining routing information according to an embodiment of the present disclosure is shown;

[0015] FIG. 4 A structural schematic diagram of a second virtual warehouse shown by an embodiment of the present disclosure;

[0016] FIG. 5 A method flowchart of an operation method shown by an embodiment of the present disclosure;

[0017] FIG. 6 A program product applied to the operation method shown by an embodiment of the present disclosure;

[0018] FIG. 7 A structural schematic diagram of an electronic device shown by an embodiment of the present disclosure.

[0019] In the drawings, the same or corresponding reference numbers denote the same or corresponding parts. DETAILED DESCRIPTION

[0020] The principles and spirits of the present disclosure will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are given only to enable those skilled in the art to better understand and implement the present disclosure, and do not limit the scope of the present disclosure in any way. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0021] Those skilled in the art know that the embodiments of the present disclosure can be implemented as a system, device, apparatus, method or computer program product. Therefore, the present disclosure can be embodied in the form of entire hardware, entire software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0022] In addition, any number of elements in the drawings is used for example and not limitation, and any naming is only for distinction and does not have any limiting meaning.

[0023] According to the embodiments of the present disclosure, a virtual warehouse system, an operation method, a medium and an electronic device are proposed.

[0024] The principles and spirits of the present disclosure will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are given only to enable those skilled in the art to better understand and implement the present disclosure, and do not limit the scope of the present disclosure in any way. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art. SUMMARY

[0026] In a first aspect, the inventors find that maintaining resource groups for computing nodes and distribution nodes can conveniently maintain the correspondence between the computing nodes and the distribution nodes.

[0027] Please refer to FIG. 2 , wherein the first distribution node 231 has a corresponding relationship with the computing nodes 241 to 244. FIG. 2In the schematic first virtual warehouse 211, the first distribution node 231 can be maintained as a resource group 221 with the computing nodes 241-244, so that the correspondence between the computing nodes and the distribution nodes can be conveniently maintained.

[0028] If the resource groups are further corresponded to the virtual warehouses, the virtual warehouses can be allocated with the computing resources. When the virtual warehouses need to increase or decrease the computing resources, the resource groups can be correspondingly increased or decreased, so that the flexible allocation of the computing resources can be realized.

[0029] Therefore, the inventors associate the virtual warehouses with the distribution nodes and the computing nodes (computing resources) through the resource groups as intermediaries.

[0030] In the second aspect, the inventors find that if the computing nodes are to be isolated from each other, the isolation can be performed from the entrances of the computing nodes.

[0031] Therefore, the inventors can avoid the mutual interference between the computing nodes of different resource groups by causing the first distribution node to distribute the operation request to the computing nodes corresponding to the same resource group as the first distribution node upon receiving the operation request.

[0032] In the third aspect, the inventors find that if the computing nodes can be shared by at least two virtual warehouses, the computing nodes included in the virtual warehouse with less operation requests and concentrated operation execution time can be shared, so that the situation that the computing nodes are easily idle can be avoided, and the resource utilization rate is improved.

[0033] Therefore, the inventors cause at least part of the computing nodes to correspond to the resource groups included in at least two virtual warehouses, so that the part of the computing nodes can correspond to the at least two virtual warehouses, i.e., the part of the computing nodes can execute the operation requests for the at least two virtual warehouses, and the purpose of sharing the computing resources in the virtual warehouses is achieved.

[0034] After introducing the basic principles of the present disclosure, various non-limiting embodiments of the present disclosure will be specifically introduced.

[0035] OVERVIEW OF APPLICATION SCENARIOS

[0036] Please refer to FIG. 1 , FIG. 1 An application scenario schematic diagram of a virtual warehouse system according to an embodiment of the present disclosure is shown.

[0037] As FIG. 1 shown, the foregoing application scenario can include a client device, a virtual warehouse system 12, and a data storage system 13.

[0038] The client devices can include a client device 111 and a client device 112. The client devices can be a notebook computer, a computer, a mobile phone, a PDA (Personal Digital Assistant), or the like. The present disclosure does not limit the specific type of the client devices.

[0039] The virtual warehouse system 12 can include a virtual warehouse 121, a virtual warehouse 122, and a virtual warehouse 123. It can be understood that in actual applications, the virtual warehouse system can include a plurality of virtual warehouses. Hereinafter, the virtual warehouse system 12 is taken as an example to include the virtual warehouse 121, the virtual warehouse 122, and the virtual warehouse 123 for illustration.

[0040] Different virtual warehouses can be used for different business scenarios. For example, the virtual warehouse 121 can be used for BI business, the virtual warehouse 122 can be used for Ad hoc business, and the virtual warehouse 123 can be used for online business. The virtual warehouses can include a plurality of server devices, and the present disclosure does not limit the type of the server devices.

[0041] The configuration of the client device can be adjusted according to different business scenarios, so that the client device can access the virtual warehouse corresponding to the business scenario. For example, for the BI business scenario, the configuration of the client device can be changed so that the client device can access the virtual warehouse 121. For the Ad hoc business, the configuration of the client device can be changed so that the client device can access the virtual warehouse 122.

[0042] The data storage system 13 can be used to store data.

[0043] Taking the BI business to read data as an example.

[0044] The user A can initiate a data query request through the client device 111 corresponding to the user A. The client device 111 can send the data query request to the corresponding virtual warehouse 121. The virtual warehouse 121 can execute the data query request, interact with the data storage system, obtain target data, and return the target data to the client device 111. The client device 111 can provide the target data to the user A.

[0045] EXEMPLARY SYSTEM

[0046] The present disclosure provides a virtual warehouse system.

[0047] The virtual warehouse system comprises a plurality of first virtual warehouses; wherein the first virtual warehouse comprises a pre-divided resource group; the resource group comprises a first distribution node and a first computing node; at least part of the first computing nodes correspond to resource groups comprised by at least two first virtual warehouses;

[0048] The first distribution node is configured to distribute, in response to receiving a first operation request for the first virtual warehouse corresponding to the first distribution node, the first operation request to a first target computing node; the first target computing node corresponds to the same resource group as the first distribution node.

[0049] The first target computing node is configured to execute the received first operation request.

[0050] In the foregoing system, first, the virtual warehouse is associated with the distribution node and the computing node (computing resource) through the resource group as an intermediary, when the virtual warehouse needs to change the computing resource, only the resource group associated with the virtual warehouse needs to be changed, so that the flexible allocation of the computing resource can be realized.

[0051] Second, the first distribution node distributes the operation request to the computing node corresponding to the same resource group as itself after receiving the operation request, so that the computing node can only receive the operation request sent by the distribution node belonging to the same resource group as itself, and the mutual interference between the computing nodes of different resource groups can be avoided.

[0052] Third, at least part of the computing nodes correspond to the resource groups comprised by at least two virtual warehouses, so that the part of the computing nodes correspond to at least two first virtual warehouses, that is, the part of the computing nodes can execute the operation request for the at least two virtual warehouses, so that the purpose of sharing the computing resource in the virtual warehouse is achieved. Compared with the related art, the disclosed method can share the computing nodes contained in the virtual warehouse with less operation request amount and more concentrated operation execution time, so as to avoid the situation that the computing node is often in an idle state, and improve the resource utilization rate.

[0053] The system will be described below in combination with the accompanying drawings. Please refer to FIG. 2 , FIG. 2 FIG. 1 is a system structure diagram of a virtual warehouse system according to an embodiment of the present disclosure.

[0054] As shown in FIG. 2 , the virtual warehouse system 200 can comprise a plurality of first virtual warehouses. It should be noted that the present disclosure does not limit the number of first virtual warehouses.

[0055] The virtual warehouse system 200 can provide online analytical processing services for clients as a service end. The virtual warehouse system 200 can be built based on different data query engines. In some embodiments, the virtual warehouse system can be built based on an Impala cluster.

[0056] The Impala is an MPP (Massively Parallel Processing) query engine open sourced by the Apache Foundation. The biggest feature of the Impala is also fast compared with other query engines. In terms of architecture, there is a separate Catalog process to cache the metadata and statistical information required for queries for query acceleration. Thus, the advantage of fast query speed of the Impala can be used to improve the data processing efficiency of the virtual warehouse system.

[0057] The present disclosure will distinguish the virtual warehouse with an entrance as a first virtual warehouse. In the first virtual warehouse, the entrances of the first computing nodes are distinguished by resource groups, and the first computing nodes can only receive operation requests sent by the first distribution nodes belonging to the same resource group as themselves, so as to avoid mutual interference between the first computing nodes of different resource groups.

[0058] Different first virtual warehouses can be used for different business scenarios. For example, the first virtual warehouse 211 can be used for BI business, and the first virtual warehouse 212 can be used for Ad hoc business.

[0059] The first virtual warehouse includes pre-divided resource groups. For example, as shown in FIG. 2 The first virtual warehouse 211 includes a resource group 221, and the first virtual warehouse 212 includes a resource group 222.

[0060] In some embodiments, the number of resource groups can be allocated to the first virtual warehouse according to the business demand. In this way, the virtual warehouse can be associated with the distribution nodes and the computing nodes (computing resources) through the resource groups as intermediaries. When the virtual warehouse needs to change the computing resources, only the resource groups associated with the virtual warehouse need to be changed, so that the flexible allocation of computing resources can be realized.

[0061] For example, if the first virtual warehouse has a large amount of data processing, more resource groups can be allocated, and if the amount of data processing is small, fewer resource groups can be allocated. In this way, the flexible allocation of computing resources can be realized.

[0062] The resource group can include the first distribution node and the first computing node. The resource group can provide computing power for a virtual warehouse through the first distribution node and the first computing node to complete a data operation request. In a virtual warehouse system constructed based on an Impala cluster, the distribution node can include a coordinator node, and the first computing node can include an executor node. The present disclosure is not limited to the device types of the distribution node and the computing node.

[0063] In some embodiments, a warehouse group (group) parameter can be added. By configuring the group parameters of the first distribution node and the first computing node, the resource group to which the distribution node and the computing node belong can be determined. For example, as shown in FIG. 2 The group of the first distribution node 231 and the first computing nodes 241 to 244 can be configured as group 1.

[0064] The first distribution node can distribute a first operation request for a first virtual warehouse corresponding to the first distribution node to a first target computing node in response to receiving the first operation request, the first target computing node and the first distribution node corresponding to the same resource group.

[0065] The first target computing node is configured to execute the received first operation request.

[0066] After receiving the operation request, the first distribution node distributes the operation request to the computing nodes corresponding to the same resource group as itself, so that the computing nodes can only receive the operation request from the distribution nodes belonging to the same resource group as themselves, and the mutual interference between the computing nodes of different resource groups can be avoided.

[0067] For example, as shown in FIG. 2 After receiving the operation request for the first virtual warehouse 211, the first distribution node 231 can distribute the operation request to the first computing nodes 241 to 244 (first target computing nodes) belonging to the same resource group 221. The first computing nodes 241 to 244 can execute the operation request and return the execution result to the client through the first distribution node. That is, the computing nodes in the resource group 221 will not be disturbed by other resource groups (such as the resource group 22n).

[0068] At least part of the first computing nodes correspond to resource groups included by at least two first virtual warehouses. In this way, the part of the computing nodes correspond to at least two virtual warehouses, that is, the part of the computing nodes can execute operation requests for the at least two virtual warehouses, achieving the purpose of sharing computing resources in virtual warehouses. Compared with the related art, the method of the present disclosure can share the computing nodes of the virtual warehouse containing less operation requests and the time of executing operations, thereby avoiding the situation that the computing nodes are easily in an idle state, and improving the resource utilization.

[0069] For example, the first virtual warehouse 211 corresponds to a BI service of automatically querying data for 24 hours. The first virtual warehouse 212 corresponds to an ad hoc service in working hours (8 am to 5 pm). FIG. 2 The computing node 241 and the computing node 242 shown belong to both the resource group 221 included by the first virtual warehouse 211 and the resource group 222 included by the first virtual warehouse 212. The computing node 241 and the computing node 242 can also process the BI service in non-working hours, avoiding the situation that the two nodes are in an idle state in non-working hours, and improving the resource utilization.

[0070] The present disclosure also provides a virtual warehouse system. The structure of the system can refer to the virtual warehouse system 200, and details are not described herein. The system can distribute operation requests according to the load of the computing nodes, thereby balancing the load of the computing nodes and avoiding the situation that the load of the computing nodes is too high, which affects the query performance of the system as a whole.

[0071] Specifically, the first distribution node can periodically acquire the load of the first target computing node, and distribute the first operation request to the first target computing node whose load does not reach a preset threshold in response to receiving the first operation request.

[0072] The load can include the load of various resources of the computing node, such as a central processing unit (CPU), memory, network, and disk. In the present disclosure, some load measurement indicators can be designed, and the load of the computing node can be measured through the load measurement indicators. The present disclosure does not limit the specific type of the load measurement indicator. For example, the indicator can include a resource occupancy rate. The higher the load of the computing node, the more saturated the workload, and the workload of the computing node needs to be reduced.

[0073] The preset threshold can be set according to the service demand.

[0074] In some embodiments, a message topic mechanism can be employed for the transmission of the load amount. Specifically, a topic related to the load amount can be set in a message service end. The first computing node can periodically upload its load amount to the topic related to the load amount. The first distribution node can subscribe to the relevant messages under the load amount topic and periodically obtain the load amount of each first computing node from the load amount topic. After receiving the first operation request, the first distribution node can determine whether the load amount of each first computing node reaches a preset threshold according to the obtained load amount, and distribute the first operation request to the computing node that does not reach the preset threshold.

[0075] Thus, the distribution of operation requests can be performed according to the load amount of the computing node, so as to balance the load amount of each computing node and avoid the situation that the computing node is overloaded, thereby affecting the query performance of the system as a whole.

[0076] The present disclosure also proposes a virtual warehouse system. The structure of the system can refer to the virtual warehouse system 200, which will not be described in detail here. The system also includes a first routing service end for routing service. The system can provide the node information of the first target distribution node corresponding to the target virtual warehouse to be accessed by the client through the first routing service end, so that the client can establish a connection with the first target distribution node through the node information and realize the access of the target virtual warehouse.

[0077] In the present disclosure, the routing service end that provides routing service for the first virtual warehouse is referred to as the first routing service end.

[0078] Please refer to FIG. 3 , FIG. 3 A flowchart for obtaining routing information is shown for the embodiments of the present disclosure. FIG. 3 The first routing service end shown can be part of the virtual warehouse system. FIG. 3 In the figure, the resource group 32k and the resource group 32j belong to the virtual warehouse 31k, and the resource group 32m belongs to the resource virtual warehouse 31m.

[0079] The first routing service end includes routing information corresponding to each of the first virtual warehouses. As shown in FIG. 3 The first routing service end includes routing information 35k corresponding to the virtual warehouse 31k and routing information 35m corresponding to the virtual warehouse 31m.

[0080] The routing information includes the node information of each first distribution node included in the first virtual warehouse. As shown in FIG. 3As shown, the routing information 35k includes the first distribution node 33k and the first distribution node 33j in the first virtual warehouse 31k, and the routing information 35m includes the first distribution node 33m in the first virtual warehouse 31m.

[0081] In some embodiments, the first routing server provides a routing registration service, and each routing information is stored in a directory corresponding to the routing registration service. The routing registration service includes a zookeeper service. Thus, the performance of the virtual warehouse system can be improved by using the simple, efficient and stable characteristics of the zookeeper operation.

[0082] In some embodiments, the configuration of the client can be adjusted for different business scenarios, so that the client can access the target routing information. The target routing information is the routing information corresponding to the virtual warehouse corresponding to the business scenario.

[0083] The client can send a routing acquisition request for a target virtual warehouse to the first routing server.

[0084] The first routing server can, in response to receiving a routing acquisition request initiated by the client for a target virtual warehouse, obtain target node information in target routing information corresponding to the target virtual warehouse, and send the target node information to the client.

[0085] In some embodiments, the first routing server can randomly select target node information from the node information included in the target routing information.

[0086] In some embodiments, in order to balance the workload of the distribution nodes, the number of calls for each distribution node can be maintained. The first routing server can obtain the number of calls corresponding to the first distribution node included in the target routing information. Then, the distribution node with the smallest number of calls is selected, and the node information of the distribution node is returned to the client as the target node information.

[0087] The client is configured to send the first operation request to the first target distribution node indicated by the target node information based on the target node information, so that the first target distribution node distributes the first operation request.

[0088] Thus, by the first routing server, routing information can be maintained for each first virtual warehouse, so that each first virtual warehouse has an independent access portal, avoiding mutual interference of portals.

[0089] For example, data is read through the virtual warehouse 31k. As shown in FIG. 3, the first routing server 32k can obtain the routing information 35k of the first virtual warehouse 31k, and the routing information 35k includes the first distribution node 33k and the first distribution node 33j in the first virtual warehouse 31k. FIG. 3As shown, the client can send a routing acquisition request initiated for the virtual warehouse 31k to the first service end. The first service end can randomly select a first distribution node from the routing information 35k as a first target distribution node. Assuming that the first distribution node 33k is selected, the first distribution node 33k can be taken as the first target distribution node, and the node information of the first distribution node 33k is returned. The node information can include information such as the IP address of the first distribution node 33k.

[0090] After receiving the returned node information, the client can establish a connection (for example, a TCP connection) with the first distribution node 33k through the node information, and send a data reading request to the first distribution node 33k, so that the first distribution node 33k distributes the data reading request to the first computing node 34k, and completes the data reading task.

[0091] In this way, the entrance of the virtual warehouse 31k can be isolated from the entrances of other virtual warehouses, and will not be affected by other entrances.

[0092] In some embodiments, the first routing service end is configured to, in response to the first target distribution node being in a target state, acquire any other node information except the target node information from the target routing information, and send the other node information to the client, so that the client accesses the first virtual warehouse through other nodes indicated by the other node information.

[0093] The target state can be set according to business requirements. For example, the target state can be a down state, an IP address change state, etc. When the distribution node is in the target state, it can be indicated that the distribution node can not work normally.

[0094] The first routing service end can provide node information of other nodes to the client when the first target distribution node is in the target state, so as to ensure that the client can still establish a connection with the virtual warehouse, normally access data, and realize service high availability of the virtual warehouse system. In addition, the process of switching the distribution node can be completed through the client and the first routing service end, and the user has no perception, which can improve the user experience.

[0095] As shown in FIG. 33, FIG. 4 As shown, assuming that the first target distribution node 33k is in a down state, the first routing service end can return the node information of the first distribution node 33j in the virtual warehouse 31k to the client. The client can continue to maintain the connection with the virtual warehouse 31k through the first distribution node 33j, so as to continue to normally access data, and realize service high availability of the virtual warehouse system.

[0096] In some embodiments, the first routing server is configured to, in response to an increase in the resource group, add node information of a first distribution node included in the increased resource group to routing information corresponding to the first virtual warehouse.

[0097] Thus, the first routing server can automatically update the routing information and keep the routing information of the virtual warehouse up to date.

[0098] For example, if the virtual warehouse 31m needs to increase the resource group 32h, the first routing server can register the first distribution node 33h included in the resource group 32h in the routing information 35m, so that the client can obtain the node information of the first distribution node 33h and establish a connection with the virtual warehouse 31m through the node. Thus, the first routing server can automatically update the routing information of the virtual warehouse 31m and keep the routing information up to date.

[0099] In the foregoing system, the configuration information of the client needs to be modified to enable the client to access the routing information of the target virtual warehouse and achieve isolation of the computing resources. For an online service, modifying the configuration information of the client can affect user experience.

[0100] The present disclosure also provides a virtual warehouse system. The system can include a second virtual warehouse without distinguishing an entry. Unlike the first virtual warehouse, the second virtual warehouse groups the computing nodes to obtain node groups without distinguishing an entry of the node groups. That is, the node groups do not distinguish an entry and can receive operation requests distributed by any distribution node. Thus, isolation of the computing resources is achieved through the node groups, and since there is no entry of the node groups (computing resources), the configuration information of the client does not need to be modified, and user experience is not affected.

[0101] The second virtual warehouse includes second distribution nodes and a plurality of node groups pre-divided. Each node group includes at least one second computing node. The disclosure refers to the distribution nodes included in the second virtual warehouse as second distribution nodes and the computing nodes as second computing nodes.

[0102] Please refer to FIG. 4 , FIG. 4 FIG. 4 shows a structure of a second virtual warehouse according to an embodiment of the present disclosure.

[0103] For example, as shown in FIG. 1 FIG. 4, the second virtual warehouse 410 includes second distribution nodes 421-42n and node groups 431-43n. Each node group can include at least one second computing node.

[0104] Different node groups can be used for different types of online service scenarios. For example, node group 431 can be used for online BI services, and node group 432 can be used for ad hoc services.

[0105] The client can obtain node information of any second distribution node through a second routing server. The second routing server includes a directory. The directory registers node information of all second distribution nodes of the second virtual warehouse. In the online service, the client accesses the directory by default, so the client configuration does not need to be modified.

[0106] The client can establish a connection with any second distribution node through the obtained node information, thereby establishing a connection with the second virtual warehouse. The client can add a group identifier of a target node group to be accessed to a second operation request and send the second operation request to the second distribution node.

[0107] The second distribution node can receive a second operation request for the second virtual warehouse, the second operation request including a group identifier. In response to receiving the second operation request, the second distribution node distributes the second operation request to a second target computing node; the second target computing node is a computing node in a target node group indicated by the group identifier.

[0108] The second target computing node is configured to execute the received second operation request.

[0109] The group identifier is a session-level parameter, which can be adjusted conveniently when initiating an operation request.

[0110] By introducing a session-level group identifier parameter, the operation request of the client can be linked to the corresponding node group without modifying the client configuration information, thereby forming mutual isolation between node groups (computing resources) and avoiding mutual interference between node groups, so that the isolation of computing resources can also be quickly and conveniently realized for the online service, without affecting the user experience.

[0111] The following embodiments will be described in conjunction with FIG. 1 application scenarios. It should be noted that the foregoing application scenarios are only shown for the purpose of facilitating understanding of the spirit and principles of the present disclosure, and the embodiments of the present disclosure are not limited in this respect. On the contrary, the embodiments of the present disclosure can be applied to any applicable scenario.

[0112] As FIG. 2 shown, the foregoing application scenarios can include a client device, a virtual warehouse system 12, and a data storage system 13.

[0113] The virtual warehouse system 12 can include a routing service, a virtual warehouse 121, a virtual warehouse 122, and a virtual warehouse 123. The virtual warehouse system can be compatible with multiple types of virtual warehouses. The virtual warehouse 121 and the virtual warehouse 122 can be first virtual warehouses. The virtual warehouse 123 can be a second virtual warehouse.

[0114] The warehouse structure of the virtual warehouse 121 and the virtual warehouse 122 can refer to FIG. 4 . The virtual warehouse 121 can be used for BI business, and the virtual warehouse 122 can be used for Ad hoc business.

[0115] The virtual warehouse 121 includes a resource group 221, a first distribution node 231, and first computing nodes 241-244. The group parameters of the first distribution node 231 and the first computing nodes 241-244 can be configured as group1, so that the first distribution node 231 and the first computing nodes 241-244 belong to the resource group 221, and the first distribution node only caches the IP addresses of the first computing nodes 241-244 in the memory.

[0116] The virtual warehouse 122 includes a resource group 222, a first distribution node 232, and first computing nodes 241 and 242. The group parameter of the first distribution node 232 can be configured as group2, and the group parameters of the first computing nodes 241 and 242 can be configured as group1 and group2, so that the first computing nodes 241 and 242 belong to both the resource group 221 and the resource group 222, and the first distribution node 232 only belongs to the resource group 222. The first distribution node 232 only saves the IP addresses of the first computing nodes 241 and 242.

[0117] The warehouse structure of the virtual warehouse 123 can refer to EXEMPLARY METHOD . The virtual warehouse 123 includes second distribution nodes 421-42n and node groups 431-43n. Each node group includes a plurality of second computing nodes. The virtual warehouse 123 can be used for online business. Different node groups can implement different types of online business. The group parameters of the second distribution nodes and the second computing nodes can be configured as a preset identifier (such as default), indicating that the resource groups are not divided for the second distribution nodes and the second computing nodes, and the second distribution nodes can store the IP addresses of each second computing node.

[0118] In this example, the virtual warehouse system 12 is a system based on an Impala cluster. In the following, the distribution node is referred to as a coordinator node, and the computing node is referred to as an executor node. The routing service provides a zookeeper service.

[0119] The routing service end includes a first directory corresponding to the virtual warehouse 121, a second directory corresponding to the virtual warehouse 122, and a third directory corresponding to the online business. The first directory registers the first coordinator node 231 under the virtual warehouse 121. The second directory registers the first coordinator node 232 under the virtual warehouse 122. The third directory registers all the second coordinator nodes under the virtual warehouse 123.

[0120] The default configuration of the client can access the third directory corresponding to the online business. If it is necessary to access the first directory or the second directory, the configuration information of the client needs to be modified.

[0121] Taking reading data as an example.

[0122] For the 24-hour BI business, the configuration information of the client can be modified so that the client can access the first directory. The client initiates a routing request to the zookeeper service end. The zookeeper service end can return the IP address of the first coordinator node 231 (hereinafter referred to as the first node 231) under the first directory to the client.

[0123] The client can establish a TCP connection with the first node 231 through the obtained IP address, and send a data reading request to the first node 231.

[0124] The first node 231 can obtain the IP addresses of the first executor nodes 241 to 244 from the memory, and send a data reading request to the four executor nodes to complete the data reading operation.

[0125] For the working time ad hoc business, the configuration information of the client can be modified so that the client can access the second directory. The client initiates a routing request to the zookeeper service end. The zookeeper service end can return the IP address of the first coordinator node 232 (hereinafter referred to as the first node 232) under the second directory to the client.

[0126] The client can establish a TCP connection with the first node 232 through the obtained IP address, and send a data reading request to the first node 232.

[0127] The first node 232 can obtain the IP addresses of the first executor nodes 241 and 242 from the memory, and send a data reading request to the two executor nodes to complete the data reading operation.

[0128] For the online business, the client can access the third directory by default without modifying the client configuration information. The client initiates a request for obtaining a route to the zookeeper server. The zookeeper server can return any second coordinator node under the third directory to the client, and assume that the IP address of the second coordinator node 421 (hereinafter referred to as the second node 421) is returned.

[0129] The client can establish a TCP connection with the second node 421 through the obtained IP address, and send a data read request to the second node 421. The data read request can include a session-level parameter group identifier. Assume that the group identifier is group 431.

[0130] The second node 421 can obtain the IP addresses of the second executor nodes 441-443 in the group 431 from the memory, and send a data read request to the three executor nodes to complete the data read operation.

[0131] In the above virtual warehouse system, first, the first virtual warehouse and the second virtual warehouse can be compatible, so that the system can take into account the online business, realize the isolation of computing resources, avoid mutual interference of computing resources, and improve customer experience.

[0132] Second, in the first virtual warehouse, the resource group is used as an intermediary to associate the virtual warehouse with the coordinator node and the executor node (computing resource). When the first virtual warehouse needs to change the computing resource, only the resource group associated with the first virtual warehouse needs to be changed, so that the flexible allocation of computing resources can be realized.

[0133] Third, the routing server maintains a corresponding directory for different first virtual warehouses, so that the first virtual warehouse has an independent access entry, avoiding mutual interference of the entries.

[0134] Fourth, when the first virtual warehouse registers multiple first coordinator nodes under the directory, if a coordinator node enters a target state (such as being down), the routing server can also automatically switch other coordinator nodes, realizing the service high availability of the virtual warehouse system, and the process is user-agnostic, improving user experience.

[0135] Fifthly, in the first virtual warehouse, the first coordinator node distributes the operation request to the executor node corresponding to the same resource group as itself after receiving the operation request, so that the executor node can only receive the operation request from the coordinator node belonging to the same resource group as itself, and mutual interference between the executor nodes of different resource groups can be avoided.

[0136] Sixthly, the first executor node 241 and the first executor node 242 correspond to the virtual warehouse 121 and the virtual warehouse 122. That is, the two executor nodes can execute the operation request for the virtual warehouse 121 and the virtual warehouse 122 in the non-working time, achieving the purpose of sharing the computing resource in the virtual warehouse, and avoiding the situation that the executor node is easily in the idle state, and improving the resource utilization.

[0137] Seventhly, in the second virtual warehouse, by introducing the grouping identification parameter at the session level, the operation request of the client can be linked to the corresponding node group without modifying the client configuration information, mutual isolation between the node groups (computing resources) is formed, mutual interference between the node groups is avoided, and thus the isolation of the computing resource can be quickly and conveniently realized for the online BI business.

[0138] FIG. 5

[0139] Please refer to FIG. 5 , FIG. 5 The method flowchart of the operation method shown in the embodiments of the present disclosure is shown.

[0140] FIG. 2 The operation method shown can be applied to the virtual warehouse system. Wherein, the structure of the virtual warehouse system can refer to FIG. 5 .

[0141] The virtual warehouse system includes a plurality of first virtual warehouses; wherein the first virtual warehouse includes a pre-divided resource group; the resource group includes a first distribution node and a first computing node; at least part of the first computing nodes in the first computing node correspond to the resource groups included by at least two first virtual warehouses.

[0142] As EXEMPLARY MEDIUM shown, the operation method can include S502-S504.

[0143] S502, distributing, by the first distribution node, the first operation request to a first target computing node in response to receiving the first operation request for a first virtual warehouse corresponding to the first distribution node; the first target computing node and the first distribution node corresponding to a same resource group.

[0144] S504, executing, by the first target computing node, the received first operation request.

[0145] The description of the method example can refer to the system example, which is not repeated here.

[0146] In the foregoing technical solutions, first, the resource group is used as an intermediary to associate the virtual warehouse with the distribution node and the computing node (computing resource), and when the computing resource needs to be changed, the resource group associated with the virtual warehouse only needs to be changed, so that the flexible allocation of the computing resource can be realized.

[0147] Second, the first distribution node distributes the operation request to the computing node corresponding to the same resource group after receiving the operation request, so that the computing node can only receive the operation request sent by the distribution node belonging to the same resource group, and the mutual interference between the computing nodes of different resource groups can be avoided.

[0148] Third, the resource group corresponding to at least two virtual warehouses is included in at least part of the computing nodes, so that the part of the computing nodes can correspond to at least two first virtual warehouses, that is, the part of the computing nodes can execute the operation request for the at least two virtual warehouses, so that the purpose of sharing the computing resource in the virtual warehouse is achieved. Compared with related technologies, the disclosed method can share the computing nodes included in the virtual warehouse with less operation request amount and concentrated operation time, so as to avoid the situation that the computing node is often in an idle state, and improve the resource utilization rate.

[0149] In some embodiments, the operation method further includes:

[0150] Periodically acquiring, by the first distribution node, a load amount of the first target computing node;

[0151] The distributing the first operation request to the first target computing node includes:

[0152] Distributing the first operation request to the first target computing node whose load amount does not reach a preset threshold.

[0153] The description of the method example can refer to the system example, which is not repeated here.

[0154] Therefore, the distribution of operation requests can be performed according to the load of the computing nodes, so that the load of each computing node is balanced, and the situation that the computing nodes are overloaded is avoided, and the query performance of the system as a whole is affected.

[0155] In some embodiments, the virtual warehouse system further comprises a first routing server; the first routing server comprises routing information corresponding to each of the first virtual warehouses respectively; the routing information comprises node information of each of the first distribution nodes included in the first virtual warehouse;

[0156] The method further comprises:

[0157] Through the first routing server, in response to receiving a routing acquisition request initiated by a client for a target virtual warehouse, target node information is acquired in target routing information corresponding to the target virtual warehouse, and the target node information is sent to the client;

[0158] Through the client, based on the target node information, the first operation request is sent to a first target distribution node indicated by the target node information, so that the first target distribution node distributes the first operation request.

[0159] The description of the method example can refer to the system example, which is not repeated here.

[0160] Therefore, through the first routing server, routing information can be maintained for each of the first virtual warehouses respectively, so that each of the first virtual warehouses has an independent access portal, and mutual interference of portals is avoided.

[0161] In some embodiments, the method further comprises:

[0162] Through the first routing server, in response to the first target distribution node being in a target state, any other node information except the target node information is acquired from the target routing information, and the other node information is sent to the client, so that the client accesses the first virtual warehouse through other nodes indicated by the other node information.

[0163] The description of the method example can refer to the system example, which is not repeated here.

[0164] Therefore, in the case that the first target distribution node is in a target state, the first routing server can provide node information of other nodes to the client to ensure that the client can still establish a connection with the virtual warehouse and normally access data, and the service high availability of the virtual warehouse system is realized. In addition, the process of switching the distribution node can be completed through the client and the first routing server, and the user has no perception, and the user experience can be improved.

[0165] In some embodiments, the method further comprises:

[0166] The first routing service end, in response to an increase in resource groups of any first virtual warehouse, adds node information of a first distribution node included in the increased resource groups to routing information corresponding to the first virtual warehouse.

[0167] The description of the method example can refer to the system example, which is not repeated here.

[0168] Thus, the first routing service end can realize automatic updating of the routing information of the virtual warehouse, and keep the routing information up to date.

[0169] In some embodiments, the first routing service end provides a routing registration service, and each routing information is stored in a directory corresponding to the routing registration service; the routing registration service includes a zookeeper service. Thus, the performance of the virtual warehouse system can be improved by using the simple operation, high performance and stable function of the zookeeper.

[0170] In some embodiments, the virtual warehouse system includes at least one second virtual warehouse; wherein the second virtual warehouse includes a second distribution node and a plurality of node groups divided in advance; each node group includes at least one second computing node;

[0171] The method further comprises:

[0172] The second distribution node receives a second operation request for the second virtual warehouse, the second operation request including a group identifier, and in response to receiving the second operation request, distributes the second operation request to a second target computing node; the second target computing node is a computing node in a target node group indicated by the group identifier;

[0173] The second target computing node executes the received second operation request.

[0174] The description of the method example can refer to the system example, which is not repeated here.

[0175] Thus, by introducing the session-level group identifier parameter, the operation request of the client can be linked to the corresponding node group without modifying the client configuration information, forming mutual isolation between node groups (computing resources), avoiding mutual interference between node groups, and thus for the online business, the isolation of computing resources can also be quickly and conveniently realized, without affecting the customer experience.

[0176] In some embodiments, the virtual warehouse system is built based on an Impala cluster, the first distribution node and the second distribution node comprise coordinator nodes, and the first computing node and the second computing node comprise executor nodes. In this way, the advantage of fast Impala query speed can be utilized to improve the data processing efficiency of the virtual warehouse system.

[0177] The application also provides an operation method applied to a first distribution node in a virtual warehouse system. The virtual warehouse system comprises a plurality of first virtual warehouses; wherein the first virtual warehouse comprises a pre-divided resource group; the resource group comprises a first distribution node and a first computing node; and at least part of the first computing nodes correspond to the resource groups comprised by at least two first virtual warehouses.

[0178] The method can comprise:

[0179] In response to receiving a first operation request for the first virtual warehouse corresponding to the first distribution node, the first operation request is distributed to a first target computing node corresponding to the same resource group as the first distribution node, so that the first target computing node executes the received first operation request.

[0180] In some embodiments, the distribution of the first operation request to the first target computing node comprises:

[0181] Periodically acquiring the load of the first target computing node;

[0182] The first operation request is distributed to the first target computing node whose load does not reach a preset threshold.

[0183] In some embodiments, the virtual warehouse system is built based on an Impala cluster, the first distribution node comprises a coordinator node, and the first computing node comprises an executor node.

[0184] In the foregoing technical solutions, first, the resource group is used as an intermediary to associate the virtual warehouse with the distribution node and the computing node (computing resource), and when the virtual warehouse needs to change the computing resource, only the resource group associated with the virtual warehouse needs to be changed, so that the flexible allocation of the computing resource can be realized.

[0185] Secondly, after receiving the operation request, the first distribution node distributes the operation request to the computing node corresponding to the same resource group as itself, so that the computing node can only receive the operation request from the distribution node belonging to the same resource group as itself, and the mutual interference between the computing nodes of different resource groups can be avoided.

[0186] Thirdly, grouping the resources included in the resource groups corresponding to at least part of the computing nodes to at least two first virtual warehouses, the at least part of the computing nodes can execute operation requests for the at least two virtual warehouses, that is, the at least part of the computing nodes can perform operations for the at least two virtual warehouses, so as to achieve the purpose of sharing computing resources in virtual warehouses. Compared with the related art, the method of the present disclosure can share the computing nodes included in the virtual warehouse which has less operation requests and performs operations in a concentrated time, thereby avoiding the situation that the computing nodes are often in an idle state, and improving the resource utilization.

[0187] The present application also provides an operation method applied to a first routing server in a virtual warehouse system. The virtual warehouse system includes a plurality of first virtual warehouses; wherein the first virtual warehouses include pre-divided resource groups; the resource groups include first distribution nodes and first computing nodes; at least part of the first computing nodes correspond to resource groups included in at least two first virtual warehouses. The virtual warehouse system further includes a first routing server; the first routing server includes routing information corresponding to each of the first virtual warehouses respectively; the routing information includes node information of each of the first distribution nodes included in the first virtual warehouses.

[0188] The method can include:

[0189] In response to receiving a routing acquisition request initiated by a client for a target virtual warehouse, obtaining target node information in target routing information corresponding to the target virtual warehouse, and sending the target node information to the client, so that the client sends the first operation request to the first target distribution node indicated by the target node information based on the target node information.

[0190] In some embodiments, the method further includes:

[0191] Through the first routing server, in response to the first target distribution node being in a target state, obtaining any other node information except the target node information from the target routing information, and sending the other node information to the client, so that the client accesses the first virtual warehouse through other nodes indicated by the other node information.

[0192] In some embodiments, the method further includes:

[0193] In response to adding a resource group to any first virtual warehouse, adding node information of the first distribution node included in the added resource group to the routing information corresponding to the first virtual warehouse.

[0194] In the foregoing scheme, first, by the first routing service end, routing information can be maintained for each first virtual warehouse, so that each first virtual warehouse has an independent access portal, avoiding mutual interference of portals.

[0195] Second, in the case that the first target distribution node is in a target state, the first routing service end can provide node information of other nodes to the client to ensure that the client can still establish a connection with the virtual warehouse, normally access data, and implement high availability of virtual warehouse system services. In addition, the process of switching the distribution node can be completed through the client and the first routing service end, and the user has no perception, which can improve user experience.

[0196] Third, the first routing service end can realize automatic updating of the routing information of the virtual warehouse, and keep the routing information in the latest state.

[0197] The application also proposes an operation method applied to a second distribution node in a virtual warehouse system. The virtual warehouse system includes at least one second virtual warehouse; wherein the second virtual warehouse includes a second distribution node and a plurality of node groups divided in advance; each node group includes at least one second computing node. The method can include:

[0198] Receiving a second operation request for the second virtual warehouse, the second operation request including a group identifier, in response to receiving the second operation request, distributing the second operation request to a second target computing node, the second target computing node being a computing node in a target node group indicated by the group identifier, so that the second target computing node executes the received second operation request.

[0199] In some embodiments, the virtual warehouse system is based on an Impala cluster, the second distribution node includes a coordinator node, and the second computing node includes an executor node.

[0200] In the foregoing scheme, by introducing a session-level group identifier parameter, without modifying the client configuration information, the operation request of the client can be linked to the corresponding node group, forming mutual isolation between node groups (computing resources), avoiding mutual interference between node groups, so that for the business that has been online, the isolation of computing resources can also be quickly and conveniently realized, without affecting the user experience.

[0201] FIG. 6

[0202] After introducing the method and device of the exemplary embodiments of the present disclosure, next, reference is made to FIG. 6A readable storage medium of an example embodiment of the present disclosure is described. The storage medium stores a computer program for causing a processor to perform the operation method shown in any of the preceding embodiments.

[0203] See FIG. 6 , EXEMPLARY ELECTRONIC DEVICE A program product 60 applied to the operation method shown in an example embodiment of the present disclosure.

[0204] In some embodiments shown, the operation method shown in any of the preceding embodiments can be implemented by the program product 60, for example, a portable compact disc read-only memory (CD-ROM) including program codes and can be run on a device, for example, a personal computer. However, the program product of the present disclosure is not limited to this, and in this document, the readable storage medium can be any tangible medium containing or storing a program which can be used by or in conjunction with an instruction execution system, apparatus or device.

[0205] The program product can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium may, for example, be but is not limited to an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include an electrical connection having one or more wires, a portable disc, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0206] The computer readable signal medium can include a data signal propagated in a baseband or as part of a carrier wave, in which readable program codes are borne. Such a propagated data signal can take on multiple forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The readable signal medium can also be any readable medium other than the readable storage medium, which can send, propagate or transmit the program for use by or in conjunction with an instruction execution system, apparatus or device.

[0207] The program codes contained on the readable medium can be transmitted by any suitable medium, including but not limited to wireless, wired, optical cable, RE, etc., or any suitable combination of the above.

[0208] The program code can be executed by one or more programmable processors, digital signal processors, ASICs, FPGAs, or combinations thereof. The program code can execute entirely on the user's electronic device, partly on the user's electronic device, as a stand-alone software package, partly on the user's electronic device and partly on a remote electronic device or entirely on the remote electronic device or server. In the latter scenario, the remote electronic device can be connected to the user's electronic device through any kind of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external electronic device (for example, through the Internet using an Internet Service Provider).

[0209] FIG. 7

[0210] After introducing the method, device and medium of the exemplary embodiments of the present disclosure, next, reference is made to FIG. 7 An electronic device according to an exemplary embodiment of the present disclosure is described. The device includes a processor, a memory for storing processor-executable instructions, wherein the processor implements the operation method shown in any of the preceding embodiments by running the executable instructions.

[0211] Please refer to FIG. 7 , FIG. 7 A structural schematic diagram of an electronic device according to an exemplary embodiment of the present disclosure is shown.

[0212] FIG. 7 The electronic device 700 shown is merely an example, and should not bring any limitation to the functions and use range of the embodiments of the present disclosure.

[0213] As shown in FIG. 7 , the electronic device 700 is in the form of a general electronic device. The components of the electronic device 700 can include, but are not limited to, the aforementioned at least one processor 701, the aforementioned at least one memory 702, and a bus 703 connecting different system components, including the processor 701 and the memory 702.

[0214] The bus 703 includes a data bus, a control bus, and an address bus.

[0215] The memory 702 can include a readable medium in the form of a volatile memory, such as a random access memory (RAM) 7021 and / or a cache memory 7022, and can further include a readable medium in the form of a non-volatile memory, such as a read-only memory (ROM) 7023.

[0216] The memory processor 702 also includes a program / utility 7025 having a set (at least one) of program modules 7024, including an operating system, one or more application programs, other program modules, and program data, each of which

[0217] The electronic device 700 can also communicate with one or more external devices 704 such as a keyboard or a pointing device, among others.

[0218] Such communication can occur via Input / Output (I / O) interfaces 705. Still yet, the electronic device 700 can communicate with one or more networks, such as a local area network (LAN), a wide area network (WAN), and / or the Internet, through a network adapter 706. As ​ illustrated, the network adapter 706 is communicatively coupled to the other components of the electronic device 700 through a bus 703. It should be appreciated that the

[0219] It should be noted that although several units / modules or sub-units / modules of the text processing apparatus are mentioned in the above detailed description, such division is merely exemplary and not mandatory. Indeed, according to embodiments of the present disclosure, features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, features and functions of one unit / module described above can be further divided into units / modules for embodiment.

[0220] Moreover, while operations of the methods of the present disclosure are described in a particular order in the drawings, this is not required or implied, and the desired results can be achieved without performing all of the operations shown, or performing the operations in the particular order shown. Additionally or alternatively, certain steps can be omitted, combined into a single step, and / or split into multiple steps.

[0221] While the spirit and principles of the present disclosure have been described with reference to several specific embodiments, it is to be understood that the present disclosure is not limited to the specific embodiments disclosed and that the division of aspects is not meant to imply that features from one aspect cannot be combined with features from another aspect to advantage, but is merely for convenience of description. The present disclosure is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the claims appended hereto.

Claims

1. A virtual warehouse system, characterized by The virtual warehouse system comprises a plurality of first virtual warehouses with distinguished entrances and at least one second virtual warehouse without distinguished entrances; wherein the first virtual warehouses comprise pre-divided computing resource groups; the computing resource groups comprise first distribution nodes and first computing nodes; at least part of the first computing nodes correspond to computing resource groups comprised by at least two first virtual warehouses; the second virtual warehouse comprises second distribution nodes and pre-divided node groups; each node group comprises at least one second computing node; The first distribution node is configured to distribute the first operation request to a first target computing node in response to receiving the first operation request for the first virtual warehouse corresponding to the first distribution node; the first target computing node corresponds to the same computing resource group as the first distribution node; The first target computing node is configured to execute the received first operation request; The second distribution node is configured to receive a second operation request for the second virtual warehouse, the second operation request comprising a group identifier, and distribute the second operation request to a second target computing node in response to receiving the second operation request; the second target computing node is a computing node in a target node group indicated by the group identifier; The second target computing node is configured to execute the received second operation request.

2. The system of claim 1, wherein, The first distribution node is configured to periodically acquire a load amount of the first target computing node, and distribute the first operation request to a first target computing node whose load amount does not reach a preset threshold in response to receiving the first operation request.

3. The system of claim 1, wherein, The virtual warehouse system further comprises a first routing server; the first routing server comprises routing information corresponding to each first virtual warehouse respectively; the routing information comprises node information of each first distribution node comprised by the first virtual warehouse; The first routing server is configured to, in response to receiving a routing acquisition request initiated by a client for a target virtual warehouse, acquire target node information in target routing information corresponding to the target virtual warehouse, and send the target node information to the client; The client is configured to, based on the target node information, send the first operation request to a first target distribution node indicated by the target node information, so that the first target distribution node distributes the first operation request.

4. The system of claim 3, wherein, The first routing server is configured to, in response to the first target distribution node being in a target state, acquire any other node information except the target node information from the target routing information, and send the other node information to the client, so that the client accesses the first virtual warehouse through other nodes indicated by the other node information.

5. The system of claim 3, wherein, The first routing server is configured to, in response to any first virtual warehouse adding a resource group, add node information of a first distribution node comprised by the added resource group to routing information corresponding to the first virtual warehouse.

6. The system according to any of claims 3-5, characterized in that, The first routing server provides a routing registration service, and each routing information is stored in a directory corresponding to the routing registration service. The routing registration service comprises a zookeeper service.

7. The system of claim 1, wherein, The virtual warehouse system is constructed based on an Impala cluster, the first distribution node and the second distribution node comprise a coordinator node, and the first computing node and the second computing node comprise an executor node.

8. An operation method applied to a virtual warehouse system, characterized by, The virtual warehouse system comprises a plurality of first virtual warehouses with distinguished entrances and at least one second virtual warehouse without distinguished entrances; wherein the first virtual warehouse comprises a pre-divided computing resource group; the computing resource group comprises a first distribution node and a first computing node; at least part of the first computing nodes in the first computing node correspond to the computing resource groups comprised by at least two first virtual warehouses; the second virtual warehouse comprises a second distribution node and a plurality of pre-divided node groups; each node group comprises at least one second computing node. The method comprises: Through the first distribution node, in response to receiving a first operation request for the first virtual warehouse corresponding to the first distribution node, the first operation request is distributed to a first target computing node; the first target computing node and the first distribution node correspond to the same computing resource group; Through the first target computing node, the received first operation request is executed; Through the second distribution node, a second operation request for the second virtual warehouse is received, the second operation request comprises a group identifier, in response to receiving the second operation request, the second operation request is distributed to a second target computing node; the second target computing node is a computing node in a target node group indicated by the group identifier; Through the second target computing node, the received second operation request is executed.

9. The method of claim 8, wherein, The method further comprises: Through the first distribution node, the load of the first target computing node is periodically acquired; The first operation request is distributed to the first target computing node, comprising: The first operation request is distributed to the first target computing node whose load does not reach a preset threshold.

10. The method of claim 8, wherein, The virtual warehouse system further comprises a first routing server; the first routing server comprises routing information corresponding to each first virtual warehouse respectively; The routing information comprises node information of each first distribution node contained in the first virtual warehouse; The method further comprises: Through the first routing server, in response to receiving a routing acquisition request initiated by a client for a target virtual warehouse, target node information is acquired in target routing information corresponding to the target virtual warehouse, and the target node information is sent to the client; Through the client, based on the target node information, the first operation request is sent to a first target distribution node indicated by the target node information, so that the first target distribution node distributes the first operation request.

11. The method of claim 10, wherein, The method further comprises: The first routing server acquires any other node information except the target node information from the target routing information and sends the other node information to the client in response to the first target distribution node being in a target state, so that the client accesses the first virtual warehouse through other nodes indicated by the other node information.

12. The method of claim 10, wherein, The method further comprises: The first routing server provides a routing registration service, and each routing information is stored in a directory corresponding to the routing registration service.

13. The method according to any of claims 10 to 12, characterized in that, The routing registration service comprises a zookeeper service. The virtual warehouse system is constructed based on an Impala cluster, and the first distribution node and the second distribution node comprise coordinator nodes, and the first computing node and the second computing node comprise executor nodes.

14. The method of claim 8, wherein, The device comprises:

15. An electronic device, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor implements the operation method of any one of claims 8-14 by running the executable instructions. The storage medium stores a computer program for causing the processor to execute the operation method of any one of claims 8-14.

16. A computer readable storage medium characterized by: ​

Images