Data exploration system, method and device and storage medium

Abstract

The invention discloses a data exploration system, method and device and a storage medium, and relates to the technical field of operation and maintaining.The system comprises a management control module used for sending a first path instruction to a routing function module when it is detected that idle resource instances exist in a cluster; the routing function module is used for establishing a communication link between the newly-added probing request and the idle resource instance so as to respond to the newly-added probing request through the idle resource instance; the parameter management module is used for sending the first performance parameter to the resource management module; and the resource management module is used for executing a resource instance capacity expansion operation or a resource instance capacity reduction operation according to the first performance parameter. According to the technical scheme provided by the embodiment of the invention, the data exploration efficiency is improved, the pre-configuration of the resource instance is avoided, and meanwhile, the large-scale exploration requirement is met, the long-time occupation of the node resource is avoided and the utilization rate of the node resource is improved through the capacity expansion and shrinkage operation of the resource instance.

Description

Technical Field

[0001] The present invention relates to the field of operation and maintenance technology, and in particular to a data exploration system, method, device and storage medium. Background Art

[0002] For cloud computing infrastructure, the exploration of performance data, operation data, and fault data in the equipment cluster is the primary prerequisite for ensuring its safe operation.

[0003] In the prior art, in order to ensure effective exploration of various data in the cluster, a fixed working node is usually pre-selected in the cluster, and the working node is specifically used to perform exploration operations. At the same time, a fixed resource instance is pre-set in the working node, and the resource instance is specifically used to provide corresponding system resources for the exploration request, thereby realizing the exploration of various data in the cluster.

[0004] However, this data exploration method not only has low exploration efficiency and cannot adapt to large-scale exploration needs, but also requires the pre-configuration of a large number of resource instances, which occupies too much system resources of the working nodes. Summary of the invention

[0005] The present invention provides a data exploration system, method, device and storage medium to solve the problem that data exploration efficiency is low and more node resources are occupied.

[0006] According to one aspect of the present invention, there is provided a data exploration system, which is applied to a management node in a cluster, and includes: a management control module, a routing function module, a parameter management module and a resource management module;

[0007] The management control module is connected to the routing function module and is used to issue a first path instruction to the routing function module when detecting that there is an idle resource instance in the cluster;

[0008] The routing function module is used to establish a communication link between the newly added probe request and the idle resource instance according to the first path instruction, so as to respond to the newly added probe request through the idle resource instance;

[0009] The parameter management module is connected to the resource management module and is used to obtain a first performance parameter and send the first performance parameter to the resource management module;

[0010] The resource management module is used to perform a resource instance expansion operation or a resource instance reduction operation according to the first performance parameter.

[0011] The management control module is also used to send a second path instruction to the routing function module when it is detected that there is no idle resource instance in the cluster; the routing function module is also used to establish a communication link between the newly added probe request and the parameter management module according to the second path instruction; the parameter management module is also used to store the acquired newly added probe request, and send the second performance parameter to the resource management module to perform the resource instance expansion operation through the resource management module, and send the stored newly added probe request to the newly added resource instance to respond to the stored newly added probe request through the newly added resource instance. In this way, the expansion operation of the resource instance is realized, which not only ensures the orderly access of the cluster to the probe request, avoids the loss of the probe request, but also ensures the timely response to the probe request.

[0012] The parameter management module is further configured to send the third performance parameter to the resource management module if no new probe request is detected by the management control module within the first preset time, so as to perform resource instance scaling processing through the resource management module. The resource management module performs the scaling operation of the resource instance according to the third performance parameter, thereby avoiding the existence of redundant resource instances in the cluster, saving computer resources of the cluster, and improving the utilization rate of node resources.

[0013] The routing function module is specifically used to establish a communication link between a newly added probe request of the first target type and a matching idle resource instance according to the first path instruction; and / or to establish a communication link between a newly added probe request of the second target type and the parameter management module according to the second path instruction. The data exploration system not only enables the current cluster to meet the access requirements of multiple types of resource instances, but also realizes the classified management and classified transmission of different types of probe requests by establishing different communication links, namely, the communication link between the probe interface and the idle resource instance, and the communication link between the probe interface and the parameter management module, which ensures the rapid response of the probe request for the idle resource instance and avoids the loss of the probe request for the non-idle resource instance.

[0014] The parameter management module is also used to send the current resource occupancy rate to the resource management module if it is detected that the resource occupancy rate of the first resource instance reaches the current resource allocation threshold, so as to expand the capacity of the first resource instance through the resource management module until the resource occupancy rate of the first resource instance reaches the preset resource allocation threshold. Thus, by expanding the capacity of the first resource instance itself and configuring the preset resource allocation threshold, not only the expansion processing of the resource instance itself is realized, the timely response of the resource instance to the complex exploration request task is ensured, and the stable operation of the first resource instance is ensured.

[0015] The data exploration system also includes an exploration parsing module; the exploration parsing module is connected to the management control module, and is used to receive a newly added exploration request, parse and obtain the resource instance type corresponding to the newly added exploration request, and send the newly added exploration request to the parameter management module or the idle resource instance according to the communication link matching the resource instance type. When different types of exploration requests enter the management node through the same exploration interface, the data exploration system implements the parsing and acquisition of the exploration request type, which not only meets the access requirements of multiple types of resource instances, but also ensures the classified management and classified transmission of different types of exploration requests.

[0016] The data exploration system also includes a host management module; the host management module is connected to the management control module, and is used to elect the exploration master node in the cluster through a consensus mechanism, and when the current management node is determined to be the exploration master node, guide the management control module to complete the startup, and guide the routing function module, parameter management module and resource management module to complete the startup through the management control module. In this way, when the current exploration master node fails, the exploration master node is replaced, ensuring the high availability of the data exploration system.

[0017] According to another aspect of the present invention, a data exploration method is provided, which is applied to the data exploration system described in any embodiment of the present invention, comprising:

[0018] When the management control module detects that there is an idle resource instance in the cluster, the management control module sends a first path instruction to the routing function module;

[0019] The routing function module establishes a communication link between the newly added probe request and the idle resource instance according to the first path instruction, so as to respond to the newly added probe request through the idle resource instance;

[0020] The parameter management module obtains a first performance parameter and sends the first performance parameter to the resource management module;

[0021] The resource management module executes a resource instance expansion operation or a resource instance reduction operation according to the first performance parameter.

[0022] According to another aspect of the present invention, there is provided a data exploration device, which is applied to the data exploration system described in any embodiment of the present invention, comprising:

[0023] A path instruction issuing module, configured in the management control module, is used to issue a first path instruction to the routing function module when detecting that there is an idle resource instance in the cluster;

[0024] A communication link establishing module, configured in the routing function module, for establishing a communication link between the newly added probe request and the idle resource instance according to the first path instruction, so as to respond to the newly added probe request through the idle resource instance;

[0025] A performance parameter sending module, configured in the parameter management module, for acquiring a first performance parameter and sending the first performance parameter to the resource management module;

[0026] The resource instance execution module is configured in the resource management module and is used to execute a resource instance expansion operation or a resource instance reduction operation according to the first performance parameter.

[0027] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the data exploration method described in any embodiment of the present invention when executed.

[0028] The technical solution of the embodiment of the present invention is that when the management control module detects that there is an idle resource instance in the cluster, it sends a first path instruction to the routing function module, and the routing function module establishes a communication link between the newly added exploration request and the idle resource instance according to the first path instruction, so as to respond to the newly added exploration request through the idle resource instance; the parameter management module sends the first performance parameter to the resource management module, so as to perform a resource instance expansion operation or a resource instance reduction operation through the resource management module. The data exploration system not only improves the data exploration efficiency, but also avoids the pre-configuration of the resource instance. At the same time, through the expansion and reduction operations of the resource instance, it not only meets the large-scale exploration needs, but also avoids the long-term occupation of the node resources, thereby improving the utilization rate of the node resources.

[0029] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0031] Figure 1 is a structural diagram of a data exploration system provided according to Embodiment 1 of the present invention;

[0032] Figure 2is a structural diagram of another data exploration system provided according to the second embodiment of the present invention;

[0033] Figure 3A is a schematic diagram of the structure of another data exploration system provided according to Embodiment 3 of the present invention;

[0034] Figure 3B is a flow chart of a data exploration system performing data exploration according to the first application scenario of the present invention;

[0035] Figure 4 is a flow chart of a data exploration method provided according to a fourth embodiment of the present invention;

[0036] Figure 5 It is a structural diagram of a data exploration device provided according to Embodiment 5 of the present invention. DETAILED DESCRIPTION

[0037] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0038] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0039] Embodiment 1

[0040] Figure 1 A schematic diagram of the structure of a data exploration system provided in the first embodiment of the present invention is shown in FIG. Figure 1 As shown, the data exploration system includes a management control module 100 , a routing function module 200 , a parameter management module 300 and a resource management module 400 .

[0041] The management control module 100 is connected to the routing function module 200, and is used to issue a first path instruction to the routing function module 200 when an idle resource instance is detected in the cluster; the routing function module 200 is used to establish a communication link between the new probe request and the idle resource instance according to the first path instruction, so as to respond to the new probe request through the idle resource instance.

[0042] Specifically, a node refers to a specific computer device that is connected to a network with an independent address and has the function of transmitting or receiving data; a node can be a terminal device, or a server or other electronic device connected to the network; a management node is used to manage the operation and maintenance operations of each working node in the cluster; the number of management nodes and working nodes in the cluster can be one or more, and the management node itself can also carry the relevant functions of the working node; in the embodiments of the present invention, there is no specific limitation on the device types of the management nodes and working nodes in the cluster.

[0043] Resource instances are system resources used to run computing tasks in a computer, such as processes, containers, and virtual machines. Resource instances can be created in some or all nodes in a cluster, and one or more resource instances can be created in each node. For a cluster, the types of resource instances it provides to the outside world when executing exploration tasks vary depending on the business functions it carries. Therefore, after the cluster configuration is completed, the cluster can only provide fixed types of resource instances to the outside world. For example, the current cluster only provides processes as resource instances to respond to exploration requests.

[0044] The management control module 100 is used to monitor the usage of resource instances of each node in the cluster. If there are idle resource instances in the cluster, the management control module 100 sends a first path instruction to the routing function module 200; the first path instruction is used to instruct the routing function module 200 to establish a communication link directly between the probe request subsequently obtained by the management node, that is, the newly added probe request, and the idle resource instance in the cluster; for example, the management node can obtain the probe request issued by the external device through a designated probe interface, and the routing function module 200 establishes a communication link directly connected between the probe interface and the idle resource instance, so that the probe interface can transmit the newly added probe request subsequently obtained directly to the designated idle resource instance through the communication link.

[0045] At the same time, if the number of idle resource instances is large, the routing function module 200 can also establish multiple communication links between the detection interface and each idle resource instance, so that each new detection request subsequently obtained by the detection interface can reach the specified resource instance in sequence according to the construction order of each communication link; the idle resource instance in the embodiment of the present invention can be a completely unused resource instance, or it can be a resource instance that has been used but has a low resource occupancy rate; for example, if the central processing unit (CPU) occupancy rate of a resource instance is low, that is, the CPU occupancy rate is less than or equal to a preset usage threshold, while the resource instance responds to the original detection request, it also has the ability to continue to respond to other new detection requests, then the resource instance can also be used as an idle resource instance.

[0046] The parameter management module 300 is connected to the resource management module 400, and is used to obtain a first performance parameter and send the first performance parameter to the resource management module 400; the resource management module 400 is used to perform a resource instance expansion operation or a resource instance reduction operation according to the first performance parameter. The first performance parameter may include the resource occupancy rate of each resource instance in the cluster, such as the CPU occupancy rate and the memory occupancy rate, and may also include the number of idle resource instances, the number of non-idle resource instances, and the number of probe requests being processed in the cluster; the parameter management module 300 may obtain the above performance parameters by directly accessing each resource instance.

[0047] When the parameter management module 300 determines that there are idle resource instances, it can send the first performance parameter to the resource management module 400 in a periodic manner according to a preset detection cycle; after obtaining the first performance parameter, the resource management module 400 determines whether to perform an expansion operation or a reduction operation based on the usage status of each resource instance; for example, if the number of idle resource instances is small and the total number of resource instances is small, the expansion operation can be performed accordingly to ensure timely response to the probe request; if the number of idle resource instances is large or the total number of resource instances is large, the reduction operation can be performed accordingly to save computer resources of the cluster.

[0048] Optionally, in an embodiment of the present invention, the management control module 100 is further used to issue a second path instruction to the routing function module 200 when it is detected that there are no idle resource instances in the cluster; the routing function module 200 is further used to establish a communication link between the new probe request and the parameter management module 300 according to the second path instruction; the parameter management module 300 is further used to store the acquired new probe request and send the second performance parameter to the resource management module 400 so as to perform the resource instance expansion operation through the resource management module 400, and send the stored new probe request to the new resource instance so as to respond to the stored new probe request through the new resource instance.

[0049] Specifically, if the management control module 100 detects that there are no idle resource instances in the cluster, it sends a second path instruction to the routing function module 200; the second path instruction is used to instruct the routing function module 200 to establish a communication link between the probe request subsequently obtained by the management node and the parameter management module 300; taking the above technical solution as an example, the routing function module 200 establishes a communication link conducted by the probe interface and the parameter management module 300, so that the probe interface can transmit the newly obtained new probe request directly to the parameter management module 300 through the communication link. When the parameter management module 300 obtains the new probe request, it first stores the new probe request locally, and then sends the second performance parameter to the resource management module 400.

[0050] In addition to the parameter items in the first performance parameter, the second performance parameter also includes the number of unexecuted probe requests stored in the parameter management module 300, that is, the number of accumulated probe requests. The resource management module 400 performs a capacity expansion operation according to the acquired second performance parameter to create a new resource instance; after detecting that the above-mentioned capacity expansion operation is completed, the parameter management module 300 sends the locally stored probe request to the newly added resource instance, thereby implementing the capacity expansion operation of the resource instance, ensuring the orderly access of the cluster to the probe request, avoiding the loss of the probe request, and ensuring the timely response to the probe request.

[0051] Optionally, in an embodiment of the present invention, the parameter management module 300 is also used to send a third performance parameter to the resource management module 400 if no new probe request is detected by the management control module within the first preset time, so as to perform resource instance scaling through the resource management module 400. Specifically, if the parameter management module 300 does not detect that a new probe request is received in the probe interface through the management control module 100 within a long period of time, the third performance parameter is sent to the resource management module 400; wherein, in addition to the parameter items in the first performance parameter, the third performance parameter also includes a specific value of the first preset time; the resource management module 400 performs the scaling operation of the resource instance according to the third performance parameter, thereby avoiding the existence of redundant resource instances in the cluster, saving the computer resources of the cluster, and improving the utilization rate of the node resources.

[0052] Optionally, in an embodiment of the present invention, the routing function module 200 is specifically used to establish a communication link between a new probe request of a first target type and a matching idle resource instance according to the first path instruction; and / or to establish a communication link between a new probe request of a second target type and the parameter management module 300 according to the second path instruction.

[0053] Specifically, the cluster can also provide multiple types of resource instances to the outside. For example, the cluster can simultaneously provide processes, containers, and virtual machines as resource instances that respond to probe requests to meet the probe needs of different parties; the management node can obtain different types of probe requests through different probe interfaces, and different types of probe requests require access to different types of resource instances. For example, when there are idle processes but no idle containers in the cluster, for probe requests to access processes (i.e., newly added probe requests of the first target type), a probe interface corresponding to this type of probe request can be established, and a communication link can be established between the idle process; for probe requests to access containers (i.e., newly added probe requests of the second target type), a probe interface corresponding to this type of probe request can be established, and a communication link can be established between the parameter management module 300.

[0054] Therefore, the data exploration system not only enables the current cluster to meet the access requirements of multiple types of resource instances, but also realizes the classified management and classified transmission of different types of exploration requests by establishing different communication links, namely the communication link between the exploration interface and the idle resource instance, and the communication link between the exploration interface and the parameter management module 300, thereby ensuring a rapid response to the exploration request for the idle resource instance and avoiding the loss of the exploration request for the non-idle resource instance.

[0055] Optionally, in an embodiment of the present invention, the parameter management module 300 is also used to send the current resource occupancy rate to the resource management module 400 if it is detected that the resource occupancy rate of the first resource instance reaches the current resource allocation threshold, so as to expand the capacity of the first resource instance through the resource management module 400 until the resource occupancy rate of the first resource instance reaches the preset resource allocation threshold.

[0056] Specifically, the node allocates different resource thresholds to each of its own resource instances. When the parameter management module 300 detects that the resource occupancy rate of the first resource instance reaches the current resource allocation threshold, it indicates that the exploration request task executed in the resource instance is relatively complex and requires more node resources to provide task support for it. At this time, the first resource instance itself is expanded, that is, the resource threshold allocated to the first resource instance is increased; and the preset resource allocation threshold is the upper limit threshold of the resource occupancy rate corresponding to the first resource instance. If the resource occupancy rate of the first resource instance reaches the preset resource allocation threshold, the first resource instance will no longer be expanded.

[0057] If the resource occupancy rate of the first resource instance is too high, it will not only affect the operation of other resource instances in the node, but also affect the effective operation of the business services provided by the node itself, and will also cause the operation of the first resource instance itself to be unstable, for example, running slowly, etc. Therefore, by expanding the capacity of the first resource instance itself and configuring the preset resource allocation threshold, not only the expansion processing of the resource instance itself is achieved, ensuring the timely response of the resource instance to complex probe request tasks, but also ensuring the stable operation of the first resource instance.

[0058] The technical solution of the embodiment of the present invention is that when the management control module detects that there is an idle resource instance in the cluster, it sends a first path instruction to the routing function module, and the routing function module establishes a communication link between the newly added exploration request and the idle resource instance according to the first path instruction, so as to respond to the newly added exploration request through the idle resource instance; the parameter management module sends the first performance parameter to the resource management module, so as to perform a resource instance expansion operation or a resource instance reduction operation through the resource management module. The data exploration system not only improves the data exploration efficiency, but also avoids the pre-configuration of the resource instance. At the same time, through the expansion and reduction operations of the resource instance, it not only meets the large-scale exploration needs, but also avoids the long-term occupation of the node resources, thereby improving the utilization rate of the node resources.

[0059] Embodiment 2

[0060] Figure 2 A structural diagram of a data exploration system provided in Embodiment 2 of the present invention is shown in FIG. Figure 2As shown, the data exploration system also includes a host management module 500; the host management module 500 is connected to the management control module 100, and is used to elect an exploration master node in the cluster through a consensus mechanism, and when it is determined that the current management node is the exploration master node, guide the management control module 100 to complete the startup, and guide the routing function module 200, the parameter management module 300 and the resource management module 400 to complete the startup through the management control module 100.

[0061] Specifically, the consensus mechanism is an algorithm for reaching distributed consensus on blockchain transactions. In an embodiment of the present invention, each node elects one or more management nodes as the exploration master node of the data exploration system through voting through the consensus mechanism; wherein the consensus mechanism may include blockchain-related proof of work, proof of stake, and distributed consensus algorithm (RAFT), etc.

[0062] Taking the consensus mechanism as proof of stake as an example, the remaining resources of the exploration master node are regarded as the "amount of currency" held, and the continuous idle time of the remaining resources is regarded as "time". The product of "amount of currency" and "time" is the "coin age". For example, the number of remaining CPU cores is 10, and the idle time of each core is 60 minutes, then the corresponding "coin age" is 600. When the remaining resources change, the corresponding resources are deducted for recalculation. If the remaining resources decrease, the "coin age" is doubled. Finally, one or more management nodes are selected as the providers of the exploration service according to the sorting algorithm, that is, the exploration master node.

[0063] If the host management module 500 determines that the current management node is elected as the exploration master node, it first guides the management control module 100 to complete the startup, and guides the routing function module 200, the parameter management module 300 and the resource management module 400 to complete the startup through the management control module 100, thereby completing the replacement of the exploration master node; in particular, the host management module 500 will also search for other available exploration master nodes in the cluster area, and copy and install the current data exploration system to the newly discovered exploration master node. For example, if the current management node has a communication failure and cannot process the exploration request normally, or based on the above consensus mechanism, the elected When other management nodes replace the current management node as the exploration master node, the host management module 500 will copy the complete data exploration system, including the above-mentioned management control module 100, routing function module 200, parameter management module 300, resource management module 400 and the host management module 500 itself to the new exploration master node, and complete the installation of the data exploration system in the new exploration master node; at the same time, after the installation is completed, the host management module 500 in the new exploration master node is guided to complete the startup, so as to realize the replacement of the exploration master node, and realize the transfer of the exploration master node under communication failure under the premise of achieving high availability of the exploration system.

[0064] According to the technical solution of the embodiment of the present invention, the host management module elects the exploration master node in the cluster through a consensus mechanism, and when determining that the current management node is the exploration master node, guides the management control module to complete the startup, and guides the routing function module, the parameter management module and the resource management module to complete the startup through the management control module, thereby realizing the replacement of the exploration master node when the current exploration master node encounters a failure such as downtime, thereby ensuring the high availability of the data exploration system.

[0065] Embodiment 3

[0066] Figure 3A A structural diagram of a data exploration system provided in Embodiment 3 of the present invention is shown in FIG. Figure 3A As shown, the data exploration system also includes an exploration parsing module 600; the exploration parsing module 600 is connected to the management control module 100, and is used to receive a new exploration request, and parse and obtain the resource instance type corresponding to the new exploration request, and send the new exploration request to the parameter management module 300 or the idle resource instance according to the communication link matching the resource instance type.

[0067] Specifically, when the management node obtains different types of probe requests through a probe interface, since the resource instance types required for each probe request are different, if there are different types of idle resource instances, it is obviously necessary to establish communication links between the different types of probe requests and the corresponding idle resource instances respectively. Therefore, a probe parsing module 600 connected to the probe interface can also be configured in the data probe system to obtain the probe request from the probe interface and parse it to obtain the resource instance type corresponding to the probe request.

[0068] The other side of the probe and analysis module 600 includes multiple transmission ports, and different transmission ports are used to transmit different types of probe requests. In particular, the type of probe request can be obtained by obtaining the identity information of the sender of the probe request, that is, different request senders correspond to different types of resource instances. It can also be obtained according to the specific type of data probed in the probe request, for example, performance data probe requests, operation data probe requests and fault data probe requests are respectively configured to be executed through containers, processes and virtual machines.

[0069] If there is a certain type of idle resource instance in the cluster, then the transmission port in the detection and analysis module 600 that matches the idle resource instance has established a communication link with the idle resource instance, and the detection and analysis module 600 sends a new detection request of the matching type to the idle resource instance through the transmission port; if there is no certain type of idle resource instance in the cluster, then the transmission port in the detection and analysis module 600 that matches the idle resource instance has established a communication link with the parameter management module 300, and the detection and analysis module 600 sends a new detection request of the matching type to the parameter management module 300 through the transmission port.

[0070] According to the technical solution of the embodiment of the present invention, the exploration and parsing module receives the newly added exploration request and parses to obtain the resource instance type corresponding to the newly added exploration request, and sends the newly added exploration request to the parameter management module or the idle resource instance according to the communication link matching the resource instance type, so that when different types of exploration requests enter the management node through the same exploration interface, the data exploration system realizes the analysis and acquisition of the exploration request type, which not only meets the access requirements of multiple types of resource instances, but also ensures the classified management and classified transmission of different types of exploration requests.

[0071] Application scenario 1

[0072] Figure 3B The flowchart of the data exploration system performing data exploration provided in the first application scenario of the present invention is as follows: Figure 3B As shown:

[0073] When the management control module detects that there are idle resource instances in the cluster, the management control module sends a first path instruction to the routing function module; the routing function module establishes a communication link between the new probe request and the idle resource instance according to the first path instruction, so as to respond to the new probe request through the idle resource instance; the routing function module sends a first notification to the parameter management module to notify the parameter management module that the above-mentioned communication link has been established; the parameter management module obtains a first performance parameter and sends the first performance parameter to the resource management module; the resource management module executes a resource instance expansion operation or a resource instance reduction operation according to the first performance parameter.

[0074] When the management control module detects that there are no idle resource instances in the cluster, the management control module sends a second path instruction to the routing function module; the routing function module establishes a communication link between the new probe request and the parameter management module according to the second path instruction; the routing function module sends a second notification to the parameter management module to notify the parameter management module that the above-mentioned communication link has been established; the parameter management module obtains the new probe request; the parameter management module stores the new probe request; the parameter management module obtains the second performance parameter and sends the second performance parameter to the resource management module; the resource management module executes the resource instance expansion operation according to the second performance parameter; the parameter management module sends the stored new probe request to the new resource instance to respond to the stored new probe request through the new resource instance.

[0075] If the management control module does not detect any new probe request within the first preset time, it will send a third notification to the parameter management module to notify the parameter management module that the data probe system is in an idle state; the parameter management module obtains the third performance parameter and sends the third performance parameter to the resource management module; the resource management module executes the resource instance scaling down operation according to the third performance parameter.

[0076] According to the technical solution of the embodiment of the present invention, when the data exploration system detects that there are idle resource instances in the cluster, the data exploration system responds to the new exploration request by establishing a communication link between the new exploration request and the idle resource instance, thereby improving the data exploration efficiency; when the data exploration system detects that there are no idle resource instances in the cluster, the data exploration system responds to the new exploration request based on the new resource instance generated by the resource instance expansion operation by establishing a communication link between the new exploration request and the parameter management module, thereby ensuring the orderly access of the cluster to the exploration request, avoiding the loss of the exploration request, and ensuring the timely response to the exploration request; when the data exploration system detects that there are no new exploration requests within the first preset time, the data exploration system executes the scaling down processing of the resource instance, thereby avoiding the existence of redundant resource instances in the cluster, saving the computer resources of the cluster, and improving the utilization rate of the node resources.

[0077] Embodiment 4

[0078] Figure 4This is a flowchart of a data exploration method provided in the fourth embodiment of the present invention. This embodiment is applicable to the case where a new exploration request is responded to through an idle resource instance. The method can be executed by a data exploration device. The data exploration device can be implemented in the form of hardware and / or software. The data exploration device can be configured in the data exploration system of any embodiment of the present invention. Figure 4 As shown, the method includes:

[0079] S401. When the management control module detects that there is an idle resource instance in the cluster, the management control module sends a first path instruction to the routing function module.

[0080] S402: The routing function module establishes a communication link between the newly added probe request and the idle resource instance according to the first path instruction, so as to respond to the newly added probe request through the idle resource instance.

[0081] S403: The parameter management module obtains a first performance parameter, and sends the first performance parameter to the resource management module.

[0082] S404: The resource management module executes a resource instance expansion operation or a resource instance reduction operation according to the first performance parameter.

[0083] The technical solution of the embodiment of the present invention is that when the management control module detects that there is an idle resource instance in the cluster, it sends a first path instruction to the routing function module, and the routing function module establishes a communication link between the newly added exploration request and the idle resource instance according to the first path instruction, so as to respond to the newly added exploration request through the idle resource instance; the parameter management module sends the first performance parameter to the resource management module, so as to perform a resource instance expansion operation or a resource instance reduction operation through the resource management module. The data exploration system not only improves the data exploration efficiency, but also avoids the pre-configuration of the resource instance. At the same time, through the expansion and reduction operations of the resource instance, it not only meets the large-scale exploration needs, but also avoids the long-term occupation of the node resources, thereby improving the utilization rate of the node resources.

[0084] Embodiment 5

[0085] Figure 5 : is a structural block diagram of a data exploration device provided by Embodiment 5 of the present invention, and the device specifically comprises:

[0086] The path instruction issuing module 501 is configured in the management control module and is used to issue a first path instruction to the routing function module when detecting that there is an idle resource instance in the cluster;

[0087] A communication link establishing module 502, configured in the routing function module, is used to establish a communication link between the newly added probe request and the idle resource instance according to the first path instruction, so as to respond to the newly added probe request through the idle resource instance;

[0088] The performance parameter sending module 503 is configured in the parameter management module, and is used to obtain a first performance parameter and send the first performance parameter to the resource management module;

[0089] The resource instance execution module 504 is configured in the resource management module, and is used to execute a resource instance expansion operation or a resource instance reduction operation according to the first performance parameter.

[0090] The technical solution of the embodiment of the present invention is that when the management control module detects that there is an idle resource instance in the cluster, it sends a first path instruction to the routing function module, and the routing function module establishes a communication link between the newly added exploration request and the idle resource instance according to the first path instruction, so as to respond to the newly added exploration request through the idle resource instance; the parameter management module sends the first performance parameter to the resource management module, so as to perform a resource instance expansion operation or a resource instance reduction operation through the resource management module. The data exploration system not only improves the data exploration efficiency, but also avoids the pre-configuration of the resource instance. At the same time, through the expansion and reduction operations of the resource instance, it not only meets the large-scale exploration needs, but also avoids the long-term occupation of the node resources, thereby improving the utilization rate of the node resources.

[0091] The above device can execute the data exploration method provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method. For technical details not described in detail in this embodiment, please refer to the data exploration method provided by any embodiment of the present invention.

[0092] Embodiment 6

[0093] In some embodiments, the data exploration method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit. In some embodiments, part or all of the computer program may be loaded and / or installed on a heterogeneous hardware accelerator via a ROM and / or a communication unit. When the computer program is loaded into RAM and executed by a processor, one or more steps of the data exploration method described above may be performed. Alternatively, in other embodiments, the processor may be configured to perform the data exploration method in any other appropriate manner (e.g., by means of firmware).

[0094] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0095] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0096] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0097] To provide interaction with a user, the systems and techniques described herein may be implemented on a heterogeneous hardware accelerator having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and pointing device (e.g., a mouse or trackball) through which a user can provide input to the heterogeneous hardware accelerator. Other types of devices may also be used to provide interaction with a user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0098] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0099] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0100] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0101] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A data exploration system, characterized in that: Applied to the management node in the cluster, including: management control module, routing function module, parameter management module and resource management module; The management control module is connected to the routing function module and is used to issue a first path instruction to the routing function module when detecting that there is an idle resource instance in the cluster; The routing function module is used to establish a communication link between the newly added probe request and the idle resource instance according to the first path instruction, so as to respond to the newly added probe request through the idle resource instance; The parameter management module is connected to the resource management module and is used to obtain a first performance parameter and send the first performance parameter to the resource management module; The resource management module is used to perform a resource instance expansion operation or a resource instance reduction operation according to the first performance parameter.

2. The data exploration system according to claim 1, characterized in that: The management control module is further configured to issue a second path instruction to the routing function module when detecting that there is no idle resource instance in the cluster; The routing function module is further used to establish a communication link between the newly added probe request and the parameter management module according to the second path instruction; The parameter management module is also used to store the acquired new probe request and send the second performance parameter to the resource management module so as to perform the resource instance expansion operation through the resource management module, and send the stored new probe request to the new resource instance so as to respond to the stored new probe request through the new resource instance.

3. The data exploration system according to claim 1, characterized in that: The parameter management module is also used to send the third performance parameter to the resource management module if no new probe request is detected by the management control module within the first preset time, so as to perform resource instance scaling processing through the resource management module.

4. The data exploration system according to claim 2, characterized in that: The routing function module is specifically used to establish a communication link between a new probe request of a first target type and a matching idle resource instance according to the first path instruction; and / or to establish a communication link between a new probe request of a second target type and the parameter management module according to the second path instruction.

5. The data exploration system according to claim 1, characterized in that: The parameter management module is also used to send the current resource occupancy rate to the resource management module if it detects that the resource occupancy rate of the first resource instance reaches the current resource allocation threshold, so as to expand the capacity of the first resource instance through the resource management module until the resource occupancy rate of the first resource instance reaches the preset resource allocation threshold.

6. The data exploration system according to claim 1, characterized in that: The data exploration system also includes an exploration and analysis module; The probe parsing module and the connection management control module are used to receive a new probe request, parse and obtain the resource instance type corresponding to the new probe request, and send the new probe request to the parameter management module or the idle resource instance according to the communication link matching the resource instance type.

7. The data exploration system according to claim 1, characterized in that: The data exploration system also includes a host management module; The host management module is connected to the management control module and is used to elect the exploration master node in the cluster through a consensus mechanism, and when the current management node is determined to be the exploration master node, guide the management control module to complete startup, and guide the routing function module, parameter management module and resource management module to complete startup through the management control module.

8. A data exploration method, characterized in that: The data exploration system used in any one of claims 1 to 7 comprises: When the management control module detects that there is an idle resource instance in the cluster, the management control module sends a first path instruction to the routing function module; The routing function module establishes a communication link between the newly added probe request and the idle resource instance according to the first path instruction, so as to respond to the newly added probe request through the idle resource instance; The parameter management module obtains a first performance parameter and sends the first performance parameter to the resource management module; The resource management module executes a resource instance expansion operation or a resource instance reduction operation according to the first performance parameter.

9. A data exploration device, characterized in that: The data exploration system used in any one of claims 1 to 7 comprises: A path instruction issuing module, configured in the management control module, is used to issue a first path instruction to the routing function module when detecting that there is an idle resource instance in the cluster; A communication link establishing module, configured in the routing function module, for establishing a communication link between the newly added probe request and the idle resource instance according to the first path instruction, so as to respond to the newly added probe request through the idle resource instance; A performance parameter sending module, configured in the parameter management module, for acquiring a first performance parameter and sending the first performance parameter to the resource management module; The resource instance execution module is configured in the resource management module and is used to execute a resource instance expansion operation or a resource instance reduction operation according to the first performance parameter.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the data exploration method of claim 8 when executed.

Images