Resource peak-shifting sharing method, device, distributed system and storage medium

Abstract

The present invention relates to distributed deployment and provides a method, device, distributed system, and storage medium for staggered resource sharing. The method detects idle periods of a resource management cluster and busy periods of a data query engine, identifies the intersection periods, and queries the running resource containers of each management server during the intersection periods. Based on the number of tasks in the data query engine and the total memory of the running resource containers, the target server is identified and controlled to process query tasks in the data query engine. This method can fully utilize idle computing resources when there are few running tasks in a YARN cluster. Furthermore, the present invention relates to blockchain technology, and the intersection periods can be stored in a blockchain.

Description

Technical Field

[0001] The present invention relates to the field of distributed deployment technology, and in particular to a resource off-peak sharing method, device, distributed system and storage medium. Background Art

[0002] The number of tasks running in a YARN cluster typically fluctuates significantly. When a YARN cluster processes a large number of batch tasks, its computing resources are limited. When a YARN cluster processes a small number of batch tasks, its computing resources are often surplus. When a YARN cluster runs only a few tasks, a large number of computing resources are idle, resulting in a waste of computing resources.

[0003] Therefore, when there are few tasks running in a YARN cluster, how to make full use of idle computing resources has become a technical problem that needs to be solved urgently. Summary of the Invention

[0004] In view of the above, it is necessary to provide a resource off-peak sharing method, device, distributed system and storage medium that can solve the technical problem of how to fully utilize idle computing resources when there are few tasks running in the YARN cluster.

[0005] In one aspect, the present invention provides a resource staggered sharing method, which is applied to a distributed system. The distributed system includes a resource management cluster and a data query engine. The resource staggered sharing method includes:

[0006] Detecting a management idle period of the resource management cluster and a query busy period of the data query engine;

[0007] Identifying an intersection period of the management idle period and the query busy period;

[0008] In the intersection period, querying the running resource container of each management server in the resource management cluster;

[0009] Identifying a target server from the resource management cluster based on the number of tasks of the data query engine and the total amount of memory of the running resource container;

[0010] Based on preset commands, the target server is controlled to process query tasks in the data query engine, so that staggered sharing is achieved between the target server and the data query engine.

[0011] According to a preferred embodiment of the present invention, detecting the management idle period of the resource management cluster includes:

[0012] Get multiple configuration periods and multiple testing days;

[0013] Based on each configuration period in the plurality of test days, monitoring the number of idle threads in the resource management cluster to obtain a plurality of thread numbers in each configuration period;

[0014] Determining a configuration period in which the number of the plurality of threads is greater than a preset number threshold as an initial period;

[0015] If there are multiple initial time periods, then calculating the quantity variance values ​​of multiple thread quantities corresponding to each initial time period;

[0016] An initial period in which the quantity variance value is less than a preset variance threshold is determined as the management idle period.

[0017] According to a preferred embodiment of the present invention, detecting the query busy period of the data query engine includes:

[0018] Counting the number of tasks to be processed in the data query engine as the number of tasks;

[0019] Identify the time period during which the number of tasks is greater than a set threshold, and calculate the duration of the time period;

[0020] The time period whose duration is longer than the preset duration is determined as the query busy period.

[0021] According to a preferred embodiment of the present invention, querying the running resource container of each management server in the resource management cluster during the intersection period includes:

[0022] At any time during the intersection period, obtaining a service queue of each management server in the resource management cluster;

[0023] Detecting whether there is a task change in the service queue;

[0024] If there is a task change in the service queue, the resource container corresponding to the service queue is determined as the running resource container.

[0025] According to a preferred embodiment of the present invention, identifying a target server from the resource management cluster according to the number of tasks of the data query engine and the total memory of the running resource container includes:

[0026] Calculate the product of the preset score and the number of tasks to obtain the target number;

[0027] The target servers are selected in sequence from the plurality of management servers according to the total amount of memory in ascending order, and the number of the target servers is equal to the target number.

[0028] According to a preferred embodiment of the present invention, controlling the target server to process query tasks in the data query engine based on a preset command so as to achieve staggered sharing between the target server and the data query engine includes:

[0029] Invoking the preset command to adjust the container information in all resource containers in the target server to the initial configuration information;

[0030] The query task is intercepted from the data query engine and sent to the target server for processing.

[0031] According to a preferred embodiment of the present invention, the method further includes:

[0032] identifying a target time period based on the intersection time period;

[0033] During the target period, shutting down the query service of the target server;

[0034] The preset command is called to adjust the container information in all resource containers in the target server to the target configuration information.

[0035] On the other hand, the present invention further proposes a resource peak-shifting sharing device, which runs in a distributed system. The distributed system includes a resource management cluster and a data query engine. The resource peak-shifting sharing device includes:

[0036] A detection unit, configured to detect a management idle period of the resource management cluster and a query busy period of the data query engine;

[0037] an identification unit, configured to identify an intersection period between the management idle period and the query busy period;

[0038] A query unit, configured to query the running resource container of each management server in the resource management cluster during the intersection period;

[0039] The identification unit is further configured to identify a target server from the resource management cluster based on the number of tasks of the data query engine and the total memory of the running resource container;

[0040] The control unit is used to control the target server to process the query tasks in the data query engine based on preset commands, so as to achieve staggered sharing between the target server and the data query engine.

[0041] On the other hand, the present invention further provides a distributed system, comprising:

[0042] a memory storing computer-readable instructions; and

[0043] The processor executes the computer-readable instructions stored in the memory to implement the resource peak-shifting sharing method.

[0044] On the other hand, the present invention further proposes a computer-readable storage medium, in which computer-readable instructions are stored. The computer-readable instructions are executed by a processor in a distributed system to implement the resource staggered sharing method.

[0045] It can be seen from the above technical solutions that this application combines the management idle period of the resource management cluster and the query busy period of the data query engine to accurately identify the period when there is surplus computing resources in the resource management cluster and the resources are tight in the data query engine, avoiding sharing the resources in the resource management cluster to the resource management cluster when there are surplus resources in the data query engine, thereby improving the rationality of identifying the intersection period, and then combining the number of tasks of the data query engine and the total memory of the running resource container to select the target server, which can accurately determine the specific server for resource sharing, and then control the target server to process the query tasks in the data query engine, alleviating the problem of resource shortage in the data query engine, and at the same time, avoiding resource waste caused by the target server being idle during the intersection period. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is a flow chart of a preferred embodiment of the resource peak-shifting sharing method of the present invention.

[0047] Figure 2 It is a schematic diagram of target servers of the resource management cluster sharing resources to the data query engine in the present invention.

[0048] Figure 3 It is a functional module diagram of a preferred embodiment of the resource peak-shifting sharing device of the present invention.

[0049] Figure 4 It is a structural diagram of a distributed system of a preferred embodiment of the method for implementing resource peak-shifting sharing of the present invention. DETAILED DESCRIPTION

[0050] In order to make the objectives, technical solutions and advantages of the present invention more clear, the present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0051] like Figure 1 FIG. 1 is a flow chart of a preferred embodiment of the resource peak-shifting sharing method of the present invention. According to different requirements, the order of the steps in the flow chart can be changed, and some steps can be omitted.

[0052] The resource-sharing method can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results.

[0053] Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0054] The resource staggered sharing method is applied to one or more distributed systems, where the distributed system is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored computer-readable instructions, and its hardware includes but is not limited to microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0055] The distributed system can be any electronic product that can interact with a user, such as a personal computer, a tablet computer, a smart phone, a personal digital assistant (PDA), a game console, an interactive network television (IPTV), a smart wearable device, etc.

[0056] The distributed system may include network devices and / or user devices, wherein the network devices include, but are not limited to, a single network distributed system, a distributed system group consisting of multiple network distributed systems, or a cloud based on cloud computing consisting of a large number of hosts or network distributed systems.

[0057] The network where the distributed system is located includes, but is not limited to: the Internet, a wide area network, a metropolitan area network, a local area network, a virtual private network (VPN), etc.

[0058] The distributed system includes a resource management cluster and a data query engine.

[0059] 101 : Detecting a management idle period of the resource management cluster and detecting a query busy period of the data query engine.

[0060] In at least one embodiment of the present invention, the resource management cluster is a general resource management system, and the resource management cluster can provide unified resource management and scheduling for upper-layer applications.

[0061] The management idle period refers to a period when the resource management cluster is in an idle state. For example, the management idle period may be 8:00-23:00.

[0062] The data query engine may be a Presto engine. The query busy period refers to a period when the data query engine is busy. For example, the query busy period may be 8:00-20:00.

[0063] In at least one embodiment of the present invention, the distributed system detecting the management idle period of the resource management cluster includes:

[0064] Get multiple configuration periods and multiple testing days;

[0065] Based on each configuration period in the plurality of test days, monitoring the number of idle threads in the resource management cluster to obtain a plurality of thread numbers in each configuration period;

[0066] Determining a configuration period in which the number of the plurality of threads is greater than a preset number threshold as an initial period;

[0067] If there are multiple initial time periods, then calculating the quantity variance values ​​of multiple thread quantities corresponding to each initial time period;

[0068] An initial period in which the quantity variance value is less than a preset variance threshold is determined as the management idle period.

[0069] The multiple configured time periods refer to time periods obtained by dividing a preset time period in a day into multiple times based on the number of requests of the resource management cluster. The preset time period includes a first time endpoint and a second time endpoint. Specifically, the distributed system monitors the number of requests of all requests in the resource management cluster with the first time endpoint as the starting point. When the number of requests monitored is greater than or equal to the preset request threshold, the monitored time point is determined as a third time endpoint, and any configured time period is determined based on the time interval formed by the first time endpoint and the third time endpoint. The system continues to monitor the number of requests of all requests in the resource management cluster with the third time endpoint as the starting point until the second time endpoint is included in the multiple configured time periods, and stops monitoring the number of requests of all requests in the resource management cluster. For example, the multiple configured time periods may include, but are not limited to: 0:00-1:00, 1:00-3:00, 3:00-4:00, etc. The length of each configured time period may be equal or different.

[0070] The multiple test days refer to specific test dates with relatively stable task loads. For example, the multiple test days may include July 5, July 6, etc.

[0071] The preset quantity threshold and the preset variance threshold can be set according to actual needs.

[0072] By comparing the number of the multiple threads with the preset number threshold, and by comparing the number variance values ​​of the multiple threads with the preset variance threshold, the stability of the management idle period can be ensured.

[0073] Specifically, the distributed system acquires multiple test days including:

[0074] Get the task volume for each preset period from the preset log library;

[0075] A preset period during which the task amount is less than a preset task threshold is determined as the plurality of test days.

[0076] The preset period is usually a time period configured in days.

[0077] The preset task threshold can be set according to actual needs.

[0078] By determining the preset period in which the task amount is less than the preset task threshold as the multiple test days, it is possible to avoid the multiple test days obtained being meaningless due to special dates, thereby improving the rationality of obtaining the multiple test days.

[0079] In other embodiments, if there is only one initial time period, the initial time period is determined as the management idle time period.

[0080] In at least one embodiment of the present invention, the distributed system detecting the query busy period of the data query engine includes:

[0081] Counting the number of tasks to be processed in the data query engine as the number of tasks;

[0082] Identify the time period during which the number of tasks is greater than a set threshold, and calculate the duration of the time period;

[0083] The time period whose duration is longer than the preset duration is determined as the query busy period.

[0084] The set threshold and the preset time duration can be set according to actual needs.

[0085] By comparing the number of tasks with the set threshold, and comparing the duration with the preset duration, since the duration during which the number of tasks is greater than the set threshold is greater than the preset duration, that is, the data query engine cannot process the pending tasks in a timely manner, the rationality of determining the query busy period can be improved.

[0086] 102. Identify an intersection period between the management idle period and the query busy period.

[0087] It should be emphasized that in order to further ensure the privacy and security of the above-mentioned intersection period, the above-mentioned intersection period can also be stored in a node of a blockchain.

[0088] In at least one embodiment of the present invention, the intersection period is the intersection of the management idle period and the query busy period. For example, if the management idle period is 8:00-23:00 and the query busy period is 8:00-20:00, then the intersection period is 8:00-20:00.

[0089] 103 : In the intersection period, query the running resource container of each management server in the resource management cluster.

[0090] In at least one embodiment of the present invention, the running resource container refers to a resource container running on each management server during the intersection period.

[0091] In at least one embodiment of the present invention, the distributed system querying the running resource container of each management server in the resource management cluster during the intersection period includes:

[0092] At any time during the intersection period, obtaining a service queue of each management server in the resource management cluster;

[0093] Detecting whether there is a task change in the service queue;

[0094] If there is a task change in the service queue, the resource container corresponding to the service queue is determined as the running resource container.

[0095] The service queue stores task information that each management server is about to process.

[0096] There is a task change in the service queue, that is, the task information to be processed in the service queue has changed.

[0097] By detecting the service list, the running resource container can be quickly identified.

[0098] 104 : Identify a target server from the resource management cluster according to the number of tasks of the data query engine and the total memory of the running resource container.

[0099] In at least one embodiment of the present invention, the number of tasks refers to the number of tasks to be processed in the data query engine.

[0100] The total memory capacity refers to the actual memory capacity in the running resource container.

[0101] The target server refers to the management server with the smaller total memory.

[0102] In at least one embodiment of the present invention, the distributed system identifying a target server from the resource management cluster according to the number of tasks of the data query engine and the total memory of the running resource container includes:

[0103] Calculate the product of the preset score and the number of tasks to obtain the target number;

[0104] The target servers are selected in sequence from the plurality of management servers according to the total amount of memory in ascending order, and the number of the target servers is equal to the target number.

[0105] The preset score can be set according to actual needs.

[0106] A reasonable number of targets can be determined through the number of tasks, thereby improving the rationality of identifying the target server.

[0107] 105. Based on a preset command, control the target server to process the query task in the data query engine, so that staggered sharing is achieved between the target server and the data query engine.

[0108] In at least one embodiment of the present invention, the query task refers to a task that is intercepted by the distributed system when sent by the user terminal to the data query engine.

[0109] In at least one embodiment of the present invention, the distributed system controls the target server to process query tasks in the data query engine based on preset commands, so as to achieve staggered sharing between the target server and the data query engine, including:

[0110] Invoking the preset command to adjust the container information in all resource containers in the target server to the initial configuration information;

[0111] The query task is intercepted from the data query engine and sent to the target server for processing.

[0112] The preset command may be yarn rmadmin-updateNodeResource [NodeID] [MemSize] [vCores], or may be an interface command.

[0113] The container information includes, but is not limited to: memory capacity and number of CPU cores.

[0114] The configuration information may be 0.

[0115] By adjusting the container information to the initial configuration information, it is possible to control the target server to process the query task. By intercepting the query task from the data query engine, it is possible to share the query task in the data query engine with the target server during the intersection period, thereby solving the technical problem of how to make full use of idle computing resources when there are few tasks running in the YARN cluster.

[0116] like Figure 2 As shown, the resource management cluster includes Svr1-100 and Svr101-200, and the data query engine includes Svr201-210. From 8:00 to 23:00, the distributed system shares the target servers Svr101-200 with the data query engine Presto for use.

[0117] In at least one embodiment of the present invention, the method further comprises:

[0118] identifying a target time period based on the intersection time period;

[0119] During the target period, shutting down the query service of the target server;

[0120] The preset command is called to adjust the container information in all resource containers in the target server to the target configuration information.

[0121] The target time period refers to the time interval excluding the intersection time period.

[0122] The target configuration information is greater than the initial configuration information.

[0123] Through the above implementation, the target server can be returned to the resource management cluster in a timely manner for resource allocation.

[0124] It can be seen from the above technical solutions that this application combines the management idle period of the resource management cluster and the query busy period of the data query engine to accurately identify the period when there is surplus computing resources in the resource management cluster and the resources are tight in the data query engine, avoiding sharing the resources in the resource management cluster to the resource management cluster when there are surplus resources in the data query engine, thereby improving the rationality of identifying the intersection period, and then combining the number of tasks of the data query engine and the total memory of the running resource container to select the target server, which can accurately determine the specific server for resource sharing, and then control the target server to process the query tasks in the data query engine, alleviating the problem of resource shortage in the data query engine, and at the same time, avoiding resource waste caused by the target server being idle during the intersection period.

[0125] like Figure 3 , which is a functional block diagram of a preferred embodiment of a resource peak-shifting sharing device according to the present invention. The resource peak-shifting sharing device 11 comprises a detection unit 110, an identification unit 111, a query unit 112, a control unit 113, a shutdown unit 114, and an adjustment unit 115. A module / unit as referred to herein refers to a series of computer-readable instruction segments that can be accessed by the processor 13 and perform a fixed function, and is stored in the memory 12. In this embodiment, the functions of each module / unit will be described in detail in subsequent embodiments.

[0126] The distributed system includes a resource management cluster and a data query engine.

[0127] The detection unit 110 detects a management idle period of the resource management cluster and detects a query busy period of the data query engine.

[0128] In at least one embodiment of the present invention, the resource management cluster is a general resource management system, and the resource management cluster can provide unified resource management and scheduling for upper-layer applications.

[0129] The management idle period refers to a period when the resource management cluster is in an idle state. For example, the management idle period may be 8:00-23:00.

[0130] The data query engine may be a Presto engine. The query busy period refers to a period when the data query engine is busy. For example, the query busy period may be 8:00-20:00.

[0131] In at least one embodiment of the present invention, the detecting unit 110 detecting the management idle period of the resource management cluster includes:

[0132] Get multiple configuration periods and multiple testing days;

[0133] Based on each configuration period in the plurality of test days, monitoring the number of idle threads in the resource management cluster to obtain a plurality of thread numbers in each configuration period;

[0134] Determining a configuration period in which the number of the plurality of threads is greater than a preset number threshold as an initial period;

[0135] If there are multiple initial time periods, then calculating the quantity variance values ​​of multiple thread quantities corresponding to each initial time period;

[0136] An initial period in which the quantity variance value is less than a preset variance threshold is determined as the management idle period.

[0137] The multiple configured time periods refer to time periods obtained by dividing a preset time period in a day into multiple times based on the number of requests in the resource management cluster. The preset time period includes a first time endpoint and a second time endpoint. Specifically, the detection unit 110 monitors the number of requests of all requests in the resource management cluster with the first time endpoint as the starting point. When the number of requests detected is greater than or equal to the preset request threshold, the detected time point is determined as a third time endpoint. The time interval formed by the first time endpoint and the third time endpoint is determined as any configured time period. The detection unit 110 continues to monitor the number of requests of all requests in the resource management cluster with the third time endpoint as the starting point until the second time endpoint is included in the multiple configured time periods, and stops monitoring the number of requests of all requests in the resource management cluster. For example, the multiple configured time periods may include, but are not limited to: 0:00-1:00, 1:00-3:00, 3:00-4:00, etc. The length of each configured time period may be equal or different.

[0138] The multiple test days refer to specific test dates with relatively stable task loads. For example, the multiple test days may include July 5, July 6, etc.

[0139] The preset quantity threshold and the preset variance threshold can be set according to actual needs.

[0140] By comparing the number of the multiple threads with the preset number threshold, and by comparing the number variance values ​​of the multiple threads with the preset variance threshold, the stability of the management idle period can be ensured.

[0141] Specifically, the detecting unit 110 acquires a plurality of test days including:

[0142] Get the task volume for each preset period from the preset log library;

[0143] A preset period during which the task amount is less than a preset task threshold is determined as the plurality of test days.

[0144] The preset period is usually a time period configured in days.

[0145] The preset task threshold can be set according to actual needs.

[0146] By determining the preset period in which the task amount is less than the preset task threshold as the multiple test days, it is possible to avoid the multiple test days obtained being meaningless due to special dates, thereby improving the rationality of obtaining the multiple test days.

[0147] In other embodiments, if there is only one initial time period, the initial time period is determined as the management idle time period.

[0148] In at least one embodiment of the present invention, the detecting unit 110 detecting the query busy period of the data query engine includes:

[0149] Counting the number of tasks to be processed in the data query engine as the number of tasks;

[0150] Identify the time period during which the number of tasks is greater than a set threshold, and calculate the duration of the time period;

[0151] The time period whose duration is longer than the preset duration is determined as the query busy period.

[0152] The set threshold and the preset time duration can be set according to actual needs.

[0153] By comparing the number of tasks with the set threshold, and comparing the duration with the preset duration, since the duration during which the number of tasks is greater than the set threshold is greater than the preset duration, that is, the data query engine cannot process the pending tasks in a timely manner, the rationality of determining the query busy period can be improved.

[0154] The identifying unit 111 identifies an intersection period between the management idle period and the query busy period.

[0155] It should be emphasized that in order to further ensure the privacy and security of the above-mentioned intersection period, the above-mentioned intersection period can also be stored in a node of a blockchain.

[0156] In at least one embodiment of the present invention, the intersection period is the intersection of the management idle period and the query busy period. For example, if the management idle period is 8:00-23:00 and the query busy period is 8:00-20:00, then the intersection period is 8:00-20:00.

[0157] The query unit 112 queries the running resource container of each management server in the resource management cluster during the intersection period.

[0158] In at least one embodiment of the present invention, the running resource container refers to a resource container running on each management server during the intersection period.

[0159] In at least one embodiment of the present invention, the query unit 112 queries the running resource container of each management server in the resource management cluster during the intersection period, including:

[0160] At any time during the intersection period, obtaining a service queue of each management server in the resource management cluster;

[0161] Detecting whether there is a task change in the service queue;

[0162] If there is a task change in the service queue, the resource container corresponding to the service queue is determined as the running resource container.

[0163] The service queue stores task information that each management server is about to process.

[0164] There is a task change in the service queue, that is, the task information to be processed in the service queue has changed.

[0165] By detecting the service list, the running resource container can be quickly identified.

[0166] The identification unit 111 identifies a target server from the resource management cluster according to the number of tasks of the data query engine and the total amount of memory of the running resource container.

[0167] In at least one embodiment of the present invention, the number of tasks refers to the number of tasks to be processed in the data query engine.

[0168] The total memory capacity refers to the actual memory capacity in the running resource container.

[0169] The target server refers to the management server with the smaller total memory.

[0170] In at least one embodiment of the present invention, the identifying unit 111 identifies the target server from the resource management cluster according to the number of tasks of the data query engine and the total memory of the running resource container, including:

[0171] Calculate the product of the preset score and the number of tasks to obtain the target number;

[0172] The target servers are selected in sequence from the plurality of management servers according to the total amount of memory in ascending order, and the number of the target servers is equal to the target number.

[0173] The preset score can be set according to actual needs.

[0174] A reasonable number of targets can be determined through the number of tasks, thereby improving the rationality of identifying the target server.

[0175] The control unit 113 controls the target server to process the query task in the data query engine based on a preset command, so that the target server and the data query engine can achieve staggered sharing.

[0176] In at least one embodiment of the present invention, the query task refers to a task that is intercepted by the distributed system when sent by the user terminal to the data query engine.

[0177] In at least one embodiment of the present invention, the control unit 113 controls the target server to process the query task in the data query engine based on a preset command, so that the target server and the data query engine can achieve staggered sharing, including:

[0178] Invoking the preset command to adjust the container information in all resource containers in the target server to the initial configuration information;

[0179] The query task is intercepted from the data query engine and sent to the target server for processing.

[0180] The preset command may be yarn rmadmin-updateNodeResource [NodeID] [MemSize] [vCores], or may be an interface command.

[0181] The container information includes, but is not limited to: memory capacity and number of CPU cores.

[0182] The configuration information may be 0.

[0183] By adjusting the container information to the initial configuration information, it is possible to control the target server to process the query task. By intercepting the query task from the data query engine, it is possible to share the query task in the data query engine with the target server during the intersection period, thereby solving the technical problem of how to make full use of idle computing resources when there are few tasks running in the YARN cluster.

[0184] like Figure 2 As shown, the resource management cluster includes Svr1-100 and Svr101-200, and the data query engine includes Svr201-210. From 8:00 to 23:00, the distributed system shares the target servers Svr101-200 with the data query engine Presto for use.

[0185] In at least one embodiment of the present invention, the identification unit 111 identifies a target time period based on the intersection time period;

[0186] The closing unit 114 closes the query service of the target server during the target period;

[0187] The adjusting unit 115 calls the preset command to adjust the container information in all resource containers in the target server to the target configuration information.

[0188] The target time period refers to the time interval excluding the intersection time period.

[0189] The target configuration information is greater than the initial configuration information.

[0190] Through the above implementation, the target server can be returned to the resource management cluster in a timely manner for resource allocation.

[0191] It can be seen from the above technical solutions that this application combines the management idle period of the resource management cluster and the query busy period of the data query engine to accurately identify the period when there is surplus computing resources in the resource management cluster and the resources are tight in the data query engine, avoiding sharing the resources in the resource management cluster to the resource management cluster when there are surplus resources in the data query engine, thereby improving the rationality of identifying the intersection period, and then combining the number of tasks of the data query engine and the total memory of the running resource container to select the target server, which can accurately determine the specific server for resource sharing, and then control the target server to process the query tasks in the data query engine, alleviating the problem of resource shortage in the data query engine, and at the same time, avoiding resource waste caused by the target server being idle during the intersection period.

[0192] like Figure 4 FIG. 1 is a structural diagram of a distributed system of a preferred embodiment of the method for implementing resource peak-shifting sharing according to the present invention.

[0193] In one embodiment of the present invention, the distributed system 1 includes, but is not limited to, a memory 12, a processor 13, and computer-readable instructions stored in the memory 12 and executable on the processor 13, such as a resource peak-shifting sharing program.

[0194] Those skilled in the art will understand that the schematic diagram is merely an example of the distributed system 1 and does not constitute a limitation on the distributed system 1. The distributed system 1 may include more or fewer components than shown in the diagram, or a combination of certain components, or different components. For example, the distributed system 1 may also include input and output devices, network access devices, buses, etc.

[0195] The processor 13 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The processor 13 is the computing core and control center of the distributed system 1, connecting various parts of the entire distributed system 1 using various interfaces and lines, and executing the operating system of the distributed system 1 as well as various installed applications, program codes, etc.

[0196] Exemplarily, the computer-readable instructions may be divided into one or more modules / units, which are stored in the memory 12 and executed by the processor 13 to implement the present invention. The one or more modules / units may be a series of computer-readable instruction segments capable of performing specific functions, and the computer-readable instruction segments are used to describe the execution process of the computer-readable instructions in the distributed system 1. For example, the computer-readable instructions may be divided into a detection unit 110, an identification unit 111, a query unit 112, a control unit 113, a shutdown unit 114, and an adjustment unit 115.

[0197] The memory 12 can be used to store the computer-readable instructions and / or modules. The processor 13 implements the various functions of the distributed system 1 by running or executing the computer-readable instructions and / or modules stored in the memory 12 and calling the data stored in the memory 12. The memory 12 can mainly include a program storage area and a data storage area. The program storage area can store an operating system and at least one application required for a function (such as a sound playback function, an image playback function, etc.); the data storage area can store data created based on the use of the distributed system. The memory 12 can include non-volatile and volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, at least one disk storage device, a flash memory device, or other storage devices.

[0198] The memory 12 may be an external memory and / or an internal memory of the distributed system 1. Furthermore, the memory 12 may be a physical memory, such as a memory stick, a TF card (Trans-flash Card), and the like.

[0199] If the modules / units integrated in the distributed system 1 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the present invention can also implement all or part of the processes in the above-mentioned method embodiments by instructing related hardware through computer-readable instructions. The computer-readable instructions can be stored in a computer-readable storage medium. When executed by a processor, the computer-readable instructions can implement the steps of each of the above-mentioned method embodiments.

[0200] The computer-readable instructions include computer-readable instruction codes, which may be in source code form, object code form, executable file, or some intermediate form. The computer-readable medium may include any entity or device capable of carrying the computer-readable instruction codes, a recording medium, a USB flash drive, a mobile hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), or a random access memory (RAM).

[0201] Blockchain, as used in this article, refers to a novel application model for computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Blockchain is essentially a decentralized database, a series of data blocks generated using cryptographic methods. Each block contains information about a batch of online transactions, used to verify the validity of this information (to prevent counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, the platform product service layer, and the application service layer.

[0202] Combine Figure 1 The memory 12 in the distributed system 1 stores computer-readable instructions to implement a resource peak-shifting sharing method, and the processor 13 can execute the computer-readable instructions to implement:

[0203] Detect idle periods of the resource management cluster and busy periods of the data query engine;

[0204] Identifying an intersection period of the management idle period and the query busy period;

[0205] In the intersection period, querying the running resource container of each management server in the resource management cluster;

[0206] Identifying a target server from the resource management cluster based on the number of tasks of the data query engine and the total amount of memory of the running resource container;

[0207] Based on preset commands, the target server is controlled to process query tasks in the data query engine, so that staggered sharing is achieved between the target server and the data query engine.

[0208] Specifically, the specific implementation method of the processor 13 for the above computer readable instructions can refer to Figure 1 The description of the relevant steps in the corresponding embodiments will not be repeated here.

[0209] In the several embodiments provided herein, it should be understood that the disclosed systems, devices, and methods may be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the module division is merely a logical functional division, and actual implementation may employ other division methods.

[0210] The computer-readable storage medium stores computer-readable instructions, wherein the computer-readable instructions, when executed by the processor 13, are used to implement the following steps:

[0211] Detect idle periods of the resource management cluster and busy periods of the data query engine;

[0212] Identifying an intersection period of the management idle period and the query busy period;

[0213] In the intersection period, querying the running resource container of each management server in the resource management cluster;

[0214] Identifying a target server from the resource management cluster based on the number of tasks of the data query engine and the total amount of memory of the running resource container;

[0215] Based on preset commands, the target server is controlled to process query tasks in the data query engine, so that staggered sharing is achieved between the target server and the data query engine.

[0216] The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed across multiple network elements. Some or all of the modules may be selected to achieve the purpose of the solution of this embodiment according to actual needs.

[0217] In addition, the functional modules in various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or hardware plus software functional modules.

[0218] Therefore, the embodiments should be considered in all respects as illustrative and non-restrictive, and the scope of the invention is defined by the appended claims rather than the foregoing description, and all changes that come within the meaning and range of equivalents of the claims are intended to be embraced therein. Any reference to a figure in a claim should not be construed as limiting the claim to which it relates.

[0219] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. The multiple units or devices described may also be implemented by a single unit or device through software or hardware. Terms such as first and second are used to indicate names and do not imply any particular order.

[0220] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not limiting. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A resource staggered sharing method, applied to a distributed system, wherein the distributed system includes a resource management cluster and a data query engine, characterized in that: The resource peak-shifting sharing method includes: Detecting the query busy period of the data query engine and detecting the management idle period of the resource management cluster, including: obtaining multiple configuration time periods and obtaining multiple test days; based on each configuration time period in the multiple test days, monitoring the number of idle threads in the resource management cluster to obtain multiple thread numbers in each configuration time period; determining a configuration time period in which the number of multiple threads is greater than a preset number threshold as an initial time period; if there are multiple initial time periods, calculating the number variance value of the number of multiple threads corresponding to each initial time period; determining an initial time period in which the number variance value is less than a preset variance threshold as the management idle period; Identifying an intersection period of the management idle period and the query busy period; In the intersection period, querying the running resource container of each management server in the resource management cluster includes: obtaining the service queue of each management server in the resource management cluster at any time in the intersection period; detecting whether there is a task change in the service queue; if there is a task change in the service queue, determining the resource container corresponding to the service queue as the running resource container; Identifying a target server from the resource management cluster based on the number of tasks of the data query engine and the total memory of the running resource container, including: selecting the target servers from the plurality of management servers in ascending order based on the total memory, wherein the number of the target servers is equal to the target number; Based on a preset command, controlling the target server to process the query task in the data query engine so as to achieve staggered sharing between the target server and the data query engine, including: intercepting the query task from the data query engine and sending the query task to the target server for processing; The method further includes: identifying a target period based on the intersection period; closing the query service of the target server in the target period; and calling a preset command to adjust the container information in all resource containers in the target server to the target configuration information.

2. The resource staggered sharing method according to claim 1, wherein: Detecting the query busy period of the data query engine includes: Counting the number of tasks to be processed in the data query engine as the number of tasks; Identify the time period during which the number of tasks is greater than a set threshold, and calculate the duration of the time period; The time period whose duration is longer than the preset duration is determined as the query busy period.

3. The resource staggered sharing method according to claim 1, wherein: The step of identifying a target server from the resource management cluster according to the number of tasks of the data query engine and the total memory of the running resource container further comprises: The product of the preset score and the number of tasks is calculated to obtain the target number.

4. The resource staggered sharing method according to claim 1, wherein: The controlling the target server to process the query task in the data query engine based on the preset command so as to achieve staggered sharing between the target server and the data query engine further includes: The preset command is called to adjust the container information in all resource containers in the target server to the initial configuration information.

5. A resource peak-shifting sharing device, running in a distributed system, the distributed system including a resource management cluster and a data query engine, characterized in that: The resource peak-shifting sharing device includes: A detection unit, configured to detect a query busy period of the data query engine and a management idle period of the resource management cluster, comprising: obtaining a plurality of configuration time periods and a plurality of test days; monitoring the number of idle threads in the resource management cluster based on each configuration time period in the plurality of test days to obtain a plurality of thread numbers in each configuration time period; determining a configuration time period in which the plurality of thread numbers are all greater than a preset number threshold as an initial time period; if there are multiple initial time periods, calculating a quantity variance value of the plurality of thread numbers corresponding to each initial time period; and determining an initial time period in which the quantity variance value is less than a preset variance threshold as the management idle period; an identification unit, configured to identify an intersection period between the management idle period and the query busy period; a query unit, configured to query the running resource container of each management server in the resource management cluster during the intersection period, comprising: obtaining a service queue of each management server in the resource management cluster at any time during the intersection period; detecting whether there is a task change in the service queue; and if there is a task change in the service queue, determining the resource container corresponding to the service queue as the running resource container; The identification unit is further configured to identify a target server from the resource management cluster based on the number of tasks of the data query engine and the total memory of the running resource container, including: sequentially selecting the target servers from the plurality of management servers in ascending order based on the total memory, wherein the number of the target servers is equal to the target number; A control unit, configured to control the target server to process query tasks in the data query engine based on preset commands, so as to achieve staggered sharing between the target server and the data query engine, including: intercepting the query tasks from the data query engine and sending the query tasks to the target server for processing; The identification unit is further used to identify the target time period based on the intersection time period; the closing unit is used to close the query service of the target server during the target time period; the adjustment unit is used to call a preset command to adjust the container information in all resource containers in the target server to the target configuration information.

6. A distributed system, characterized in that: The distributed system includes: a memory storing computer-readable instructions; and A processor executes computer-readable instructions stored in the memory to implement the resource peak-shifting sharing method as described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions are executed by a processor in a distributed system to implement the resource peak-shifting sharing method according to any one of claims 1 to 4.

Images