Off-line resource hybrid deployment method, device, equipment and medium

Abstract

The invention relates to the field of data processing technology and financial science and technology, and discloses an off-line resource hybrid deployment method, device and equipment and a storage medium, the method comprises the following steps: obtaining resource deployment information of all nodes corresponding to an off-line task in a preset management platform, and carrying out performance monitoring on the resource deployment information, obtaining a resource space utilization rate; calculating a resource load value of the offline task according to the resource space utilization rate; when the resource load value is lower than a preset capacity expansion threshold value, carrying out capacity expansion on the offline task to obtain target resource deployment information of a node corresponding to the offline task; and when the resource load value is higher than a preset capacity reduction threshold value, carrying out capacity reduction on the offline task to obtain target resource deployment information of a node corresponding to the offline task. According to the invention, the resource utilization rate of offline resources can be effectively improved.

Description

Technical Field

[0001] The present invention relates to data processing technology and financial technology fields, and in particular to a method, device, equipment and medium for hybrid deployment of offline resources. Background Art

[0002] With the rapid development of the financial technology industry, data centers are carrying more and more online services and offline jobs. Online tasks are usually user-oriented, with long running time, latency sensitivity, and time-varying load characteristics, such as web browsing, online payment, social media, etc. Offline tasks can usually tolerate larger runtime delays and only need to ensure that they are completed within the deadline, such as data analysis. This situation where online services are idle at night and offline jobs are idle during the day causes a waste of server resources.

[0003] Many studies have shown that the average resource utilization of most data centers is low, usually less than 40%. Existing technologies use containerized deployment to deploy online services and offline jobs on a part of the server in a time-sharing hybrid manner, deploying online applications during the day and offline applications at night. However, due to the tidal phenomenon of online applications, when the peak period of online applications arrives, offline applications still occupy server resources, which will cause the response time of online applications to become longer, affecting the user service experience. At the same time, due to user access patterns and unpredictable peaks, the load of tasks varies greatly. Usually, the amount of resources allocated by the data center exceeds the actual amount of resources consumed. Before the peak load arrives, a large amount of resources are idle. For example, in the stock trading scene, online resources enable traders to view stock price changes in real time and quickly obtain market trends and news information. Offline resources can avoid Internet security issues such as hacker attacks and data leaks, thereby protecting the personal information and funds of traders.

[0004] Therefore, how to improve the resource utilization rate of offline resources has become an urgent problem to be solved. Summary of the invention

[0005] The present invention provides a method, device, equipment and medium for hybrid deployment of offline resources, the main purpose of which is to solve the problem of low resource utilization of offline resources.

[0006] In a first aspect, to achieve the above-mentioned purpose, the present invention provides a method for hybrid deployment of offline resources, comprising:

[0007] Obtain resource deployment information of all nodes corresponding to offline tasks in a preset management platform, and perform performance monitoring on the resource deployment information to obtain resource space utilization rate;

[0008] Calculating the resource load value of the offline task according to the resource space usage rate;

[0009] When the resource load value is lower than a preset expansion threshold, the offline task is expanded to obtain target resource deployment information of the node corresponding to the offline task;

[0010] When the resource load value is higher than a preset shrinking threshold, the offline task is shrunk to obtain target resource deployment information of the node corresponding to the offline task.

[0011] In a second aspect, the present invention further provides an offline resource hybrid deployment system, characterized in that the system comprises:

[0012] The performance monitoring module is used to obtain the resource deployment information of all nodes corresponding to offline tasks in the preset management platform, and to perform performance monitoring on the resource deployment information to obtain the resource space utilization rate;

[0013] A resource load value calculation module, used to calculate the resource load value of the offline task according to the resource space utilization rate;

[0014] An expansion module, configured to expand the capacity of the offline task when the resource load value is lower than a preset expansion threshold, and obtain target resource deployment information of the node corresponding to the offline task;

[0015] The shrinking module is used to shrink the offline task when the resource load value is higher than a preset shrinking threshold, and obtain the target resource deployment information of the node corresponding to the offline task.

[0016] In a third aspect, the present invention further provides an electronic device, the electronic device comprising:

[0017] at least one processor; and,

[0018] a memory communicatively connected to the at least one processor; wherein,

[0019] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the above-mentioned method for hybrid deployment of offline resources.

[0020] In a fourth aspect, the present invention further provides a computer-readable storage medium, wherein at least one computer program is stored in the computer-readable storage medium, and the at least one computer program is executed by a processor in an electronic device to implement the above-mentioned method for hybrid deployment of offline resources.

[0021] By acquiring the resource deployment information of all nodes corresponding to offline tasks, the present invention can fully understand the allocation and use of offline resources. On this basis, performance monitoring can track the dynamic changes of resources in real time to ensure that resources are fully utilized; by calculating the resource load value, the present invention can clearly understand the actual occupancy of the resources corresponding to each node, thereby avoiding excessive allocation or idleness of resources, and dynamically adjusting resources according to the load value can ensure that resources are used in the most reasonable way, improve the operating efficiency of the entire system, and thus achieve load balancing; by dynamic capacity expansion, it can flexibly respond to changes in task loads and ensure that tasks run in an environment with sufficient resources, which helps to improve the scalability of the management platform and enable it to handle larger-scale tasks and higher loads; by reducing the number of Pods for capacity reduction, the consumption of computing resources can be reduced, thereby reducing energy consumption and operating costs, avoiding over-deployment, and thus improving the resource utilization of offline resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0023] Figure 1 This is a schematic diagram of an application environment of a hybrid deployment method of offline resources in one embodiment of the present invention;

[0024] Figure 2 A schematic diagram of a flow chart of a method for hybrid deployment of offline resources provided by an embodiment of the present invention;

[0025] Figure 3 A schematic diagram of a module of an offline resource hybrid deployment system provided by an embodiment of the present invention;

[0026] Figure 4 A schematic diagram of the structure of an electronic device for implementing a method for hybrid deployment of offline resources provided by an embodiment of the present invention;

[0027] Figure 5 It is another structural schematic diagram of an electronic device for implementing a method for hybrid deployment of offline resources provided by an embodiment of the present invention.

[0028] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0029] In order to enable those skilled in the art to better understand the technical solution of the present disclosure, and to fully understand and implement how the present disclosure applies technical means to solve technical problems and achieve the corresponding technical effects, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only embodiments of a part of the present disclosure, not all of the embodiments. The embodiments of the present disclosure and the various features in the embodiments can be combined with each other without conflict, and the technical solutions formed are all within the scope of protection of the present disclosure. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative work should fall within the scope of protection of the present disclosure.

[0030] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, device, product or equipment that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0031] An embodiment of the present application provides a method for hybrid deployment of offline resources, and the execution subject of the method for hybrid deployment of offline resources includes but is not limited to at least one of the electronic devices such as a server and a terminal that can be configured to execute the device provided by the embodiment of the present application. In other words, the method for hybrid deployment of offline resources can be executed by software or hardware installed on a terminal device or a server device. The server includes but is not limited to: a single server, a server cluster, a cloud server or a cloud server cluster, etc. The server can be an independent server, or it can be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), and big data and artificial intelligence platforms.

[0032] The embodiment of the present invention provides a method for hybrid deployment of offline resources, which can be applied in the following situations: Figure 1In the application environment. Among them, the client communicates with the server through the network. The server can obtain the resource deployment information of all nodes corresponding to the offline task through the client, and can fully understand the allocation and use of offline resources. On this basis, performance monitoring can track the dynamic changes of resources in real time to ensure that resources are fully utilized; the present invention can clearly understand the actual occupancy of the corresponding resources of each node by calculating the resource load value, thereby avoiding excessive allocation or idleness of resources, and dynamically adjusting resources according to the load value, which can ensure that resources are used most reasonably and improve the operating efficiency of the entire system, thereby achieving load balancing; through dynamic expansion, it can flexibly respond to changes in task loads and ensure that tasks run in an environment with sufficient resources, which helps to improve the scalability of the management platform and enable it to handle larger-scale tasks and higher loads; by reducing the number of Pods to reduce capacity, the consumption of computing resources can be reduced, thereby reducing energy consumption and operating costs, avoiding over-deployment, thereby improving the resource utilization of offline resources, and feeding back the final resource deployment information to the client. Among them, the client can be but not limited to various personal computers, laptops, smart phones, tablet computers and portable wearable devices. The server can be implemented with an independent server or a server cluster consisting of multiple servers. The present invention is described in detail below through specific embodiments.

[0033] Reference Figure 2 FIG. 1 is a flow chart of a hybrid deployment method for offline resources provided by an embodiment of the present invention. In this embodiment, the hybrid deployment method for offline resources includes:

[0034] S1. Obtain resource deployment information of all nodes corresponding to offline tasks in a preset management platform, and perform performance monitoring on the resource deployment information to obtain resource space utilization rate.

[0035] In an embodiment of the present invention, the offline tasks refer to online tasks and offline tasks, the online tasks refer to user-oriented applications with service quality requirements; the offline tasks refer to batch processing applications that do not need to be user-oriented; the resource space utilization rate includes the CPU (Central Processing Unit) utilization rate of the preset management platform occupied by the client in the preset management platform. Specifically, the resource space utilization rate is the percentage of CPU resources of the preset management platform occupied by the client during operation. In this embodiment, the CPU utilization rate of each client in the preset management platform can be queried through a custom query statement.

[0036] In an embodiment of the present invention, the resource deployment information of the node corresponding to the offline task refers to the computing resources or server usage required to execute the offline task and the detailed information of the task deployment in the preset management platform. The present invention can extract the resource deployment information from the preset management platform by using an API interface or database query, etc., to ensure that the extracted information contains the task ID, node ID and specific usage.

[0037] In the embodiment of the present invention, the performance monitoring of the resource deployment information to obtain the resource space utilization rate includes:

[0038] Obtaining the average utilization rate of each CPU core in the resource deployment information and the number of cores of the preset client;

[0039] Multiply the average usage rate by the number of cores of the client to obtain the CPU usage rate of the management platform occupied by the client;

[0040] The CPU usage is used as the resource space usage.

[0041] In detail, first obtain the average current utilization rate of each CPU core in the resource deployment information and the number of cores of the client, where the number of cores refers to the number of cores of the preset client, and then multiply the two to obtain the CPU utilization rate of each client in the preset management platform.

[0042] Furthermore, if the preset management platform CPU has two cores, that is, a dual-core CPU, and their current utilization rates are 30% and 55% respectively, then the average current utilization rate of each core of the management platform CPU is: (30%+55%) / 2=42.5%. When the number of cores of client A is 2 (that is, dual-core), the CPU utilization rate of client A in the target system is 42.5%*2=85%.

[0043] In an embodiment of the present invention, by obtaining the resource deployment information of all nodes corresponding to offline tasks, the allocation and use of offline resources can be fully understood. On this basis, performance monitoring can track the dynamic changes of resources in real time to ensure that resources are fully utilized. When it is found that the resource utilization rate of a node is low, corresponding measures can be taken subsequently, such as migrating tasks to other nodes or releasing idle resources, thereby improving the overall resource utilization.

[0044] S2. Calculate the resource load value of the offline task according to the resource space usage rate.

[0045] In an embodiment of the present invention, the resource load value is used to reflect the load status of the preset management platform, and can intuitively display the processing status of offline tasks currently being processed and waiting to be processed by the preset management platform. According to the resource load value, the preset management platform can allocate system resources more reasonably, such as adjusting the number of CPU cores, increasing memory capacity or optimizing disk storage, etc., thereby helping to improve the overall performance and response speed of the management platform.

[0046] In the embodiment of the present invention, the calculating the resource load value of the offline task according to the resource space usage rate includes:

[0047] Determining the number of clients in the preset management platform;

[0048] Calculating the sum of resource space usage rates of all clients in the management platform;

[0049] The resource load value of the offline task is calculated according to the number of the clients and the sum value.

[0050] Specifically, the following formula may be used to perform a sum calculation to obtain the sum value corresponding to the resource space utilization rate of all clients:

[0051]

[0052] Wherein, K represents the resource load value of the preset management platform, k A , k B , k C ,...k n It represents the sum of resource space usage rates of preset clients A, B, ..., and n, where n represents the number of preset clients.

[0053] In detail, you can use network scanning tools (such as nmap, etc.) to scan the network of the management platform to discover the client devices connected to it. You can also use performance testing tools (such as JMeter, LoadRunner, etc.) to simulate concurrent user requests to evaluate the performance of the management platform under a specific load, which helps to determine the maximum number of concurrent users that the system can support, thereby indirectly understanding the number of clients.

[0054] In an embodiment of the present invention, for example, the preset management platform is composed of three clients, including client A, client B and client C respectively. The resource space utilization rates of all clients are counted, and the resource space utilization rate of each client is obtained and the sum value corresponding to the resource space utilization rate of all clients is calculated through a summation function. The resource load value of the offline task is calculated according to the number of the clients and the sum value.

[0055] In the embodiment of the present invention, by calculating the resource load value, the actual occupancy of the corresponding resources of each node can be clearly understood, thereby avoiding excessive allocation or idleness of resources. Dynamic adjustment of resources according to the load value can ensure the most reasonable utilization of resources, improve the operating efficiency of the entire system, and thus achieve load balancing.

[0056] S3. When the resource load value is lower than a preset expansion threshold, the offline task is expanded to obtain target resource deployment information of the node corresponding to the offline task.

[0057] In the embodiment of the present invention, it is necessary to pre-set a capacity expansion threshold for the management platform. When the calculated resource load value of the management platform is lower than the preset capacity expansion threshold of the management platform, it is considered that the management platform meets the resource capacity expansion conditions and needs to expand the resources of the management platform. The specific value of the preset capacity expansion threshold is not limited here, and its specific value can be set according to actual needs. For example, if the current resource load value of all clients is 30%, which is less than the preset capacity expansion threshold of 40%, it is determined that the management platform meets the resource capacity expansion conditions and needs to expand the resources of the management platform.

[0058] In the embodiment of the present invention, the step of expanding the offline task to obtain target resource deployment information of the node corresponding to the offline task includes:

[0059] Calculate the required expansion capacity in the expansion scenario according to the resource load value and the preset expansion threshold;

[0060] The offline task is expanded according to the expansion capacity to obtain target resource deployment information of the node corresponding to the offline task.

[0061] In detail, based on the current resource load value of the offline task (i.e. the actual amount of resources occupied by the task) and the preset capacity expansion threshold (i.e. the critical value of the resource load that triggers the capacity expansion operation), when the resource load value is lower than the preset capacity expansion threshold, it indicates that the resources currently required offline are already lower than the lower limit of the resources that can be provided by the currently deployed nodes, so capacity expansion is needed.

[0062] Furthermore, the calculation of the expansion capacity may involve multiple factors, such as adding new nodes or improving the resources of existing nodes (such as CPU, memory, storage, etc.) to ensure that the resource deployment after expansion can meet the task requirements in the future.

[0063] In the embodiment of the present invention, the step of calculating the required expansion capacity in the expansion scenario according to the resource load value and the preset reduction threshold value includes:

[0064] Utilizing the resource load value to obtain the dependency of the resource load required for the offline task, and obtaining a dependency ranking;

[0065] Performing weight calculation according to the dependency ranking to obtain a weight value of the resource load required for the offline task;

[0066] The weight value is used to obtain the expansion weight in the expansion scenario, and the required expansion capacity in the expansion scenario is determined according to the expansion weight.

[0067] Specifically, the following formula may be used to calculate the dependency to obtain the dependency of the resource load required for the offline task:

[0068]

[0069] Among them, Y i represents the dependency of the resource load required by the offline task, i represents the corresponding offline task, K represents the resource load value, Indicates the average value of the resource load value.

[0070] Furthermore, a division operation is performed on the dependency ranking corresponding to the offline task and the total number of offline tasks in the preset management platform to obtain the weight corresponding to the offline task, and the weight is used as the expansion weight in the expansion scenario, and the required expansion capacity in the expansion scenario is calculated based on this, and the resource load value is divided by the preset expansion threshold to obtain a quotient value, and the quotient value is rounded according to a preset rounding-up function and then multiplied by the expansion weight to obtain the expansion capacity.

[0071] In the embodiment of the present invention, for example, the resource load value is 30%, the preset expansion threshold is 40%, and the expansion weight is 0.5. The quotient value can be (30% / 40%)*0.5, and the expansion capacity is calculated to be 1, indicating that when the preset expansion threshold is 40%, the number of resources that need to be expanded is 1.

[0072] In the embodiment of the present invention, dynamic capacity expansion can flexibly respond to changes in task loads and ensure that tasks are run in an environment with sufficient resources. This helps to improve the scalability of the management platform so that it can handle larger-scale tasks and higher loads; it avoids over-allocation or idleness of resources, helps to optimize resource utilization and reduce operating costs.

[0073] S4. When the resource load value is higher than a preset shrinking threshold, shrink the offline task to obtain target resource deployment information of the node corresponding to the offline task.

[0074] In the embodiment of the present invention, it is necessary to pre-set a shrinking threshold for the management platform. When the calculated resource load value of the management platform exceeds the preset shrinking threshold of the management platform, it is considered that the management platform meets the resource shrinking conditions and needs to shrink the resources of the management platform. The specific value of the preset shrinking threshold is not limited here, and its specific value can be set according to actual needs. For example, if the current resource load value of all clients is 90%, which is greater than the preset shrinking threshold of 80%, it is determined that the management platform meets the resource shrinking conditions and needs to shrink the resources of the management platform.

[0075] In the embodiment of the present invention, the scaling down of the offline task to obtain target resource deployment information of the node corresponding to the offline task includes:

[0076] Calculate the required shrinkage capacity in the shrinkage scenario according to the resource load value and the preset shrinkage threshold;

[0077] The offline task is scaled down according to the scaled-down capacity to obtain target resource deployment information of the node corresponding to the offline task.

[0078] In detail, based on the resource load value of the current offline task (i.e. the amount of resources actually occupied by the task) and the preset shrinking threshold (i.e. the critical value of the resource load that triggers the shrinking operation), when the resource load value exceeds the preset shrinking threshold, it indicates that the resources currently required for offline have exceeded the lower limit of the resources that the currently deployed nodes can provide, so shrinking is required. Shrinking includes reducing the number of Pods, shutting down some nodes, or migrating tasks to nodes with fewer resources. After the shrinking operation is completed, the resource deployment information of the node corresponding to the offline task is obtained and updated.

[0079] Furthermore, the present invention obtains a quotient by dividing the resource load value by the preset shrinkage threshold, and rounds the quotient according to a preset rounding-up function to obtain the shrinkage capacity. For example, the resource load value is 90%, and the preset shrinkage threshold is 80%. Substituting the above values ​​into the preset algorithm, a quotient of (90% / 80%) can be obtained, and the shrinkage capacity is calculated to be 1, indicating that when the preset shrinkage threshold is 80%, the number of resources that need to be shrunk is 1.

[0080] In the embodiment of the present invention, the step of scaling down the offline task according to the scaled-down capacity to obtain target resource deployment information of the node corresponding to the offline task includes:

[0081] Get the number of container replicas in the scaling-down scenario;

[0082] The offline task is scaled down according to the scaled-down capacity and the number of container replicas to obtain target resource deployment information of the node corresponding to the offline task.

[0083] Specifically, the following formula can be used to calculate the number of container replicas in a scaled-down scenario:

[0084]

[0085] Among them, r represents the number of container copies, t represents the total duration of the management platform accessing offline tasks in the scale-down scenario, n represents the total number of times the management platform accesses offline tasks within the duration of t, and Q represents the number of requests per second for applying one container copy.

[0086] Furthermore, the container replica refers to a Pod replica, which is the smallest deployment unit in the open source container cluster management system of the cloud platform, which can contain one or more containers. The number of Pod replicas indicates the number of instances of the Pod running in the cluster. By running multiple Pod replicas, requests can be dispersed to different Pod instances, thereby improving the throughput and response speed of the preset management platform.

[0087] The present invention calculates the number of Pod replicas in a scaling-down scenario, and the HPA (Horizontal Pod Autoscaler) in the preset management platform initiates a scaling-down operation by accessing the Pod's replica controller (such as Deployment), thereby reducing the number of Pod replicas, thereby adjusting the number of Pod replicas to update resource deployment information.

[0088] In the embodiment of the present invention, by reducing the number of Pods to perform capacity reduction, the consumption of computing resources can be reduced, thereby reducing energy consumption and operating costs, avoiding over-deployment, and thus improving resource utilization of offline resources.

[0089] In detail, the development of financial technology has promoted the digitization and intelligence of financial services. Through the rational deployment of online and offline resources, the efficiency of financial services can be further improved. For example, the application of new models such as smart payment, smart outlets, and digital financing has made financial services more convenient and efficient, and the coverage is also wider.

[0090] Through technologies such as big data and artificial intelligence, financial technology can achieve the optimal allocation of financial resources, namely offline resources, and intelligent risk management. The deployment of online and offline resources can better balance the use of resources, ensure that necessary data processing and risk management can be performed offline, and reduce the impact of network interruptions or system failures.

[0091] Furthermore, the deployment of online and offline resources can ensure that users can still use some financial services in an offline state, thereby improving the user experience. At the same time, this deployment method can also ensure that the business can continue to run in extreme situations, thereby enhancing business continuity and stability.

[0092] By acquiring the resource deployment information of all nodes corresponding to offline tasks, the present invention can fully understand the allocation and use of offline resources. On this basis, performance monitoring can track the dynamic changes of resources in real time to ensure that resources are fully utilized; by calculating the resource load value, the present invention can clearly understand the actual occupancy of the resources corresponding to each node, thereby avoiding excessive allocation or idleness of resources, and dynamically adjusting resources according to the load value can ensure that resources are used in the most reasonable way, improve the operating efficiency of the entire system, and thus achieve load balancing; by dynamic capacity expansion, it can flexibly respond to changes in task loads and ensure that tasks run in an environment with sufficient resources, which helps to improve the scalability of the management platform and enable it to handle larger-scale tasks and higher loads; by reducing the number of Pods for capacity reduction, the consumption of computing resources can be reduced, thereby reducing energy consumption and operating costs, avoiding over-deployment, and thus improving the resource utilization of offline resources.

[0093] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0094] like Figure 3 , which is a functional module diagram of an offline resource hybrid deployment system provided by an embodiment of the present invention.

[0095] In an embodiment of the present disclosure, a hybrid deployment system for offline resources is provided, and the hybrid deployment system for offline resources corresponds one-to-one to the hybrid deployment method for offline resources in the above embodiment. Figure 3 As shown, the offline resource hybrid deployment system 100 can be installed in an electronic device. According to the functions to be implemented, the offline resource hybrid deployment system 100 includes a performance monitoring module 101, a resource load value calculation module 102, a capacity expansion module 103, and a capacity reduction module 104. The functional modules are described in detail as follows:

[0096] The performance monitoring module 101 is used to obtain resource deployment information of all nodes corresponding to offline tasks in a preset management platform, and to perform performance monitoring on the resource deployment information to obtain resource space utilization rate;

[0097] A resource load value calculation module 102, configured to calculate the resource load value of the offline task according to the resource space usage rate;

[0098] An expansion module 103 is used to expand the capacity of the offline task when the resource load value is lower than a preset expansion threshold, and obtain target resource deployment information of the node corresponding to the offline task;

[0099] The shrinking module 104 is used to shrink the offline task when the resource load value is higher than a preset shrinking threshold, and obtain target resource deployment information of the node corresponding to the offline task.

[0100] In one embodiment, when the performance monitoring module 101 performs performance monitoring on the resource deployment information to obtain the resource space usage rate, it is used to:

[0101] Obtaining the average utilization rate of each CPU core in the resource deployment information and the number of cores of the preset client;

[0102] Multiply the average usage rate by the number of cores of the client to obtain the CPU usage rate of the management platform occupied by the client;

[0103] The CPU usage is used as the resource space usage.

[0104] In one embodiment, when calculating the resource load value of the offline task according to the resource space usage rate, the resource load value calculation module 102 is used to:

[0105] Determining the number of clients in the preset management platform;

[0106] Calculating the sum of resource space usage rates of all clients in the management platform;

[0107] The resource load value of the offline task is calculated according to the number of the clients and the sum value.

[0108] In one embodiment, when the capacity expansion module 103 expands the capacity of the offline task and obtains the target resource deployment information of the node corresponding to the offline task, it is used to:

[0109] Calculate the required expansion capacity in the expansion scenario according to the resource load value and the preset expansion threshold;

[0110] The offline task is expanded according to the expansion capacity to obtain target resource deployment information of the node corresponding to the offline task.

[0111] In one embodiment, when calculating the required expansion capacity in the expansion scenario according to the resource load value and the preset shrinking threshold, the expansion module 103 is used to:

[0112] Utilizing the resource load value to obtain the dependency of the resource load required for the offline task, and obtaining a dependency ranking;

[0113] Performing weight calculation according to the dependency ranking to obtain a weight value of the resource load required for the offline task;

[0114] The weight value is used to obtain the expansion weight in the expansion scenario, and the required expansion capacity in the expansion scenario is determined according to the expansion weight.

[0115] In one embodiment, when the scaling-down module 104 performs scaling-down on the offline task and obtains the target resource deployment information of the node corresponding to the offline task, it is used to:

[0116] Calculate the required shrinkage capacity in the shrinkage scenario according to the resource load value and the preset shrinkage threshold;

[0117] The offline task is scaled down according to the scaled-down capacity to obtain target resource deployment information of the node corresponding to the offline task.

[0118] In one embodiment, when the capacity reduction module 104 reduces the capacity of the offline task according to the reduced capacity and obtains the target resource deployment information of the node corresponding to the offline task, it is used to:

[0119] Get the number of container replicas in the scaling-down scenario;

[0120] The offline task is scaled down according to the scaled-down capacity and the number of container replicas to obtain target resource deployment information of the node corresponding to the offline task.

[0121] In the present invention, for a hybrid deployment of offline resources, first of all, the present invention can fully understand the allocation and use of offline resources by obtaining the resource deployment information of all nodes corresponding to offline tasks. On this basis, performance monitoring can track the dynamic changes of resources in real time to ensure that resources are fully utilized; the present invention can clearly understand the actual occupancy of resources corresponding to each node by calculating the resource load value, thereby avoiding excessive allocation or idleness of resources, and dynamically adjusting resources according to the load value, which can ensure that resources are used most reasonably and improve the operating efficiency of the entire system, thereby achieving load balancing; dynamic expansion can flexibly respond to changes in task loads and ensure that tasks run in an environment with sufficient resources, which helps to improve the scalability of the management platform and enable it to handle larger-scale tasks and higher loads; by reducing the number of Pods for shrinking, the consumption of computing resources can be reduced, thereby reducing energy consumption and operating costs, avoiding over-deployment, and thus improving the resource utilization of offline resources. For the specific definition of a hybrid deployment system for offline resources, please refer to the definition of a hybrid deployment method for offline resources above, which will not be repeated here. Each module in the above-mentioned data visualization configuration device can be implemented in whole or in part by software, hardware and a combination thereof. The above modules may be embedded in or independent of a processor in a computer device in the form of hardware, or may be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to the above modules.

[0122] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 4 As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external client via a network connection. When the computer program is executed by the processor, a function or step on the server side of the offline resource hybrid deployment method is implemented.

[0123] In one embodiment, a computer device is provided. The computer device may be a client, and its internal structure diagram may be as follows: Figure 5As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server through a network connection. When the computer program is executed by the processor, a function or step on the client side of the offline resource hybrid deployment method is implemented.

[0124] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0125] Obtain resource deployment information of all nodes corresponding to offline tasks in a preset management platform, and perform performance monitoring on the resource deployment information to obtain resource space utilization rate;

[0126] Calculating the resource load value of the offline task according to the resource space usage rate;

[0127] When the resource load value is lower than a preset expansion threshold, the offline task is expanded to obtain target resource deployment information of the node corresponding to the offline task;

[0128] When the resource load value is higher than a preset shrinking threshold, the offline task is shrunk to obtain target resource deployment information of the node corresponding to the offline task.

[0129] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and apparatuses can be implemented in other ways. For example, the system embodiments described above are only schematic, for example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation.

[0130] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional modules.

[0131] Therefore, no matter from which point of view, the embodiments should be regarded as illustrative and non-restrictive, and the scope of the present invention is limited by the appended claims rather than the above description, so it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention. Any attached figure mark in the claims should not be regarded as limiting the claims involved.

[0132] It is obvious to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0133] In some implementations of this embodiment, a computer-readable storage medium is provided, on which a computer program is stored, characterized in that when the computer program is executed by a processor, the steps of the method described in the above embodiment are implemented.

[0134] The readable storage medium of the present invention stores a computer program, and when the computer program is executed by a processor of an electronic device, the computer program can achieve:

[0135] Obtain resource deployment information of all nodes corresponding to offline tasks in a preset management platform, and perform performance monitoring on the resource deployment information to obtain resource space utilization rate;

[0136] Calculating the resource load value of the offline task according to the resource space usage rate;

[0137] When the resource load value is lower than a preset expansion threshold, the offline task is expanded to obtain target resource deployment information of the node corresponding to the offline task;

[0138] When the resource load value is higher than a preset shrinking threshold, the offline task is shrunk to obtain target resource deployment information of the node corresponding to the offline task.

[0139] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can refer to the relevant descriptions on the server side and the client side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0140] The computer-readable storage medium may also store at least one computer executable program / instruction, which may be, for example, a computer-readable instruction. The computer-readable storage medium includes, but is not limited to, for example, a volatile memory and / or a non-volatile memory. The volatile memory may include, for example, a random access memory (RAM) and / or a cache memory (cache), etc. The computer-readable storage medium may include, for example, a read-only memory (ROM), a hard disk, a flash memory, etc. For example, a non-transitory computer-readable storage medium may be connected to a computing device such as a computer, and then, when the computing device runs the computer-readable instructions stored on the computer-readable storage medium, the various methods described above may be performed.

[0141] In addition, the computer device may also include (but not limited to) a data bus, an input / output (I / O) bus, a display, and input / output devices (eg, keyboard, mouse, speaker, etc.), etc.

[0142] The processor may communicate with external devices via an I / O bus via a wired or wireless network.

[0143] In one embodiment, the at least one computer executable instruction may also be compiled into or constitute a software product / computer program product, wherein one or more computer executable instructions are executed by a processor to perform the various functions and / or method steps in the embodiments described in the present technology.

[0144] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0145] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0146] In the embodiments provided in the present disclosure, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the above-mentioned module, program segment or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0147] It should be noted that in the present disclosure, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element limited by the sentence "includes a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0148] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

[0149] It should be noted that if software tools or components other than those of our company appear in the embodiments of the present application, they are only used for illustrative purposes and do not represent actual use.

Claims

1. A hybrid deployment method for offline resources, characterized in that: The method comprises: Obtain resource deployment information of all nodes corresponding to offline tasks in a preset management platform, and perform performance monitoring on the resource deployment information to obtain resource space utilization rate; Calculating the resource load value of the offline task according to the resource space usage rate; When the resource load value is lower than a preset expansion threshold, the offline task is expanded to obtain target resource deployment information of the node corresponding to the offline task; When the resource load value is higher than a preset shrinking threshold, the offline task is shrunk to obtain target resource deployment information of the node corresponding to the offline task.

2. The offline resource hybrid deployment method according to claim 1, characterized in that: The performing performance monitoring on the resource deployment information to obtain resource space utilization rate includes: Obtaining the average utilization rate of each CPU core in the resource deployment information and the number of cores of the preset client; Multiply the average usage rate by the number of cores of the client to obtain the CPU usage rate of the management platform occupied by the client; The CPU usage is used as the resource space usage.

3. The offline resource hybrid deployment method according to claim 1, characterized in that: The calculating the resource load value of the offline task according to the resource space usage rate includes: Determining the number of clients in the preset management platform; Calculating the sum of resource space usage rates of all clients in the management platform; The resource load value of the offline task is calculated according to the number of the clients and the sum value.

4. The offline resource hybrid deployment method according to claim 1, characterized in that: The step of expanding the offline task to obtain target resource deployment information of the node corresponding to the offline task includes: Calculate the required expansion capacity in the expansion scenario according to the resource load value and the preset expansion threshold; The offline task is expanded according to the expansion capacity to obtain target resource deployment information of the node corresponding to the offline task.

5. The offline resource hybrid deployment method according to claim 4, characterized in that: The calculating the required expansion capacity in the expansion scenario according to the resource load value and the preset shrinking threshold value includes: Utilizing the resource load value to obtain the dependency of the resource load required for the offline task, and obtaining a dependency ranking; Performing weight calculation according to the dependency ranking to obtain a weight value of the resource load required for the offline task; The weight value is used to obtain the expansion weight in the expansion scenario, and the required expansion capacity in the expansion scenario is determined according to the expansion weight.

6. The offline resource hybrid deployment method according to claim 1, characterized in that: The step of scaling down the offline task to obtain target resource deployment information of the node corresponding to the offline task includes: Calculate the required shrinkage capacity in the shrinkage scenario according to the resource load value and the preset shrinkage threshold; The offline task is scaled down according to the scaled-down capacity to obtain target resource deployment information of the node corresponding to the offline task.

7. The offline resource hybrid deployment method according to claim 6, characterized in that: The step of reducing the offline task according to the reduced capacity to obtain target resource deployment information of the node corresponding to the offline task includes: Get the number of container replicas in the scaling-down scenario; The offline task is scaled down according to the scaled-down capacity and the number of container replicas to obtain target resource deployment information of the node corresponding to the offline task.

8. A hybrid deployment system for offline resources, characterized in that: The system comprises: The performance monitoring module is used to obtain the resource deployment information of all nodes corresponding to offline tasks in the preset management platform, and to perform performance monitoring on the resource deployment information to obtain the resource space utilization rate; A resource load value calculation module, used to calculate the resource load value of the offline task according to the resource space utilization rate; An expansion module, configured to expand the capacity of the offline task when the resource load value is lower than a preset expansion threshold, and obtain target resource deployment information of the node corresponding to the offline task; The shrinking module is used to shrink the offline task when the resource load value is higher than a preset shrinking threshold, and obtain the target resource deployment information of the node corresponding to the offline task.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute a method for hybrid deployment of offline resources as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for hybrid deployment of offline resources as described in any one of claims 1 to 7 is implemented.

Images