Job migration method and device, storage medium and program product

Abstract

The invention discloses a job migration method and device, a storage medium and a program product, and relates to the technical field of computers, and the method comprises the steps: obtaining multi-dimensional feature data of each arithmetic unit, preprocessing the multi-dimensional feature data into normalized resource state data, determining a to-be-migrated job in combination with a resource demand parameter of a running job, and carrying out the migration of the to-be-migrated job; the limitation that a migration decision is triggered based on a single resource index in the prior art is solved, the resource condition of each operation unit can be comprehensively considered, and new problems caused by insufficient other resources of the migrated target migration calculation unit are avoided. Meanwhile, neighborhood operation unit sets corresponding to different neighborhood radiuses are generated with the source operation unit as the center, the target migration operation units are screened through the preset target function, the traditional mode of indiscriminate traversal or random selection of the target migration operation units is changed, the search range is accurately narrowed, the search time consumption is reduced, the migration delay is reduced, and the search efficiency is improved. The problem of unbalanced resource allocation is effectively relieved, and the overall cluster utilization rate is improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a job migration method, device, storage medium, and program product. Background Art

[0002] With the development of technologies such as intelligent workers, high-performance computing, and cloud computing, clusters have become the core infrastructure for data processing and task execution. In actual operations, the efficiency of job migration between computing units directly affects the utilization of cluster resources. Lagging or irrational migration mechanisms can lead to unbalanced resource allocation, becoming a key bottleneck restricting cluster performance.

[0003] Currently, the job migration mechanism of a cluster is mostly based on a single resource indicator to trigger migration decisions. As a result, the target migration computing unit may cause new problems due to insufficient other resources after migration. In addition, when searching for the target migration computing unit, indiscriminate traversal or random selection is often adopted, which makes the search time too long and the migration delay high, exacerbating the imbalance in resource allocation and causing low overall cluster utilization. Summary of the Invention

[0004] The present application provides a job migration method, device, storage medium, and program product to at least solve the problem of low overall cluster utilization in related technologies.

[0005] The present application provides a job migration method, including: obtaining multidimensional feature data of each computing unit in a cluster, and obtaining resource status data from the multidimensional feature data; calculating the load intensity of each computing unit based on the resource status data; obtaining resource requirement parameters for running jobs on each computing unit; determining the job to be migrated based on the load intensity of each computing unit and the resource requirement parameters for running jobs on each computing unit; generating a set of neighborhood computing units corresponding to different neighborhood radii with the source computing unit where the job to be migrated is located as the center, wherein the neighborhood radius is determined based on the number of network hops between each computing unit; determining a set of neighborhood computing units corresponding to a target neighborhood radius from the sets of neighborhood computing units corresponding to different neighborhood radii; within the set of neighborhood computing units corresponding to the target neighborhood radius, screening a target migration computing unit by using a preset objective function; and executing a migration operation of the job to be migrated from the source computing unit to the target migration computing unit.

[0006] The present application also provides a job migration device, comprising: a multidimensional feature data processing module for acquiring multidimensional feature data of each computing unit in a cluster and obtaining resource status data from the multidimensional feature data; a load intensity calculation module for calculating the load intensity of each computing unit based on the resource status data; a resource requirement parameter acquisition module for acquiring the resource requirement parameters of the job running on each computing unit; a job to be migrated determination module for determining the job to be migrated based on the load intensity of each computing unit and the resource requirement parameters of the job running on each computing unit; a neighborhood set generation module for generating a set of neighborhood computing units corresponding to different neighborhood radii with the source computing unit where the job to be migrated is located as the center, wherein the neighborhood radius is determined based on the number of network hops between the computing units; a target neighborhood set determination module for determining a set of neighborhood computing units corresponding to a target neighborhood radius from the sets of neighborhood computing units corresponding to different neighborhood radii; a target migration computing unit screening module for screening a target migration computing unit within the set of neighborhood computing units corresponding to the target neighborhood radius by using a preset objective function; and a job migration module for performing a migration operation of the job to be migrated from the source computing unit to the target migration computing unit.

[0007] The present application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the steps of any of the above-mentioned job migration methods when executing the computer program.

[0008] The present application also provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned job migration methods are implemented.

[0009] The present application also provides a computer program product, including a computer program, which implements the steps of any of the above-mentioned job migration methods when executed by a processor.

[0010] The job migration method, device, storage medium and program product of the present application obtain the multi-dimensional feature data of each computing unit and pre-process it into normalized resource status data, and determine the job to be migrated in combination with the resource demand parameters of the running job. This solves the limitation of the traditional migration decision-making based on a single resource indicator, and can comprehensively consider the resource status of each computing unit to avoid new problems caused by insufficient other resources in the target migration computing unit after migration. At the same time, a set of neighborhood computing units corresponding to different neighborhood radii is generated with the source computing unit as the center, and the target migration computing unit is screened by a preset objective function, which changes the traditional method of indiscriminate traversal or random selection of target migration computing units, accurately narrows the search range, reduces search time, reduces migration delay, effectively alleviates the problem of imbalanced resource allocation, and improves the overall utilization of the cluster. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0012] Figure 1 A schematic diagram of an application scenario architecture of the job migration method provided in an embodiment of the present application;

[0013] Figure 2 A flowchart of a job migration method provided in an embodiment of the present application;

[0014] Figure 3 A schematic diagram of the structure of a job migration device provided in an embodiment of the present application;

[0015] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0016] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0017] It should be noted that, in the description of this application, the terms "comprises," "includes," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. The terms "first," "second," etc., in this application are used to distinguish similar objects, and are not used to describe a particular order or sequence.

[0018] In order to clearly understand the technical solution of the present application, we first introduce the solution of the prior art in detail. With the development of technologies such as artificial intelligence, high-performance computing, and cloud computing, clusters have become the core infrastructure for data processing and task execution. This type of cluster integrates computing, storage, and network resources through a distributed architecture, but its efficient operation depends on precise resource scheduling and load balancing. However, in actual operation, the efficiency of job migration between computing units directly affects the utilization efficiency of cluster resources. The lag or unreasonableness of the migration mechanism will lead to an imbalance in resource allocation, becoming a key bottleneck restricting cluster performance. At present, the job migration mechanism of the cluster is mostly based on a single resource indicator to trigger the migration decision, resulting in the target migration computing unit after migration may cause new problems due to insufficient other resources. In addition, the search for the target migration computing unit often adopts an indiscriminate traversal or random selection method without considering the characteristics of the cluster network structure, resulting in excessively long search times and high migration delays, which ultimately aggravates the imbalance in resource allocation, causing some computing units to be overloaded while other node resources are idle, and the overall cluster utilization is low.

[0019] In order to solve the above technical problems, the inventors proposed a decision-making method based on multi-dimensional resource status data to drive decision-making in response to the limitations of single resource indicator decision-making. By collecting multi-dimensional feature data such as processor, memory, storage, network bandwidth, etc. of the computing unit, and forming resource status data reflecting the resource status after normalization preprocessing, the job to be migrated is identified in combination with the load intensity of each computing unit and the resource demand parameters of the job itself, avoiding the imbalance of resources of the target migration computing unit caused by misjudgment of a single indicator. Secondly, in response to the problem of low search efficiency of the target migration computing unit, the neighborhood radius is introduced. With the source computing unit as the center, the neighborhood computing unit sets of different ranges are divided according to the number of network hops. The target migration computing unit is preferentially screened within the range of the near hop number, which narrows the search range. The target migration computing unit is accurately located from the neighborhood computing unit set corresponding to the target neighborhood radius through a preset objective function, which reduces the time consumption caused by indiscriminate traversal, reduces the migration delay, and improves the overall utilization of the cluster.

[0020] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0021] In conjunction with the specific application environment architecture or specific hardware architecture on which the execution of the job migration method depends, the specific application environment architecture or specific hardware architecture is described here.

[0022] refer to Figure 1 , Figure 1 This is a schematic diagram of the application scenario architecture of the job migration method provided in the embodiment of the present application. Figure 1 As shown, the specific application scenario architecture of this application includes: a receiving device 101, a processing device 102 and a display device 103.

[0023] It is understood that the structure illustrated in the embodiments of this application does not constitute a specific limitation on the job migration method. In other feasible implementations of this application, the above architecture may include more or fewer components than shown, or may combine or split certain components, or arrange the components differently. The specific configuration can be determined based on the actual application scenario and is not limited here. Figure 1 The components shown may be implemented in hardware, software, or a combination of software and hardware.

[0024] In a specific implementation process, the receiving device 101 can be an input / output interface or a communication interface, which is used to obtain multidimensional feature data of each computing unit in the cluster and resource requirement parameters of the job running on each computing unit.

[0025] The processing device 102 is used to obtain resource status data from multidimensional feature data; calculate the load intensity of each operation unit based on the resource status data; determine the job to be migrated according to the load intensity of each operation unit and the resource requirement parameters of the job running on each operation unit; generate a set of neighborhood operation units corresponding to different neighborhood radii with the source operation unit where the job to be migrated is located as the center, wherein the neighborhood radius is determined based on the number of network hops between each operation unit; determine a set of neighborhood operation units corresponding to a target neighborhood radius from the sets of neighborhood operation units corresponding to different neighborhood radii; within the set of neighborhood operation units corresponding to the target neighborhood radius, screen the target migration operation unit by using a preset objective function; and execute the migration operation of the job to be migrated from the source operation unit to the target migration operation unit.

[0026] The display device 103 is used to display the result of the migration operation.

[0027] In addition, the network architecture and business scenarios described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Ordinary technicians in this field can know that with the evolution of network architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0028] Figure 2 A flowchart of the job migration method provided in the embodiment of the present application is shown in FIG. Figure 2 As shown, an embodiment of the present application provides a job migration method, which is described in detail as follows:

[0029] S201: Acquire multidimensional feature data of each computing unit in the cluster, and obtain resource status data from the multidimensional feature data.

[0030] Specifically, multi-dimensional feature data from each computing unit is collected in real time, covering dimensions such as processor, memory, storage, network bandwidth, and input / output bandwidth. The collection interval is set to 30 seconds to 2 minutes. This multi-dimensional feature data is preprocessed, and missing data is first processed. When a single dimension is occasionally missing, the sliding average of the last five data points of the same dimension is used to fill the missing data. If a dimension is missing for three or more consecutive times, the computing unit is marked as an abnormal unit and excluded from subsequent calculations. Finally, normalization is performed.

[0031] Specifically, the normalization process includes: calculating the processor utilization, memory remaining rate, storage remaining rate, network bandwidth remaining rate and input / output bandwidth remaining rate based on multi-dimensional feature data; and determining the processor utilization, memory remaining rate, storage remaining rate, network bandwidth remaining rate and input / output bandwidth remaining rate as resource status data.

[0032] Specifically, the average processor load, the number of processor physical cores, available memory, total memory, used storage, total storage, disk utilization, current network bandwidth occupancy, theoretical bandwidth of the network interface, current input and output throughput, and maximum storage throughput are extracted from the multidimensional feature data; the processor utilization is determined based on the average processor load and the number of processor physical cores; the memory surplus rate is determined based on the available memory and the total memory; the storage surplus rate is determined based on the used storage, total storage, and disk utilization; the network bandwidth surplus rate is determined based on the current network bandwidth occupancy and the theoretical bandwidth of the network interface; the input and output bandwidth surplus rate is determined based on the current input and output throughput and the maximum storage throughput; the processor utilization, memory surplus rate, storage surplus rate, network bandwidth surplus rate, and input and output bandwidth surplus rate are determined as normalized resource status data.

[0033] Specifically, resource status data is defined as a five-dimensional vector ,in Represents the resource status data of the jth operation unit, represents the processor utilization of the jth operation unit, Indicates the memory remaining rate of the j-th operation unit, represents the storage surplus rate of the jth operation unit, represents the network bandwidth surplus rate of the j-th computing unit, It represents the input and output bandwidth surplus rate of the j-th operation unit.

[0034] The calculation formula for each dimension of the resource status data of the j-th operation unit is:

[0035]

[0036] Where max(·) is the maximum value function, the average processor load is the average processor load in the past minute, and when the average processor load exceeds the number of physical cores of the processor, the processor utilization is forced to 0.

[0037]

[0038]

[0039] Where min(·) is the minimum function.

[0040]

[0041]

[0042] S202: Calculate the load intensity of each computing unit based on the resource status data.

[0043] Specifically, the weight coefficient of each dimension in the resource status data of each operation unit is determined; a weighted accumulation algorithm is used to multiply the data of each dimension in the resource status data by the corresponding weight coefficient and then sum them to obtain a comprehensive resource value; based on the comprehensive resource value of each operation unit, the load intensity of each operation unit is determined.

[0044] Specifically, the weight coefficient of each dimension , where the sum of the weight coefficients of each dimension is 1, and the load intensity of each computing unit is defined as:

[0045]

[0046] Where, Indicates the load intensity of the jth computing unit, represents the weight coefficient of the i-th dimension, Represents the resource status data of the i-th dimension of the j-th computing unit.

[0047] Specifically, the process of determining the weight coefficient of each dimension in the resource status data of each computing unit includes: collecting the execution data of historical jobs in the cluster on each computing unit, and extracting the actual consumption ratio of the historical jobs on the processor, memory, storage, network bandwidth and input and output bandwidth from the execution data; using the average consumption ratio of resources in each dimension as the initial weight coefficient, and based on the actual consumption ratio, using an online learning algorithm to dynamically update the initial weight coefficient to generate a dynamic weight vector; generating a feature coding vector based on the type of historical jobs of each computing unit, wherein the feature coding vector is a multidimensional vector that quantifies the resource demand characteristics of the historical job type on each computing unit; matching the corresponding weight template according to the feature coding vector, and using the weight template as the basic weight vector; determining the weight coefficient of the data of each dimension based on the dynamic weight vector and the basic weight vector.

[0048] Specifically, with the help of the cluster log system, the execution data of historical jobs in the past 6 months on each computing unit are collected from the cluster's job scheduling module and resource monitoring module, including job start time, end time, processor occupancy rate at each time point, memory usage rate, storage read and write volume, network bandwidth flow rate, input and output bandwidth throughput, etc. The collection frequency is once per hour and stored in the distributed database. For each historical job, the total processor consumption time, total memory occupancy, total storage read and write volume, total network bandwidth transmission volume, and total input and output bandwidth throughput throughout its entire life cycle are counted. Then, the proportion of resource consumption of each dimension of each job on each computing unit to the total resource capacity of the corresponding dimension of the computing unit is calculated, and the consumption ratio of the same dimension of all historical jobs is averaged to obtain the actual consumption ratio set of each dimension resource for different computing units.

[0049] Specifically, the average consumption ratio of resources in each dimension is used as the initial weight coefficient. That is, the average actual consumption ratio of resources in each dimension, such as processors and memory, is normalized so that its sum is 1 as the initial weight distribution. Then, based on the actual consumption ratio of newly collected historical jobs, online learning algorithms such as stochastic gradient descent are used to iteratively adjust the initial weight coefficient to adapt to changes in resource consumption patterns, and finally generate a dynamic weight vector that reflects the current resource consumption characteristics.

[0050] Specifically, the category attribute information of historical jobs is first sorted out. The category attribute information includes the job type identifier, the typical task processing scale of this type of job, the regular data input and output volume, and the resource preference dimension of this type of job during historical execution. Then, through encoding rules, this information is converted into quantifiable feature codes to reflect the resource demand characteristics of different types of jobs. Based on the generated feature codes, a match is performed in the preset weight template library to find the weight template corresponding to the feature code. This weight template is a pre-set basic weight allocation scheme for each dimension of different types of jobs. It can reflect the basic dependence of this type of job on each resource dimension and is used as the basic weight vector. The basic weight vector is then weighted and fused with the previously generated dynamic weight vector to finally determine the weight coefficient of each dimension of data.

[0051] S203: Obtain resource requirement parameters for running jobs on each computing unit.

[0052] Specifically, the resource requirement parameters of the currently running jobs on each computing unit are extracted in batches. The resource requirement parameters include static requirement parameters such as the number of processor cores, memory capacity, storage space, network bandwidth lower limit, and input and output throughput lower limit of each running job. For jobs that support dynamic scaling, the resource requirement thresholds that are dynamically adjusted during their operation are additionally collected.

[0053] S204: Determine the job to be migrated based on the load intensity of each computing unit and the resource requirement parameters of the job running on each computing unit.

[0054] Specifically, a computing unit with a preset load intensity is selected from each computing unit; resource requirement parameters of each running job on the computing unit with the preset load intensity, as well as resource status data of the computing unit with the preset load intensity, are extracted, wherein the resource requirement parameters include the required values ​​for processor utilization, memory surplus rate, storage surplus rate, network bandwidth surplus rate, and input and output bandwidth surplus rate; the resource requirement parameters of each dimension of each running job are compared item by item with the resource status data of the same dimension corresponding to the computing unit with the preset load intensity; if the resource requirement parameter of any dimension of any running job is higher than the resource status data of the corresponding dimension of the computing unit with the preset load intensity, the running job is determined as a job to be migrated.

[0055] Specifically, the load intensity data of all computing units in the cluster are traversed, and the computing unit with the largest load intensity is determined as the computing unit that meets the preset load intensity. The resource requirement parameters of all running jobs on the computing unit with the preset load intensity are obtained, including processor utilization requirement, memory surplus rate requirement, storage surplus rate requirement, network bandwidth surplus rate requirement and input / output bandwidth surplus rate requirement. At the same time, the real-time resource status data of these preset load intensity computing units are extracted from the cluster monitoring system, including the current processor utilization, memory surplus rate, storage surplus rate, network bandwidth surplus rate and input / output bandwidth surplus rate. For each running job on the preset load intensity computing unit, a comparison is made between the multi-dimensional resource requirements and the actual resource status of the node. If the resource requirement parameter of a running job in at least one dimension is higher than the corresponding dimensional resource status data of the preset load intensity computing unit, the job is marked as a job to be migrated.

[0056] S205: Taking the source computing unit where the job to be migrated is located as the center, generate a set of neighboring computing units corresponding to different neighborhood radii, wherein the neighborhood radius is determined based on the number of network hops between the computing units.

[0057] The source computing unit refers to the computing unit where the job to be migrated is located before migration.

[0058] Specifically, the cluster is converted into an undirected graph, where each node corresponds to each computing unit in the cluster, the edges correspond to the network connections between the computing units, and the edge weights correspond to the network hop counts between the computing units. The network hop counts between the source computing unit and the other computing units in the undirected graph are obtained. With the source computing unit as the center, based on the network hop count values, the computing units with the same network hop count as the source computing unit are divided into a set of neighboring computing units corresponding to the same neighborhood radius.

[0059] Specifically, the network connection relationship of each computing unit is first scanned through the cluster network topology discovery tool to generate an undirected graph model with the computing unit as the node. Each node is marked with the unique hardware identifier of the computing unit, and the physical network link between the nodes is used as the edge of the undirected graph, and the weight of each edge is fixed to 1. Secondly, starting from the source computing unit where the job to be migrated is located, the breadth-first search algorithm is used to traverse the undirected graph. The BFS algorithm starts from the source node and visits the adjacent nodes layer by layer, recording the number of edges contained in the shortest path between each node and the source node. The number of edges is the corresponding network hop count. Finally, with the source computing unit as the center, the computing units with the same network hop count value are classified into the same set. Each set corresponds to a neighborhood radius, where the neighborhood radius value is equal to the network hop count value.

[0060] S206: Determine a neighborhood operation unit set corresponding to the target neighborhood radius from the neighborhood operation unit sets corresponding to different neighborhood radii.

[0061] Specifically, in order from small to large neighborhood radius, the neighborhood operation unit sets corresponding to each neighborhood radius are checked in turn; for each neighborhood operation unit set, it is determined whether there is an operation unit that meets the preset resource conditions, wherein the preset resource conditions are that each dimension of the resource status data of the operation unit is not lower than the resource demand parameters of the corresponding dimensions of the job to be migrated; if there is an operation unit that meets the preset resource conditions in the neighborhood operation unit set corresponding to the neighborhood radius currently being checked, the neighborhood radius currently being checked is determined as the target neighborhood radius, and each operation unit that meets the preset resource conditions is determined as the neighborhood operation unit set corresponding to the target neighborhood radius; if there is no operation unit that meets the preset resource conditions in the neighborhood operation unit set corresponding to the neighborhood radius currently being checked, the neighborhood operation unit set corresponding to the next neighborhood radius is continued to be checked until the neighborhood operation unit set corresponding to the target neighborhood radius is found or the preset neighborhood radius is reached.

[0062] Specifically, from the set of neighborhood operation units generated by the source operation unit, all neighborhood radius values ​​are extracted and sorted in ascending order to form a check sequence. Through an iterative checking mechanism, starting with the first value in the sequence (neighborhood radius 1), the neighborhood operation unit set data corresponding to each neighborhood radius is called in sequence, ensuring that the check is carried out in order from closest to farthest from the source operation unit, with priority given to finding suitable operation units in closer neighborhoods.

[0063] Specifically, for the set of neighborhood operation units currently being checked, traverse each operation unit in the set and compare its resource status data with the resource requirement parameters of the job to be migrated one by one. Specifically, the real-time data of the five dimensions of processor utilization, memory remaining rate, storage remaining rate, network bandwidth remaining rate, and input and output bandwidth remaining rate of the operation unit are compared with the demand values ​​of the corresponding dimensions of the job to be migrated. For example, if the job to be migrated requires that the processor utilization rate is not less than 30% and the memory remaining rate is not less than 20%, it is necessary to check whether the actual processor utilization rate of the operation unit is ≥30% and whether the memory remaining rate is ≥20%, and at the same time ensure that the actual data of the other three dimensions also meet the corresponding requirements. Only operation units that meet all the standards in five dimensions are judged to meet the preset resource conditions.

[0064] Specifically, when checking the set of the current neighborhood radius, if it is found that at least one operation unit meets the preset resource conditions of all dimensions, the inspection process of the subsequent neighborhood radius is terminated immediately. The neighborhood radius is marked as the target neighborhood radius, and the set composed of the operation units that meet the preset resource conditions is determined as the neighborhood operation unit set corresponding to the target neighborhood radius. When all the operation units in the set of the current neighborhood radius do not meet the preset resource conditions, it automatically switches to the set corresponding to the next neighborhood radius in the inspection sequence, and repeatedly determines whether there is an operation unit that meets the preset resource conditions in the set corresponding to the neighborhood radius. If no operation unit that meets the conditions is found after traversing all the preset neighborhood radii, the exception handling mechanism is triggered, or the operation unit with resources closest to the requirements is selected as an alternative.

[0065] S207: Filtering target migration operation units within the neighborhood operation unit set corresponding to the target neighborhood radius by using a preset objective function.

[0066] Specifically, the resource status data of each operation unit in the neighborhood operation unit corresponding to the target neighborhood radius, as well as the number of network hops between each operation unit and the source operation unit, are obtained; the resource status data of each operation unit, as well as the number of network hops between each operation unit and the source operation unit are substituted into the preset objective function to calculate the objective function value of each operation unit; the objective function value of each operation unit is sorted, and the operation unit corresponding to the objective function value at the preset position is determined as the target migration operation unit.

[0067] Specifically, real-time resource status data is collected in batches from all computing units within the set of neighboring computing units corresponding to the target neighborhood radius. This includes normalized multi-dimensional data such as processor utilization, memory remaining rate, storage remaining rate, network bandwidth remaining rate, and input and output bandwidth remaining rate. Furthermore, the number of network hops between each computing unit and the source computing unit is extracted from the metadata of the cluster network topology map.

[0068] Among them, the preset objective function is:

[0069]

[0070] Where, Represents the jth candidate operation unit in the neighborhood operation unit set corresponding to the target neighborhood radius; represents the weight coefficient assigned to the i-th dimension in the resource status data; Represents the jth candidate operation unit Resource status data of the i-th dimension; Indicates the source computing unit where the job to be migrated is located; Represents the number of network hops between the source operation unit and the jth candidate operation unit.

[0071] Specifically, a descending sorting algorithm is used to sort the objective function values ​​of all operation units in descending order, and the operation unit with the highest objective function value is used as the target migration operation unit.

[0072] Specifically, after determining the target migration operation unit, a secondary resource verification is performed on the target migration operation unit to check whether the resource status data of the target migration operation unit still meets the resource requirement parameters of the job to be migrated; if the secondary resource verification passes, a distributed lock is added to the target migration operation unit; if the secondary resource verification fails, the target migration operation unit is added to the temporary unavailable list, and returned to the set of neighborhood operation units corresponding to the target neighborhood radius, and the target migration operation unit is re-screened through the preset objective function.

[0073] Specifically, after the target migration computing unit is selected through the preset objective function, the secondary resource verification process is immediately triggered. Through the cluster real-time monitoring interface, the latest resource status data of the target migration computing unit is collected again, including processor utilization, memory remaining rate, storage remaining rate, network bandwidth remaining rate, and input and output bandwidth remaining rate. This real-time data is compared item by item with the resource requirement parameters of the job to be migrated. If all dimensions meet the requirements, the secondary resource verification passes; if any dimension does not meet the requirements, the verification fails.

[0074] Specifically, when the secondary resource verification is passed, an exclusive lock is added to the target migration operation unit through the distributed lock service. The specific operation is to create a lock node with the unique identifier of the operation unit as the key in the distributed lock service, set the expiration time of the lock, and ensure that other job migration processes cannot select the operation unit during the validity period of the lock to prevent resource competition due to concurrent migration. At the same time, the lock holder information is recorded to facilitate permission verification when the lock is released later. If the secondary resource verification fails, the unique identifier of the operation unit, the dimension that failed the verification, and the specific data are recorded in a temporary unavailable list. The list is set with an expiration time, during which the unit will be excluded from the candidate range. Subsequently, the unit is removed from the set of neighborhood operation units corresponding to the target neighborhood radius, and the preset objective function is re-called to calculate and sort the remaining operation units, screen out a new target migration operation unit, and repeat the screening and secondary verification process until a target unit that meets the conditions is found or there are no available units in the neighborhood set.

[0075] S208: Execute a migration operation of the job to be migrated from the source computing unit to the target migration computing unit.

[0076] Specifically, a job migration instruction is sent to the source computing unit to suspend the operation of the job to be migrated and save the job operation status; the saved job operation status is transmitted to the target migration computing unit; the operation status of the job to be migrated is restored in the target migration computing unit, and the job to be migrated is started; when it is monitored that the job to be migrated is running normally in the target migration computing unit, the resources related to the job to be migrated in the source computing unit are released, and the distributed lock of the target migration computing unit is released.

[0077] Specifically, a migration instruction containing the identifier of the job to be migrated is sent to the source computing unit. After receiving the instruction, the source computing unit suspends the job process to ensure that the job execution status does not change. At the same time, for containerized jobs, a container snapshot tool is used to generate an image snapshot containing memory data, process context, and network connection status. For native process jobs, a process freezing tool is used to serialize the process status and store it in a temporary file. The snapshot file contains key information such as register values, virtual memory mapping, and open file descriptors.

[0078] Specifically, a dedicated data transmission channel is started between the source computing unit and the target migration computing unit, and the remote direct memory access protocol is used preferentially for low-latency transmission; if the cluster does not support the remote direct memory access protocol, the TCP / IP protocol is used and software acceleration is enabled. The snapshot file is integrity checked before transmission, and the checksum is compared in real time during the transmission process to ensure that the data has not been tampered with. For large files, a fragmented transmission strategy is adopted, and a breakpoint resume mark is performed every 100MB of transmission to avoid retransmission overhead caused by network interruption. After receiving the data, the target migration computing unit stores the snapshot file in a local temporary directory and returns a reception completion confirmation signal, where the local temporary directory is consistent with the source unit path.

[0079] Specifically, after the target migration computing unit receives the complete snapshot file, it first verifies that the file hash value is consistent with that sent by the source unit, and then starts the state recovery process. For containerized jobs, the snapshot image is loaded during the container runtime, the network namespace, mount point, and other operating environments are rebuilt, and the process state within the container is restored. For native process jobs, the process context is restored from the snapshot file, the virtual memory address is remapped, and the network connection is rebuilt. After the recovery is complete, the job is started, and the successful startup of the job is monitored. If the startup fails, the rollback mechanism is triggered, the snapshot is retransmitted, and a recovery attempt is made.

[0080] Specifically, by continuously monitoring and confirming that the job to be migrated is running normally on the target computing unit, a resource release instruction is sent to the source computing unit, reclaiming the processor cores, memory pages, disk space, and other resources occupied by the job, and updating the resource status data of the source unit. Simultaneously, the lock node of the target computing unit is deleted through the distributed lock service, releasing the resource lock and allowing the computing unit to participate in subsequent job scheduling.

[0081] In summary, by obtaining the multidimensional feature data of each computing unit and preprocessing it into normalized resource status data, and then combining it with the resource requirement parameters of the running job to determine the job to be migrated, this overcomes the limitations of traditional migration decisions triggered by a single resource indicator. It can comprehensively consider the resource status of each computing unit and avoid new problems caused by insufficient other resources in the target migration computing unit after migration. At the same time, a set of neighborhood computing units corresponding to different neighborhood radii is generated with the source computing unit as the center. The target migration computing unit is selected through a preset objective function, which changes the traditional method of indiscriminate traversal or random selection of target migration computing units. It accurately narrows the search range, reduces search time, reduces migration delays, effectively alleviates resource allocation imbalances, and improves the overall utilization of the cluster.

[0082] At the same time, by performing secondary resource verification on the target migration computing unit, we can effectively avoid the problem of insufficient resources in the target unit caused by other jobs grabbing resources between screening and migration, ensure that the target unit truly has the resource capacity to undertake the migration job, and reduce the risk of migration failure.

[0083] In another embodiment provided by the present application, in the process of determining the target migration operation unit, if there are multiple operation units corresponding to the objective function value at the preset position, the following steps are performed:

[0084] Sa: The network transmission delay between the source operation unit and the operation unit corresponding to the target function value at the preset position.

[0085] Sb: The computing unit with the smallest network transmission delay is selected as the target migration computing unit.

[0086] Sc: If there are computing units with the same network transmission delay, compare the historical job migration success rates of each computing unit and determine the computing unit with the highest historical job migration success rate as the target migration computing unit.

[0087] Sd: If there are multiple computing units with the highest historical job migration success rate, the computing unit with the smallest load fluctuation is selected as the target migration computing unit based on the node load history of each computing unit.

[0088] In summary, when there are multiple computing units corresponding to the objective function values ​​at preset positions, the target migration computing unit is determined by comparing the network transmission delay, historical job migration success rate, and node load history in turn. While ensuring the fairness of screening, it can give priority to computing units with better network performance, higher migration reliability, and more stable load. It effectively avoids the uncertainty brought by random selection, reduces job operation problems caused by high network delay, low migration success rate or large load fluctuations after migration, further improves the accuracy of job migration and the stability of cluster operation, and ensures the efficiency of resource scheduling of large-scale computing clusters.

[0089] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method.

[0090] Figure 3 This is a schematic diagram of the structure of the job migration device provided in the embodiment of the present application. Figure 3 As shown, an embodiment of the present application also provides a job migration device, including: a multi-dimensional feature data processing module 301, a load intensity calculation module 302, a resource requirement parameter acquisition module 303, a job to be migrated determination module 304, a neighborhood set generation module 305, a target neighborhood set determination module 306, a target migration operation unit screening module 307 and a job migration module 308.

[0091] The multi-dimensional feature data processing module 301 is configured to acquire multi-dimensional feature data of each operation unit in the cluster, and obtain resource state data from the multi-dimensional feature data.

[0092] The load intensity calculation module 302 is configured to calculate the load intensity of each operation unit based on the resource state data.

[0093] The resource requirement parameter acquisition module 303 is configured to acquire resource requirement parameters of jobs running on each operation unit.

[0094] The job to be migrated determination module 304 is configured to determine the job to be migrated according to the load intensity of each operation unit and the resource requirement parameters of the jobs running on each operation unit.

[0095] The neighborhood set generation module 305 is configured to generate neighborhood operation unit sets corresponding to different neighborhood radii with the source operation unit where the job to be migrated is located as the center, wherein the neighborhood radii are determined based on network hops between operation units.

[0096] The target neighborhood set determination module 306 is configured to determine a neighborhood operation unit set corresponding to a target neighborhood radius from the neighborhood operation unit sets corresponding to different neighborhood radii.

[0097] The target migration operation unit screening module 307 is configured to screen a target migration operation unit from the neighborhood operation unit set corresponding to the target neighborhood radius by using a preset target function.

[0098] The job migration module 308 is configured to perform a migration operation of the job to be migrated from the source operation unit to the target migration operation unit.

[0099] In a possible implementation, the multi-dimensional feature data processing module 301 is specifically configured to calculate processor utilization, memory remaining rate, storage remaining rate, network bandwidth remaining rate, and input / output bandwidth remaining rate from the multi-dimensional feature data, and determine the processor utilization, the memory remaining rate, the storage remaining rate, the network bandwidth remaining rate, and the input / output bandwidth remaining rate as the resource state data.

[0100] In one possible implementation, the module 304 for determining jobs to be migrated is specifically used to screen out computing units with a preset load intensity from various computing units; extract resource requirement parameters of each running job on the computing unit with the preset load intensity, as well as resource status data of the computing unit with the preset load intensity, wherein the resource requirement parameters include the required values ​​for processor utilization, memory remaining rate, storage remaining rate, network bandwidth remaining rate, and input and output bandwidth remaining rate; compare the resource requirement parameters of each dimension of each running job with the resource status data of the same dimension corresponding to the computing unit with the preset load intensity item by item; if there is any running job whose resource requirement parameter of any dimension is higher than the resource status data of the corresponding dimension of the computing unit with the preset load intensity, the running job is determined as a job to be migrated.

[0101] In one possible implementation, the neighborhood set generation module 305 is specifically used to convert the cluster into an undirected graph, wherein each node of the undirected graph corresponds to each operation unit in the cluster, the edge of the undirected graph corresponds to the network connection between each operation unit, and the weight of the edge corresponds to the number of network hops between each operation unit; obtain the network hop count value between the source operation unit and other operation units in the undirected graph; with the source operation unit as the center, based on the network hop count value, divide the operation units with the same network hop count value as the source operation unit into a set of neighborhood operation units corresponding to the same neighborhood radius.

[0102] In one possible implementation, the target neighborhood set determination module 306 is specifically used to check the neighborhood operation unit sets corresponding to each neighborhood radius in order from small to large; for each neighborhood operation unit set, determine whether there is an operation unit that meets the preset resource conditions, wherein the preset resource conditions are that each dimension of the resource status data of the operation unit is not lower than the resource requirement parameters of the corresponding dimensions of the job to be migrated; if there is an operation unit that meets the preset resource conditions in the neighborhood operation unit set corresponding to the neighborhood radius currently being checked, then the neighborhood radius currently being checked is determined as the target neighborhood radius, and each operation unit that meets the preset resource conditions is determined as the neighborhood operation unit set corresponding to the target neighborhood radius; if there is no operation unit that meets the preset resource conditions in the neighborhood operation unit set corresponding to the neighborhood radius currently being checked, then continue to check the neighborhood operation unit set corresponding to the next neighborhood radius until the neighborhood operation unit set corresponding to the target neighborhood radius is found or the preset neighborhood radius is reached.

[0103] In a possible implementation, the target migration operation unit screening module 307 is specifically configured to: acquire resource state data of each operation unit in the neighbor operation unit corresponding to the target neighbor radius, and network hop numbers of each operation unit and the source operation unit; substitute the resource state data of each operation unit and the network hop numbers of each operation unit and the source operation unit into a preset target function, and calculate target function values of each operation unit; sort the target function values of each operation unit, and determine an operation unit corresponding to a target function value at a preset position as a target migration operation unit.

[0104] In a possible implementation, the job migration apparatus further includes an average rule breaking module, which is configured to: if the operation unit corresponding to the target function value at the preset position includes multiple operation units, acquire network transmission delays between the source operation unit and the operation unit corresponding to the target function value at the preset position; and determine an operation unit with a minimum network transmission delay as the target migration operation unit.

[0105] In a possible implementation, the preset target function in the target migration operation unit screening module 307 is as follows:

[0106]

[0107] In the formula, R represents the target neighbor radius; R j represents the j th candidate operation unit in the neighbor operation unit set corresponding to the target neighbor radius; w i represents a weight coefficient allocated in the resource state data; r i represents the i th dimension resource state data of the j th candidate operation unit; d represents the source operation unit of the job to be migrated; and h j represents the network hop number between the source operation unit and the j th candidate operation unit.

[0108] In a possible implementation, the job migration apparatus further includes a secondary verification module, which is configured to: perform secondary resource verification on the target migration operation unit, and check whether the resource state data of the target migration operation unit still satisfies the resource requirement parameter of the job to be migrated; if the secondary resource verification is passed, add a distributed lock to the target migration operation unit; if the secondary resource verification is not passed, add the target migration operation unit to a temporary unavailable list, and return to the neighbor operation unit set corresponding to the target neighbor radius, and re-screen the target migration operation unit through the preset target function.

[0109] ​​​​​​In one possible implementation, the job migration module 308 is specifically used to send a job migration instruction to the source computing unit, suspend the operation of the job to be migrated and save the job operation status; transmit the saved job operation status to the target migration computing unit; restore the operation status of the job to be migrated in the target migration computing unit, and start the job to be migrated; when it is monitored that the job to be migrated is running normally in the target migration computing unit, release the resources related to the job to be migrated in the source computing unit, and release the distributed lock of the target migration computing unit.

[0110] In one possible implementation, the load intensity calculation module 302 is specifically used to determine the weight coefficient of each dimension in the resource status data of each operation unit; using a weighted accumulation algorithm, the data of each dimension in the resource status data is multiplied by the corresponding weight coefficient and then summed to obtain a comprehensive resource value; based on the comprehensive resource value of each operation unit, the load intensity of each operation unit is determined.

[0111] In one possible implementation, the load intensity calculation module 302 is also used to collect execution data of historical jobs on each computing unit in the cluster, and extract the actual consumption ratio of the historical jobs on the processor, memory, storage, network bandwidth and input and output bandwidth from the execution data; use the average consumption ratio of each dimension of resources as the initial weight coefficient, and based on the actual consumption ratio, use an online learning algorithm to dynamically update the initial weight coefficient to generate a dynamic weight vector; generate a feature coding vector based on the type of historical jobs of each computing unit, wherein the feature coding vector is a multidimensional vector that quantifies the resource demand characteristics of the historical job type on each computing unit; match the corresponding weight template according to the feature coding vector, and use the weight template as the basic weight vector; determine the weight coefficient of each dimension of data based on the dynamic weight vector and the basic weight vector.

[0112] For the description of the features in the embodiment corresponding to the job migration apparatus, reference can be made to the relevant description of the embodiment corresponding to the job migration method, which will not be repeated here.

[0113] Figure 4 This is a schematic diagram of the structure of the electronic device provided in the embodiment of the present application. Figure 4 As shown, the electronic device provided by this embodiment includes: at least one processor 401 and a memory 402. Optionally, the electronic device further includes a communication component 403. The processor 401, the memory 402 and the communication component 403 are connected via a bus.

[0114] During the specific implementation process, at least one processor 401 executes the computer-executable instructions stored in the memory 402 , so that the at least one processor 401 executes the above-mentioned job migration method embodiment.

[0115] The specific implementation process of the processor 401 can be found in the above method embodiment. Its implementation principle and technical effects are similar and will not be repeated here in this embodiment.

[0116] In the above embodiments, it should be understood that the processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the application may be directly executed by a hardware processor or by a combination of hardware and software modules within the processor.

[0117] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage.

[0118] A bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. Buses can be categorized as address buses, data buses, and control buses. For ease of illustration, the buses in the drawings of this application are not limited to just one bus or just one type of bus.

[0119] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored. The computer program is configured to execute the steps of any of the above-mentioned job migration method embodiments when running.

[0120] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0121] The embodiment of the present application further provides a computer program product comprising a computer program, the computer program being executed by a processor to implement the steps in any of the job migration method embodiments.

[0122] The embodiment of the present application further provides another computer program product comprising a non-volatile computer readable storage medium, the non-volatile computer readable storage medium storing a computer program, the computer program being executed by a processor to implement the steps in any of the job migration method embodiments.

[0123] Those skilled in the art will further appreciate that the units and algorithm steps of the examples described in connection with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the various examples have been described generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the particular application and design constraints imposed on the overall system. Those skilled in the art can implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

[0124] The above provides a job migration method, device, storage medium and program product. The principles and implementation modes of the present application are described by applying specific examples. The above description of the examples is only applicable to help understand the method and core idea of the present application. It should be pointed out that, for those skilled in the art, without departing from the principles of the present application, some improvements and modifications can be made to the present application, and these improvements and modifications also fall within the protection scope of the claims of the present application.

Claims

1. A method for job migration, characterized in that: include: Acquire multidimensional feature data of each computing unit in the cluster, and obtain resource status data from the multidimensional feature data; Calculating the load intensity of each computing unit based on the resource status data; Obtain resource requirement parameters for running jobs on each of the computing units; Determining jobs to be migrated based on the load intensity of each computing unit and resource requirement parameters of jobs running on each computing unit; Taking the source computing unit where the job to be migrated is located as the center, generating a set of neighboring computing units corresponding to different neighborhood radii, wherein the neighborhood radius is determined based on the number of network hops between the computing units; Determining a neighborhood operation unit set corresponding to a target neighborhood radius from the neighborhood operation unit sets corresponding to the different neighborhood radii; In the neighborhood operation unit set corresponding to the target neighborhood radius, a target migration operation unit is selected by a preset objective function; Perform a migration operation of the job to be migrated from the source computing unit to the target migration computing unit.

2. The job migration method according to claim 1, wherein: The obtaining of resource status data from the multi-dimensional feature data includes: Calculating the processor utilization rate, memory remaining rate, storage remaining rate, network bandwidth remaining rate, and input and output bandwidth remaining rate based on the multi-dimensional feature data; The processor utilization rate, the memory surplus rate, the storage surplus rate, the network bandwidth surplus rate, and the input / output bandwidth surplus rate are determined as the resource status data.

3. The job migration method according to claim 2, wherein: The step of determining the job to be migrated based on the load intensity of each computing unit and the resource requirement parameters of the job running on each computing unit includes: Filtering out a computing unit with a preset load intensity from the computing units; Extracting resource requirement parameters of each running job on the computing unit of the preset load intensity, and resource status data of the computing unit of the preset load intensity, wherein the resource requirement parameters include demand values ​​for the processor utilization rate, the memory surplus rate, the storage surplus rate, the network bandwidth surplus rate, and the input and output bandwidth surplus rate; Comparing the resource requirement parameters of each dimension of each running job with the resource status data of the same dimension corresponding to the computing unit of the preset load intensity item by item; If the resource requirement parameter of any dimension of any running job is higher than the resource status data of the dimension corresponding to the computing unit of the preset load intensity, the running job is determined as the job to be migrated.

4. The job migration method according to claim 1, wherein: The generating of neighborhood operation unit sets corresponding to different neighborhood radii with the source operation unit where the job to be migrated is located as the center includes: Converting the cluster into an undirected graph, wherein each node of the undirected graph corresponds to each computing unit in the cluster, an edge of the undirected graph corresponds to a network connection between the computing units, and a weight of the edge corresponds to the number of network hops between the computing units; Obtaining the network hop count between the source operation unit and other operation units in the undirected graph; Taking the source operation unit as the center, based on the network hop count value, operation units having the same network hop count value as the source operation unit are divided into a set of neighborhood operation units corresponding to the same neighborhood radius.

5. The job migration method according to claim 1, wherein: The determining, from the neighborhood operation unit sets corresponding to the different neighborhood radii, a neighborhood operation unit set corresponding to the target neighborhood radius, includes: In order of the neighborhood radius from small to large, check the neighborhood operation unit set corresponding to each neighborhood radius in turn; For each set of neighboring computing units, determine whether there is a computing unit that meets a preset resource condition, where the preset resource condition is that each dimension of the resource status data of the computing unit is not less than the resource requirement parameter of the corresponding dimension of the job to be migrated; If there is an operation unit that meets the preset resource condition in the neighborhood operation unit set corresponding to the neighborhood radius currently being checked, the neighborhood radius currently being checked is determined as the target neighborhood radius, and each operation unit that meets the preset resource condition is determined as the neighborhood operation unit set corresponding to the target neighborhood radius; If there is no operation unit that meets the preset resource condition in the neighborhood operation unit set corresponding to the neighborhood radius currently being checked, continue to check the neighborhood operation unit set corresponding to the next neighborhood radius until the neighborhood operation unit set corresponding to the target neighborhood radius is found or the preset neighborhood radius is reached.

6. The job migration method according to claim 2, wherein: The step of screening the target migration operation unit within the neighborhood operation unit set corresponding to the target neighborhood radius by using a preset objective function includes: Acquire resource status data of each computing unit in the neighborhood computing unit corresponding to the target neighborhood radius, and the number of network hops between each computing unit and the source computing unit; Substituting the resource status data of each computing unit and the number of network hops between each computing unit and the source computing unit into the preset objective function, and calculating the objective function value of each computing unit; The objective function values ​​of the operation units are sorted, and the operation unit corresponding to the objective function value at a preset position is determined as the target migration operation unit.

7. The job migration method according to claim 6, characterized in that: After determining the operation unit corresponding to the objective function value at the preset position as the target migration operation unit, the method further includes: If there are multiple operation units corresponding to the target function value at the preset position, obtaining a network transmission delay between the source operation unit and the operation unit corresponding to the target function value at the preset position; The computing unit with the smallest network transmission delay is used as the target migration computing unit.

8. The job migration method according to claim 6, wherein: The preset objective function is: Where, represents the jth candidate operation unit in the neighborhood operation unit set corresponding to the target neighborhood radius; represents the weight coefficient assigned to the i-th dimension in the resource status data; Represents the jth candidate operation unit Resource status data of the i-th dimension; Indicates the source computing unit where the job to be migrated is located; represents the number of network hops between the source operation unit and the j-th candidate operation unit.

9. The job migration method according to claim 1, wherein: After the target migration operation unit is selected by the preset objective function, the method further includes: Performing secondary resource verification on the target migration computing unit to check whether the resource status data of the target migration computing unit still meets the resource requirement parameters of the job to be migrated; If the secondary resource verification passes, adding a distributed lock to the target migration computing unit; If the secondary resource verification fails, the target migration operation unit is added to a temporary unavailable list and returned to the neighborhood operation unit set corresponding to the target neighborhood radius, and the target migration operation unit is re-screened by the preset objective function.

10. The job migration method according to claim 9, characterized in that: The performing of the migration operation of the job to be migrated from the source computing unit to the target migration computing unit includes: Sending a job migration instruction to the source computing unit to suspend the running of the job to be migrated and save the job running status; Transmitting the saved job running status to the target migration computing unit; Restoring the running state of the job to be migrated in the target migration computing unit and starting the job to be migrated; When it is monitored that the job to be migrated is running normally in the target migration computing unit, resources related to the job to be migrated in the source computing unit are released, and the distributed lock of the target migration computing unit is released.

11. The job migration method according to claim 2, wherein: The calculating the load intensity of each computing unit based on the resource status data includes: Determining the weight coefficient of each dimension in the resource status data of each computing unit; Using a weighted accumulation algorithm, the data of each dimension in the resource status data is multiplied by the corresponding weight coefficient and then summed to obtain a comprehensive resource value; The load intensity of each computing unit is determined according to the comprehensive resource value of each computing unit.

12. The job migration method according to claim 11, wherein: Determining the weight coefficient of each dimension in the resource status data of each computing unit includes: Collecting execution data of historical jobs on each computing unit in the cluster, and extracting the actual consumption ratio of the historical jobs on the processor, memory, storage, network bandwidth, and input and output bandwidth from the execution data; Taking the average consumption ratio of resources in each dimension as the initial weight coefficient, based on the actual consumption ratio, an online learning algorithm is used to dynamically update the initial weight coefficient to generate a dynamic weight vector; generating a feature coding vector based on the type of historical jobs of each computing unit, wherein the feature coding vector is a multidimensional vector that quantifies the resource demand characteristics of the historical job type on each computing unit; Matching a corresponding weight template according to the feature coding vector, and using the weight template as a basic weight vector; The weight coefficients of the data of each dimension are determined according to the dynamic weight vector and the basic weight vector.

13. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the job migration method according to any one of claims 1 to 12 when executing the computer program.

14. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the job migration method according to any one of claims 1 to 12.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the job migration method according to any one of claims 1 to 12 are implemented.

Images