Gpu cluster load balancing system based on microservice architecture

Abstract

The present application relates to the technical field of micro-service architecture, and particularly to a GPU cluster load balancing system based on micro-service architecture, which comprises a micro-service topology construction module for real-time capturing source service identifiers and target service identifiers in requests according to remote procedure call request streams generated by components in the micro-service architecture. In the present application, the original cross-node remote network communication is converted into high-speed memory access within the node, the network bandwidth pressure and communication delay within the cluster are reduced, the idle state of the physical node is locked and the health degree is evaluated and screened, the tasks are ensured to be placed on the most stable computing resources, the independent computing units and address spaces are divided according to the memory requirements of the tasks by using the GPU multi-instance segmentation interface, the hardware-level isolation of resources is realized at the physical level, and the concurrent processing capacity and overall resource utilization of the GPU cluster are improved.

Description

Technical Field

[0001] This invention relates to the field of microservice architecture technology, and in particular to a GPU cluster load balancing system based on microservice architecture. Background Technology

[0002] Microservice architecture is an architectural style that develops a single application as a collection of small, independent, and independently deployable services, where each service is built around a specific business capability and runs in its own process.

[0003] Existing microservice architectures, when dealing with high-performance computing scenarios involving large-scale tensor operations or image processing, experience a sudden surge in network bandwidth bottlenecks. This increases end-to-end request-response latency and can even lead to service timeouts due to network congestion. Furthermore, current technologies often treat GPU cards as indivisible resources allocated to single containers, resulting in wasted hardware resources even for services requiring minimal computing power, leading to low overall cluster utilization. When multiple services share the same physical node, memory overflow contention or disordered competition for CUDA cores can easily occur between different service processes, causing performance fluctuations in critical business applications and failing to meet the stability and real-time requirements of high-performance computing applications. Therefore, improvements are needed. Summary of the Invention

[0004] The purpose of this invention is to address the shortcomings of existing technologies by proposing a GPU cluster load balancing system based on a microservice architecture.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: a GPU cluster load balancing system based on a microservice architecture includes:

[0006] The microservice topology construction module is used to capture the source service identifier and target service identifier in the remote procedure call request stream generated by each component in the microservice architecture in real time, measure the amount of data transmission bits and network communication round-trip time between service nodes for a single request, calculate the link communication weighting value, establish a directed connection structure based on the pointing relationship between the source service identifier and the target service identifier, and assign the link communication weighting value to the corresponding directed connection edge to generate a weighted service dependency graph.

[0007] The graph segmentation and clustering module is used to traverse the weight values ​​of all connecting edges in the weighted service dependency graph, identify connecting edges with weight values ​​lower than the preset segmentation limit, disconnect the identified connecting edges, generate independent service subgraph clusters, scan all computing task instructions included in the independent service subgraph clusters, encapsulate computing task instructions belonging to the same subgraph cluster into an indivisible whole unit, and generate an affinity task set.

[0008] The node stability scoring module is used to identify idle physical computing nodes in the current cluster based on the type of GPU computing resources required by the affinity task set, and to calculate and generate a list of healthy node associations.

[0009] The resource sharding mapping module is used to divide independent computing units and video memory address spaces on physical computing nodes according to the physical computing nodes and corresponding task units determined in the health node association list, generate MIG instance sharding parameters, load the microservice container images in the affinity task to-do set into the physical address range pointed to by the MIG instance sharding parameters, and generate microservice scheduling execution instructions.

[0010] Preferably, the step of obtaining the link communication weighting value is as follows:

[0011] Based on the remote procedure call request flow, parse the header and payload fields of each request, locate the source service identifier field and the target service identifier field, record the source service identifier and target service identifier of each request in chronological order and remove duplicates to form a pairing list of source service identifier and target service identifier.

[0012] Based on the source service identifier and target service identifier pairing list, locate the request message length field and response message length field corresponding to each pair, calculate the difference bit value between the request message length and the response message length as the data transmission bit amount, record the network communication round-trip time value of the same pair, and generate a single link communication weighted value by multiplying the data transmission bit amount by the network communication round-trip time value.

[0013] Preferably, the steps for obtaining the weighted service dependency graph are as follows:

[0014] Based on the link communication weighting value, and according to the pointing relationship between the source service identifier and the target service identifier pairing list, a directed connection edge is established for each pairing. The link communication weighting value is written into the weight field of the directed connection edge, and the edges are connected in the pairing order to form a directed connection structure, thereby generating a weighted service dependency graph.

[0015] Preferably, the step of obtaining the independent service subgraph cluster is as follows:

[0016] Based on the weighted service dependency graph, the weight values ​​of each connection edge are read one by one. The weight values ​​are compared with the preset segmentation limit. Connection edges with weight values ​​less than the preset segmentation limit are marked. A list of connection edges that need to be disconnected is generated. The corresponding connection edges are deleted according to the list of connection edges that need to be disconnected, forming a directed connection structure with low-weight connection edges removed.

[0017] Based on the directed connection structure with low-weighted edges removed, the local connection structure is divided according to the reachability of nodes. The node set and edge set of each local connection structure are recorded. A node set number index is established and local connection structures with consistent boundaries are merged to generate independent service subgraph clusters.

[0018] Preferably, the steps for obtaining the affinity task set are as follows:

[0019] Based on the independent service subgraph clusters, the computation task instructions are scanned cluster by cluster, the service node identifiers associated with the computation task instructions are located, the clusters are sorted and spliced ​​into a single execution sequence, an indivisible marker and cluster number are written, and the sequence is encapsulated into an indivisible whole unit to generate an affinity task set.

[0020] Preferably, the step of obtaining the list of associated healthy nodes is as follows:

[0021] Based on the type of GPU computing resources required by the affinity task backlog, lock the idle status flag of the physical computing node, extract the time consumption sample data of the same type of task in the current sliding window, calculate the average time consumption and time consumption standard deviation of each physical computing node, and obtain the node latency variation coefficient.

[0022] Calculate the health score of the physical computing node based on the list of node latency variation coefficients;

[0023] The health score is compared with the admission score, and physical computing nodes with a health score not less than the admission score are selected. The selected physical computing nodes are mapped according to the task requirement type in the affinity task to-do set to generate a list of associated health nodes.

[0024] Preferably, the steps for obtaining the MIG instance segmentation parameters are as follows:

[0025] Based on the physical computing nodes and corresponding task units determined in the health node association list, read the memory requirement value of the task unit, pull the GPU device number, splittable instance specification, starting address of remaining memory block, length of remaining memory block and isolation policy flag of the physical computing node, compare the splittable instance specification with the memory requirement value of the task unit and record the matching result to form a mapping list of physical computing nodes and task units.

[0026] Based on the physical computing node and task unit mapping list, the GPU multi-instance splitting interface is called to divide each matching result into an independent computing unit and video memory address space on the physical computing node. The starting address and length of the video memory address space are fixed and bound to the GPU device number. The instance specification, starting address, length and isolation policy flag are written to generate MIG instance splitting parameters.

[0027] Preferably, the step of obtaining the microservice scheduling execution instruction is as follows:

[0028] Based on the MIG instance splitting parameters, the microservice container image is located from the affinity task to-do set, the microservice container image is loaded according to the physical address range pointed to by the MIG instance splitting parameters and the loading completion mark is verified, the container running process is started and the container running process identifier and GPU device number are recorded, the execution entry point and network configuration of the corresponding task unit are output, and the microservice scheduling execution instruction is generated.

[0029] Compared with the prior art, the advantages and positive effects of the present invention are as follows:

[0030] In this invention, source and target service identifiers are captured in real time based on the remote procedure call request streams generated by each component. The data transmission bit volume and network communication round-trip time are measured and calculated to quantify the link communication cost to establish a weighted service dependency graph. This can capture the real data interaction intensity and latency bottleneck between microservices. The connection edge weight values ​​are traversed and connections below the preset limit are disconnected. The retained high-frequency interactive local connection structures are clustered into independent service subgraph clusters. Then, the computing tasks belonging to the same subgraph cluster are encapsulated into an affinity task set. This intelligent aggregation strategy based on communication overhead ensures that service components with high-frequency data dependencies are coordinated and scheduled to the same physical node. The original cross-node remote network communication is transformed into high-speed memory access within the node, reducing the network bandwidth pressure and communication latency within the cluster. Combined with locking the idle state of physical nodes and health assessment and screening, it ensures that tasks are placed on the most stable computing resources. The GPU multi-instance partitioning interface is used to divide independent computing units and address spaces according to the task's memory requirements, realizing hardware-level isolation of resources at the physical level. This avoids resource contention and mutual interference between multiple tasks and improves the concurrent processing capability and overall resource utilization of the GPU cluster. Attached Figure Description

[0031] Figure 1 This is a system flowchart of the present invention. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0033] Please see Figure 1 This invention provides a technical solution: a GPU cluster load balancing system based on a microservice architecture includes:

[0034] The microservice topology construction module is used to capture the source service identifier and target service identifier in the remote procedure call request stream generated by each component in the microservice architecture in real time, measure the amount of data transmission bits and network communication round-trip time between service nodes for a single request, calculate the link communication weighting value, establish a directed connection structure based on the pointing relationship between the source service identifier and the target service identifier, assign the link communication weighting value to the corresponding directed connection edge, and generate a weighted service dependency graph.

[0035] The graph segmentation and clustering module is used to traverse the weight values ​​of all connecting edges in the weighted service dependency graph, identify connecting edges with weight values ​​lower than the preset segmentation limit, disconnect the identified connecting edges, generate independent service subgraph clusters, scan all computation task instructions included in the independent service subgraph clusters, encapsulate computation task instructions belonging to the same subgraph cluster into an indivisible whole unit, and generate an affinity task set.

[0036] The node stability scoring module is used to identify idle physical computing nodes in the current cluster based on the type of GPU computing resources required by the affinity task backlog and to calculate and generate a list of healthy node associations.

[0037] The resource sharding mapping module is used to divide independent computing units and video memory address spaces on physical computing nodes based on the physical computing nodes and corresponding task units determined in the health node association list, generate MIG instance sharding parameters, load the microservice container images in the affinity task to-do set into the physical address range pointed to by the MIG instance sharding parameters, and generate microservice scheduling execution instructions.

[0038] The steps for obtaining the link communication weighting value are as follows:

[0039] Based on the remote procedure call request flow, parse the header and payload fields of each request, locate the source service identifier field and the target service identifier field, record the source service identifier and target service identifier of each request in chronological order and remove duplicates to form a pairing list of source service identifier and target service identifier.

[0040] Based on the pairing list of source service identifiers and target service identifiers, locate the request message length field and response message length field corresponding to each pair, calculate the difference in bit value between the request message length and the response message length as the data transmission bit amount, record the network communication round-trip time value of the same pair, and generate a single link communication weighting value by multiplying the data transmission bit amount by the network communication round-trip time value.

[0041] Specifically, based on the remote procedure call request flow, network packets are captured at the network access layer's traffic mirroring port using a packet capture tool. The capture buffer size is set to a predefined fixed value, such as setting the buffer capacity. To prevent data loss under high concurrency traffic, the intercepted binary data stream is decoded to identify the payload type field in the transport layer protocol header, filtering out communication messages of HTTP / 2 or gRPC protocols. The metadata area of ​​the message header is read, and key tracking identifiers are extracted based on a pre-configured field offset table. The source service ID is set at offset 0x1A with a length of 16 bytes, and the destination service ID is set at offset 0x2B with a length of 16 bytes. The extracted byte sequence is converted into UTF-8 encoded strings, representing the source and destination service identifiers respectively. Simultaneously, the timestamp register value of the data packet arriving at the network interface card is read, with precision retained to the microsecond level. The extracted triplet data is temporarily stored in a circular queue in memory. Deduplication logic is initiated, establishing a sliding time window based on a hash table and setting the window duration. In milliseconds, calculate the hash value of the currently extracted source service identifier and target service identifier combination. Within the sliding time window, search for records with the same hash value. If they exist, they are determined to be retransmissions or redundant requests and discarded. If they do not exist, the record is marked as valid, and the latest timestamp in the hash table is updated. Iterate through all records marked as valid in the circular queue, concatenate the source service identifier string and the target service identifier string, and sort them in ascending order by timestamp to form a list of source service identifier and target service identifier pairs.

[0042] Based on the source service identifier and target service identifier pairing list, and using the globally unique tracing ID recorded in the list, an index search is performed in the distributed tracing log database to locate the request message record when each call is initiated and the response message record when the response is received. The Content-Length field value and the payload length of the TCP header in the request message record are read, and the corresponding length field in the response message record is read. If the field is missing, the length is obtained by calculating the offset difference between the start and end addresses of the binary stream. The obtained byte length is multiplied by 8 to convert it to bit length, thus obtaining the bit length of the request message. and response message bit length Simultaneously read the timestamp of the request message. and the timestamp of receiving the complete response message The difference between the two values ​​is calculated to obtain the network communication round-trip time. Then, the relationship between data transmission volume and time is processed according to the preset weighted calculation logic. The calculation formula is as follows: , This represents the weighted value of a single communication link, used to characterize the overall communication overhead of this call. The bit length of the request message, The absolute difference between the two values ​​is taken as the bit length of the response message to reflect the impact of data exchange asymmetry on the buffer. The timestamp of the received response message. Both are timestamps for sending the request message, and are in milliseconds. The dimensional normalization coefficient is set to 1. This is used to adjust the numerical range to prevent overflow, retain the calculated result to four decimal places, and assign a very small non-zero default value if the calculated result is zero. To avoid division by zero errors in subsequent calculations, the above calculation process is repeated for each pair of items in the list, and a single link communication weighted value is generated by multiplying the amount of data transmitted by the network communication round-trip time value.

[0043] The steps to obtain the weighted service dependency graph are as follows:

[0044] Based on the link communication weighting value, and according to the pointing relationship between the source service identifier and the target service identifier pairing list, a directed connection edge is established for each pairing. The link communication weighting value is written into the weight field of the directed connection edge, and the edges are connected in the pairing order to form a directed connection structure, generating a weighted service dependency graph.

[0045] Specifically, based on the link communication weighting value, an in-memory graph data structure container is initialized. First, all source service identifiers and target service identifiers in the pairing list are traversed and mapped to node objects in the graph structure. A hash mapping table from service ID to node index is established. For each pairing relationship, its corresponding source node index is identified. and target node index Check if there exists a connection from the adjacency matrix or adjacency list. point to If the connection record does not exist, a new directed edge object is created; if it already exists, a reference to that edge is obtained. The link communication weighting value calculated in the previous steps is read and assigned to the weight attribute variable of the directed edge object. If multiple records pointing to the same point exist, the weighted moving average method is used to update the weight attribute. The calculation and update formula is as follows: , For the updated edge weights, This is the original weight value of the edge. The weighted value of the currently read link communication. As a factor of historical forgetting, pre-set as This is used to control the fusion ratio of new and old data. After writing the weights of all pairs, the graph structure is checked for integrity to see if there are any isolated nodes. If there are any nodes with both in-degree and out-degree of 0, they are removed. Finally, the set of nodes in memory and the set of edges with weight attributes are serialized and stored. According to the pointing relationship between the source service identifier and the target service identifier pairing list, a directed connection edge is established for each pair. The link communication weight value is written into the weight field of the directed connection edge. The edges are connected in the pairing order to form a directed connection structure and generate a weighted service dependency graph.

[0046] The steps to obtain an independent service subgraph cluster are as follows:

[0047] Based on the weighted service dependency graph, read the weight values ​​of each connection edge, compare the weight values ​​with the preset segmentation limit, mark the connection edges that are less than the preset segmentation limit, compile a list of connection edges that need to be disconnected, delete the corresponding connection edges according to the list of connection edges that need to be disconnected, and form a directed connection structure with low-weight connection edges removed.

[0048] Based on the directed connection structure after removing low-weight edges, the local connection structure is divided according to the reachability of nodes. The node set and edge set of each local connection structure are recorded. The node set number index is established and the local connection structures with consistent boundaries are merged to generate independent service subgraph clusters.

[0049] Specifically, based on the weighted service dependency graph, an empty list of unprocessed edges is first initialized. All directed edge objects stored in the graph are traversed, and the value of the weight field recorded in each edge object is read. To distinguish between strong and weak dependencies, a dynamic segmentation criterion needs to be set. The calculation steps for this criterion are as follows: The weight values ​​of all edges in the current graph are counted to form a weight set, and the arithmetic mean of this set is calculated. and standard deviation Set a preset segmentation limit ,in The preset segmentation limit is calculated. The average weight of all connected edges. The standard deviation of the weights The relaxation coefficient is set to 0.5 based on experience. For example, in a real cluster load analysis, the average weight of all edges was calculated to be 60, and the standard deviation was 20. Substituting these values ​​into the formula, we can obtain the relaxation coefficient. This means that connections with a communication weight value lower than 50 are considered non-core paths. Next, the weight value of each connection edge is compared with the calculated 50 one by one. Connection edges with all weight values ​​less than 50 are filtered out, and their unique identifiers are recorded in the low-weight edge list. After the traversal is complete, the corresponding element is located in the adjacency matrix of the graph data structure based on this list. The connection state at that position is set to disconnected, and the corresponding edge node reference is removed from the adjacency list. Related network topology pointing relationships are cleared, and strongly associated edges with weight values ​​greater than or equal to 50 are retained. The topology view of the graph is updated, forming a directed connection structure with low-weight connection edges removed.

[0050] Based on the directed connection structure after removing low-weight edges, a boolean node visit state array is initialized. The array length is the same as the total number of nodes in the graph, and the initial values ​​are all false. A depth-first search algorithm is used to traverse each node that has not yet been marked as visited. Starting from the current node, the remaining strong connection edges are recursively explored. All reachable nodes on the path are added to the current local node set in turn, and the corresponding positions in the visit state array are updated to true, until all nodes under the connected path have been visited, thus locking a complete connected component. For each identified connected component, record its set of node IDs and set of internal connecting edges. Calculate the boundary feature fingerprint of the component, which consists of the IDs of all nodes connected to the outside and their connection directions. Establish a hash index table based on the boundary feature fingerprint, mapping connected components with the same fingerprint to the same bucket. Check if multiple components exist in the bucket. If so, it indicates that these components are consistent at the logical boundary. Perform a merging operation, taking the union of the node sets and the union of the connecting edge sets of these components to eliminate redundant boundary descriptions. Encapsulate the merged new structure into an independent subgraph object, assigning a unique subgraph cluster identifier to each object, and generating an independent service subgraph cluster.

[0051] The steps to obtain the affinity task to-do list are as follows:

[0052] Based on the independent service subgraph clusters, scan the computation task instructions cluster by cluster, locate the service node identifiers associated with the computation task instructions, sort and splice them into a single execution sequence according to the clusters, write an indivisible tag and cluster number, encapsulate it into an indivisible whole unit, and generate an affinity task to-do set.

[0053] Specifically, based on the independent service subgraph clusters, each generated subgraph object is traversed, and all service node identifiers contained in the object are read. These identifiers are then used to search the pending queue in the task scheduling center to locate the specific computational task instructions that are bound to these service nodes. The GPU kernel function name, input data memory address, and dependency description carried in each instruction are parsed. A directed acyclic graph of task execution within the cluster is constructed according to the order of calls between microservices. A topological sorting algorithm is used to arrange these task instructions into a linear execution sequence, ensuring that dependent tasks are scheduled before tasks that depend on them. Next, a continuous task data packet structure is created, with the unique number of the current subgraph cluster written in the packet header. The sorted task instructions are then sequentially filled into the data area of ​​the data packet. For each instruction, an indivisible atomicity flag is set to 1, indicating that the sequence must be treated as a whole in subsequent scheduling processes, and splitting it for execution between different physical nodes is strictly prohibited to avoid cross-node memory copy overhead. Finally, the checksum of the entire data packet is calculated and appended to the tail. The encapsulated data packet is then pushed into a high-priority affinity scheduling queue to generate an affinity task set.

[0054] The steps to obtain the list of associated healthy nodes are as follows:

[0055] Based on the type of GPU computing resources required by the affinity task backlog, lock the idle status flag of the physical computing nodes, extract the time consumption sample data of similar tasks in the current sliding window, calculate the average time consumption and time consumption standard deviation of each physical computing node, and obtain the node latency variation coefficient.

[0056] Based on the list of node latency variation coefficients, the health score of the physical computing node is calculated using the following formula:

[0057] ;

[0058] in, For the first Health score of each physical computing node The time delay variation coefficient penalty weight is denoted by , which is a positive dimensionless constant that controls the sensitivity to historical stability. The real-time load penalty weight is a positive dimensionless constant that controls the sensitivity to the current running load. For the first The node latency variation coefficient of each physical computing node represents the variability of task processing time within a sliding window. It is calculated by dividing the standard deviation of node processing time by the average node processing time. For the first Enhanced real-time comprehensive load index of physical computing nodes, , For the first Current GPU computing power utilization of each physical computing node For the first Current memory utilization of each physical computing node As the main bottleneck resource load weight, As a secondary resource load weight, and and satisfy ;

[0059] The health score is compared with the admission score, and physical computing nodes with a health score not lower than the admission score are selected. The selected physical computing nodes are then mapped according to the task requirement type in the affinity task to-do set to generate a list of associated health nodes.

[0060] Specifically, based on the GPU computing resources required by the affinity task backlog, the system first connects to the cluster resource manager's status monitoring interface to read the real-time status registers of all physical computing nodes in the cluster. Nodes with status fields of "IDLE" or "READY" are filtered out to form an initial candidate node pool. Then, for each physical computing node in this candidate pool, the system accesses the time-series database, sets a sliding window spanning the past 30 minutes, and queries historical task execution logs within this window that match the task type labels in the affinity task backlog. The start and end timestamps of each historical task are extracted, and the difference between them is calculated to obtain a time consumption sample sequence. For example, if the sequence is {120ms, 125ms, 118ms, ...}, this sequence is cleaned to remove abnormally large values ​​caused by network timeouts or node failures, ensuring the sample size meets the minimum statistical requirement (e.g., no less than 30 samples). Then, based on statistical principles, the cleaned time consumption sample data is summed and divided by the total number of samples to obtain the average time consumed by the node for processing similar tasks. Simultaneously, calculate the sum of squares of the differences between the sample data and the mean, take the square root of the mean, and obtain the standard deviation of the time consumption. The standard deviation reflects the jitter of node performance. Finally, the calculated standard deviation of time consumption is divided by the average time consumption and dimensionless processing is performed to generate a numerical index that reflects the difference in node processing stability, which is the node delay variation coefficient.

[0061] In the health score calculation formula, the monotonically decreasing characteristic of the exponential function is used to transform the latency fluctuation and real-time load pressure of the node into a normalized health score. Through the exponential decay mechanism, the deterioration of any indicator will lead to a non-linear and sharp decline in health, thereby identifying sub-healthy nodes that are not suitable for scheduling.

[0062] The penalty weight for latency variation coefficient is determined by the following steps: collecting fault logs and task timeout records of all nodes in the cluster over the past three months, combining them with corresponding historical latency variation coefficient data, performing logistic regression analysis, establishing a correlation model between the variation coefficient and task failure rate, solving for the regression coefficient using maximum likelihood estimation, and mapping this coefficient to the interval [1, 5] to quantify the negative impact of historical instability on current health. For example, in a high availability scenario, regression analysis shows that historical fluctuations significantly contribute to the failure rate, so a penalty weight is set. This indicates a high sensitivity to historical fluctuations;

[0063] The real-time load penalty weight is obtained by running a stress testing tool on a test node in a non-production environment, gradually increasing the GPU computing load and memory usage, recording the change curve of the node response time, finding the inflection point where the response time begins to deteriorate exponentially, calculating the load rate and performance degradation slope before this inflection point, and determining the weight value through a lookup table method according to the business's requirements for real-time response. For example, for real-time rendering tasks, the load has a significant impact on latency, and the weight is set after testing and measurement. This is used to control the severity of the penalty for the current high load state;

[0064] For the first The node delay variation coefficient of the nth physical computing node is obtained by directly reading the list of node delay variation coefficients generated in the previous processing step. This value has been standardized and represents the performance dispersion of the node within the most recent sliding window. In this calculation example, the nth node in the list is selected. The coefficient of variation for each node, for example, if the average time taken for this node is calculated to be 200 milliseconds and the standard deviation is 30 milliseconds through previous steps, then... ;

[0065] For the first The enhanced real-time comprehensive load index of the physical computing node involves multiple sub-parameters: firstly, the load index of the first physical computing node is collected in real time using the NVIDIA DCGM monitoring tool. Current GPU computing power utilization of each node and current video memory utilization For example, the collected values ​​are respectively (i.e. 60%) and (i.e., 40%), then configure weights based on the characteristics of the tasks in the to-do list. If the task is computationally intensive, set the load weight of the main bottleneck resources. Secondary resource load weight Calculations yielded ;

[0066] Calculations based on parameters:

[0067] Substitute the obtained parameter values ​​into the main formula for calculation:

[0068] Calculate the total penalty value for the exponential part: :

[0069] Substitute the values: ;

[0070] Calculate the product term: , ;

[0071] Summation: ;

[0072] Calculate health score: ;

[0073] The calculation yields: ;

[0074] The result indicates that the first The current health score of the physical computing node is 0.2600, which is in the range of (0, 1]. A low score indicates that although the node is idle, its overall reliability is poor due to large historical performance fluctuations and a high weighted load index, even though it is not currently fully loaded. It may not be suitable for carrying high-priority core tasks.

[0075] Comparing health scores with admission scores involves first determining the admission score benchmark based on the overall Service Level Agreement (SLA) requirements of the cluster. This could be done by statistically analyzing the distribution of health scores across all nodes in the cluster and selecting the median as a dynamic benchmark, or by setting a fixed threshold based on experience, such as an admission score. Read the health score of each physical computing node. Perform a numerical comparison operation; if If a node is deemed capable of handling tasks, it is added to the available resource pool. Otherwise, it is marked as unhealthy and temporarily excluded. For the selected qualified physical computing nodes, the metadata of the task units encapsulated in the affinity task backlog is read to parse their specific GPU requirements, including memory size, number of computing units, and MIG partitioning specifications, such as "3g.20gb". The hardware attribute table of qualified nodes is scanned to match nodes with the corresponding MIG instance generation capability. A one-to-one correspondence between task unit IDs and physical node IP addresses is established. The situation of multiple tasks competing for the same node is handled by using load balancing algorithms such as round-robin or least-connection strategies for allocation. Finally, this mapping relationship is solidified to generate a list of healthy nodes.

[0076] The steps to obtain the MIG instance splitting parameters are as follows:

[0077] Based on the physical computing nodes and corresponding task units identified in the health node association list, read the memory requirement value of the task unit, pull the GPU device number, splittable instance specification, starting address of remaining memory block, length of remaining memory block and isolation policy flag of the physical computing node, compare the splittable instance specification with the memory requirement value of the task unit and record the matching result to form a mapping list of physical computing nodes and task units.

[0078] Based on the physical computing node and task unit mapping list, the GPU multi-instance splitting interface is called to divide each matching result into an independent computing unit and video memory address space on the physical computing node. The starting address and length of the video memory address space are fixed and bound to the GPU device number. The instance specification, starting address, length and isolation policy flag are written to generate MIG instance splitting parameters.

[0079] Specifically, based on the physical computing nodes and corresponding task units identified in the health node association list, each mapping record in the list is traversed. First, the metadata description file of the task unit is parsed to extract the value of the memory requirement field, which is accurate to megabytes. For example, if the task requirement is read as 4096MB, the NVIDIA ManagementLibrary (NVML) interface is called to send a status query command to the physical computing node to obtain the GPU device's Universal Unique Identifier (UUID). The current MIG (Multi-Instance GPU) configuration status of the device is read, existing instance partitioning schemes are retrieved, and the physical starting address and contiguous length of the remaining memory blocks are obtained. To ensure the alignment and efficiency of memory allocation, a memory fragmentation rate tolerance threshold is set. The threshold is set based on the following: statistical analysis of the task deployment failure rate caused by GPU memory fragmentation within the cluster over the past six months, combined with the GPU memory page size (typically 2MB or larger), to calculate the ratio of average fragment size to total GPU memory. For example, multiple tests revealed that when the fragmentation rate exceeds 5%, the failure rate of loading large models increases significantly; therefore, this threshold is set accordingly. Next, the optimal fit algorithm is executed, comparing the task's memory requirements with all available partitioned instance specifications on the physical node (e.g., 1GB, 5GB, 2GB, 10GB, etc.), and calculating the difference between each available specification and the task requirement. The formula is as follows: ,in Representing the The allowance for each specification For the first The amount of video memory provided by each instance specification Based on the actual memory requirements of the task, filter out and The specification with the smallest value is selected as the best match. If multiple physical nodes meet the conditions, the node with the smallest starting address of the remaining contiguous video memory block is selected first to reduce fragmentation of the high address space. The UUID of the successfully matched physical node, the ID of the selected instance specification, and the logical number of the target video memory block are filled into a temporary structure. Tasks that cannot meet the requirements are marked as suspended and an alarm is triggered. All matching data rows are summarized to form a mapping list of physical computing nodes and task units.

[0080] Based on the physical computing node and task unit mapping list, the configuration parameters in the mapping relationship are read one by one, the control API of the GPU driver layer is loaded, and for each target physical computing node, a device configuration lock is requested to prevent concurrent conflicts. According to the instance specifications determined in the list, the physical offset of the new instance in the video memory space is calculated. The calculation formula is as follows: , Indicates the first The starting address of the video memory of each MIG instance. This is the physical base address of the GPU's video memory, usually defined by the hardware manufacturer in the firmware, such as 0x00000000. For the first The amount of video memory occupied by each allocated instance. The size of the guard band between instances is used to prevent out-of-bounds memory access. This value is set according to the security isolation level. For example, in a financial-grade isolation scenario, the guard band is empirically set to 2MB. The system uses the calculated starting address and the length corresponding to the instance specification to divide the physical GPU into independent computing units and video memory address spaces. At the same time, it enables the ECC check bit and the NVLink isolation policy flag, and sets the isolation policy flag to 1 to enable hardware-level fault isolation, ensuring that the failure of a single instance will not affect other instances. After the split is completed, it obtains the UUID of the newly generated MIG instance, combines it with the starting address of the physical video memory, the length range (e.g., the length is set to 0x140000000 bytes), and the bound GPU device master number, formats it into a JSON object, and writes the instance specification, starting address, length, and isolation policy flag to generate the MIG instance split parameters.

[0081] The steps for obtaining microservice scheduling execution instructions are as follows:

[0082] Based on the MIG instance splitting parameters, the microservice container image is located from the affinity task backlog. The microservice container image is loaded according to the physical address range pointed to by the MIG instance splitting parameters and the loading completion mark is verified. The container running process is started and the container running process identifier and GPU device number are recorded. The execution entry point and network configuration of the corresponding task unit are output, and the microservice scheduling execution instruction is generated.

[0083] Specifically, based on the MIG instance partitioning parameters, the scheduler executor of the container orchestration system is activated. The MIG instance UUID specified in the parameters (e.g., MIG-GPU-xxxx-xxxx) is parsed, and the corresponding microservice container image hash value is indexed in the affinity task backlog using this UUID. The image layer file is pulled through the internal image repository interface, and an isolated runtime sandbox is built on the host machine. The OCI (Open Container Initiative) runtime specification is configured, and the environment variable NVIDIA_VISIBLE_DEVICES is injected into config.json, with its value set to the MIG instance UUID in the parameters. This ensures that only the specific compute slices partitioned within the container are visible. Simultaneously, cgroups resource limits are configured to map the physical address range of the GPU memory address space to the container's virtual address space, and a loading timeout threshold is set. This threshold is calculated based on the mirror size and average network bandwidth, using the following formula: ,in For the mirror volume, The average I / O throughput of the node. For safety redundancy, a value of 0.5 is used. For example, if the image is 2GB and the bandwidth is 100MB / s, the timeout is set to 30 seconds. If the loading time exceeds this value, it is considered a timeout and retry is performed. After loading is complete, the file system integrity checksum is checked, the container engine startup command (such as docker run or ctr run) is called, the process status is monitored until it enters the Running state, the container PID is read through the cgroup file system, the binding relationship between the PID and the GPU device number it occupies is recorded, standardized network routing rules are generated, virtual network card IP and port mapping are configured, the execution entry point and network configuration of the corresponding task unit are output, and microservice scheduling execution instructions are generated.

Claims

1. A GPU cluster load balancing system based on a microservice architecture, characterized in that, The system includes: The microservice topology construction module is used to capture the source service identifier and target service identifier in the remote procedure call request stream generated by each component in the microservice architecture in real time, measure the amount of data transmission bits and network communication round-trip time between service nodes for a single request, calculate the link communication weighting value, establish a directed connection structure based on the pointing relationship between the source service identifier and the target service identifier, and assign the link communication weighting value to the corresponding directed connection edge to generate a weighted service dependency graph. The graph segmentation and clustering module is used to traverse the weight values ​​of all connecting edges in the weighted service dependency graph, identify connecting edges with weight values ​​lower than the preset segmentation limit, disconnect the identified connecting edges, generate independent service subgraph clusters, scan all computing task instructions included in the independent service subgraph clusters, encapsulate computing task instructions belonging to the same subgraph cluster into an indivisible whole unit, and generate an affinity task set. The node stability scoring module is used to identify idle physical computing nodes in the current cluster based on the type of GPU computing resources required by the affinity task set, and to calculate and generate a list of healthy node associations. The resource sharding mapping module is used to divide independent computing units and video memory address spaces on physical computing nodes according to the physical computing nodes and corresponding task units determined in the health node association list, generate MIG instance sharding parameters, load the microservice container images in the affinity task to-do set into the physical address range pointed to by the MIG instance sharding parameters, and generate microservice scheduling execution instructions.

2. The GPU cluster load balancing system based on a microservice architecture according to claim 1, characterized in that, The steps for obtaining the link communication weighting value are as follows: Based on the remote procedure call request flow, parse the header and payload fields of each request, locate the source service identifier field and the target service identifier field, record the source service identifier and target service identifier of each request in chronological order and remove duplicates to form a pairing list of source service identifier and target service identifier. Based on the source service identifier and target service identifier pairing list, locate the request message length field and response message length field corresponding to each pair, calculate the difference bit value between the request message length and the response message length as the data transmission bit amount, record the network communication round-trip time value of the same pair, and generate a single link communication weighted value by multiplying the data transmission bit amount by the network communication round-trip time value.

3. The GPU cluster load balancing system based on a microservice architecture according to claim 1, characterized in that, The steps for obtaining the weighted service dependency graph are as follows: Based on the link communication weighting value, and according to the pointing relationship between the source service identifier and the target service identifier pairing list, a directed connection edge is established for each pairing. The link communication weighting value is written into the weight field of the directed connection edge, and the edges are connected in the pairing order to form a directed connection structure, thereby generating a weighted service dependency graph.

4. The GPU cluster load balancing system based on a microservice architecture according to claim 1, characterized in that, The steps for obtaining the independent service subgraph cluster are as follows: Based on the weighted service dependency graph, the weight values ​​of each connection edge are read one by one. The weight values ​​are compared with the preset segmentation limit. Connection edges with weight values ​​less than the preset segmentation limit are marked. A list of connection edges that need to be disconnected is generated. The corresponding connection edges are deleted according to the list of connection edges that need to be disconnected, forming a directed connection structure with low-weight connection edges removed. Based on the directed connection structure with low-weighted edges removed, the local connection structure is divided according to the reachability of nodes. The node set and edge set of each local connection structure are recorded. A node set number index is established and local connection structures with consistent boundaries are merged to generate independent service subgraph clusters.

5. The GPU cluster load balancing system based on a microservice architecture according to claim 1, characterized in that, The steps for obtaining the affinity task set are as follows: Based on the independent service subgraph clusters, the computation task instructions are scanned cluster by cluster, the service node identifiers associated with the computation task instructions are located, the clusters are sorted and spliced ​​into a single execution sequence, an indivisible marker and cluster number are written, and the sequence is encapsulated into an indivisible whole unit to generate an affinity task set.

6. The GPU cluster load balancing system based on a microservice architecture according to claim 1, characterized in that, The steps for obtaining the list of associated healthy nodes are as follows: Based on the type of GPU computing resources required by the affinity task backlog, lock the idle status flag of the physical computing node, extract the time consumption sample data of the same type of task in the current sliding window, calculate the average time consumption and time consumption standard deviation of each physical computing node, and obtain the node latency variation coefficient. Calculate the health score of the physical computing node based on the list of node latency variation coefficients; The health score is compared with the admission score, and physical computing nodes with a health score not less than the admission score are selected. The selected physical computing nodes are mapped according to the task requirement type in the affinity task to-do set to generate a list of associated health nodes.

7. The GPU cluster load balancing system based on microservice architecture according to claim 1, characterized in that, The steps for obtaining the MIG instance segmentation parameters are as follows: Based on the physical computing nodes and corresponding task units determined in the health node association list, read the memory requirement value of the task unit, pull the GPU device number, splittable instance specification, starting address of remaining memory block, length of remaining memory block and isolation policy flag of the physical computing node, compare the splittable instance specification with the memory requirement value of the task unit and record the matching result to form a mapping list of physical computing nodes and task units. Based on the physical computing node and task unit mapping list, the GPU multi-instance splitting interface is called to divide each matching result into an independent computing unit and video memory address space on the physical computing node. The starting address and length of the video memory address space are fixed and bound to the GPU device number. The instance specification, starting address, length and isolation policy flag are written to generate MIG instance splitting parameters.

8. The GPU cluster load balancing system based on microservice architecture according to claim 1, characterized in that, The steps for obtaining the microservice scheduling execution instruction are as follows: Based on the MIG instance splitting parameters, the microservice container image is located from the affinity task to-do set, the microservice container image is loaded according to the physical address range pointed to by the MIG instance splitting parameters and the loading completion mark is verified, the container running process is started and the container running process identifier and GPU device number are recorded, the execution entry point and network configuration of the corresponding task unit are output, and the microservice scheduling execution instruction is generated.

Images