Decentralized heterogeneous GPU inference node self-organizing scheduling method and system

Abstract

The present application relates to the technical field of resource scheduling, and more particularly to a decentralized heterogeneous GPU inference node self-organizing scheduling method and system, comprising sending heterogeneous computing power texture fingerprints with decay stamps by each inference node periodically, which can synchronize the real-time running state of GPUs in a local area network with low latency; relying on the Lamport clock version vector and the Gossip anti-entropy mechanism to identify node state divergence, based on view divergence entropy monitoring and calibration, two-way convergence synchronization is realized; through deep analysis of the model semantics and tensor characteristics of the inference request by the gateway, a high-adaptability candidate inference node portrait cluster is generated, narrowing the subsequent scheduling decision range; by carrying out inference affinity coupling field scoring, the optimal execution node with the highest adaptation degree to the current inference task is selected; by forwarding the inference request to the optimal node for execution, the node reputation evolution vector is dynamically updated through the reply deviation, and the out-of-tune nodes are automatically isolated and supported for state recovery and automatic unblocking.

Description

Technical Field

[0001] This invention relates to the field of resource scheduling technology, and in particular to a decentralized heterogeneous GPU inference node self-organizing scheduling method and system. Background Technology

[0002] In the current deployment of large language models and generative AI services, a large number of heterogeneous GPU inference nodes are often centrally networked, relying on local area network clusters to handle high-concurrency model inference requests. Existing mainstream scheduling solutions generally adopt a centralized management and control model. This model can maintain basic operation in small-scale clusters with fixed node size and uniform GPU models, but it is not suitable for the fine-grained collaboration needs of heterogeneous computing power in decentralized peer-to-peer networking scenarios.

[0003] When there are significant generational differences in GPU nodes within a cluster, and some nodes experience dynamic load fluctuations, the shortcomings of existing centralized scheduling methods are further amplified: the central node's computing power status acquisition has inherent latency differences and single-point bottlenecks, easily leading to inconsistencies in computing power views between different nodes. Furthermore, the traditional allocation logic based solely on total computing power cannot match the differentiated adaptation requirements of different inference tasks for tensor cores, memory bandwidth, and quantization precision. This can easily result in mismatches where high-computing-power nodes execute low-adaptability tasks, while low-computing-power nodes bear high-load inference loads. Ultimately, this leads to a decrease in overall cluster inference throughput and severe end-to-end latency jitter. Moreover, a centralized node failure can directly paralyze the entire inference cluster scheduling, severely limiting the robustness and service ceiling of heterogeneous GPU inference clusters. Therefore, this invention proposes a decentralized self-organizing scheduling method and system for heterogeneous GPU inference nodes. Summary of the Invention

[0004] The purpose of this invention is to solve the problems in the background art, and to propose a decentralized heterogeneous GPU inference node self-organizing scheduling method and system.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: The first aspect of this invention provides a decentralized heterogeneous GPU inference node self-organizing scheduling method, comprising: S1. Each inference node sends its heterogeneous computing power texture fingerprint to peer inference nodes within the local area network segment in the form of periodic broadcast beacons; wherein, the heterogeneous computing power texture fingerprint is the encapsulation of real-time running status parameters according to a predefined binary structure. S2. The local storage service of each inference node receives broadcast beacons and Gossip anti-entropy messages from peer inference nodes, and obtains node status data. At the same time, it identifies differences in node status data and completes bidirectional convergence synchronization of node status data. During the bidirectional convergence synchronization process, it continuously monitors the view divergence entropy between each inference node and determines whether state drift calibration is triggered. S3. When the gateway of any inference node receives an external inference request, it performs deep semantic analysis of the inference request and uses the results of the deep semantic analysis of the model to filter out the set of online inference nodes and form a candidate inference node profile cluster. S4. Based on the heterogeneous computing power texture fingerprint of each candidate inference node in the candidate inference node profile cluster, the inference affinity coupling field score of the candidate inference node is performed, and then the optimal inference node is selected. S5. Forward the received inference request to the optimal inference node to execute model inference, receive the inference result and execution status receipt returned by the optimal inference node, parse and determine whether the isolation circuit breaker mechanism is triggered.

[0006] Further, S1 includes: Collect real-time running status parameters of the GPU of this inference node at the beginning of each broadcast cycle; A beacon decay stamp is appended to the broadcast beacon header, where the beacon decay stamp records the start timestamp of this broadcast cycle and the local monotonic clock count value of the inference node; An encapsulated broadcast beacon is sent to a multicast address within the local area network segment via the UDP protocol.

[0007] Furthermore, the real-time runtime status parameters include GPU architecture generation identifier, tensor core version, memory bandwidth level, context window length, quantization precision level, inference affinity, and request queue congestion depth.

[0008] Further, S2 includes: Initialize the local state database and maintain an independent Lamport clock version vector record for each known peer inference node; wherein, the Lamport clock version vector record includes Lamport clock count values ​​with the identifier of each known peer inference node as the dimension; Upon receiving a broadcast beacon from a peer inference node, the inference node identifier and state content carried in the broadcast beacon are parsed; the parsed inference node identifier and state content are used to construct a state record; the current Lamport clock count value in the clock version vector corresponding to the inference node is used to add a causal version mark to the state record and write it to the local state database. When the Gossip anti-entropy cycle arrives, several target inference nodes are randomly selected from the peer inference node list to establish an anti-entropy session, and the locally maintained version digest vector is sent to the target inference nodes. Receive the version digest vector returned by the target inference node, and compare it with the local version digest vector dimension by dimension according to the Lamport clock count value corresponding to each peer inference node, so as to identify the state record items where the two parties disagree. Determine the causal relationship of the state records with discrepancies, and obtain the state data of the node to be synchronized. The node state data to be synchronized is pushed incrementally to peer inference nodes with lagging state data, thus completing the bidirectional convergence and synchronization of node state data.

[0009] Furthermore, S2 also includes: After each Gossip anti-entropy synchronization is completed, count the number of dimensions in the version summary vector that differ between the local state database and the target inference node state database. Based on the number of dimensions that differ and the difference in the count values ​​in the Lamport clock version vector of the difference record, the view divergence entropy value between the current peer inference node pairs is quantitatively calculated. When the view divergence entropy value between any pair of inference nodes continuously exceeds the preset convergence threshold for a preset observation period, it is determined that the pair of peer inference nodes has experienced state drift. Actively initiate a full version summary vector retrieval request to the peer inference node that has experienced state drift, and perform overlay calibration on the state records of the corresponding inference node in the local state database.

[0010] Further, S3 includes: When the gateway receives an external inference request, it performs protocol parsing on the external inference request message to extract the request body; Parse the target inference model identifier field, input tensor dimension, sequence length, and target inference precision level from the request body; The input tensor morphology feature code is generated based on the input tensor dimension and sequence length to represent the load morphology for inference. Based on the target inference model identifier, retrieve all online inference node records loaded with the target inference model from the local state database; from the retrieval results, remove offline inference node records whose beacon decay stamp time exceeds the freshness threshold from the current time, forming an online inference node set; The heterogeneous computing power texture fingerprint recorded by each inference node in the online inference node set is initially matched with the input tensor morphological feature code and the target inference accuracy level to eliminate invalid inference nodes and form a candidate inference node profile cluster.

[0011] Further, S4 includes: Extract the heterogeneous computing power texture fingerprint of each candidate inference node in the candidate inference node profile cluster; Perform inference affinity coupling field scoring to generate a comprehensive scheduling score for each candidate inference node; Candidate inference nodes are sorted in descending order of their comprehensive scheduling scores, and the candidate inference node ranked first is selected as the optimal inference node.

[0012] Further, S5 includes: The parsed external inference request is encapsulated into an inter-inference node forwarding message according to a predefined internal communication protocol; The forwarded message is sent to the target inference node through a long connection channel established with the optimal inference node to execute the model inference task; After the inference task is completed, the optimal inference node will send back the inference result and the execution status receipt along the reverse link of the external inference request forwarding path; The execution indicators in the execution status receipt are analyzed and compared with the expected execution indicators estimated based on the heterogeneous computing power texture fingerprint of the inference node. The reputation evolution vector of the corresponding inference node in the local state database is updated incrementally based on the deviation comparison results.

[0013] Furthermore, S5 also includes: Continuously monitor the changing trend of the reputation evolution vector of each inference node in the local state database to determine whether the inference node has entered a detuned state; For inference nodes that enter a detuned state, trigger the isolation circuit breaker mechanism and set a temporary isolation flag for the inference node in the local state database; In the subsequent generation of candidate inference node profile clusters, inference nodes carrying temporary isolation markers are excluded from the candidate range; During the isolation period, the broadcast beacon of the inference node is continuously monitored. When the heterogeneous computing power texture fingerprint of the inference node is detected to have stabilized and the reputation evolution vector has returned to the normal range, the temporary isolation mark is automatically lifted.

[0014] A second aspect of the present invention provides a decentralized heterogeneous GPU inference node self-organizing scheduling system, comprising: Computing power texture fingerprint broadcasting module: Each inference node sends its heterogeneous computing power texture fingerprint to peer inference nodes within the local area network segment in the form of periodic broadcast beacons; Anti-entropy convergence synchronization module: The local storage service of each inference node receives broadcast beacons and Gossip anti-entropy messages from peer inference nodes, obtains node status data, identifies differences in node status data, and completes bidirectional convergence synchronization of node status data. Inference Request Parsing and Filtering Module: When the gateway of any inference node receives an external inference request, it performs deep model semantic parsing on the inference request, and uses the results of deep model semantic parsing as search conditions to filter out the set of online inference nodes, forming a candidate inference node profile cluster; Optimal inference node selection module: Based on the heterogeneous computing power texture fingerprint of each candidate inference node in the candidate inference node profile cluster, the inference affinity coupling field score of the candidate inference nodes is performed, and then the optimal inference node is selected. The inference forwarding receipt analysis module forwards the received inference request to the optimal inference node to execute model inference, receives the inference results and execution status receipts returned by the optimal inference node, analyzes and determines whether the isolation circuit breaker mechanism is triggered.

[0015] Compared with existing technologies, the beneficial effects of this invention in providing a decentralized heterogeneous GPU inference node self-organizing scheduling method and system are as follows: 1) By using a lightweight beacon mechanism with UDP multicast, core computing power characteristics such as GPU architecture generation, memory bandwidth, and queue congestion depth are encapsulated into standardized texture fingerprints. Combined with beacon decay stamps, low-latency synchronization of node states is achieved. This allows all peer nodes in the local area network to grasp the dynamic details of surrounding computing power in real time without relying on a central scheduler, reducing the communication overhead and latency of computing power state acquisition. By relying on Lamport clock version vector and Gossip anti-entropy mechanism, the divergence of state data between nodes is identified. Based on view divergence entropy quantization, state drift is monitored and automatic calibration is triggered, achieving eventual consistency of the state data of all nodes in the network without the endorsement of a central node. 2) By extracting key features such as model identifier, input tensor dimension, and precision level from inference requests, a unique input tensor morphological feature code is generated. This code first filters out online nodes that load the target model, then removes invalid nodes with mismatched computing power features to form a candidate profile cluster, narrowing the decision range for subsequent scheduling. This avoids assigning inference tasks to nodes with poor computing power adaptability from the source, improving the pre-screening efficiency of the overall scheduling. By constructing an inference affinity coupling field based on heterogeneous computing power texture fingerprints, a multi-dimensional feature of GPU tensor core capabilities and quantization precision adaptability is used to complete quantization scoring, selecting the optimal node with the highest matching degree to the current inference task. This allows heterogeneous GPUs with different architectures to leverage their computing power advantages, improving the inference execution efficiency of a single task. Efficient forwarding of inference requests is achieved through long-connection channels. The node reputation evolution vector is dynamically updated by comparing the deviation between the execution receipt and the expected indicators. Defective nodes are automatically temporarily isolated, and the marking is automatically removed after the state recovers. Faulty nodes are automatically removed without interrupting the overall scheduling service, improving the overall fault tolerance and operational stability of the decentralized inference cluster. Attached Figure Description

[0016] Figure 1 This is a flowchart of a decentralized heterogeneous GPU inference node self-organizing scheduling method proposed in this invention.

[0017] Figure 2 This is a block diagram of a decentralized heterogeneous GPU inference node self-organizing scheduling system proposed in this invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] Please see Figure 1 This invention provides a decentralized heterogeneous GPU inference node self-organizing scheduling method, comprising: S1. Each inference node sends its heterogeneous computing power texture fingerprint to peer inference nodes within the local area network segment in the form of periodic broadcast beacons. S2. The local storage service of each inference node receives broadcast beacons and Gossip anti-entropy messages from peer inference nodes, obtains node status data, identifies differences in node status data, and completes bidirectional convergence and synchronization of node status data. S3. When the gateway of any inference node receives an external inference request, it performs deep semantic analysis of the inference request and uses the results of the deep semantic analysis of the model to filter out the set of online inference nodes and form a candidate inference node profile cluster. S4. Based on the heterogeneous computing power texture fingerprint of each candidate inference node in the candidate inference node profile cluster, the inference affinity coupling field score of the candidate inference node is performed, and then the optimal inference node is selected. S5. Forward the received inference request to the optimal inference node to execute model inference, receive the inference result and execution status receipt returned by the optimal inference node, parse and determine whether the isolation circuit breaker mechanism is triggered.

[0020] In this embodiment of the invention, the specific implementation method of each inference node in S1 sending its heterogeneous computing power texture fingerprint to peer inference nodes within the local area network segment in the form of periodic broadcast beacons is as follows: S11. Collect the real-time running status parameters of the GPU of this inference node at the beginning of each broadcast cycle; Specifically, the system identifies the corresponding architecture generation, tensor core version, and memory bandwidth level based on the GPU device model; it enumerates all model instances currently loaded in the GPU memory, records the context window length and quantization precision level of each model instance, and marks the inference affinity of the model instance based on the average first-word latency and throughput performance of historical inference requests on that model instance; and it counts the number of inference requests currently waiting to be processed as the request queue congestion depth. Specifically, during the inference node startup phase, the GPU model string in the local PCI device list is read. If NVIDIA H100 is detected, the model is mapped to the architecture generation identifier 90; if NVIDIA A100 is detected, it is mapped to 80; and if NVIDIA RTX 4090 is detected, it is mapped to 89. The tensor core version is read from the unified computing device architecture device attribute interface. H100 returns the major version number 4 and the minor version number 0, and 0x40 is written during packaging. The memory bandwidth level is calculated based on the memory type and memory interface width. The HBM3 memory frequency is 3.2Gbps and the width is 5120 bits, so the bandwidth calculation result is 2048GB / s, and the write level is 204. When enumerating all model instances currently loaded in the GPU memory, the inference runtime interface is called to obtain a list of loaded model instances. For each model instance, the context window length of the instance is recorded. The context window length of the Llama-3-8B-Instruct instance is 8192 tokens, and the quantization precision level is INT8. The average first-word latency and throughput performance of the model instance in the last 100 historical inference requests are recorded. The average first-word latency is 230 milliseconds, and the throughput is 35 tokens per second. The inference affinity of a model instance is determined by a combination of average first-word latency and throughput performance: if the average first-word latency is less than 200 milliseconds and the throughput is greater than 40 tokens per second, the inference affinity is marked as 180 or higher; if the average first-word latency is greater than 800 milliseconds or the throughput is less than 10 tokens per second, the inference affinity is marked as 60 or lower; otherwise, the inference affinity is mapped to an integer between 60 and 180 based on the combined ranking of latency and throughput. When counting the number of inference requests currently waiting to be processed, the number of tasks that have not yet been allocated to the GPU executor is read from the local inference task scheduling queue. The number of requests currently waiting to be processed is 7. This value is written as the request queue congestion depth into the corresponding field of the heterogeneous computing power texture fingerprint.

[0021] S12. Encapsulate the real-time running status parameters into a heterogeneous computing power texture fingerprint according to a predefined binary structure, including: First, allocate a continuous memory buffer of fixed length 64 bytes and set all bytes of the buffer to 0; write 4 bytes of fixed magic number to the starting address of the buffer. The magic number is 0x47505546 in hexadecimal, which is used to quickly identify the binary structure as a heterogeneous computing power texture fingerprint on the receiving side. Following the magic number, the following fields are written sequentially: The first byte contains the GPU architecture generation identifier, using an enumerated value to represent the architecture: Ampere architecture is written as 80, Hopper architecture as 90, Adelaide architecture as 89, and Blackwell architecture as 100; The second byte contains the tensor core version number, with the lower 4 bits containing the major version number and the higher 4 bits containing the minor version number; for the 4th generation tensor core, 0x40 is written; The third and fourth bytes form a 16-bit unsigned integer containing the memory bandwidth level, written in GB / s, divided by 10 and rounded down; 2048GB / s is written as 204; The fifth and sixth bytes form a 16-bit unsigned integer containing the context window length, written in 100... The tokens are written in units of integers, with 32768 tokens written as 327; the 7th byte writes the quantization precision level, using an enumerated value: 0 for FP32, 1 for FP16, 2 for BF16, 3 for INT8, 4 for INT4, 5 for FP8, 6 for GPTQ, and 7 for AWQ; the 8th byte writes the inference affinity, using an integer between 0 and 255, with higher values ​​indicating higher inference affinity for the model instance; the 9th and 10th bytes form a 16-bit unsigned integer to write the request queue congestion depth; bytes 11 to 18 write the real-time GPU temperature sampling value; and bytes 19 and 20 write the memory fragmentation level. The last 4 bytes are written with a CRC32 checksum, covering all fields from the magic number to the memory fragmentation level; all multi-byte fields are arranged in little-endian byte order; if some fields in the buffer are not used, they are kept at 0; this binary structure is the heterogeneous computing power texture fingerprint, which is then concatenated with the beacon decay stamp.

[0022] S13. Add a beacon decay stamp to the broadcast beacon header, wherein the beacon decay stamp records the start timestamp of this broadcast cycle and the local monotonic clock count value of the inference node.

[0023] S14. Send an encapsulated broadcast beacon to a multicast address within the local area network segment via the UDP protocol.

[0024] In this embodiment of the invention, the detailed implementation steps for the local storage service of each inference node in S2 to receive broadcast beacons and Gossip anti-entropy messages from peer inference nodes, acquire node state data, identify differences in node state data, and complete bidirectional convergence synchronization of node state data include: S21. Initialize the local state database and maintain an independent Lamport clock version vector record for each known peer inference node; wherein, the Lamport clock version vector record includes Lamport clock count values ​​with the identifier of each known peer inference node as the dimension; Specifically, the local state database adopts a key-value storage structure. The key is composed of the inference node identifier and the state type string, and the value includes the state content, causal version flag, and update time. First, the initial known peer inference node identifiers are read from the preset node discovery list or configuration file. The first 8 bytes of the node public key hash are taken as the unique identifier for each node identifier. An independent Lamport clock version vector record is established for each known peer inference node. This vector is divided into dimensions based on all known peer inference node identifiers, and each dimension stores a 4-byte unsigned integer count value. During the initialization phase, all dimension counts are set to 0, and the count of the corresponding dimension of the local inference node is set to 1, indicating that the local inference node has generated version 1 of the state. When a new peer inference node is added during the operation, the local inference node will add a new dimension to all existing version vectors and initialize the count of the new dimension to 0. At the same time, it will create an independent version vector record for the new peer inference node. When a state record is written to the local state database, the corresponding version vector is stored as a causal version marker to provide a comparison basis for subsequent Gossip anti-entropy synchronization.

[0025] S22. Upon receiving a broadcast beacon sent by a peer inference node, parse the inference node identifier and state content carried in the broadcast beacon; construct a state record from the parsed inference node identifier and state content; and use the current Lamport clock count value in the clock version vector corresponding to the inference node maintained locally to add a causal version mark to the state record and write it to the local state database. Specifically, the local inference node first parses the inference node identifier field and the state content field in the broadcast beacon; constructs a state record by combining the parsed inference node identifier and state content; extracts the sender's 8-byte node identifier and variable-length heterogeneous computing power texture fingerprint; then finds the corresponding sender's inference node's Lamport clock version vector from the locally maintained version vector record set, reads the current Lamport clock count value of the sender node dimension in the vector, which is currently 5, and increments the count value to 6; and uses the incremented complete version vector as the causal version marker of the state record, and writes it into the local state database along with the parsed state content, sender node identifier, and local reception time. Before writing, a version comparison is performed. If an old record of the same sender and state type already exists locally, the causal version tags of the new and old records are compared: if the version vector of the new record is strictly greater than that of the old record in the sender dimension, the new record is used to overwrite the old record; if the version vector of the new record is not strictly greater than that of the old record, the old record is retained and duplicate beacons are discarded. This writing process ensures that the state update carried by each broadcast beacon has a traceable causal version tag in the local database.

[0026] S23. When the Gossip anti-entropy cycle arrives, randomly select several target inference nodes from the peer inference node list to establish an anti-entropy session and send the locally maintained version summary vector to the target inference nodes. Understandably, the local inference node randomly selects several target inference nodes from the peer inference node list to establish an anti-entropy session; the peer inference node list consists of peer inference nodes that have remained active in the most recent Gossip cycle in the local state database, and each peer inference node list item contains node identifier, IP address and port number; the number of currently known active nodes is 24, so the number of randomly selected nodes is no more than 5 and at least 1; local inference nodes and peer inference nodes in an isolated state are excluded during selection; After selection, the local inference node sends a locally maintained version digest vector to each target inference node. The version digest vector consists of the current count value in the Lamport clock version vector corresponding to each peer inference node that is maintained locally. The versions digest vector is arranged in ascending order according to the peer inference node identifier. Each count value occupies 4 bytes. The first part of the vector contains 4 bytes of dimension count, and the last part contains 4 bytes of CRC32 checksum. The version digest vector of the local inference node contains 24 dimensions, with 24 in the dimension count field, followed by 96 bytes of count value, and finally 4 bytes of checksum. The transmission uses either UDP or TCP channels, with priority given to existing reliable transmission channels.

[0027] S24. Receive the version digest vector returned by the target inference node, and compare it with the local version digest vector dimension by dimension according to the Lamport clock count value corresponding to each peer inference node, so as to identify the state record items where the two parties have disagreements. Specifically, the local inference node first verifies the CRC32 at the end, then parses the dimension count field; it compares the target version digest vector and the local version digest vector dimension by dimension according to the Lamport clock count value corresponding to each peer inference node identifier; if the count value of inference node A in the local version digest vector is 15, and the count value of peer inference node A in the target version digest vector is 12, then the local inference node has 3 more updates than the target inference node in the dimension of peer inference node A, and the target inference node is missing 3 status records of peer inference node A (13, 14, and 15); if the count value of peer inference node B in the target version digest vector is 9, and the count value in the local version digest vector is 7, then the target inference node has 2 more updates than the local inference node in the dimension of peer inference node B, and the local inference node is missing 2 status records of peer inference node B (8 and 9). During the comparison process, the direction of the difference in each dimension is recorded, which is divided into three types: local inference node leading, local inference node lagging, and consistent. All dimensions that are leading in local inference nodes and those that are leading in the target inference node, along with their count differences, are summarized into a list of divergence status records. The status record corresponding to the dimension that is leading in local inference nodes is the incremental data that the local inference node needs to push to the target inference node. The status record corresponding to the dimension that is leading in the target inference node is the data that the local inference node needs to pull from the target inference node.

[0028] S25. Based on the causal version vector pruning mechanism, determine the causal relationship of the state record items with discrepancies and obtain the state data of the node to be synchronized: determine whether there is a clear causal sequence relationship between the two state records according to the Lamport clock version vectors corresponding to the two state records; if there is a causal sequence relationship, only the newer version record with the causal successor relationship is retained as the state data of the node to be synchronized; if it is a concurrent conflict relationship, select the record with the higher dimension of the Lamport clock version vector as the state data of the node to be synchronized.

[0029] S26. Push the state data of the node to be synchronized to the peer inference node with state lag in an incremental manner through a reliable transmission channel to complete the bidirectional convergence synchronization of the node state data, including: The sender reads the state records to be pushed one by one from the local state database according to the list of divergent state record items. Each state record is encapsulated into a synchronization message, which includes node identifier, state type, causal version marker, state content length and state content body. The synchronization message is sent through TCP long connection or QUIC reliable channel, and a 4-byte message total length field and a 4-byte CRC32 check value are added before sending. After receiving the synchronization message, the receiver first performs verification, and then performs a causal comparison with the corresponding local record based on the causal version tag. It only writes when the local record is lagging behind. The sender does not push the entire state database to the target inference node, but only pushes the state records in the local version digest vector that are ahead of the target version digest vector, thereby achieving incremental synchronization. After a batch of pushes is completed, the receiver sends back a confirmation message, the sender updates the synchronization status of its local inference node and the target inference node, and enters the bidirectional convergence synchronization process; if there is a leading dimension in both directions, both parties simultaneously push their respective leading status records to each other to achieve bidirectional convergence.

[0030] Among these measures, during the bidirectional convergence synchronization process, the view divergence entropy between each inference node is continuously monitored to determine whether state drift calibration is triggered, including: W1. After each Gossip anti-entropy synchronization is completed, count the number of dimensions in which the version summary vector differs between the local state database and the target inference node state database. Specifically, the local inference node retains a snapshot of the version digest vector returned by the target inference node in the most recent anti-entropy session. This snapshot is compared with the current local version digest vector dimension by dimension. When the count value on a node identifier dimension is not equal, that dimension is included in the difference dimension. Given that the total number of peer nodes is 24, the local inference node has 15 and the target inference node has 12 in the dimension of peer inference node A, and the local inference node has 7 and the target inference node has 9 in the dimension of peer inference node B. The count values ​​of the remaining 22 dimensions are the same, so the number of difference dimensions is 2. After the statistics are completed, the number of difference dimensions, together with the difference in the count values ​​of each difference dimension, are cached for subsequent quantification of view divergence entropy value. This statistics do not depend on the message payload content, but only on the count value in the version digest vector.

[0031] W2. Based on the number of differing dimensions and the difference in count values ​​in the Lamport clock version vector, the view divergence entropy value between the current peer inference node pairs is quantified, including: First, for each dimension where there is a difference, calculate the absolute value of the difference between the count values ​​of the two versions of the summary vector in that dimension. The absolute value of the difference for peer inference node A is 3, and the absolute value of the difference for peer inference node B is 2. Preset difference segmentation weights: These are obtained during initialization based on the statistical impact of different count value differences on the state convergence speed during historical synchronization. The larger the difference, the more severe the state divergence, and the higher the corresponding single-dimensional divergence contribution value. Then, according to the preset difference segmentation weights, map the count value difference of each difference dimension to a single-dimensional divergence contribution value. Differences between 1 and 2 are mapped to 0.2, differences between 3 and 5 are mapped to 0.5, and differences between 6 and 10 are mapped to... The value is 1, and a difference greater than 10 is mapped to 1.5; the sum of the single-dimensional divergence contribution values ​​of all difference dimensions is used to obtain the original total divergence; the original total divergence is then divided by the current known total number of peer inference nodes and multiplied by 100 to obtain the view divergence entropy value between the current peer inference node pairs; the total number of known peer inference nodes is 24, the single-dimensional divergence contribution values ​​of the two difference dimensions are 0.5 and 0.2 respectively, the original total divergence is 0.7, and the view divergence entropy value is equal to 0.7 divided by 24 and then multiplied by 100, which is approximately 2.9; where, the higher the view divergence entropy value, the greater the divergence between the two states; if the number of difference dimensions is 0, the view divergence entropy value is directly set to 0.

[0032] W3. When the view divergence entropy value between any pair of inference nodes is detected to continuously exceed the preset convergence threshold and reach the preset observation period, it is determined that the pair of peer inference nodes has experienced state drift. Understandably, the preset convergence threshold is obtained by statistically analyzing the historical synchronization records during the stable operation phase of the local state database. The upper quartile of the view divergence entropy value obtained from 100 consecutive rounds of Gossip anti-entropy synchronization process is selected. The upper quartile of the historical stable period is 18, so the convergence threshold is set to 20. Preset observation period: set to 5 consecutive Gossip anti-entropy periods; when the view divergence entropy value between the local inference node and a certain target inference node exceeds the convergence threshold of 20 for 5 consecutive Gossip anti-entropy periods, it is determined that the peer inference node pair has experienced state drift and enters the coverage calibration process.

[0033] W4. Proactively initiate a full version summary vector retrieval request to the peer inference node that has experienced state drift, and perform overlay calibration on the state record of the corresponding inference node in the local state database. Specifically, the local inference node proactively initiates a full version digest vector retrieval request to the peer inference node experiencing state drift. Upon receiving the request, the target inference node returns its local complete version digest vector and the corresponding state record metadata for each dimension. The local inference node compares the returned version digest vector with its local version digest vector dimension by dimension, identifies all missing or outdated state records, and requests the complete content of these state records from the target inference node. The target inference node sends the complete state records to the local node in batches via a reliable transmission channel. The local inference node performs overlay calibration on the state records of the corresponding inference node, that is, for each returned state record, it directly uses the target version vector and state content to overwrite the old record with the same key value in the local state database, without merging fields or retaining the old local version. After the overlay is completed, the local inference node replaces the version vector corresponding to the target inference node with the version vector returned by the target and resets the view divergence entropy statistics of the peer inference node pair to 0. After calibration, it re-enters regular Gossip anti-entropy synchronization until the view divergence entropy values ​​between all inference node pairs in the entire network are lower than the convergence threshold, so as to achieve the final consistency convergence of the state view of the entire network.

[0034] In this embodiment of the invention, the specific implementation of step S3, whereby when the gateway of any inference node receives an external inference request, the inference request is subjected to deep model semantic parsing, and the online inference node set is selected based on the results of the deep model semantic parsing as retrieval conditions to form a candidate inference node profile cluster, includes: S31. When the gateway receives an external inference request, it performs protocol parsing on the external inference request message to extract the request body.

[0035] S32. The specific implementation method for parsing the target inference model identifier field, input tensor dimension, sequence length, and target inference precision level from the request body is as follows: If the external inference request is sent via HTTP, the HTTP request header and request body are read. The Content-Type field in the request header indicates whether the payload format is JSON or Protobuf. If it is sent via gRPC, the request body is deserialized according to the predefined service description file. The request body is converted from a binary request body into structured fields. Here, gRPC is a remote procedure call framework, and the predefined service description file is a protocol buffer file that defines the structure of request and response messages. The extracted request body enters the field parsing process: The target inference model identifier field is parsed from the request body; the `model_name` field in the JSON object has the value `llama-3-8b-instruct`. The input tensor dimension field is parsed; the `input_shape` field has the value of a 1-dimensional array containing 1 and 4096, indicating a batch size of 1 and a sequence length of 4096. The sequence length field is parsed; the `sequence_length` field has the value of 2048. The target inference precision tier field is parsed; the `precision_tier` field has the value of FP16 or INT8. If the request body does not contain a sequence length field, the sequence length is inferred from the last dimension of the input tensor dimension array; if the request body does not contain a target inference precision level field, the default value BF16 is used; after parsing, the target inference model identifier, input tensor dimensions, sequence length, and target inference precision level are written to the structured request context object.

[0036] S33. The specific process of generating the input tensor morphological feature code for representation inference and calculating the load morphology based on the input tensor dimension and sequence length includes: First, concatenate the values ​​of each dimension in the input tensor dimension array sequentially into a string separated by dots. For example, the 1-dimensional array 1 and 4096 are concatenated into the string 1.4096. If the input tensor dimension is a 4-dimensional array 2, 3, 512, 512, then it is concatenated into the string 2.3.512.512. Then, concatenate this dimension string with the sequence length string. The sequence length string is directly converted from the value of the sequence length field to a string. For example, if the sequence length field is 2048, then the sequence length string is 2048. Concatenate the dimension string 1.4096 and the sequence length string 2048 with dots to form the complete load pattern string 1.4096.2048. The complete payload form string is input into a pre-defined lightweight hash function, FNV-1a, a 64-bit hash function, which outputs a 64-bit unsigned integer as the input tensor form feature code. This input tensor form feature code can distinguish different inference payloads such as text sequence form, image form, and video form, and provides a unified numerical index for subsequent matching with the context window length and computation form adaptation field in the heterogeneous computing power texture fingerprint. At the same time, after the input tensor form feature code is generated, the original input tensor dimension and sequence length are retained in the request context, so that the original information is not lost due to hash mapping.

[0037] S34. Based on the target inference model identifier, retrieve all online inference node records loaded with the target inference model in the local state database; from the retrieval results, remove offline inference node records whose beacon decay stamp time is more than the freshness threshold from the current time, and form an online inference node set; Specifically, using the target inference model identifier as an index prefix, a prefix scan is performed in the local state database to find node identifiers in all state records whose loaded model instance list contains the target inference model identifier. The target inference model identifier is llama-3-8b-instruct. After scanning, three node identifiers are obtained: peer inference node A, peer inference node B, and peer inference node C. Then, for each node identifier, the online status field and beacon decay stamp in its most recent broadcast beacon are read from the node state record. Peer inference nodes that have declared themselves offline or whose beacon decay stamps exceed the freshness threshold are removed to obtain the set of online inference nodes. The freshness threshold is determined based on twice the sum of the maximum transmission delay and processing delay of the peer inference node broadcast beacon in the local area network. In this embodiment, it is set to 3 seconds.

[0038] S35. Perform preliminary matching between the heterogeneous computing power texture fingerprint recorded by each inference node in the online inference node set and the input tensor morphological feature code and target inference precision level, eliminate invalid inference nodes, and form a candidate inference node profile cluster; among them, invalid inference nodes are inference nodes that cannot carry the current request due to insufficient video memory or incompatibility of quantization precision. Specifically, firstly, a heterogeneous computing power texture fingerprint is extracted from the state record of each candidate inference node. The quantization precision level field, context window length field, memory margin field, and tensor core version bit field are read from the fingerprint. The quantization precision level field is compared with the target inference precision level. If the quantization precision level list of the target model instance loaded by the candidate inference node does not contain the target inference precision level, the candidate inference node is directly eliminated. If the target inference precision level is INT8 and the target model instance loaded by inference node A only contains FP16 and BF16, then inference node A is eliminated. Then, the context window length field is compared with the input sequence length. If the context window length of the candidate inference node that has loaded the target model instance is less than the input sequence length, the inference node is removed. Next, the required GPU memory for the current request is estimated based on the input tensor dimension and the sequence length. When the sequence length is 4096, the estimated activation GPU memory requirement is 2.1GB. If the GPU memory of the candidate inference node is less than 2.1GB, the candidate inference node is removed. Finally, the tensor core version of the candidate inference node is compared with the minimum tensor core version required by the target inference model. The target inference model requires a 4th generation tensor core, while inference node B is a 3rd generation, so inference node B is removed. After removing all invalid inference nodes, the remaining inference nodes constitute the candidate inference node profile cluster.

[0039] In this embodiment of the invention, the detailed implementation steps of S4, which involves scoring the inference affinity coupling field of candidate inference nodes based on the heterogeneous computing power texture fingerprint of each candidate inference node in the candidate inference node profile cluster, and then selecting the optimal inference node, include: S41. Extract the heterogeneous computing power texture fingerprint of each candidate inference node in the candidate inference node portrait cluster; Specifically, the process iterates through each candidate inference node identifier in the candidate inference node profile cluster, reads the state record of the most recent successful write to the candidate inference node in the local state database, and extracts the heterogeneous computing power texture fingerprint field from the state record. This field is a binary structure, and during parsing, it reads fields such as architecture generation identifier, tensor core version bit, memory bandwidth level, context window length, quantization precision level, inference affinity, request queue congestion depth, GPU temperature, and memory fragmentation level in sequence according to the defined binary structure. If the most recent state record of the candidate inference node in the local state database is missing or exceeds the freshness threshold, the heterogeneous computing power texture fingerprint carried in the most recent broadcast beacon of the node is directly parsed as a supplement.

[0040] S42. Perform inference affinity coupling field scoring to generate a comprehensive scheduling score for each candidate inference node, including: S421. For each candidate inference node, the specific process for evaluating the congestion margin elasticity based on the deviation between its request queue congestion depth and the maximum concurrency that the inference node can historically handle is as follows: The request queue congestion depth is read from the heterogeneous computing power texture fingerprint of the candidate inference node, and the current value is 26. The upper quartile of the number of tasks processed by the inference node in the past 100 Gossip anti-entropy cycles is read from the historical running records of the local state database as the historical maximum concurrency, which is 32. The remaining concurrency capacity is obtained by subtracting the current request queue congestion depth from the historical maximum concurrency. If the current queue congestion depth is less than 50% of the historical maximum concurrency, the congestion margin elasticity is marked as high, with a value of 1; if the current queue depth is between 50% and 80% of the historical maximum concurrency, the congestion margin elasticity is marked as medium, with a value of 0.6; if the current queue depth exceeds 80% of the historical maximum concurrency but does not reach 100%, the congestion margin elasticity is marked as low, with a value of 0.3; if the current queue depth is greater than or equal to the historical maximum concurrency, the congestion margin elasticity is marked as very low, with a value of 0.1. For example, the current request queue congestion depth is 26, the historical maximum concurrency is 32, accounting for approximately 81%, and the congestion margin elasticity is set to 0.3. The marginal elasticity threshold is obtained as follows: 90% of the historical maximum concurrency of this inference node is set as the marginal elasticity threshold. 90% of the historical maximum concurrency of 32 is 28.8, so the marginal elasticity threshold is set to 28. The current request queue congestion depth of 26 is close to the marginal elasticity threshold of 28, further reducing the first scheduling score of this candidate inference node. This congestion margin elasticity is used as the input for adjusting the congestion penalty of the subsequent first scheduling score.

[0041] S422. The specific process for determining the temperature throttling precursor warning level based on the difference between the real-time GPU temperature sample value carried in the broadcast beacon of each candidate inference node and the temperature throttling trigger threshold corresponding to the GPU architecture is as follows: The real-time GPU temperature sample value is read from the heterogeneous computing power texture fingerprint carried in the candidate inference node's most recent broadcast beacon, and is read as 82 degrees Celsius. Based on the GPU architecture generation identifier of the candidate inference node, the locally pre-configured GPU architecture temperature throttling trigger threshold table is looked up. This temperature throttling trigger threshold table is extracted from the architecture specification documents published by the GPU manufacturer and pre-configured in the local configuration file during inference node deployment. It is used to query the temperature throttling trigger threshold for the corresponding architecture based on the GPU architecture generation identifier. The temperature throttling trigger threshold for the Hopper architecture is 86 degrees Celsius, for the Ampere architecture it is 83 degrees Celsius, and for the Adidas architecture it is 85 degrees Celsius. Calculate the difference between the current temperature and the temperature throttling trigger threshold. For example, the difference between 82 degrees Celsius and 86 degrees Celsius is 4 degrees Celsius. Determine the temperature throttling warning level based on the magnitude of the difference: a difference greater than 10 degrees Celsius is set to level 0, a difference between 5 and 10 degrees Celsius is set to level 1, a difference between 2 and 5 degrees Celsius is set to level 2, and a difference less than 2 degrees Celsius is set to level 3; a difference of 4 degrees Celsius corresponds to warning level 2. The higher the warning level, the closer the candidate inference node is to the temperature throttling trigger point, and the greater the risk of GPU frequency reduction and inference latency increase. This temperature throttling warning level will be used as input for temperature risk reduction in the subsequent scoring process.

[0042] S423. The specific process for evaluating memory fragmentation based on the distribution span of loaded model instances in video memory and the size of contiguous video memory blocks for each candidate inference node is as follows: Read the distribution span and contiguous memory block size of loaded model instances in video memory from the heterogeneous computing power texture fingerprint of the candidate inference node; the distribution span refers to the total length of the address range occupied by the model instance in the video memory of the candidate inference node from the start address to the end address, and the contiguous memory block size refers to the length of the largest contiguous free address region in the video memory. The total video memory capacity of the candidate inference node is 80GB. Three model instances have been loaded, occupying starting address offsets of 0-8GB, 12-24GB, and 30-35GB respectively, with a distribution span of 35GB. The length of the largest contiguous free address region is 6GB. The ratio of the length of the largest contiguous free address region to the total free video memory length is calculated. The total free video memory length is 45GB, and the length of the largest contiguous free address region is 6GB, which is approximately 13.3%. If the ratio is less than 40%, the video memory fragmentation is marked as high; if the ratio is between 40% and 70%, it is marked as medium; if the ratio is greater than 70%, it is marked as low. High fragmentation indicates that the subsequent video memory continuity correction coefficient will take a smaller value, reducing the fourth scheduling score of the candidate inference node.

[0043] S424. Based on the time corresponding to the beacon decay stamp of the most recent state record of each candidate inference node, calculate the time decay degree between that time and the current time. The specific process for calculating the Gossip synchronization freshness decay value is as follows: The beacon decay stamp field in the most recent state record of the candidate inference node is read, and the broadcast cycle start timestamp and local monotonic clock count value are parsed out. The state aging duration is obtained by subtracting the broadcast cycle start timestamp from the current time. The time difference between the current time and the corresponding beacon decay stamp is 0.8 seconds. If the time difference is less than 1 second, the freshness decay value is set to 0. If the time difference is between 1 and 3 seconds, the freshness decay value is set to 0.2. If the time difference is between 3 and 10 seconds, the freshness decay value is set to 0.5. If the time difference exceeds 10 seconds, the freshness decay value is set to 0.8. A time difference of 0.8 seconds corresponds to a freshness decay value of 0. The higher the freshness decay value, the more outdated the state information of the candidate inference node is, and the greater the discount in subsequent scoring, thus avoiding scheduling requests to candidate inference nodes with lagging state updates.

[0044] S425. The specific process of using the GPU architecture generation identifier of the candidate inference node and the hardware acceleration adaptation of the tensor core version to the target inference model as the first scheduling score is as follows: A local hardware compatibility table for the target inference model is maintained, recording the optimal GPU architecture generation identifier and the lowest tensor core version number supported by the target model. The target inference model is llama-3-8b-instruct, the optimal GPU architecture generation identifier is 90, and the lowest tensor core version number is 4th generation. The architecture generation identifier of the candidate inference node is compared with the optimal architecture generation identifier of the target model. If they are completely identical, the base score for hardware acceleration adaptation is 90; if they differ by 1 generation, the base score is 75; if they differ by 2 generations or more, the base score is 55. Then, the tensor core version number of the candidate inference node is compared with the lowest tensor core version number of the target inference model. If the tensor core version number of the candidate node is higher than the lowest tensor core version number, the base score is increased by 5 points; if it is equal to the lowest tensor core version number, the base score remains unchanged; if it is lower than the lowest tensor core version number, the base score is decreased by 20 points. If the architecture generation identifier of the candidate node is 90 and the tensor core version number is 4th generation, the first scheduling score is 95.

[0045] S426. The specific process of adjusting the first scheduling score value by applying congestion penalty using the congestion margin to obtain the second scheduling score value is as follows: After obtaining the first scheduling score, the first scheduling score is multiplied by the congestion margin elasticity margin to obtain the second scheduling score. If the first scheduling score is 95 and the congestion margin elasticity margin is 0.3, then the second scheduling score is 28.5. If the congestion depth of the request queue is close to the marginal elasticity threshold, the penalty weight is further increased. Specifically, before the multiplicative adjustment, if the difference between the current request queue congestion depth and the marginal elasticity threshold is less than or equal to 2, then the original congestion margin elasticity margin is multiplied by 0.5; if the difference between the current request queue congestion depth and the marginal elasticity threshold is less than or equal to 1, then the original congestion margin elasticity margin is multiplied by 0.3. For example, if the current request queue congestion depth is 26, the marginal elasticity threshold is 28, and the difference is 2, then the congestion marginal elasticity margin decreases from 0.3 to 0.15, and the second scheduling score further decreases to 14.25. Through this penalty adjustment, the second scheduling score of candidate inference nodes that are severely congested and close to the capacity limit is significantly reduced, thereby reducing the probability of being selected.

[0046] S427. The specific process of reducing the temperature risk of the second scheduling score value to obtain the third scheduling score value based on the temperature throttling precursor warning level is as follows: During initialization, the temperature risk reduction coefficient is set according to the relationship curve between the GPU temperature throttling trigger threshold and the inference task latency degradation. Temperature throttling warning level 0 corresponds to temperature risk reduction coefficient 1, temperature throttling warning level 1 corresponds to temperature risk reduction coefficient 0.9, temperature throttling warning level 2 corresponds to temperature risk reduction coefficient 0.7, and temperature throttling warning level 3 corresponds to temperature risk reduction coefficient 0.5. The second scheduling score is multiplied by the temperature risk reduction factor corresponding to the current candidate inference node's temperature throttling precursor warning level to obtain the third scheduling score. For example, if the second scheduling score is 14.25, the current temperature throttling precursor warning level is 2, and the temperature risk reduction factor is 0.7, then the third scheduling score is 9.975. As the temperature throttling precursor warning level increases, the temperature risk reduction factor decreases progressively. For each level increase in the temperature throttling precursor warning level, the temperature risk reduction increases, putting candidate inference nodes whose temperatures are close to the throttling threshold at a significant disadvantage in the scoring. If the GPU temperature of a candidate inference node has reached or exceeded the temperature throttling trigger threshold, the third scheduling score of that candidate inference node is directly set to 0 and it is not included in the subsequent ranking.

[0047] S428. The specific process of correcting the third scheduling score for memory continuity based on memory fragmentation and disorder to obtain the fourth scheduling score is as follows: A preset memory continuity correction coefficient is used, which is obtained based on the impact of memory fragmentation on the model loading success rate. Low memory fragmentation corresponds to a memory continuity correction coefficient of 1, medium memory fragmentation corresponds to a memory continuity correction coefficient of 0.95, and high memory fragmentation corresponds to a memory continuity correction coefficient of 0.85. The third scheduling score is multiplied by the memory continuity correction coefficient corresponding to the memory fragmentation to obtain the fourth scheduling score. For example, if the third scheduling score is 9.975, the memory fragmentation is high, and the memory continuity correction coefficient is 0.85, then the fourth scheduling score is 8.48. If the memory fragmentation of a candidate inference node reaches a high level, it indicates that there are many discontinuous free areas in the memory of the candidate inference node. When loading new model instances or allocating large continuous activation tensors, there may be a risk of memory allocation failure, thus reducing its score. During the correction process, the higher the memory fragmentation, the smaller the memory continuity correction coefficient, and the lower the score after correction. S429. The specific process of using the Gossip synchronization freshness decay value to apply a freshness discount to the fourth scheduling score value to obtain the comprehensive scheduling score value of the candidate inference node is as follows: The freshness discount coefficient is obtained based on the statistical impact of state staleness on scheduling hit rate. The freshness discount coefficient is 1 when the Gossip synchronization freshness decay value is 0, 0.9 when it is 0.2, 0.75 when it is 0.5, and 0.5 when it is 0.8. The more severe the freshness decay, the greater the discount. The fourth scheduling score is multiplied by the freshness discount coefficient corresponding to the Gossip synchronization freshness decay value of the current candidate inference node to obtain the comprehensive scheduling score of that candidate inference node. For example, if the fourth scheduling score is 8.48, the Gossip synchronization freshness decay value is 0, and the freshness discount coefficient is 1, then the overall scheduling score remains 8.48. If the Gossip synchronization freshness decay value of another candidate inference node is 0.5, then its fourth scheduling score is multiplied by 0.75, and the overall scheduling score is further reduced. This discount puts candidate inference nodes with lagging state updates at a disadvantage in the ranking, thus prioritizing the selection of candidate inference nodes with higher state freshness to execute inference requests.

[0048] S43. Sort the candidate inference nodes in descending order of their comprehensive scheduling score, and select the candidate inference node with the highest score as the optimal inference node.

[0049] In this embodiment of the invention, the specific implementation of step S5—forwarding the received inference request to the optimal inference node for model inference, receiving the inference result and execution status receipt returned by the optimal inference node, and parsing and determining whether to trigger the isolation circuit breaker mechanism—is as follows: S51. Encapsulate the parsed external inference request into an inter-inference node forwarding message according to a predefined internal communication protocol, including: The internal communication protocol message consists of a fixed header and a variable payload. The fixed header contains a 2-byte protocol identifier (value 0x4750); a 1-byte version number; a 16-byte request identifier (UUID); an 8-byte source node identifier; an 8-byte target node identifier; a 1-byte message type (inference request type, value 0x01); and a 4-byte total payload length. The variable payload is encoded field-by-field using a type-length-value format, with each field consisting of a 1-byte field type, a 2-byte field length, and a field value. Fields requiring encapsulation include the target inference model identifier field. The segment type is 0x01; the input tensor dimension field is 0x02; the sequence length field is 0x03; the target inference precision level field is 0x04; the input data reference or input binary content field is 0x05; the request priority field is 0x06; the sending timestamp field is 0x07; all multi-byte fields are encoded in little-endian byte order; after encoding, a 4-byte CRC32 checksum is appended to the end of the message to generate a complete inference node forwarding message.

[0050] S52. The specific method for executing the model inference task is as follows: The forwarded message is sent to the target inference node through the long connection channel established with the optimal inference node. The local gateway maintains a pool of long-lived connections to each inference node, with each connection corresponding to a target inference node. The connection protocol is either TCP or QUIC. Before sending, it checks whether there is a valid connection to the optimal inference node in the long-lived connection pool. If the connection has been broken or not established, it recreates the connection and completes the handshake. The encapsulated forwarding message is written to the connection sending buffer, and the sending timestamp and message request identifier are recorded after the sending is completed. The sending process adopts an asynchronous non-blocking mode, and the local gateway can continue to process other external inference requests while waiting for the return result. After receiving the forwarding message, the target inference node parses and executes the model inference task according to the internal communication protocol.

[0051] S53. After the inference task is completed, the optimal inference node will send back the inference result and the execution status receipt along the reverse link of the external inference request forwarding path.

[0052] S54. Analyze the execution metrics in the execution status receipt and compare them with the expected execution metrics estimated based on the heterogeneous computing power texture fingerprint of the inference node. Among them, the execution metrics in the execution status receipt include at least the actual first-word latency of this inference task, the total inference time, the peak actual GPU utilization, and whether any abnormal retry occurred during the execution process. After receiving the execution status receipt, the local gateway first parses the actual first-word latency value, which is 340 milliseconds; the actual total inference time is 2700 milliseconds; the actual peak GPU utilization is 72%; and the abnormal retry flag is 0. Based on the heterogeneous computing power texture fingerprint of this inference node and the input tensor morphological feature code of this inference request, the expected execution indicators are read (obtained by offline regression of the historical execution records of this inference node): the expected first-word latency is 300 milliseconds, the expected total inference time is 2500 milliseconds, and the expected peak GPU utilization is 78%. The actual values ​​are compared with the expected values ​​one by one, and the deviation of the first-word latency is calculated to be 40 milliseconds, the deviation of the total inference time is 200 milliseconds, and the deviation of the peak GPU utilization is 6 percentage points. If the abnormal retry flag is 1, an abnormal retry event is recorded. The deviation comparison result is converted into a standardized deviation value, which is used as the input for subsequent reputation evolution vector updates.

[0053] S55. Update the reputation evolution vector of the corresponding inference node in the local state database in an incremental manner based on the deviation comparison results. Understandably, the reputation evolution vector records the average performance deviation and anomaly frequency of the inference node in recent inference tasks. The average performance deviation is updated using an exponentially weighted moving average. The old average performance deviation value of this inference node is stored locally. The old average performance deviation value is 0.12. The current comprehensive deviation is normalized to 0.18. When updating, the old average performance deviation value is multiplied by 0.8 and the current deviation is multiplied by 0.2 to obtain the new average performance deviation of 0.132. The anomaly frequency is counted using a sliding window method, with the window length set to the last 100 inference tasks. If the anomaly retry flag is 1 in the current execution status receipt, the anomaly event count within the window is incremented by 1, and the anomaly frequency is equal to the anomaly event count divided by the window length. If there are no anomalies in the current execution, the anomaly event count within the window remains unchanged. The updated average performance deviation and anomaly frequency, along with the update timestamp, are written back to the reputation evolution vector record corresponding to the inference node in the local status database.

[0054] S56. Continuously monitor the changing trend of the reputation evolution vector of each inference node in the local state database and determine whether the inference node has entered a detuned state: when the abnormal frequency index in the reputation evolution vector of any inference node exceeds the preset abnormal frequency threshold, or the average performance deviation continues to drift in the direction of deterioration and exceeds the preset drift amplitude threshold, the inference node is determined to have entered a detuned state. Among them, the preset anomaly frequency threshold is obtained from the statistics of the local stable operation phase. It is the sum of the average value of the anomaly frequency of the most recent 500 inference tasks and 3 times the standard deviation. The average value is 0.03 and the standard deviation is 0.02, so the anomaly frequency threshold is 0.09. The preset drift amplitude threshold is obtained from the sum of the average performance deviation during the local stable operation phase and twice the standard deviation. The average performance deviation has a mean of 0.15 and a standard deviation of 0.04, so the drift amplitude threshold is 0.23. When the frequency of anomalies in the reputation evolution vector exceeds the anomaly frequency threshold of 0.09 or the average performance deviation continues to drift in the direction of deterioration and exceeds the drift amplitude threshold of 0.23, the inference node is determined to have entered a detuned state.

[0055] S57. The process of triggering the isolation circuit breaker mechanism for an inference node that enters a detuned state and setting a temporary isolation flag for the inference node in the local state database is as follows: Add an isolation flag field to the node status record of the inference node in the local state database, set the field to 1, and write the isolation trigger timestamp, isolation reason, and reputation evolution vector snapshot at the time of detuning to the same status record; the isolation trigger timestamp is the local current time, the isolation reason is specified as the abnormal frequency exceeding the limit or the average performance deviation drift exceeding the limit, and the reputation evolution vector snapshot saves the average performance deviation value and abnormal frequency value at the time of triggering; at the same time, mark the detuned inference node as unschedulable from the peer inference node list, and directly exclude peer inference nodes carrying the isolation flag in the subsequent candidate inference node profile cluster generation process.

[0056] S58. In the subsequent generation of candidate inference node profile clusters, inference nodes carrying temporary isolation markers are excluded from the candidate range.

[0057] S59. During the isolation period, continuously monitor the broadcast beacon of the inference node. When the heterogeneous computing power texture fingerprint of the inference node is detected to have stabilized and the reputation evolution vector has returned to the normal range, automatically remove the temporary isolation mark. The normal range is obtained by taking the mean of each indicator of the reputation evolution vector during the stable operation phase before the isolation of the inference node as the center, and using the center plus or minus 1.5 times the standard deviation as the upper and lower bounds of the normal range. The mean of the average performance deviation before isolation is 0.15 and the standard deviation is 0.04, so the normal range of the average performance deviation is 0.09 to 0.21. The mean of the abnormal frequency before isolation is 0.02 and the standard deviation is 0.01, so the normal range of the abnormal frequency is 0 to 0.035. During the isolation period, the broadcast beacon of the inference node is continuously monitored. When it is detected that the heterogeneous computing power texture fingerprint of the inference node remains stable for 30 consecutive broadcast cycles, and the mean performance deviation and abnormal frequency in the reputation evolution vector of the inference node have returned to the normal range, the temporary isolation mark is automatically removed, the isolation mark field is set to 0, and the inference node re-participates in the scheduling candidate.

[0058] Please see Figure 2 This invention provides a decentralized heterogeneous GPU inference node self-organizing scheduling system, comprising: Computing power texture fingerprint broadcasting module: Each inference node sends its heterogeneous computing power texture fingerprint to peer inference nodes within the local area network segment in the form of periodic broadcast beacons; Anti-entropy convergence synchronization module: The local storage service of each inference node receives broadcast beacons and Gossip anti-entropy messages from peer inference nodes, obtains node status data, identifies differences in node status data, and completes bidirectional convergence synchronization of node status data. Inference Request Parsing and Filtering Module: When the gateway of any inference node receives an external inference request, it performs deep model semantic parsing on the inference request, and uses the results of deep model semantic parsing as search conditions to filter out the set of online inference nodes, forming a candidate inference node profile cluster; Optimal inference node selection module: Based on the heterogeneous computing power texture fingerprint of each candidate inference node in the candidate inference node profile cluster, the inference affinity coupling field score of the candidate inference nodes is performed, and then the optimal inference node is selected. The inference forwarding receipt analysis module forwards the received inference request to the optimal inference node to execute model inference, receives the inference results and execution status receipts returned by the optimal inference node, analyzes and determines whether the isolation circuit breaker mechanism is triggered.

[0059] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Various changes made within the scope of knowledge possessed by those skilled in the art without departing from the concept of the present invention still fall within the scope of protection of the present invention.

[0060] It should be noted that all formulas in this manual are calculated by removing dimensions and taking their numerical values. The formulas are derived from software simulations based on a large amount of collected data to obtain the most recent real-world results. The preset parameters and thresholds in the formulas are set by those skilled in the art according to the actual situation.

[0061] Although embodiments of the present invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of the present invention. All equivalent changes and improvements made within the scope of the present invention should still fall within the scope of the present invention.

Claims

1. A decentralized, heterogeneous GPU inference node self-organizing scheduling method, characterized in that, include: S1. Each inference node sends its heterogeneous computing power texture fingerprint to peer inference nodes within the local area network segment in the form of periodic broadcast beacons; wherein, the heterogeneous computing power texture fingerprint is the encapsulation of real-time running status parameters according to a predefined binary structure. S2. The local storage service of each inference node receives broadcast beacons and Gossip anti-entropy messages from peer inference nodes, and obtains node status data. At the same time, it identifies differences in node status data and completes bidirectional convergence synchronization of node status data. During the bidirectional convergence synchronization process, it continuously monitors the view divergence entropy between each inference node and determines whether state drift calibration is triggered. S3. When the gateway of any inference node receives an external inference request, it performs deep semantic analysis of the inference request and uses the results of the deep semantic analysis of the model to filter out the set of online inference nodes and form a candidate inference node profile cluster. S4. Based on the heterogeneous computing power texture fingerprint of each candidate inference node in the candidate inference node profile cluster, the inference affinity coupling field score of the candidate inference node is performed, and then the optimal inference node is selected. S5. Forward the received inference request to the optimal inference node to execute model inference, receive the inference result and execution status receipt returned by the optimal inference node, parse and determine whether the isolation circuit breaker mechanism is triggered.

2. The method according to claim 1, characterized in that, S1 includes: Collect real-time running status parameters of the GPU of this inference node at the beginning of each broadcast cycle; A beacon decay stamp is appended to the broadcast beacon header, where the beacon decay stamp records the start timestamp of this broadcast cycle and the local monotonic clock count value of the inference node; An encapsulated broadcast beacon is sent to a multicast address within the local area network segment via the UDP protocol.

3. The method according to claim 2, characterized in that, The real-time running status parameters include GPU architecture generation identifier, tensor core version, memory bandwidth level, context window length, quantization precision level, inference affinity, and request queue congestion depth.

4. The method according to claim 1, characterized in that, S2 includes: Initialize the local state database and maintain an independent Lamport clock version vector record for each known peer inference node; wherein, the Lamport clock version vector record includes Lamport clock count values ​​with the identifier of each known peer inference node as the dimension; Upon receiving a broadcast beacon from a peer inference node, the inference node identifier and state content carried in the broadcast beacon are parsed; the parsed inference node identifier and state content are used to construct a state record; the current Lamport clock count value in the clock version vector corresponding to the inference node is used to add a causal version mark to the state record and write it to the local state database. When the Gossip anti-entropy cycle arrives, several target inference nodes are randomly selected from the peer inference node list to establish an anti-entropy session, and the locally maintained version digest vector is sent to the target inference nodes. Receive the version digest vector returned by the target inference node, and compare it with the local version digest vector dimension by dimension according to the Lamport clock count value corresponding to each peer inference node, so as to identify the state record items where the two parties disagree. Determine the causal relationship of the state records with discrepancies, and obtain the state data of the node to be synchronized. The node state data to be synchronized is pushed incrementally to peer inference nodes with lagging state data, thus completing the bidirectional convergence and synchronization of node state data.

5. The method according to claim 1, characterized in that, S2 further includes: After each Gossip anti-entropy synchronization is completed, count the number of dimensions in the version summary vector that differ between the local state database and the target inference node state database. Based on the number of dimensions that differ and the difference in the count values ​​in the Lamport clock version vector of the difference record, the view divergence entropy value between the current peer inference node pairs is quantitatively calculated. When the view divergence entropy value between any pair of inference nodes continuously exceeds the preset convergence threshold for a preset observation period, it is determined that the pair of peer inference nodes has experienced state drift. Actively initiate a full version summary vector retrieval request to the peer inference node that has experienced state drift, and perform overlay calibration on the state records of the corresponding inference node in the local state database.

6. The method according to claim 1, characterized in that, S3 includes: When the gateway receives an external inference request, it performs protocol parsing on the external inference request message to extract the request body; Parse the target inference model identifier field, input tensor dimension, sequence length, and target inference precision level from the request body; The input tensor morphology feature code is generated based on the input tensor dimension and sequence length to represent the load morphology for inference. Based on the target inference model identifier, retrieve all online inference node records loaded with the target inference model from the local state database; from the retrieval results, remove offline inference node records whose beacon decay stamp time exceeds the freshness threshold from the current time, forming an online inference node set; The heterogeneous computing power texture fingerprint recorded by each inference node in the online inference node set is initially matched with the input tensor morphological feature code and the target inference accuracy level to eliminate invalid inference nodes and form a candidate inference node profile cluster.

7. The method according to claim 1, characterized in that, S4 includes: Extract the heterogeneous computing power texture fingerprint of each candidate inference node in the candidate inference node profile cluster; Perform inference affinity coupling field scoring to generate a comprehensive scheduling score for each candidate inference node; Candidate inference nodes are sorted in descending order of their comprehensive scheduling scores, and the candidate inference node ranked first is selected as the optimal inference node.

8. The method according to claim 1, characterized in that, S5 includes: The parsed external inference request is encapsulated into an inter-inference node forwarding message according to a predefined internal communication protocol; The forwarded message is sent to the target inference node through a long connection channel established with the optimal inference node to execute the model inference task; After the inference task is completed, the optimal inference node will send back the inference result and the execution status receipt along the reverse link of the external inference request forwarding path; The execution indicators in the execution status receipt are analyzed and compared with the expected execution indicators estimated based on the heterogeneous computing power texture fingerprint of the inference node. The reputation evolution vector of the corresponding inference node in the local state database is updated incrementally based on the deviation comparison results.

9. The method according to claim 1, characterized in that, The S5 also includes: Continuously monitor the changing trend of the reputation evolution vector of each inference node in the local state database to determine whether the inference node has entered a detuned state; For inference nodes that enter a detuned state, trigger the isolation circuit breaker mechanism and set a temporary isolation flag for the inference node in the local state database; In the subsequent generation of candidate inference node profile clusters, inference nodes carrying temporary isolation markers are excluded from the candidate range; During the isolation period, the broadcast beacon of the inference node is continuously monitored. When the heterogeneous computing power texture fingerprint of the inference node is detected to have stabilized and the reputation evolution vector has returned to the normal range, the temporary isolation mark is automatically lifted.

10. A decentralized heterogeneous GPU inference node self-organizing scheduling system, characterized in that, The system, applied to the method of any one of claims 1-9, comprises: Computing power texture fingerprint broadcasting module: Each inference node sends its heterogeneous computing power texture fingerprint to peer inference nodes within the local area network segment in the form of periodic broadcast beacons; Anti-entropy convergence synchronization module: The local storage service of each inference node receives broadcast beacons and Gossip anti-entropy messages from peer inference nodes, obtains node status data, identifies differences in node status data, and completes bidirectional convergence synchronization of node status data. Inference Request Parsing and Filtering Module: When the gateway of any inference node receives an external inference request, it performs deep model semantic parsing on the inference request, and uses the results of deep model semantic parsing as search conditions to filter out the set of online inference nodes, forming a candidate inference node profile cluster; Optimal inference node selection module: Based on the heterogeneous computing power texture fingerprint of each candidate inference node in the candidate inference node profile cluster, the inference affinity coupling field score of the candidate inference nodes is performed, and then the optimal inference node is selected. The inference forwarding receipt analysis module forwards the received inference request to the optimal inference node to execute model inference, receives the inference results and execution status receipts returned by the optimal inference node, analyzes and determines whether the isolation circuit breaker mechanism is triggered.

Images