Method and device for switching node roles of inference cluster, equipment and storage medium

Abstract

The method, device and equipment for switching the node role of an inference cluster and the storage medium comprise the following steps: identifying a target node in which a target role is changed, emptying online business request traffic corresponding to an old role, querying a current state of a large model in an inference container group, switching a target role model corresponding to the target role to a foreground serviceable state through a local state conversion in the inference container group, and integrating the target node into a corresponding service pool. The present application is based on a traffic isolation timing sequence of first removing an old identifier and then mounting a new identifier, a node non-serviceable window is extremely short, service jitter in a matching adjustment process is effectively inhibited, a cluster can quickly respond to business load changes through a declarative target topology combined with autonomous convergence of a node local state machine, a redistribution between pre-filled resources and decoded resources is timely completed, and overall resource utilization and online service stability are improved.

Description

Technical Field

[0001] This invention relates to the field of large-scale model inference service technology, and in particular to a method, apparatus, device, and storage medium for switching node roles in an inference cluster. Background Technology

[0002] In the pre-filling or decoding separation architecture of large model inference services, pre-filling instances and decoding instances constitute independent resource pools to address the significant differences in computing resources, GPU memory bandwidth, and key-value cache requirements between the two types of tasks. When changes in business load necessitate dynamic adjustments to the ratio of pre-filling to decoding resources, existing solutions typically embed the pre-filling or decoding roles into the container group's startup parameters, environment variables, service tags, or workload templates. Under such solutions, changing the inference role of a node means modifying the specifications of the corresponding workload, with the controller executing the deletion of the old container group and the creation of the new one. Since large model inference container groups need to complete operations such as model weight loading, inference engine initialization, communication channel establishment, and key-value cache structure preparation during startup, the aforementioned rebuilding adjustment process is time-consuming, repeatedly consuming GPU computing power and GPU memory bandwidth, and the adjusted node cannot provide inference services during the entire rebuilding period, resulting in fluctuations in the cluster's available capacity. Furthermore, due to the time overhead of container group scheduling and model startup, rebuilding adjustments cannot quickly keep up with real-time changes in business load, easily leading to an imbalance where one side has resource shortages while the other side is idle.

[0003] Therefore, the existing technology has problems and needs further improvement and development. Summary of the Invention

[0004] (I) Purpose of the invention: In order to solve the problems existing in the prior art, the purpose of the present invention is to provide a node role switching method, device, electronic device and computer-readable storage medium for inference clusters that can solve the problems of long switching time, large resource consumption, inability to quickly respond to changes in business load and poor service continuity, which depend on the pre-filling or decoding ratio adjustment of inference instance reconstruction.

[0005] (II) Technical Solution: To solve the above-mentioned technical problems, this technical solution provides a method for switching node roles in an inference cluster, including:

[0006] In response to the target topology adjustment of the large model inference cluster, the target node whose target role has changed is identified. The large model inference cluster is deployed with a pre-filled service pool and a decoding service pool.

[0007] While maintaining the normal existence of the inference container group on the target node, remove the service identifier corresponding to the old role of the target node before the change from the inference container group, so as to drain the online business request traffic corresponding to the old role;

[0008] The current state of the large model within the inference container group is queried through the local switching agent unit of the target node, wherein the large model includes a pre-filled role model and a decoding role model.

[0009] Based on the current state of the large model, a target convergence action is issued to the switching agent unit to switch the target role model corresponding to the target role to the foreground serviceable state through local state transition within the inference container group, and to switch the peer role model corresponding to the old role to the background state.

[0010] After confirming that the target role model is in the foreground serviceable state and the peer role model is in the background state, the service discovery identifier corresponding to the target role is attached to the inference container group to merge the target node into the corresponding service pool.

[0011] Furthermore, the global target state is managed through a role topology orchestration unit in the large model inference cluster, and an independent node role convergence unit is assigned to each node in the cluster.

[0012] The target nodes whose roles have changed include:

[0013] The updated target topology is read by the role topology orchestration unit, the target node whose target role has changed is identified, and the target role and target service identifier in the node role convergence unit corresponding to the target node are updated.

[0014] By comparing the current service role with the target role through the node role convergence unit corresponding to the target node, if it is determined that the two are inconsistent and the configuration template of the inference container group has not changed, the in-situ role switching path is triggered.

[0015] Furthermore, the current state obtained through the switching agent unit query includes at least one of the following: not loaded, foreground, background, error, and transitioning.

[0016] Furthermore, the step of issuing the target convergence action to the switching agent unit based on the current state of the large model includes executing at least one of the following state convergence rules:

[0017] If the target role model or the peer role model is in an error state, the cleanup interface of the switching proxy unit is called to perform cleanup.

[0018] If the peer role model is in the foreground state, a switch to background action is issued to call the switch agent unit's switch to background interface;

[0019] If the peer role model is not loaded and the system is configured with a peer role background backup strategy, the startup interface is called first to load the peer role model to a runnable state, and then the background switch interface is called.

[0020] If the target role model is not loaded, a startup action is issued to call the startup interface of the switching agent unit;

[0021] If the target role model is in the background, a foreground switch action is issued to call the foreground switch interface of the switching proxy unit;

[0022] If any model is in transition or its state is unknown, the node role convergence unit controls the state query to re-execute after a preset time.

[0023] Furthermore, when the backend interface is invoked, the target node performs the following operations at the hardware level:

[0024] Pause the computation flow or thread allocation corresponding to the peer role;

[0025] The active key-value cache (KV Cache) occupied by the aforementioned counterparty character is directly discarded, and the corresponding video memory space is released.

[0026] Release the high-frequency read / write area of ​​the target node's current graphics processor (GPU) memory, and release the model weights of the large model from the GPU memory and transfer them to the host memory.

[0027] Furthermore, when the foreground interface is invoked, the target node performs the following operations at the hardware level:

[0028] Reactivate the computational stream or thread corresponding to the target role;

[0029] Allocate a new blank key-value cache block for the target role and directly discard the original key-value cache corresponding to the old role;

[0030] Switch the request receiving queue of the large model inference engine to the computing path corresponding to the target role.

[0031] Furthermore, the role topology orchestration unit maintains an availability budget, which is used to limit the number of nodes in the large model inference cluster that are in the non-ready stage at the same time.

[0032] Before triggering the entry into the on-the-spot character switching path, it also includes:

[0033] The role topology orchestration unit verifies whether the number of nodes in the current cluster that are in the non-ready stage has reached the upper limit of the availability budget.

[0034] If the upper limit is reached, the process of pushing the target node into the traffic draining process will be temporarily suspended until the node that is switching over completes the switchover or enters a clear failure state.

[0035] Furthermore, this also includes fault isolation and failure handling:

[0036] When the target convergence action fails, the number of retries corresponding to the target node is increased, and the backoff time is calculated based on the number of retries.

[0037] After the backoff time is reached, the local status within the inference container group is queried again. If a failed or timed-out action is found, the cleanup interface is called first.

[0038] When the number of retries exceeds a preset threshold, the target node is marked as a failure and the failure status is independently reported to the role topology orchestration unit to limit the failure of a single node to the range of the target node.

[0039] Furthermore, after removing the service identifier corresponding to the old role from the inference container group and before querying the current state through the switching agent unit, the process also includes: entering the pre-switching extension phase, invoking idempotent external extension logic to wait for the external storage side, cache side, or traffic side to complete dependency preparation.

[0040] After confirming that the target role model meets the foreground serviceable state and the peer role model is in the background state, and before attaching the service discovery identifier corresponding to the target role to the inference container group, the process further includes: entering the post-switch extension phase to wait for the external system to confirm that the target role meets the external service conditions.

[0041] Furthermore, it also includes closed-loop state tracking:

[0042] The node role convergence unit maintains and records the dynamic switching context of the target node. The dynamic switching context includes at least the current stage, the target role model state, the peer role model state, and the most recent action record.

[0043] When performing the target convergence action, the node role convergence unit updates the action type, start time, end time and execution result in the most recent action record;

[0044] The target model key in the most recent action record is obtained by combining the model identifier of the large model with the corresponding business role;

[0045] When the execution result of the most recent action record is detected as a failure, based on the dynamic switching context, the inference container group is rolled back to the most recent stable role configuration by prioritizing the use of the target role model state and the peer role model state to ensure node serviceability.

[0046] A node role switching device for an inference cluster, comprising:

[0047] The role topology orchestration unit is used to express the global target state and manage the target topology, and in response to target topology adjustments, it identifies target nodes whose target roles have changed;

[0048] The node role convergence unit is bound to the target node and is used to remove the service identifier corresponding to the old role of the target node before the change from the inference container group to drain traffic while keeping the inference container group on the target node in normal existence. It is also responsible for driving the target node to converge its role status.

[0049] The local switching agent unit is deployed within the inference container group. It is used to allow the node role convergence unit to query the current status of the large model within the inference container group, and to switch the target role model corresponding to the target role to the foreground serviceable state according to the target convergence action issued by the node role convergence unit, and to switch the peer role model corresponding to the old role to the background state.

[0050] Specifically, after confirming that the target role model is in the foreground serviceable state and the peer role model is in the background state, the node role convergence unit attaches the service discovery identifier corresponding to the target role to the inference container group.

[0051] An electronic device includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the node role switching method of the inference cluster.

[0052] A computer storage medium storing a computer program, which, when executed by a processor, implements the node role switching method of the inference cluster.

[0053] (III) Beneficial Effects: When node roles need to be adjusted, this invention, while maintaining the normal existence of existing inference container groups on the target node, removes the old role service identifier to drain traffic. Then, the node's local switching agent unit performs a local state transition based on the current state of the pre-filled role model and the decoded role model, switching the target role model to a foreground serviceable state and the peer role model to a background state. After confirming state convergence, the new role service identifier is attached to achieve service publication. This invention preloads commonly used model weights into the host memory. During the state transition process, model weights are only moved at high speed between the host memory and the graphics processor's video memory. Compared to the conventional setting that requires reloading model weights from disk for each role switch, this invention eliminates the extremely high disk read / write time. At the same time, combined with the design that the inference engine does not need to be re-initialized, it avoids container scheduling latency caused by deleting and rebuilding container groups, significantly shortening the role switch time. Furthermore, based on the traffic isolation timing of first removing the old identifier and then attaching the new identifier, the node's unserviceable window is extremely short, effectively suppressing service jitter during the allocation adjustment process. Furthermore, by combining declarative target topology with the autonomous convergence of node local state machines, the cluster can quickly respond to changes in business load and promptly redistribute pre-filled and decoded resources, thereby improving overall resource utilization and online service stability. Attached Figure Description

[0054] Figure 1 This is a flowchart illustrating a node role switching method for an inference cluster according to the present invention.

[0055] Figure 2 This is a flowchart illustrating the state convergence rule of the present invention;

[0056] Figure 3 This is a schematic diagram of the structure of a node role switching device for an inference cluster according to the present invention;

[0057] Figure 4 This is a schematic diagram of the architecture of the large model inference cluster target topology adjusted from 2P2D to 3P1D in this invention. Detailed Implementation

[0058] The present invention will be further described in detail below with reference to preferred embodiments. More details are set forth in the following description in order to provide a full understanding of the present invention. However, the present invention can obviously be implemented in many other ways different from those described herein. Those skilled in the art can make similar extensions and derivations based on actual application situations without departing from the spirit of the present invention. Therefore, the scope of protection of the present invention should not be limited by the content of this specific embodiment.

[0059] The accompanying drawings are schematic diagrams of embodiments of the present invention. It should be noted that these drawings are for illustrative purposes only and are not drawn to scale, and should not be construed as limiting the actual scope of protection of the present invention.

[0060] like Figure 3 As shown, a node role switching device for an inference cluster includes a role topology orchestration unit, a node role convergence unit, and a local switching agent unit.

[0061] The role topology orchestration unit is used to express the global target state of the large model inference cluster and manage the target topology. Responding to target topology adjustment operations, the role topology orchestration unit identifies target nodes whose target roles have changed. In one embodiment, the role topology orchestration unit maintains a target topology data structure, records the correspondence between each node name in the cluster and its target role, and identifies nodes whose target roles have changed by periodically or event-drivenly comparing the difference between the target topology and the current actual state of the cluster. The role topology orchestration unit can also maintain availability budget information to limit the number of nodes in the cluster that are simultaneously in the non-ready phase, ensuring the overall service stability of the cluster during the switchover process.

[0062] The node role convergence unit is a single-node control component independently generated or managed by the role topology orchestration unit for each node according to the target topology. Each node participating in pre-filling or decoding separation inference corresponds to a node role convergence unit, and there is a one-to-one binding relationship between the two. The node role convergence unit is used to remove the service identifier corresponding to the old role of the target node from the inference container group while maintaining the normal existence of the inference container group on the target node, so as to drain the online business request traffic corresponding to the old role. The node role convergence unit is also responsible for driving the inference container group on its bound node to complete the state convergence process from the current service role to the target role. Specifically, this includes: querying the current state of the large model in the inference container group by calling the interface provided by the local switching proxy unit, making a decision and issuing the target convergence action based on the current state, and after confirming that the target role model is in a foreground serviceable state and the peer role model is in a background state, attaching the service discovery identifier corresponding to the target role to the inference container group.

[0063] In one embodiment, the node role convergence unit internally maintains a state machine, whose states include pending creation, creation, probing, starting, ready, emptying, switching, attaching service identifier, retrying, and failure. The node role convergence unit transitions between the states of the state machine according to the current stage and external triggering conditions, driving the node to complete the full switching process from the old role to the new role.

[0064] The node role convergence unit also maintains a dynamic switching context, including the current stage, the target role model state, the peer role model state, and the most recent action record. The most recent action record includes the action type, target model key, start time, end time, and execution result, where the target model key is generated by combining the model identifier of the large model with the corresponding business role.

[0065] The local switching agent unit is deployed inside the inference container group on the target node, sharing the container group network environment with the main inference container. The local switching agent unit provides standardized control interfaces to the node role convergence unit, including a status query interface, a startup interface, a background switch interface, a foreground switch interface, and a cleanup interface. The status query interface returns the current status of the pre-populated role model and the decoding role model within the inference container group; the startup interface loads models in an unloaded state into a runnable state; the background switch interface converts models in a foreground state to a background state; the foreground switch interface converts models in a background state to a foreground serviceable state; and the cleanup interface performs cleanup operations on models in error, timeout, or uncertain states.

[0066] The local switching agent unit executes specific model state transition operations within the inference container group based on the target convergence action issued by the node role convergence unit. For example, when receiving a background action instruction, the local switching agent unit drives the target node's hardware to perform operations such as pausing the peer role's computation flow or thread allocation, directly discarding the active key-value cache occupied by the peer role, releasing the high-frequency read / write area of ​​the current graphics processor's video memory, and releasing the model weights from video memory and transferring them to host memory. When receiving a foreground action instruction, the local switching agent unit drives the target node's hardware to perform operations such as reactivating the target role's computation flow or thread, allocating a new blank key-value cache block for the target role, and switching the inference engine request receiving queue to the target role's computation path. The commonly used model weights of the large model can be pre-loaded in the host memory.

[0067] It should be noted that the active key-value cache refers to the key-value cache occupied by the online business request session that the peer role is currently processing and has not yet ended when switching roles in place. In this invention, in order to achieve millisecond-level local ultra-fast convergence, this part of the residual active key-value cache does not need to be moved to main memory or have its state preserved. Instead, a hardware control strategy of directly discarding and physically releasing video memory space is adopted, thereby quickly freeing up video memory space for the target role.

[0068] The data maintained by the role topology orchestration unit includes model identifiers, target topologies, role service identifiers, inference container group templates, availability budgets, and summary status. Model identifiers identify the large models being served; target topologies represent the correspondence between node names in the cluster and target roles; role service identifiers correspond to the pre-populated service pool and the decoding service pool, respectively, for use by the service discovery mechanism; inference container group templates are used to create inference container groups on nodes; availability budgets limit the maximum number of nodes in the role switching process at the same time; and summary status records the readiness status, switching status, and the current overall pre-population or decoding ratio of each node.

[0069] In one Kubernetes-based implementation, the role topology orchestration unit can be instantiated using a custom resource, such as a cluster-wide custom resource object to hold the aforementioned data. Cluster administrators or upper-level scheduling systems can declare the desired role distribution state of the cluster by modifying the target topology field in this custom resource. The controller of the role topology orchestration unit continuously monitors changes to this resource and drives the lower-level node role convergence unit to converge towards the target state.

[0070] The data maintained by the node role convergence unit includes at least: bound node name, model identifier, target role with values ​​of pre-filled or decoded, current service role, current stage, current inference container group reference, target role model status, peer role model status, most recent action record, and retries and next retry time. The current stage includes pending creation, creation, probing, starting, ready, emptying, switching, mounting service identifier, retrying, and failed. The current inference container group reference includes the container group name, container group unique identifier, and container group network address. The most recent action record includes action type, target model key, start time, end time, and error information.

[0071] Using Kubernetes as an example again, a node role convergence unit can be implemented as another custom resource object. The lifecycle of this custom resource object can be managed by the controller of the role topology orchestration unit or by a separate controller. The name or label of each node role convergence unit can include the identifier of its bound node to facilitate a one-to-one correspondence.

[0072] like Figure 1 As shown, a method for switching node roles in an inference cluster includes,

[0073] In response to target topology adjustments in the large model inference cluster, the system identifies target nodes whose target roles have changed. The large model inference cluster deploys a pre-filled service pool and a decoding service pool, each composed of multiple nodes assuming corresponding roles. The target role refers to the type of service role the system expects the node to ultimately assume, based on the target topology in the role topology orchestration unit; the value is either pre-filled or decoding.

[0074] While maintaining the normal operation of the inference container group on the target node, the service identifier corresponding to the old role before the target node's change is removed from the inference container group to drain the online business request traffic corresponding to the old role. This is used to cut off the external traffic entry point of the old role before the model state transition, preventing nodes in the intermediate switching state from receiving inference requests for incompatible roles.

[0075] The target node uses a local switching agent unit to query the current state of the large model within the inference container group. This large model includes a pre-populated role model and a decoded role model. The retrieved state information is used to support subsequent action decisions.

[0076] Based on the current state of the large model, the target convergence action is issued to the switching agent unit. Through local state transitions within the inference container group, the target role model corresponding to the target role is switched to a foreground serviceable state, and the peer role model corresponding to the old role is switched to a background state. Based on dynamic action decisions of the current state, rather than a preset fixed process, it has the ability to self-heal and converge from various abnormal intermediate states.

[0077] The target role model refers to the model instance corresponding to the target role. When the target role is pre-filled, the target role model is the pre-filled role model; when the target role is decoded, the target role model is the decoded role model. The peer role model refers to the model instance corresponding to the non-target role. When the target role is pre-filled, the peer role model is the decoded role model; when the target role is decoded, the peer role model is the pre-filled role model.

[0078] After confirming that the target role model is in a foreground serviceable state and the peer role model is in a background state, the service discovery identifier corresponding to the target role is attached to the inference container group to merge the target node into the corresponding service pool. This is used to implement service publication for the new role, enabling the node to reconnect to the cluster and provide services externally with the new role identity.

[0079] Cluster administrators or upper-level scheduling systems create role topology orchestration units, defining initial information such as model identifiers, target topologies, role service identifiers, and inference container group templates. The target topology declares the target roles for each node, such as... Figure 4As shown, the target roles of nodes a and b are pre-filling, and the target roles of nodes c and d are decoding, forming an initial 2P2D configuration.

[0080] After reading the target topology, the controller of the role topology orchestration unit creates a corresponding node role convergence unit for each topology item and writes information such as the bound node name, model identifier, target role, target service identifier, and inference container group template into the node role convergence unit.

[0081] The controller of the node role convergence unit checks whether a reasoning container group belonging to itself already exists on the node it is bound to. If it does not exist, a new reasoning container group is created on the bound node according to the reasoning container group template, and scheduling constraints such as node affinity are used to ensure that the container group runs fixedly on the bound node. The reasoning container group contains at least a main reasoning container and a local switching agent unit.

[0082] After the inference container group enters the running state, the node role convergence unit calls the status query interface of the local switching agent unit to read the current status of the pre-filled role model and the decoded role model in the inference container group.

[0083] The node role convergence unit performs local convergence actions based on the target role. Taking a node with a pre-filled target role as an example, the node role convergence unit sequentially issues instructions to the local switching agent unit, causing the pre-filled role model, i.e., the target role model, to enter the foreground serviceable state, and causing the decoding role model, i.e., the peer role model, to enter the background state.

[0084] The foreground serviceable state refers to the target role model having completed the activation of the computation flow or thread, allocated or restored the required key-value cache blocks, and switched the inference engine's request receiving queue to the computation path corresponding to that role. The model now possesses the complete capability to receive and process inference requests of the corresponding type for that role. The service identifier corresponding to the model in the foreground serviceable state has been attached to the inference container group, enabling the service discovery mechanism to recognize the container group as a member of the role's service pool, and allowing external requests to be normally routed to that container group.

[0085] The background state refers to the paused computation flow or thread of the target role model, the direct discarding of its active key-value cache to release the corresponding video memory space, the release of the high-frequency read / write area of ​​the graphics processor's video memory occupied by the role, and the transfer of the model's weights to the host memory. Models in the background state do not receive or process inference requests and do not occupy the high-frequency read / write bandwidth of the graphics processor. However, because the model weights are resident in the host memory, they can be quickly read into video memory and awakened to transition to a foreground, serviceable state.

[0086] After confirming that the target role model is in a foreground serviceable state, the node role convergence unit writes the service discovery identifier corresponding to the target role, such as a pre-populated service tag, to the inference container group. Subsequently, the service discovery mechanism can identify the inference container group as a member of the pre-populated service pool and route pre-populated inference requests to that container group.

[0087] The node role convergence unit updates its own stage to the ready stage and reports information such as the current service role, the status of each model, and the inference container group reference to the role topology orchestration unit. The role topology orchestration unit summarizes the reported status of all nodes to form an overall cluster status view. The current service role refers to the type of service role that the node currently provides to the outside world, that is, the role type corresponding to the service identifier currently attached to the inference container group of the node, and the value is either pre-filled or decoded.

[0088] When the workload changes, such as an increase in the proportion of long context requests, and the cluster needs more pre-filled computing power, the user or the upper-level scheduling system can modify the target topology in the role topology orchestration unit to adjust the cluster from 2P2D to 3P1D.

[0089] Specifically, the user or the upper-level scheduling system modifies the corresponding entry of the target topology in the role topology orchestration unit, changing the target role of node d from decoded to pre-filled, and simultaneously adjusting the target service identifier accordingly. The controller of the role topology orchestration unit reads the updated target topology, compares it with the current cluster state, and identifies that the target role of node d has changed.

[0090] The role topology orchestration unit then updates the target role field and target service identifier field in the node role convergence unit corresponding to node three. In the coordination loop, the controller of the node role convergence unit corresponding to node three compares the current service role recorded in that node role convergence unit with the updated target role. If they are inconsistent, it indicates that the node needs to perform a role switch.

[0091] The node role convergence unit further checks whether the inference container group template has changed. If the template has not changed, it means that the inference container group on that node does not need to be rebuilt, and the system enters the in-situ role switching path. If the template has changed, the system can enter the container group reconstruction path. This path is not the focus of protection of this invention, and simple changes in pre-filling or decoding roles do not trigger this path.

[0092] In a preferred embodiment of the present invention, before formally entering the in-situ role switching path, the controller of the role topology orchestration unit can also perform availability budget verification. The availability budget field maintained in the role topology orchestration unit is used to limit the number of nodes in the cluster that are in a non-ready stage at the same time, such as nodes in the process of draining, switching, attaching service identifiers, or retrying. If the number of nodes in the current cluster that are in a non-ready stage has reached or exceeded the upper limit of the availability budget, the role topology orchestration unit will temporarily suspend pushing node three into the switching process until a node that is already switching completes the switch and returns to a ready state, or explicitly enters a failure state and releases the budget quota. Through this availability budget mechanism, unacceptable drops in available capacity on the cluster pre-filling side or decoding side due to multiple nodes switching simultaneously can be avoided, thereby ensuring the overall stability of online services.

[0093] The node role convergence unit can go through the following stages during its lifecycle: creation, probing, ready, draining, switching, attaching service identifiers, retrying, and failure. The creation stage is the initialization stage; the probing stage is the stage where the local model status is queried for the first time or after a restart; and the ready stage is a stable state where the current service role is consistent with the target role and provides normal external service. When an inconsistency between the current service role and the target role is detected, the node role convergence unit transitions from the ready stage to the draining stage, initiating an in-situ role switching process.

[0094] Emptying stage

[0095] Once the node role convergence unit corresponding to node three enters the emptying phase, it first performs a traffic isolation operation. Specifically, the node role convergence unit sends an instruction to the inference container group to remove the service discovery identifier corresponding to the current service role from the inference container group, for example, by deleting or modifying the corresponding service label. After the service discovery mechanism detects that the inference container group no longer carries the decoding service identifier, it will stop routing decoding-type inference requests to it. After the existing requests have been processed within a reasonable time window, the node completes the emptying of traffic for the old role. Here, the current service role is the old role, which in this example is the decoding role.

[0096] Pre-switch extension phase

[0097] After service identifier removal is completed and before the formal model state switch begins, the node role convergence unit can enter the pre-switch extension phase. During this phase, the node role convergence unit invokes pre-registered external extension logic. This logic waits for dependent systems such as external storage, caching, or traffic systems to complete necessary preparatory operations, such as waiting for distributed cache consistency synchronization to complete, or waiting for upstream traffic scheduling components to confirm that the node has been removed from the old role service pool. The extension logic should be implemented with idempotency to ensure consistency of operation results in retry scenarios. If the extension logic returns an incomplete state, the node role convergence unit re-invokes it after a preset time, until the extension logic returns a complete state, or a preset timeout condition is met.

[0098] Switching phase

[0099] After the pre-switch expansion phase is completed, the node role convergence unit enters the switch phase. The goal of the switch phase is to enable the target role model to enter the foreground serviceable state and the peer role model to enter the background state through local state transitions within the inference container group.

[0100] The node role convergence unit first calls the status query interface of the local switching agent unit to obtain the current status of the pre-populated role model and the decoded role model within the inference container group. The status information returned by the status query interface may include: the status of the pre-populated role model, the status of the decoded role model, the current foreground service role, and the most recently failed or timed-out action records, etc.

[0101] After obtaining the local model state, the node role convergence unit determines the target convergence action to be executed based on preset state convergence rules. For example... Figure 2 As shown, the specific state convergence rules are as follows:

[0102] If the target role model is in an error state, a cleanup action is selected. The cleanup interface of the local switching proxy unit is called to clean up the target role model's state, restoring it to a predetermined base state, such as an unloaded state. The unloaded state means the model has not yet been loaded into video memory, or the model weights have not been initialized, and it does not yet meet the conditions for operation. The error state means that an anomaly occurred during loading, running, or state transition, rendering the model unusable. Cleanup operations are typically required before subsequent actions can proceed.

[0103] If the peer role model is in an error state, the cleanup action is also selected, and the cleanup interface is called to clean up the peer role model.

[0104] If the peer role model is in the foreground state, then select the switch to background action, call the switch to background interface of the local switching agent unit, and switch the peer role model from the foreground state to the background state.

[0105] If the peer role model is not loaded and the system is configured with a background backup strategy for the peer role, which requires the peer role model to remain loaded even in the background for possible quick switchback in the future, then first select the start action to call the start interface to load the peer role model into a runnable state, and then select the switch to background action to switch it into the background state.

[0106] If the target role model is not loaded, select the start action, call the start interface of the local switching agent unit, and load the target role model into a runnable state.

[0107] If the target role model is in the background, then select the "switch to foreground" action, call the "switch to foreground" interface of the local switching agent unit, and switch the target role model from the background state to the foreground serviceable state.

[0108] If the target role model is already in the foreground state and the counterpart role model is already in the background state, then the target state is considered satisfied, and no further model state transition action is required. The basic goal of the switching phase has been achieved.

[0109] If any model is in a transitional state or the state information returned by the state query is uncertain, no action will be issued immediately. The control node's convergence unit will wait for a preset time and then re-initiate a state query to the local switching agent unit to obtain a definite model state before making a decision. A transitional state refers to a situation where the model is performing an action and has not yet reached a steady state. At this time, the model's state is temporarily uncertain and needs to wait until the action is completed before it can be clarified.

[0110] After selecting a target convergence action to be executed according to the above rules, the node role convergence unit issues the action instruction to the local switching agent unit. In one embodiment, the execution of each action instruction is an asynchronous call. After issuing the instruction, the node role convergence unit periodically queries the action execution status. After the action is completed, the node role convergence unit records the type of this action, the target model key, the start time, the end time, and the execution result. The target model key is composed of a model identifier and a business role, such as llama-3-70b-prefill or llama-3-70b-decode, and is used to uniquely identify the operation object of this action.

[0111] If the action is executed successfully, but the status is checked and it is found that the target status has not been fully met, for example, although the target role model has been started, it has not been completed in the foreground, the node role convergence unit re-enters the status query and action selection loop, and continues to issue the next necessary action until the target status is fully achieved.

[0112] If the action fails, the node role convergence unit enters the failure handling path, the specific process of which will be described in detail later.

[0113] Example 1

[0114] When the node role convergence unit calls the backend interface to request the peer role model to be switched from the foreground state to the backend state, the local switching proxy unit drives the target node to perform the following hardware-level operations:

[0115] Pause the computation flow or thread allocation corresponding to the peer role. The local switching agent unit sends a control signal to the inference engine, causing the inference engine to suspend the scheduling of computation tasks corresponding to that role and no longer allocate execution time slices of graphics processing unit computing units to the computation path of that role.

[0116] Handle active key-value caches occupied by peer characters. Since key-value cache reuse has low value, the local switching agent unit will discard active key-value cache blocks that are being used by peer characters by default, freeing up the video memory space they occupy for foreground characters or other computing tasks.

[0117] Release the high-frequency read / write area of ​​the target node's current graphics processor memory. Since the peer no longer needs to provide inference services, the high-bandwidth memory area it originally occupied can be released for use by foreground roles or other purposes.

[0118] This invention transfers the model weights of large models to host memory. It pre-loads several commonly used model weights and keeps them resident in host memory. When a role moves to the background, the model weights of the counterpart role are released from GPU memory to host memory; when the model needs to be brought back to the foreground, the weights are directly loaded from host memory to GPU memory at high speed. Conventional model loading is extremely time-consuming and requires repeated disk reads for each model during the switching process. This invention utilizes the ultra-high read / write bandwidth between host memory and GPU memory, eliminating the time-consuming process of repeatedly loading weights from disk. Service can be quickly restored simply by reallocating key-value cache blocks and activating the computation path, greatly shortening the overall business process time.

[0119] Example 2

[0120] When the node role convergence unit calls the foreground interface to request the target role model to be switched from a background state to a foreground serviceable state, the local switching proxy unit drives the target node to perform the following hardware-level operations:

[0121] Reactivate the computational flow or thread corresponding to the target role. The local switching agent unit sends a control signal to the inference engine, causing the inference engine to resume scheduling the computational path for that role and allocate execution time slices of the graphics processing unit's computational unit to it.

[0122] Allocate a new blank key-value cache block for the target role. Since this model will begin receiving new inference requests after moving to the foreground, it needs to be allocated a completely new, blank key-value cache block for subsequent pre-filling or decoding computation. In scenarios where the model resumes from the background rather than starting afresh, the local switching agent unit directly allocates a completely new blank key-value cache block for the target role to handle new inference requests, as the old key-value cache has been discarded by default.

[0123] The request receiving queue of the large model inference engine is switched to the computation path corresponding to the target role. The local switching agent unit modifies the routing configuration of the inference engine so that subsequent inference requests arriving at this inference container group are correctly distributed to the computation path of the target role for processing.

[0124] Through the hardware-level foreground / background state transitions in Examples 1 and 2, the models of the two roles can achieve rapid reallocation of computing resources and video memory bandwidth within the same inference container group and on the same set of graphics processor hardware, without having to go through a complete model unloading and reloading process, significantly shortening the switching time and reducing cold start costs.

[0125] Post-switch expansion phase

[0126] After confirming that the target role model is in a foreground serviceable state and the peer role model is in a background state, and before attaching a new service discovery identifier to the inference container group, the node role convergence unit can enter the post-switchover expansion phase. The purpose of this phase is to wait for the external system to confirm that the target role has met all the conditions for providing external services.

[0127] Similar to the pre-switch expansion phase, the post-switch expansion phase is implemented by invoking idempotent external expansion logic. For example, it waits for external monitoring systems to confirm that the node's health check probe has returned to a normal state, or waits for the distributed caching system to confirm that the shared key-value cache data required by the target role has been synchronized. If the expansion logic returns incomplete, the node role convergence unit waits a preset time before re-invoking it until completion.

[0128] After the post-switch expansion phase is completed, the node role convergence unit enters the service identifier mounting phase. The node role convergence unit writes the service discovery identifier corresponding to the new role to the inference container group, for example, by adding or modifying service tags to make them conform to the selector rules of the pre-populated service pool. Here, the new role is the target role, which in this example is the pre-populated role.

[0129] Once the service discovery mechanism detects that the inference container group has a newly attached pre-filled service identifier, it identifies it as an available member of the pre-filled service pool and begins routing pre-filled inference requests to it. At this point, node three is removed from the decoding service pool and merged into the pre-filled service pool, completing the in-situ switch from the decoding role to the pre-filled role.

[0130] During the completion and reporting process, the node role convergence unit updates its current service role field to the target role, updates the current stage to ready, and reports the latest current service role, the status of each model, the inference container group reference, and the most recent action record to the role topology orchestration unit.

[0131] After receiving the report that node 3 has completed the switchover, the role topology orchestration unit updates the entry corresponding to node 3 in its summary status, recalculates and displays the overall pre-filling or decoding ratio of the cluster. At this point, the cluster has successfully transformed from the initial 2P2D configuration to a 3P1D configuration, enhancing the computing power of the pre-filled service pool and enabling it to better handle the increased business load from long context requests.

[0132] Example 3

[0133] During the switching phase, abnormal situations may occur such as action execution failure, timeout, or uncertain return status. This invention ensures that the switching anomaly of a single node will not affect the normal convergence of other nodes and the overall service stability of the cluster through fault isolation and failure handling.

[0134] When the node role convergence unit detects that the target convergence action has failed, such as when the call to the startup interface returns an error, the foreground interface times out or the cleanup interface returns an exception, the node role convergence unit enters the failure handling path.

[0135] The node role convergence unit increments the retry counter of the node by one and calculates the backoff time based on the current retry count. The backoff time can adopt an exponential backoff strategy, that is, as the number of retries increases, the backoff waiting time increases exponentially. For example, the first retry waits for 1 second, the second retry waits for 2 seconds, the third retry waits for 4 seconds, and so on. At the same time, a maximum backoff time limit can be set to avoid excessive waiting time.

[0136] After the backoff time is reached, the node role convergence unit re-initiates a state query with the local switching agent unit to obtain the latest model state within the current inference container group. If the state query result shows that there are failed or timed-out incomplete actions, the node role convergence unit will first call the cleanup interface of the local switching agent unit to clean up the model in the abnormal state, restoring it to the determined basic state. Then, based on the actual state after cleanup, it will re-enter the state convergence rule decision loop to select the next action to be executed.

[0137] If the target node still fails to complete role convergence after the number of retries reaches a preset threshold, the node role convergence unit marks the current stage of the node as a failure and independently reports the failure status and failure reason to the role topology orchestration unit.

[0138] Upon receiving a report of a node failure, the role topology orchestration unit marks the node as failed in its summary status. However, this does not interrupt or roll back the convergence process of other nodes that are switching over or are ready. The impact of the failed node is limited to the target node itself. Operations personnel or the upper-level scheduling system can identify the failed node by querying the summary status of the role topology orchestration unit and decide whether to manually intervene or have the scheduling system retry the node's convergence in a subsequent scheduling cycle.

[0139] Example 4

[0140] At the role topology orchestration unit level, there is also an availability budget mechanism to limit the maximum number of nodes that can perform switching operations simultaneously in the cluster.

[0141] During system operation, the role topology orchestration unit continuously monitors the current stage status of all node role convergence units in the cluster and counts the number of nodes currently in the non-ready stage. This statistic is the number of nodes currently in the failover phase.

[0142] The availability budget field defines the maximum number of nodes currently in a switchover that the system allows. For example, it can be set to 1, 2, or a certain percentage of the total number of nodes in the cluster. When the role topology orchestration unit identifies a change in the target role of a node and prepares to trigger the node role convergence unit corresponding to that node to enter the emptying phase, it first checks whether the number of nodes currently in a switchover has reached the upper limit of the availability budget.

[0143] If the limit has been reached, the role topology orchestration unit temporarily suspends issuing instructions to the role convergence unit of that node to enter the drain phase; that is, the switchover operation of that node is temporarily blocked in the trigger phase. In subsequent coordination loops, the role topology orchestration unit continuously monitors whether any nodes already in the switchover process have completed their switchover and returned to a ready state, or have clearly entered a failure state and released their budget quotas. Once the number of nodes currently in the switchover process drops below the availability budget limit, the role topology orchestration unit immediately pushes the waiting nodes into the drain process.

[0144] This embodiment uses an availability budget mechanism to ensure that the number of nodes leaving their respective service pools at any given time is within a controllable range at the system level. This avoids significant fluctuations in the available capacity of the pre-filling or decoding side due to a large number of nodes switching at the same time, thus ensuring the stability of the online inference service.

[0145] Example 5

[0146] To enhance the transparency and observability of the single-node switching process, this embodiment incorporates closed-loop state tracking in the node role convergence unit.

[0147] The node role convergence unit maintains the dynamic switching context of the node in its data structure. The dynamic switching context contains at least the following information: the current stage reflecting the node's real-time position in the state machine, the target role model state reflecting the current running state of the target role model, the peer role model state reflecting the current running state of the peer role model, and the most recent action record.

[0148] The most recent action record is a structured data field used to record detailed information about the most recently executed action. Whenever a node role convergence unit issues and executes a target convergence action, the most recent action record is updated, containing: action type, target model key, action start time, action end time, and execution result. The target model key is a unique identifier composed of a model identifier and a business role. The execution result includes success or failure; if it fails, it includes an error code and error message.

[0149] Through the aforementioned dynamic switching context and the most recent action record, operators or upper-level monitoring systems can clearly understand the current switching progress, model state distribution, most recently executed control actions, and their results for any given node by querying the status of the node role convergence unit. This addresses the shortcomings of existing workload controllers, which can only reflect macroscopic states such as whether the container group is running and ready, but cannot reflect the switching progress of pre-filling or decoding roles within the container. It significantly improves the transparency and observability of the node-level role switching process, facilitating problem localization and troubleshooting for operations and maintenance personnel.

[0150] It should be noted that, in addition to the above-described implementation method of dividing the video memory state into hot and cold data within a single physical graphics processor node, the foreground or background state transition of the present invention can also be achieved through other hardware-assisted means. Several alternative implementation methods are listed below.

[0151] In an alternative implementation, lightweight graphics processing unit (GPU) virtualization technology within a node can be utilized, such as the dynamic reconfiguration capability based on multiple instance GPUs or virtual GPUs, to dynamically adjust computing power quotas among different roles. When a peer role needs to be moved to the background, the GPU computing unit quota occupied by that role is instantly reduced or revoked through virtualization technology, and the released computing unit quota is redistributed to the target role; when the target role needs to be brought back to the foreground, the computing unit quota allocation is dynamically adjusted again. This scheme achieves role resource isolation and switching at the level of computing unit hardware partitioning, complementing the application-layer memory block reclamation action and enriching the applicability of this invention across different hardware platforms.

[0152] Regardless of which hardware-level alternative implementation method is adopted, its macro-control logic and switching process all follow the in-situ role switching framework described in this invention, which is collaboratively driven by the role topology orchestration unit, the node role convergence unit, and the local switching agent unit. Therefore, these alternative implementation methods should all fall within the protection scope of this invention.

[0153] This invention provides an electronic device that can be used to implement the node role switching method for an inference cluster. The electronic device includes a processor and a memory. The processor and the memory are connected via a bus communication connection.

[0154] The processor may be a central processing unit, a graphics processing unit, a field-programmable gate array, an application-specific integrated circuit, or other processing unit with data processing capabilities and / or instruction execution capabilities. The processor may include one or more processing cores.

[0155] The memory is used to store computer programs. The memory may include volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory, cache, etc. Non-volatile memory may include, for example, read-only memory, flash memory, hard disk, solid-state drive, etc.

[0156] The computer program contains computer-executable instructions that, when executed by a processor, cause the processor to perform the method steps described in the foregoing method embodiments. Specifically, when the processor executes the computer program, it can perform the following operations: In response to the target topology adjustment of the large model inference cluster, identify the target node whose target role has changed, wherein the large model inference cluster is deployed with a pre-filled service pool and a decoding service pool; while maintaining the normal existence of the inference container group on the target node, remove the service identifier corresponding to the old role of the target node from the inference container group to drain the online business request traffic corresponding to the old role; query the current state of the large model in the inference container group through the local switching agent unit of the target node, wherein the large model includes a pre-filled role model and a decoding role model; based on the current state of the large model, issue a target convergence action to the switching agent unit to switch the target role model corresponding to the target role to the foreground serviceable state through the local state transition within the inference container group, and switch the peer role model corresponding to the old role to the background state; after confirming that the target role model is in the foreground serviceable state and the peer role model is in the background state, attach the service discovery identifier corresponding to the target role to the inference container group to merge the target node into the corresponding service pool.

[0157] When the processor executes the computer program, it can also implement various optional implementation methods described in the foregoing method embodiments, such as availability budget verification, fault isolation and failure handling, pre-switch extension and post-switch extension, closed-loop state tracking, etc., which will not be described in detail here.

[0158] In one implementation, the electronic device can be a server or computing node deployed in a data center or cloud computing environment. In another implementation, the electronic device can also be a distributed computing cluster composed of multiple servers or computing nodes, where the role topology orchestration unit, node role convergence unit, and local switching agent unit can be deployed on different physical or virtual nodes in the cluster, and work collaboratively through network communication.

[0159] The present invention also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the processor performs all or part of the method steps described in the foregoing method embodiments.

[0160] Computer-readable storage media refers to any available medium capable of persistently and non-transitory storing computer programs, including but not limited to: semiconductor storage media, magnetic storage media, optical storage media, magneto-optical storage media, etc. The computer program can be copied or transferred from one computer-readable storage medium to another, and can also be transmitted over a network via wired, wireless, fiber optic, or other communication methods.

[0161] Specifically, when the computer program stored on the computer-readable storage medium is executed by the processor, it can implement the steps of the node role switching method for the inference cluster described in the foregoing method embodiments. The specific implementation process is consistent with the description in the foregoing method embodiments, and will not be repeated here.

[0162] This invention addresses the issue of node role adjustment. While maintaining the existing inference container group on the target node, it removes the old role service identifier to drain traffic. Then, the node's local switching agent performs a local state transition based on the current states of the pre-filled and decoded role models, switching the target role model to a foreground serviceable state and the peer role model to a background state. After confirming state convergence, the new role service identifier is attached to achieve service publication. Because the model weights remain in GPU memory during the state transition, the inference engine does not need to be reinitialized, avoiding the overhead of repeated model loading and container scheduling latency caused by deleting and rebuilding container groups. This significantly shortens role switching time and reduces the redundant consumption of GPU computing power and GPU memory bandwidth. Furthermore, based on the traffic isolation sequence of first removing the old identifier and then attaching the new identifier, the node's unserviceable window is extremely short, effectively suppressing service jitter during allocation adjustments. In addition, through declarative target topology combined with the autonomous convergence of the node's local state machine, the cluster can quickly respond to changes in business load and promptly reallocate pre-filled and decoded resources, improving overall resource utilization and online service stability.

[0163] The technical solution provided by this invention can be widely applied to the production and operation scenarios of various large-model online inference service systems, especially suitable for inference clusters with pre-filled or decoded separation architectures deployed on container orchestration platforms. By implementing this invention, cloud service providers, enterprise AI platforms, or large-model inference service operators can flexibly and quickly adjust the ratio between pre-filled and decoded resources without affecting service stability, thereby improving the overall utilization efficiency of the graphics processor cluster, reducing operation and maintenance costs and energy consumption, and demonstrating clear industrial applicability and economic value.

[0164] The above description illustrates preferred embodiments of the present invention and helps those skilled in the art to more fully understand the technical solution of the present invention. However, these embodiments are merely illustrative and should not be construed as limiting the specific implementation of the present invention to these embodiments. For those skilled in the art, several simple deductions and modifications can be made without departing from the inventive concept, and all such modifications should be considered within the protection scope of the present invention.

Claims

1. A method for switching node roles in an inference cluster, characterized in that, include: In response to the target topology adjustment of the large model inference cluster, the target node whose target role has changed is identified. The large model inference cluster is deployed with a pre-filled service pool and a decoding service pool. While maintaining the normal existence of the inference container group on the target node, remove the service identifier corresponding to the old role of the target node before the change from the inference container group, so as to drain the online business request traffic corresponding to the old role; The current state of the large model within the inference container group is queried through the local switching agent unit of the target node, wherein the large model includes a pre-filled role model and a decoding role model. Based on the current state of the large model, a target convergence action is issued to the switching agent unit to switch the target role model corresponding to the target role to the foreground serviceable state through local state transition within the inference container group, and to switch the peer role model corresponding to the old role to the background state. After confirming that the target role model is in the foreground serviceable state and the peer role model is in the background state, the service discovery identifier corresponding to the target role is attached to the inference container group to merge the target node into the corresponding service pool.

2. The node role switching method for an inference cluster according to claim 1, characterized in that, The large model inference cluster manages the global target state through a role topology orchestration unit, and assigns an independent node role convergence unit to each node in the cluster. The target nodes whose roles have changed include: The updated target topology is read by the role topology orchestration unit, the target node whose target role has changed is identified, and the target role and target service identifier in the node role convergence unit corresponding to the target node are updated. By comparing the current service role with the target role through the node role convergence unit corresponding to the target node, if it is determined that the two are inconsistent and the configuration template of the inference container group has not changed, the in-situ role switching path is triggered.

3. The node role switching method for an inference cluster according to claim 1, characterized in that, The current status obtained by querying through the switching agent unit includes at least one of the following: not loaded, foreground, background, error, and transitioning.

4. The node role switching method for an inference cluster according to claim 3, characterized in that, The step of issuing the target convergence action to the switching agent unit based on the current state of the large model includes executing at least one of the following state convergence rules: If the target role model or the peer role model is in an error state, the cleanup interface of the switching proxy unit is called to perform cleanup. If the peer role model is in the foreground state, a switch to background action is issued to call the switch agent unit's switch to background interface; If the peer role model is not loaded and the system is configured with a peer role background backup strategy, the startup interface is called first to load the peer role model to a runnable state, and then the background switch interface is called. If the target role model is not loaded, a startup action is issued to call the startup interface of the switching agent unit; If the target role model is in the background, a foreground switch action is issued to call the foreground switch interface of the switching proxy unit; If any model is in transition or its state is unknown, the control node role convergence unit will wait for a preset time and then re-execute the state query.

5. The node role switching method for an inference cluster according to claim 4, characterized in that, When the backend transfer interface is invoked, the target node performs the following operations at the hardware level: Pause the computation flow or thread allocation corresponding to the peer role; The active key-value cache (KV Cache) occupied by the aforementioned counterparty character is directly discarded, and the corresponding video memory space is released. Release the high-frequency read / write area of ​​the target node's current graphics processor (GPU) memory, and release the model weights of the large model from the GPU memory and transfer them to the host memory.

6. The node role switching method for an inference cluster according to claim 4, characterized in that, When the foreground-to-the-front interface is invoked, the target node performs the following operations at the hardware level: Reactivate the computational stream or thread corresponding to the target role; Allocate a new blank key-value cache block for the target role and directly discard the original key-value cache corresponding to the old role; Switch the request receiving queue of the large model inference engine to the computing path corresponding to the target role.

7. The node role switching method for an inference cluster according to claim 2, characterized in that, The role topology orchestration unit maintains an availability budget, which is used to limit the number of nodes in the large model inference cluster that are in the non-ready stage at the same time. Before triggering the entry into the on-the-spot character switching path, it also includes: The role topology orchestration unit verifies whether the number of nodes in the current cluster that are in the non-ready stage has reached the upper limit of the availability budget. If the upper limit is reached, the process of pushing the target node into the traffic draining process will be temporarily suspended until the node that is switching over completes the switchover or enters a clear failure state.

8. The node role switching method for an inference cluster according to claim 4, characterized in that, It also includes fault isolation and failure handling: When the target convergence action fails, the number of retries corresponding to the target node is increased, and the backoff time is calculated based on the number of retries. After the backoff time is reached, the local status within the inference container group is queried again. If a failed or timed-out action is found, the cleanup interface is called first. When the number of retries exceeds a preset threshold, the target node is marked as a failure and the failure status is independently reported to the role topology orchestration unit to limit the failure of a single node to the range of the target node.

9. The node role switching method for an inference cluster according to claim 1, characterized in that, After removing the service identifier corresponding to the old role from the inference container group and before querying the current state through the switching agent unit, the process further includes: entering the pre-switching extension phase, invoking idempotent external extension logic to wait for the external storage side, cache side, or traffic side to complete dependency preparation; After confirming that the target role model meets the foreground serviceable state and the peer role model is in the background state, and before attaching the service discovery identifier corresponding to the target role to the inference container group, the process further includes: entering the post-switch extension phase to wait for the external system to confirm that the target role meets the external service conditions.

10. The node role switching method for an inference cluster according to any one of claims 1 to 9, characterized in that, It also includes closed-loop state tracking: The node role convergence unit maintains and records the dynamic switching context of the target node. The dynamic switching context includes at least the current stage, the target role model state, the peer role model state, and the most recent action record. When performing the target convergence action, the node role convergence unit updates the action type, start time, end time and execution result in the most recent action record; The target model key in the most recent action record is obtained by combining the model identifier of the large model with the corresponding business role; When the execution result of the most recent action record is detected as a failure, based on the dynamic switching context, the inference container group is rolled back to the most recent stable role configuration by prioritizing the use of the target role model state and the peer role model state to ensure node serviceability.

11. A node role switching device for an inference cluster, characterized in that, include: The role topology orchestration unit is used to express the global target state and manage the target topology, and in response to target topology adjustments, it identifies target nodes whose target roles have changed; The node role convergence unit is bound to the target node and is used to remove the service identifier corresponding to the old role of the target node before the change from the inference container group to drain traffic while keeping the inference container group on the target node in normal existence. It is also responsible for driving the target node to converge its role status. The local switching agent unit is deployed within the inference container group. It is used to allow the node role convergence unit to query the current status of the large model within the inference container group, and to switch the target role model corresponding to the target role to the foreground serviceable state according to the target convergence action issued by the node role convergence unit, and to switch the peer role model corresponding to the old role to the background state. Specifically, after confirming that the target role model is in the foreground serviceable state and the peer role model is in the background state, the node role convergence unit attaches the service discovery identifier corresponding to the target role to the inference container group.

12. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and when the processor executes the computer program, it implements the node role switching method of the inference cluster according to any one of claims 1 to 10.

13. A computer storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the node role switching method of the inference cluster as described in any one of claims 1 to 10.

Images