Container unified arrangement and non-sensing migration method and system for heterogeneous computing power pool

Abstract

This invention discloses a method and system for unified orchestration and seamless migration of containers for heterogeneous computing power pools. The method includes establishing hierarchical state records corresponding to each inference layer of a large model; collecting resource pressure indicators of the source device and triggering a pre-replication process before the source device becomes resource saturated; responding to the triggering of the pre-replication process, serializing the running state of each inference layer of the large model into an architecture-independent intermediate representation based on the hierarchical state records and transmitting it to the target device, while recording the inference layers that are written and updated during the transmission to form a dirty layer list; on the target device, deserializing all received architecture-independent intermediate representations into the target device's native running state format, reconstructing the inference execution state of the large model's inference container and resuming operation until migration confirmation is complete. This invention compresses service interruption time to an extremely brief suspension phase that only requires retransmission of the dirty layer list, achieving near-seamless migration of inference services.

Description

Technical Field

[0001] This invention relates to the field of cloud computing migration technology, and in particular to a method and system for unified orchestration and seamless migration of containers for heterogeneous computing power pools. Background Technology

[0002] With the large-scale deployment of large-model inference services, enterprise computing infrastructure generally exhibits heterogeneity, with computing devices from different vendors and architectures often coexisting in the same computing pool.

[0003] In real-world scenarios, when an inference container on a certain device needs to be migrated to another device due to resource saturation, existing container migration solutions typically involve first stopping the source container, then exporting and transferring the container's complete state to the target device, and finally restarting it on the target device. For example, in a large model inference platform with both GPU and NPU clusters deployed, when the memory usage of a GPU node continues to rise and approaches saturation, operations personnel need to manually intervene, stop and export the inference container on that node, wait for the complete state data to be transferred to the NPU node, and then restart the service. During this entire process, the inference service is completely interrupted, which represents an unacceptable service interruption for real-time inference services aimed at users.

[0004] Furthermore, due to the fundamental differences in the native data formats of GPUs and NPUs, existing solutions often fail to restore the container state of the source device to the target device due to format incompatibility, resulting in accuracy deviations in inference results. There is a lack of technical means to automatically complete format adaptation between heterogeneous devices and achieve uninterrupted migration of inference services. Summary of the Invention

[0005] In view of the aforementioned existing problems, the present invention is proposed.

[0006] Therefore, this invention provides a unified orchestration and seamless migration method and system for containers in heterogeneous computing pools, solving the problems of existing container migration solutions requiring downtime for export, format incompatibility, and long inference service interruption time in heterogeneous computing pool scenarios.

[0007] To solve the above-mentioned technical problems, the present invention provides the following technical solution: In a first aspect, the present invention provides a unified orchestration and seamless migration method for containers in heterogeneous computing power pools, comprising: establishing a hierarchical state record corresponding to each inference layer of a large model; collecting resource pressure indicators of the source device and triggering a pre-copying process before the source device becomes resource saturated; responding to the triggering of the pre-copying process, serializing the running state of each inference layer of the large model into an architecture-independent intermediate representation according to the hierarchical state record and transmitting it to the target device, while recording the inference layers that are written and updated during the transmission process to form a dirty layer list; when the dirty layer list meets the suspension condition, suspending the source container, and serializing the running state of each dirty layer in the dirty layer list into an architecture-independent intermediate representation and transmitting it to the target device; on the target device, deserializing all received architecture-independent intermediate representations into the running state in the target device's native format, reconstructing the inference execution state of the large model inference container and resuming operation until the migration is confirmed.

[0008] As a preferred embodiment of the container unified orchestration and seamless migration method for heterogeneous computing power pools described in this invention, the step of serializing the running state of each inference layer of a large model into an architecture-independent intermediate representation includes: extracting tensor data, attention cache data, and inter-layer activation values ​​for each inference layer of the large model; encoding the tensor data, attention cache data, and inter-layer activation values ​​according to a preset architecture-independent data format to generate an architecture-independent intermediate representation corresponding to the inference layer.

[0009] As a preferred embodiment of the container unified orchestration and seamless migration method for heterogeneous computing power pools described in this invention, the step of encoding the tensor data, attention cache data, and inter-layer activation values ​​according to a preset architecture-independent data format includes: encoding the tensor data, attention cache data, and inter-layer activation values ​​of each inference layer sequentially according to the inference execution order of each inference layer of the large model; for inter-layer activation values ​​where there is a data dependency relationship between adjacent inference layers, the source inference layer index and the target inference layer index corresponding to the inter-layer activation value are appended during encoding, so that the target device can restore the data dependency relationship between each inference layer of the large model according to the source inference layer index and the target inference layer index during deserialization.

[0010] As a preferred embodiment of the container unified orchestration and seamless migration method for heterogeneous computing power pools described in this invention, the step of the dirty layer list satisfying the suspension condition includes: counting the number of dirty layers currently to be transferred in the dirty layer list, and calculating the total amount of running status data of all dirty layers in the dirty layer list; when the number of dirty layers is lower than a preset dirty layer number threshold and the total amount of running status data is lower than a preset data volume threshold, it is determined that the dirty layer list satisfies the suspension condition.

[0011] As a preferred embodiment of the container unified orchestration and seamless migration method for heterogeneous computing power pools described in this invention, the step of serializing the running state of each dirty layer in the dirty layer list into an architecture-independent intermediate representation includes: extracting tensor data, attention cache data, and inter-layer activation values ​​of each dirty layer at the time the source container is suspended, according to the layer index of each dirty layer in the dirty layer list; incrementally encoding the tensor data, attention cache data, and inter-layer activation values ​​that have been updated by writing compared to the pre-copy process transmission in each dirty layer to generate an architecture-independent intermediate representation corresponding to each dirty layer; and appending the dirty layer index corresponding to the incremental encoding to the architecture-independent intermediate representation.

[0012] As a preferred embodiment of the container unified orchestration and seamless migration method for heterogeneous computing power pools described in this invention, the step of deserializing all received architecture-independent intermediate representations into the runtime state of the target device's native format includes: reading the layer index in the architecture-independent intermediate representation of each inference layer of the large model; retrieving the runtime state description information of the corresponding inference layer from the layered state record according to the layer index; mapping the tensor data, attention cache data, and inter-layer activation values ​​in the architecture-independent intermediate representation of each inference layer to the corresponding data structure in the target device's native format according to the runtime state description information, thereby generating the runtime state of each inference layer in the target device's native format.

[0013] As a preferred embodiment of the container unified orchestration and seamless migration method for heterogeneous computing power pools described in this invention, the step of mapping to the corresponding data structure in the native format of the target device includes: for the inter-layer activation values ​​of the attached source inference layer index and the target inference layer index, restoring the data dependency relationship between the corresponding inference layers in the running state of the native format of the target device according to the source inference layer index and the target inference layer index; after the data dependency relationship of all inference layers is restored, verifying the integrity of the running state of each inference layer in the native format of the target device layer by layer according to the layered state record, and completing the deserialization.

[0014] As a preferred embodiment of the container unified orchestration and seamless migration method for heterogeneous computing power pools described in this invention, the step of collecting resource pressure indicators of the source device includes: collecting the computing unit utilization rate, video memory utilization rate, and video memory bandwidth utilization rate of the source device according to a preset collection period, as the resource pressure indicators; comparing each resource pressure indicator with the corresponding resource saturation warning threshold, and triggering the pre-replication process when any resource pressure indicator exceeds the corresponding resource saturation warning threshold.

[0015] As a preferred embodiment of the container unified orchestration and seamless migration method for heterogeneous computing power pools described in this invention, the step of comparing each resource pressure indicator with the corresponding resource saturation warning threshold includes: calculating the moving average of the computing unit utilization rate, memory utilization rate, and memory bandwidth utilization rate according to a preset time window; using the moving average as the input value for comparing each resource pressure indicator with the corresponding resource saturation warning threshold, so as to filter out the interference of instantaneous fluctuations in the resource pressure indicators on the pre-replication process triggering judgment.

[0016] Secondly, the present invention provides a unified orchestration and seamless migration system for containers in heterogeneous computing power pools, including: a data acquisition module, used to collect resource pressure indicators of source devices and trigger a pre-replication process before the source devices become resource saturated; The intermediate transmission module is used to serialize the running state of each inference layer of the large model into an architecture-independent intermediate representation according to the hierarchical state record, and transmit it to the target device. At the same time, it records the inference layers that are written and updated during the transmission process to form a dirty layer list. The conversion module is used to suspend the origin container when the dirty layer list meets the suspension condition, serialize the running state of each dirty layer in the dirty layer list into an architecture-independent intermediate representation, and transmit it to the target device; The migration module is used to deserialize all received architecture-independent intermediate representations into the target device's native format runtime state on the target device, reconstruct the inference execution state of the large model inference container and resume operation until the migration is confirmed.

[0017] Compared with existing technologies, the beneficial effects of this invention are as follows: By collecting resource pressure indicators of the source device and triggering a pre-copying process before resource saturation, the running state of each inference layer of the large model is serialized and transmitted during the continuous operation of the source container. This avoids the operation method of having to stop and export in existing solutions, compressing the service interruption time to an extremely short suspension stage that only requires retransmission of the dirty layer list, and realizing near-seamless migration of inference services. At the same time, this invention uniformly encodes the tensor data, attention cache data, and inter-layer activation values ​​of each inference layer of the large model into an architecture-independent intermediate representation, and maps the architecture-independent intermediate representation to the running state in the native format of the target device according to the layered state record during the deserialization stage. This fundamentally solves the format incompatibility problem caused by the difference in native data formats between heterogeneous devices, enabling the inference container to accurately restore the state and seamlessly continue inference execution between computing devices with different architectures. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a schematic diagram of the overall process of a container unified orchestration and seamless migration method for heterogeneous computing power pools according to an embodiment of the present invention.

[0020] Figure 2 This is a schematic diagram of heterogeneous computing pool migration, which is a container unified orchestration and seamless migration method for heterogeneous computing pools according to an embodiment of the present invention. Detailed Implementation

[0021] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of the present invention.

[0022] Example 1, referring to Figure 1 As an embodiment of the present invention, a method for unified orchestration and seamless migration of containers for heterogeneous computing power pools is provided, comprising: S1. Establish hierarchical state records corresponding to each inference layer of the large model.

[0023] S2. Collect the resource pressure indicators of the source device and trigger the pre-replication process before the source device becomes resource saturated.

[0024] S3. In response to the triggering of the pre-copy process, the running state of each inference layer of the large model is serialized into an architecture-independent intermediate representation according to the hierarchical state record and transmitted to the target device. At the same time, the inference layers that are written and updated during the transmission process are recorded to form a dirty layer list.

[0025] S4. When the dirty layer list meets the suspension condition, suspend the origin container, serialize the running state of each dirty layer in the dirty layer list into an architecture-independent intermediate representation, and transmit it to the target device.

[0026] S5. On the target device, all received architecture-independent intermediate representations are deserialized into the target device's native format running state, the inference execution state of the large model inference container is reconstructed and resumed, until the migration is confirmed.

[0027] It should be noted that in large-scale deployment scenarios of large model inference services, enterprise computing infrastructure is generally heterogeneous, with computing devices from different vendors and architectures often coexisting in the same computing pool. In actual operation scenarios, when an inference container on a certain device needs to be migrated to another device due to the device's resources becoming saturated, existing container migration solutions typically involve first stopping the source container, then exporting and transferring the container's complete state to the target device, and finally restarting it on the target device. This method completely interrupts the inference service during the entire migration process, which means an unacceptable service interruption for real-time inference services for users. In addition, due to the fundamental differences in the native data formats of computing devices with different architectures, existing solutions often fail to restore the container state from the source device to the target device due to format incompatibility, resulting in accuracy deviations in the inference results. There is a lack of technical means to automatically complete format adaptation between heterogeneous devices and achieve uninterrupted migration of inference services.

[0028] Therefore, to address the aforementioned heterogeneous migration compatibility issues and inference service interruption problems, steps S1-S5 are used to pre-establish hierarchical state records for each inference layer of the large model, providing a hierarchical structural basis for subsequent serialization and deserialization processes. By continuously collecting resource pressure indicators of the source device and proactively triggering a pre-copying process before resource saturation, the running state of each inference layer of the large model can be serialized and transmitted while the source container is continuously running. This avoids the operation method of having to stop and export in existing solutions, compressing the service interruption time to an extremely brief suspension stage that only requires retransmission of the dirty layer list, achieving near-seamless migration of inference services. At the same time, the tensor data, attention cache data, and inter-layer activation values ​​of each inference layer of the large model are uniformly encoded into an architecture-independent intermediate representation. During the deserialization stage, the architecture-independent intermediate representation is mapped to the running state in the native format of the target device according to the hierarchical state records. This fundamentally solves the format incompatibility problem caused by differences in native data formats between heterogeneous devices, enabling the inference container to accurately restore the state and seamlessly continue inference execution between computing devices with different architectures.

[0029] Example 2, refer to Figure 1 and Figure 2 As an embodiment of the present invention, based on the above embodiment, a unified orchestration and seamless migration method for containers in heterogeneous computing power pools is provided.

[0030] In the implementation of this application, S1, establish a hierarchical state record corresponding to each inference layer of the large model.

[0031] For example, in this embodiment, the object to be migrated is a large model inference container deployed on a GPU node, and the large model used is a typical large language model containing 32 Transformer inference layers. During the container startup phase, the system traverses the inference layer structure of the large model layer by layer, and records structural description information such as the layer index, layer type, tensor dimension, number of attention heads, hidden layer dimension, and data type for each inference layer, forming a hierarchical state record entry that corresponds one-to-one with each inference layer. For inference layers with skip connections or residual connections between layers, the system also records its input source layer index and output target layer index to fully describe the data dependencies between each inference layer.

[0032] In this embodiment of the application, S2 collects the resource pressure index of the source device and triggers the pre-replication process before the source device becomes resource saturated.

[0033] Specifically, in this embodiment, the collection of resource pressure indicators and the triggering of the pre-replication process adopt a multi-indicator joint monitoring strategy. First, various resource pressure indicators are continuously collected according to a preset collection cycle. Then, instantaneous fluctuation interference is eliminated by moving average filtering. Finally, the smoothed indicator values ​​are compared with the warning threshold to determine whether the pre-replication process is triggered.

[0034] The steps for collecting resource pressure indicators from source devices include S2.1 to S2.2: S2.1 Collect the computing unit utilization rate, video memory utilization rate, and video memory bandwidth utilization rate of the source device according to the preset collection cycle, and use them as the resource pressure indicators.

[0035] In this embodiment, the resource pressure index collection period is set to 500ms. By calling the hardware monitoring interface of the source device, the GPU computing unit utilization rate, video memory utilization rate and video memory bandwidth utilization rate are collected every 500ms. The collection results are written into a sliding window buffer of length 20 in timestamp order. When the buffer is full, it is updated in a first-in-first-out manner to maintain a continuous record of the resource pressure change trend in the last 10 seconds.

[0036] S2.2. Compare each resource pressure indicator with its corresponding resource saturation warning threshold. When any resource pressure indicator exceeds the corresponding resource saturation warning threshold, trigger the pre-replication process.

[0037] The steps of comparing each resource pressure indicator with its corresponding resource saturation early warning threshold include: According to the preset time window, the moving average of the computing unit utilization rate, video memory utilization rate and video memory bandwidth utilization rate are calculated respectively.

[0038] For example, the time window length is set to 10 seconds, corresponding to 20 sampling points in the sliding window buffer. After each new sampling is completed, the arithmetic mean of the computing unit utilization, video memory utilization, and video memory bandwidth utilization of the 20 sampling points in the buffer is calculated to obtain the sliding mean of each indicator within the current time window.

[0039] Taking a certain sampling moment as an example, the average memory utilization rate of the 20 sampling points in the buffer is 83.6%, the average computing unit utilization rate is 76.2%, and the average memory bandwidth utilization rate is 79.4%.

[0040] The moving average is used as the input value for comparing each resource pressure indicator with the corresponding resource saturation warning threshold, so as to filter out the interference of instantaneous fluctuations in the resource pressure indicators on the pre-replication process trigger judgment.

[0041] In this embodiment, the resource saturation warning thresholds corresponding to the computing unit utilization rate, video memory utilization rate, and video memory bandwidth utilization rate are set to 85%, 85%, and 80%, respectively. By comparing the above moving average with each threshold, it is found that the moving average of the video memory bandwidth utilization rate of 79.4% is close to its warning threshold of 80%. However, if the instantaneous sample value is used for comparison, the instantaneous value of the video memory bandwidth utilization rate at that moment is 91.3%, which far exceeds the threshold. This is an instantaneous fluctuation lasting less than 1 second. The moving average mechanism effectively avoids false triggering caused by instantaneous fluctuations.

[0042] In subsequent sampling periods, the moving average of video memory utilization continued to climb to 86.1%, exceeding the warning threshold of 85%. The system determined that the video memory utilization index triggered the warning condition and immediately started the pre-copy process. At this time, the actual utilization of the source device's video memory had not yet reached saturation, leaving sufficient time window for the pre-copy transmission process.

[0043] In this embodiment of the application, S3, in response to the triggering of the pre-copy process, the running state of each inference layer of the large model is serialized into an architecture-independent intermediate representation according to the hierarchical state record and transmitted to the target device. At the same time, the inference layers that are written and updated during the transmission process are recorded to form a dirty layer list.

[0044] Specifically, in this embodiment, after the pre-copy process is triggered, the system serializes the running status of each inference layer in the source container according to the hierarchical status record established in S1 and the inference layer index order. After serialization, the generated architecture-independent intermediate representation is transmitted to the target device through the high-speed interconnection channel between nodes. At the same time, the dirty layer tracking mechanism is started to mark and record the inference layers that are written and updated during the transmission process, and finally form a dirty layer list.

[0045] The steps of serializing the runtime states of each inference layer of the large model into architecture-independent intermediate representations include S3.1 to S3.2: S3.1 For each inference layer of the large model, extract the tensor data, attention cache data and inter-layer activation values ​​of that inference layer.

[0046] In this embodiment, for the 32 inference layers of the large model, the system performs state extraction operations sequentially from layer 0 to layer 31 according to the layer index. Taking the 8th inference layer as an example, the extracted content includes three categories: First, tensor data, which includes the weight matrix, query matrix Q, key matrix K, value matrix V, and output projection matrix of this layer, with a data precision of FP16 and a single-layer tensor data size of approximately 1.2GB; second, attention cache data, which is the KV Cache generated by this layer in the current inference step, including key cache tensors and value cache tensors, with a cache sequence length equal to the number of tokens currently processed; and third, inter-layer activation values, which are the input activation tensors received by this inference layer from the previous layer (layer 7) and the output activation tensors output to the next layer (layer 9). All three types of data are read directly from the source device's video memory, and the source container inference execution is not interrupted during the extraction process.

[0047] S3.2 Encode the tensor data, attention cache data, and inter-layer activation values ​​according to a preset architecture-independent data format to generate an architecture-independent intermediate representation corresponding to the inference layer.

[0048] It should be noted that the steps of encoding the tensor data, attention cache data, and inter-layer activation values ​​according to a preset architecture-independent data format include A1~A2: A1. Encode the tensor data, attention cache data, and inter-layer activation values ​​of each inference layer in the order of inference execution of each inference layer in the large model.

[0049] In this embodiment, the system performs encoding operations sequentially from layer 0 to layer 31 according to the inference execution order. The architecture-independent data format adopts a unified binary encoding structure. Each layer's encoding block consists of two parts: layer header information and layer data body. The layer header information includes the layer index, layer type identifier, tensor dimension description, and data precision identifier. The layer data body sequentially stores the flattened tensor data byte stream, attention cache data byte stream, and inter-layer activation value byte stream, with each field separated by a fixed-length delimiter. Taking the completion of encoding from layer 0 to layer 31 as an example, the generated architecture-independent intermediate representation has a total data volume of approximately 38.4 GB, an average encoding time of approximately 120 ms per layer, and a total encoding time of approximately 3.8 seconds for all 32 layers. The source container runs continuously throughout the process, and the inference service is uninterrupted.

[0050] A2. For inter-layer activation values ​​where there is a data dependency between adjacent inference layers, the source inference layer index and the target inference layer index corresponding to the inter-layer activation value are appended during encoding. This is used by the target device to recover the data dependency between each inference layer of the large model according to the source inference layer index and the target inference layer index during deserialization.

[0051] Taking the inter-layer activation value from layer 7 to layer 8 as an example, when encoding this activation value tensor, the source inference layer index field value "7" and the target inference layer index field value "8" are appended to the header of its corresponding data field, occupying a total of 8 bytes of index annotation space. For inference layers with residual connections, such as the output activation value of layer 0 being used as the input of both layer 1 and layer 8, two inter-layer activation value encoding entries with different target inference layer indices are generated respectively, ensuring that the target device can completely restore the cross-layer data dependency relationship according to the index annotation during the deserialization stage. After all 32 layers are encoded, the system counts the inference layers that are written and updated during transmission, forming a dirty layer list. In this embodiment, a total of 5 inference layers are written and updated during the pre-copy transmission, with corresponding layer indices of 3, 7, 12, 18, and 25. These 5 layers are recorded in the dirty layer list for use in the subsequent incremental retransmission processing in stage S4.

[0052] In this embodiment of the application, S4, when the dirty layer list meets the suspension condition, the origin container is suspended, and the running state of each dirty layer in the dirty layer list is serialized into an architecture-independent intermediate representation and transmitted to the target device.

[0053] Specifically, in this embodiment, after the pre-copy transmission in step S3 is completed, the system continuously monitors the dynamic changes of the dirty layer list, periodically evaluates the dirty layer list according to the preset suspension condition judgment logic, and immediately suspends the origin container when the dirty layer list meets the suspension condition. Then, incremental serialization processing is performed on the running status of each dirty layer in the dirty layer list to generate the corresponding architecture-independent intermediate representation and transmit it to the target device to complete the final state synchronization before migration.

[0054] The steps for ensuring that the dirty layer list meets the suspension conditions include S4.1~S4.2: S4.1 Count the number of dirty layers currently waiting to be transmitted in the dirty layer list, and calculate the total amount of running status data of all dirty layers in the dirty layer list.

[0055] In this embodiment, after the pre-copy transmission in step S3 is completed, the system performs a statistical evaluation of the dirty layer list at a period of 500ms. Taking a certain evaluation moment as an example, the currently recorded dirty layer indices to be transmitted in the dirty layer list are 3, 7, 12, 18, and 25, and the statistical result of the number of dirty layers is 5. Subsequently, the system reads the running status data of the above 5 dirty layers that have been written and updated since the pre-copy transmission was completed, calculates the amount of tensor data, attention cache data, and inter-layer activation values ​​of each dirty layer after the update, and accumulates them to obtain the total running status data of all dirty layers in the dirty layer list, which is 1.86GB.

[0056] S4.2 When the number of dirty layers is lower than a preset dirty layer number threshold and the total amount of running status data is lower than a preset data amount threshold, it is determined that the dirty layer list meets the suspension condition.

[0057] For example, the preset threshold for the number of dirty layers is set to 8 layers, and the preset threshold for the amount of data is set to 3GB. The current statistical results are compared with the above thresholds item by item: the number of dirty layers (5 layers) is lower than the threshold of 8 layers, and the total amount of running data (1.86GB) is lower than the threshold of 3GB. When both conditions are met, the system determines that the dirty layer list has met the suspension conditions and immediately sends a suspension command to the source container. The source container enters the suspension state after completing the current inference step. The suspension operation takes about 80ms.

[0058] The step of serializing the running state of each dirty layer in the dirty layer list into an architecture-independent intermediate representation includes S4.3~S4.5: S4.3. According to the layer index of each dirty layer in the dirty layer list, extract the tensor data, attention cache data and inter-layer activation values ​​of each dirty layer at the time of suspension of the source container.

[0059] After the source container is suspended, the system performs state extraction operations on each dirty layer in ascending order of the layer indices 3, 7, 12, 18, and 25 recorded in the dirty layer list.

[0060] Taking the 12th dirty layer as an example, the system extracts the complete running state of this layer at the time of suspension from the source device's video memory, including tensor data such as the weight matrix, query matrix Q, key matrix K, value matrix V, and output projection matrix, attention cache data composed of the key cache tensor and value cache tensor under the current inference step, and inter-layer activation values ​​composed of the input activation tensor from the 11th layer and the output activation tensor output to the 13th layer. The extraction operation directly reads the video memory snapshot to ensure that the extracted data is consistent with the state of the source container at the time of suspension.

[0061] S4.4 Incrementally encode the tensor data, attention cache data, and inter-layer activation values ​​that are written and updated in each dirty layer compared to the pre-copy process, and generate the architecture-independent intermediate representation for each dirty layer.

[0062] The system compares the running status of each dirty layer at the time of suspension extracted in step S4.3 with the running status during the pre-copy process transmission in step S3, and performs incremental encoding only on the fields that have been written and updated.

[0063] Taking the 7th dirty layer as an example, the comparison revealed that the key cache tensor in the attention cache data of this layer had 12 new cache entries corresponding to tokens, and the output activation tensor in the inter-layer activation values ​​had been fully updated, while the weight matrix in the tensor data had not changed. Therefore, only the incremental part of the key cache tensor and the output activation tensor were encoded, and the unupdated weight matrix field was skipped. The incremental encoded data volume was 214MB, which saved about 82% of the transmission data volume compared to the 1.2GB of full retransmission of this layer.

[0064] After all five dirty layers have completed incremental encoding, the generated architecture-independent intermediate representation has a total data size of 1.86GB and an encoding time of approximately 420ms.

[0065] S4.5. Append the dirty layer index corresponding to the incremental encoding to the architecture-independent intermediate representation.

[0066] In this embodiment, after the incremental encoding of each dirty layer is completed, the system appends the layer index identifier field of that dirty layer to the header of the encoded block of each dirty layer's architecture-independent intermediate representation. Taking the 18th dirty layer as an example, the layer index field value "18" is written to the header of its architecture-independent intermediate representation encoded block, and an incremental identifier bit is appended to distinguish that the encoded block is incrementally encoded rather than fully encoded, so that the target device can identify the layer to which the encoded block belongs and the merging method during the deserialization stage. After the layer index is appended to the architecture-independent intermediate representations of all 5 dirty layers, the system transmits all incremental encoded data to the target device through the high-speed interconnect channel between nodes. The transmission time is about 230ms. At this point, the migration operation on the source container side is completed. The total time from the source container suspension to the completion of the dirty layer transmission is about 730ms, realizing a very short interruption of the inference service.

[0067] In this embodiment of the application, S5, on the target device, all received architecture-independent intermediate representations are deserialized into the target device's native format running state, the inference execution state of the large model inference container is reconstructed and the operation is resumed until the migration confirmation is completed.

[0068] The step of deserializing all received architecture-independent intermediate representations into the target device's native format runtime state includes S5.1~S5.2: S5.1 Read the layer index in the architecture-independent intermediate representation of each inference layer of the large model, and retrieve the running state description information of the corresponding inference layer in the layered state record according to the layer index.

[0069] In this embodiment, the target device receives a total of 32 full coding blocks and 5 incremental coding blocks, for a total of 37 architecture-independent intermediate representation coding blocks.

[0070] First, the layer index field of each coding block header is read sequentially according to layer index 0 to 31. Taking the 8th layer coding block as an example, after reading the layer index field value "8", the system retrieves the corresponding running status description information of the 8th layer in the layer status record table. The retrieval results include complete description information such as the layer type identifier (multi-head attention layer), tensor dimension (hidden layer dimension 4096, number of attention heads 32), data type (FP16), and inter-layer data dependency relationship (input source layer index 7, output target layer index 9).

[0071] The same retrieval operation is then performed on the five incremental coding blocks. The system identifies the coding block type based on the incremental identifier bit, merges the updated data in the incremental coding block with the full coding block data of the corresponding layer, and forms the complete running status code of each dirty layer at the time of source container suspension, ensuring that the data used by subsequent mapping operations is consistent with the time of source container suspension.

[0072] S5.2. According to the running state description information, map the tensor data, attention cache data and inter-layer activation values ​​in the architecture-independent intermediate representation of each inference layer to the corresponding data structure in the native format of the target device, and generate the running state of each inference layer under the native format of the target device.

[0073] It should be noted that the steps of mapping to the corresponding data structure in the native format of the target device include B1~B2: B1. For the inter-layer activation values ​​with attached source inference layer index and target inference layer index, restore the data dependency relationship between the corresponding inference layers in the runtime state of the target device's native format according to the source inference layer index and target inference layer index.

[0074] In this embodiment, the native format of the target device is the NPU operator graph format, which has a different tensor storage layout than the GPU native format. The GPU native format uses a row-major storage layout, while the NPU native format uses a tiered layout. Based on the tensor dimension description information in the hierarchical state record, the storage layout conversion is automatically completed during the mapping process.

[0075] Taking the inter-layer activation value from the output of layer 7 to the input of layer 8 as an example, the system reads the source inference layer index field value "7" and the target inference layer index field value "8" attached to the header of the activation value encoding entry, locates the output node of layer 7 and the input node of layer 8 in the operator graph of the NPU native format, writes the mapped activation value tensor to the corresponding video memory address according to the NPU block storage layout, and establishes the data flow connection from the output node of layer 7 to the input node of layer 8 in the operator graph, thus completing the restoration of the inter-layer data dependency relationship.

[0076] For inference layers with residual connections, the system establishes corresponding cross-layer data flow connections in the operator graph one by one according to the source inference layer index and target inference layer index attached to each entry, based on the multiple inter-layer activation value encoding entries in the encoding block. After the inter-layer data dependency relationships of all 32 layers are restored, a total of 68 data flow connections are established in the operator graph, which are completely consistent with the inter-layer dependency relationships during the inference execution of the source device.

[0077] B2. After the data dependencies of all inference layers are restored, the integrity of the operating status of each inference layer under the native format of the target device is verified layer by layer according to the layered status record, and deserialization is completed.

[0078] In this embodiment, after all inference layer data dependencies are restored, the system performs integrity checks layer by layer in the order of layer indices 0 to 31. For each inference layer, based on the runtime status description information of that layer in the layer status record, the system verifies whether the dimension and data volume of the tensor data are consistent with the description information, whether the sequence length of the attention cache data matches the number of tokens in the current inference step, and whether the data flow connection corresponding to the inter-layer activation value has been fully established in the operator graph.

[0079] Taking the 25th layer verification as an example, the verification results show that the tensor data dimension is consistent with the description information, the attention cache sequence length is 512 tokens consistent with the source container suspension time, the inter-layer activation value data stream connection is completely established, and the 25th layer running state integrity verification is passed.

[0080] After all 32 layers have passed verification, the system completes the deserialization operation and then starts the reconstructed large model inference container on the target device. The container loads the NPU native format running state and resumes inference execution. The inference service successfully resumes running on the target device. The system sends a migration confirmation signal to the source device. After receiving the confirmation signal, the source container performs resource release operations, realizing a near-seamless migration of the inference service.

[0081] In summary, by collecting resource pressure indicators from the source device and triggering a pre-copying process before resource saturation, the running state of each inference layer of the large model is serialized and transmitted during the continuous operation of the source container. This avoids the operation method of having to stop and export in existing solutions, compressing the service interruption time to an extremely brief suspension stage that only requires retransmission of the dirty layer list, achieving near-seamless migration of inference services. At the same time, this invention uniformly encodes the tensor data, attention cache data, and inter-layer activation values ​​of each inference layer of the large model into an architecture-independent intermediate representation, and maps the architecture-independent intermediate representation to the running state in the native format of the target device according to the layered state record during the deserialization stage. This fundamentally solves the format incompatibility problem caused by the difference in native data formats between heterogeneous devices, enabling the inference container to accurately restore the state and seamlessly continue inference execution between computing devices with different architectures.

[0082] Example 3 illustrates a schematic scheme for a unified container orchestration and seamless migration method for heterogeneous computing power pools. It should be noted that the technical solution of this system for unified container orchestration and seamless migration for heterogeneous computing power pools is based on the same concept as the technical solution of the aforementioned method for unified container orchestration and seamless migration for heterogeneous computing power pools. Details not described in detail in this embodiment can be found in the description of the aforementioned method for unified container orchestration and seamless migration for heterogeneous computing power pools.

[0083] This embodiment also provides a unified orchestration and seamless migration system for containers in heterogeneous computing power pools, including: The acquisition module is used to collect resource pressure indicators of the source device and trigger the pre-replication process before the source device becomes resource saturated. The intermediate transmission module is used to serialize the running state of each inference layer of the large model into an architecture-independent intermediate representation according to the hierarchical state record, and transmit it to the target device. At the same time, it records the inference layers that are written and updated during the transmission process to form a dirty layer list. The conversion module is used to suspend the origin container when the dirty layer list meets the suspension condition, serialize the running state of each dirty layer in the dirty layer list into an architecture-independent intermediate representation, and transmit it to the target device; The migration module is used to deserialize all received architecture-independent intermediate representations into the target device's native format runtime state on the target device, reconstruct the inference execution state of the large model inference container and resume operation until the migration is confirmed.

[0084] This embodiment also provides an electronic device applicable to the unified orchestration and seamless migration of containers for heterogeneous computing power pools, including: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to realize the method for unified orchestration and seamless migration of containers for heterogeneous computing power pools as proposed in the above embodiment.

[0085] This embodiment also provides a storage medium on which a computer program is stored. When the program is executed by a processor, it implements the method for unified orchestration and seamless migration of containers for heterogeneous computing power pools as proposed in the above embodiments.

[0086] The storage medium proposed in this embodiment and the method for unified orchestration and seamless migration of containers for heterogeneous computing power pools proposed in the above embodiments belong to the same inventive concept. Technical details not described in detail in this embodiment can be found in the above embodiments, and this embodiment has the same beneficial effects as the above embodiments.

[0087] Based on the above description of the implementation methods, those skilled in the art can clearly understand that the present invention can be implemented using software and necessary general-purpose hardware, and of course, it can also be implemented using hardware. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk, or optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of the various embodiments of the present invention.

[0088] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A unified orchestration and seamless migration method for containers in heterogeneous computing power pools, characterized in that, include: Establish hierarchical state records corresponding to each inference layer of the large model; Collect resource pressure indicators of the source device and trigger a pre-replication process before the source device becomes resource saturated. In response to the triggering of the pre-copy process, the running state of each inference layer of the large model is serialized into an architecture-independent intermediate representation according to the hierarchical state record and transmitted to the target device. At the same time, the inference layers that are written and updated during the transmission process are recorded to form a dirty layer list. When the dirty layer list meets the suspension condition, the origin container is suspended, and the running state of each dirty layer in the dirty layer list is serialized into an architecture-independent intermediate representation and transmitted to the target device; On the target device, all received architecture-independent intermediate representations are deserialized into the target device's native format runtime state, the inference execution state of the large model inference container is reconstructed and resumed, until migration confirmation is completed.

2. The unified orchestration and seamless migration method for containers in heterogeneous computing pools as described in claim 1, characterized in that, The steps to serialize the runtime states of each inference layer of a large model into architecture-independent intermediate representations include: For each inference layer of the large model, extract the tensor data, attention cache data and inter-layer activation values ​​for that inference layer. The tensor data, attention cache data, and inter-layer activation values ​​are encoded according to a preset architecture-independent data format to generate an architecture-independent intermediate representation corresponding to the inference layer.

3. The unified orchestration and seamless migration method for containers in heterogeneous computing pools as described in claim 2, characterized in that, The steps of encoding the tensor data, attention cache data, and inter-layer activation values ​​according to a preset architecture-independent data format include: According to the inference execution order of each inference layer of the large model, the tensor data, attention cache data and inter-layer activation values ​​of each inference layer are encoded sequentially. For inter-layer activation values ​​where there is a data dependency between adjacent inference layers, the source inference layer index and the target inference layer index corresponding to the inter-layer activation value are appended during encoding.

4. The unified orchestration and seamless migration method for containers in heterogeneous computing pools as described in claim 3, characterized in that, The steps for the dirty layer list to meet the suspension conditions include: The number of dirty layers currently awaiting transmission in the dirty layer list is counted, and the total amount of running status data for all dirty layers in the dirty layer list is calculated. When the number of dirty layers is lower than a preset dirty layer number threshold and the total amount of running status data is lower than a preset data amount threshold, the dirty layer list is determined to meet the suspension condition.

5. The unified orchestration and seamless migration method for containers in heterogeneous computing pools as described in claim 4, characterized in that, The steps of serializing the running state of each dirty layer in the dirty layer list into an architecture-independent intermediate representation include: According to the layer index of each dirty layer in the dirty layer list, extract the tensor data, attention cache data and inter-layer activation values ​​of each dirty layer at the time of suspension of the source container. Incremental encoding is performed on the tensor data, attention cache data, and inter-layer activation values ​​that are written and updated in each dirty layer compared to the pre-copy process to generate the architecture-independent intermediate representation of each dirty layer. The dirty layer index corresponding to the incremental encoding is appended to the architecture-independent intermediate representation.

6. The method for unified orchestration and seamless migration of containers for heterogeneous computing power pools as described in claim 5, characterized in that, The steps of deserializing all received architecture-independent intermediate representations into the runtime state of the target device's native format include: Read the layer index from the architecture-independent intermediate representation of each inference layer of the large model, and retrieve the running state description information of the corresponding inference layer in the hierarchical state record according to the layer index; Based on the described running status information, the tensor data, attention cache data, and inter-layer activation values ​​in the architecture-independent intermediate representations of each inference layer are mapped to the corresponding data structures in the native format of the target device, thereby generating the running status of each inference layer under the native format of the target device.

7. The unified orchestration and seamless migration method for containers in heterogeneous computing pools as described in claim 6, characterized in that, The steps of mapping to the corresponding data structure in the native format of the target device include: For the inter-layer activation values ​​with attached source inference layer index and target inference layer index, the data dependencies between the corresponding inference layers are restored in the runtime state of the target device's native format according to the source inference layer index and target inference layer index. After the data dependencies of all inference layers are restored, the integrity of the operating state of each inference layer under the native format of the target device is verified layer by layer according to the layered state record, and deserialization is completed.

8. The method for unified orchestration and seamless migration of containers for heterogeneous computing power pools as described in claim 7, characterized in that, The steps for collecting resource stress indicators from source devices include: The computing unit utilization rate, video memory utilization rate, and video memory bandwidth utilization rate of the source device are collected according to a preset collection cycle, and used as the resource pressure indicator. Each resource pressure indicator is compared with its corresponding resource saturation warning threshold. When any resource pressure indicator exceeds the corresponding resource saturation warning threshold, the pre-replication process is triggered.

9. The method for unified orchestration and seamless migration of containers for heterogeneous computing power pools as described in claim 8, characterized in that, The steps for comparing each resource pressure indicator with its corresponding resource saturation early warning threshold include: According to a preset time window, the moving average of the computing unit utilization rate, video memory utilization rate and video memory bandwidth utilization rate are calculated respectively. The moving average is used as the input value for comparing each resource pressure indicator with the corresponding resource saturation warning threshold, so as to filter out the interference of instantaneous fluctuations in the resource pressure indicators on the pre-replication process trigger judgment.

10. A unified orchestration and seamless migration system for containers in heterogeneous computing pools, employing the method described in any one of claims 1-9, characterized in that, include: The acquisition module is used to collect resource pressure indicators of the source device and trigger the pre-replication process before the source device becomes resource saturated. The intermediate transmission module is used to serialize the running state of each inference layer of the large model into an architecture-independent intermediate representation according to the hierarchical state record, and transmit it to the target device. At the same time, it records the inference layers that are written and updated during the transmission process to form a dirty layer list. The conversion module is used to suspend the origin container when the dirty layer list meets the suspension condition, serialize the running state of each dirty layer in the dirty layer list into an architecture-independent intermediate representation, and transmit it to the target device; The migration module is used to deserialize all received architecture-independent intermediate representations into the target device's native format runtime state on the target device, reconstruct the inference execution state of the large model inference container and resume operation until the migration is confirmed.

Images