A model file processing method and an electronic device

Abstract

This disclosure provides a method for processing model files and an electronic device. The method relates to the field of model processing technology and specifically includes: acquiring the hardware parameters of the electronic device; determining the model sharding strategy corresponding to the model file of the target model based on the hardware parameters of the electronic device, and dividing the model file into multiple model shards according to the model sharding strategy; determining the storage area corresponding to each model shard in the electronic device, and storing the model shard in the storage area, the storage area including a first storage area and a second storage area, wherein the access performance of the model shard stored in the first storage area is higher than the access performance of the model shard stored in the second storage area; and if adjustment conditions are met, performing at least one adjustment operation, the adjustment operation including adjusting the model sharding strategy and allocating the storage area corresponding to the model shard.

Description

Technical Field

[0001] This disclosure relates to the field of model processing technology, and in particular to a method for processing model files and an electronic device. Background Technology

[0002] With the continuous development of artificial intelligence technology, models are being widely used in various fields. The parameter scale and computational complexity of these models are constantly increasing, placing higher demands on computing resources, storage resources, and data transmission efficiency during the inference phase. Especially on personal computing devices, edge computing devices, or other resource-constrained electronic devices, how to efficiently deploy and run models has become one of the most pressing technical problems to be solved in this field. Summary of the Invention

[0003] This disclosure provides a method for processing model files and an electronic device to at least solve the above-mentioned technical problems existing in the prior art.

[0004] According to a first aspect of this disclosure, a method for processing model files is provided, the method comprising: Obtain the hardware parameters of electronic devices; Based on the hardware parameters of the electronic device, determine the model sharding strategy corresponding to the model file of the target model, and divide the model file into multiple model shards according to the model sharding strategy; Determine the storage area corresponding to each model fragment in the electronic device, and store the model fragment in the storage area. The storage area includes a first storage area and a second storage area. The access performance of the model fragment stored in the first storage area is higher than that of the model fragment stored in the second storage area. If the adjustment conditions are met, at least one adjustment operation is performed, including adjusting the model sharding strategy and allocating the storage area corresponding to the model shard.

[0005] In one possible implementation, the model sharding strategy corresponding to the model file of the target model is determined based on the hardware parameters of the electronic device, including: The model sharding strategy corresponding to the model file is determined based on the computational granularity of processor data processing and the access granularity of memory data reading and writing; wherein, the hardware parameters include the computational granularity of processor data processing and the access granularity of memory data reading and writing.

[0006] In one possible implementation, the method further includes: Parse the model file of the target model to obtain the number of model layers and the weight matrix corresponding to each model layer; After dividing the model file into multiple model fragments according to the model fragmentation strategy corresponding to the model file, the methods include: Determine the fragmentation information for each model fragment. The fragmentation information for each model fragment includes the model layer number in the target model, the weight type of the weight matrix to which the model fragment belongs, and the fragmentation index of the model fragment in the weight matrix.

[0007] In one possible implementation, if the adjustment conditions are met, at least one adjustment operation is performed, including: If the adjustment conditions are met, adjust the model sharding strategy and perform re-sharding operation on multiple model shards to obtain multiple re-sharded model shards. Determine the amount of scheduling resources required to schedule each model fragment across different storage regions; Candidate model fragments with a scheduling resource amount less than a preset value are selected from multiple model fragments; Candidate models stored in the second storage area are sharded and scheduled to the first storage area.

[0008] The adjustment conditions include at least one of the following: The migration frequency of the target model shard between different storage regions in multiple model shards is greater than a preset frequency threshold. In one possible implementation, the access correlation between any two model shards in the multiple model shards is greater than a first correlation threshold. The resource pressure status of the electronic device meets the preset pressure triggering conditions; The computing configuration strategy of the electronic device is switched, and the computing configuration strategy includes at least one of the following: switching of computing units and switching of execution precision configuration of computing units.

[0009] In one implementation, the re-sharding operation includes a merge operation, which merges at least two model shards into one model shard; performing the re-sharding operation on multiple model shards includes: Get the access correlation between every two model shards in multiple model shards; Based on access relevance, at least two first model fragments are selected from multiple model fragments, and the access relevance of any two first model fragments is greater than the second relevance threshold. Merge at least two first-model fragments.

[0010] In one implementation, the re-sharding operation includes a splitting operation for splitting a model shard into at least two model shards; performing the re-sharding operation on multiple model shards includes: Obtain the popularity distribution information for each model fragment; the popularity distribution information represents the access popularity distribution within the model fragment. The third model segment is selected from multiple model segments based on the heat distribution information of the model segments that meets the preset heat distribution conditions. Split the third model into pieces.

[0011] In one possible implementation, determining the amount of scheduling resources required to schedule each model shard across different storage regions includes: Obtain the operating mode of the electronic device and the scheduling evaluation function corresponding to each operating mode; for the multiple evaluation indicators in the scheduling evaluation function, different weight configuration combinations are corresponding to the evaluation indicators under different operating modes; Using the scheduling evaluation function corresponding to the current operating mode, determine the amount of scheduling resources required for each model fragment to be scheduled in different storage regions.

[0012] In one possible implementation, the multiple evaluation metrics include at least energy consumption metrics, latency metrics, and write amplification metrics; When the operating mode is the first operating mode, the first weight of the energy consumption index and the third weight of the write amplification index are both less than the second weight of the delay index. When the operating mode is the second operating mode, the first weight of the energy consumption index and the third weight of the write amplification index are both greater than the second weight of the delay index. When the operating mode is the third operating mode, the weight difference between the first weight corresponding to the energy consumption index and the second weight corresponding to the delay index is less than the preset weight difference. The performance of the electronic device in the first operating mode is higher than that in the second and third operating modes.

[0013] In one embodiment, the operating mode of the electronic device and the storage parameters of the storage area corresponding to the operating mode are obtained. The storage parameters include at least one of the following: the number of storage model fragments, fragment precision, and access popularity threshold, wherein the access popularity threshold is used to filter the model fragments stored in the storage area; different operating modes correspond to different storage parameters. Configure the storage area of ​​the electronic device based on the storage parameters corresponding to the operating mode.

[0014] According to a second aspect of this disclosure, an electronic device is provided, the electronic device comprising: At least two memory locations; Multiple processors; The scheduler is used to: acquire the hardware parameters of the electronic device; determine the model sharding strategy corresponding to the model file of the target model based on the hardware parameters of the electronic device, and divide the model file into multiple model shards according to the model sharding strategy; determine the storage area corresponding to each model shard in the electronic device, and store the model shard in the storage area, which includes a first storage area and a second storage area, wherein the access performance of the model shards stored in the first storage area is higher than that of the model shards stored in the second storage area; and if the adjustment conditions are met, execute at least one adjustment operation, which includes adjusting the model sharding strategy and allocating the storage area corresponding to the model shard.

[0015] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0016] The above and other objects, features, and advantages of this disclosure will become readily apparent from the following detailed description of exemplary embodiments, taken in conjunction with the accompanying drawings. Several embodiments of this disclosure are illustrated in the drawings by way of example and not limitation, in which: In the accompanying drawings, the same or corresponding reference numerals indicate the same or corresponding parts.

[0017] Figure 1 This is a schematic diagram of the implementation flow of a model file processing method according to an embodiment of this disclosure. Figure 1 ; Figure 2 This is a schematic diagram of the implementation flow of a model file processing method according to an embodiment of this disclosure. Figure 2 ; Figure 3 This is a schematic diagram of the implementation flow of a model file processing method according to an embodiment of this disclosure. Figure 3 ; Figure 4 A schematic diagram of the composition structure of an electronic device according to an embodiment of the present disclosure is shown. Detailed Implementation

[0018] To make the objectives, features, and advantages of this disclosure more apparent and understandable, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without creative effort are within the scope of protection of this disclosure.

[0019] To address the technical problems existing in the background art, this disclosure provides a method for processing model files and an electronic device. The method for processing model files provided by this disclosure will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0020] Figure 1 This is a schematic diagram of the implementation flow of a model file processing method provided in this embodiment. Figure 1 .

[0021] like Figure 1 As shown, the model file processing method provided in this embodiment may include the following steps: S110: Obtain the hardware parameters of the electronic device.

[0022] The electronic device can be any of the devices related to the target model to be deployed. For example, the electronic device can be any of the following: personal terminal devices (laptops, desktops, tablets, or embedded devices, etc.), servers (edge ​​computing servers, cloud servers, etc.), robotic devices, and in-vehicle computing devices. No specific limitation is made here.

[0023] The hardware parameters of the electronic device can be used to characterize the hardware configuration capabilities of the electronic device, such as at least one of the processor's computing power and memory access capabilities, without being specifically limited here.

[0024] S120: Based on the hardware parameters of the electronic device, determine the model sharding strategy corresponding to the model file of the target model, and divide the model file into multiple model shards according to the model sharding strategy.

[0025] The target model can be, for example, a Generative Pre-trained Transformer (GPT) series, a Large Language Model Meta AI (LLaMA) model, or an image processing model, a speech recognition model, etc. There are no specific limitations here.

[0026] Additionally, a model partitioning strategy can be a strategy used to divide the target model's model file into multiple model partitions. This model partitioning strategy can include information such as the partitioning granularity (i.e., partition size) and partition boundaries. It should be noted that different hardware parameters can correspond to different model partitioning strategies, but no specific limitations are made here.

[0027] S130, determine the storage area corresponding to each model fragment in the electronic device, and store the model fragment in the storage area.

[0028] In some embodiments, the aforementioned storage area may include a first storage area and a second storage area, wherein the access performance of model fragments stored in the first storage area is higher than that of model fragments stored in the second storage area. This access performance may include access speed, bandwidth, latency, and the time it takes for the model to produce results, etc., which are not specifically limited here. Thus, frequently accessed model fragments can be allocated to the first storage area for storage, and infrequently accessed model fragments can be allocated to the second storage area for storage; further details are omitted here.

[0029] Furthermore, this second storage area can be further subdivided into a first sub-storage area and a second sub-storage area, wherein the access performance of the model stored in the first sub-storage area is higher than the access performance of the model fragments stored in the second sub-storage area. For example, the first storage area can be GPU memory, the first sub-storage area can be CPU memory, and the second sub-storage area can be a non-volatile memory express solid-state drive (NVMe SSD), without specific limitations here.

[0030] S140, if the adjustment conditions are met, perform at least one adjustment operation, including adjusting the model sharding strategy and allocating the storage area corresponding to the model shard.

[0031] The adjustment condition can be a pre-set condition based on the actual situation to determine whether an adjustment operation needs to be performed; no specific limitation is made here. Additionally, the adjustment operation can include adjusting the model sharding strategy (e.g., changing the sharding granularity or sharding boundaries) and reallocating the storage area corresponding to the model shard (e.g., migrating a model shard originally located in the second storage area to the first storage area); no specific limitation is made here.

[0032] Specifically, the system can acquire the hardware parameters of an electronic device and, based on these parameters, determine the model sharding strategy corresponding to the target model's model file. The model file is then divided into multiple model shards according to this strategy. Subsequently, the storage region corresponding to each model shard is determined, with some shards stored in a first storage region with higher access performance, and others stored in a second storage region with lower access performance. Based on this, during model execution, the system continuously monitors whether adjustment conditions are met. If so, at least one operation—adjusting the model sharding strategy and reallocating storage regions—is executed, thereby achieving dynamic optimization of model storage and execution.

[0033] Based on the model file processing method provided in this disclosure, the model allocation strategy corresponding to the target model's model file can be dynamically determined based on the acquired hardware parameters of the electronic device. The model file is then divided into multiple model fragments according to the allocation strategy, and each model fragment is allocated a corresponding storage area within the electronic device. Model fragments with higher access performance requirements are stored in the first storage area, while those with lower access performance requirements are stored in the second storage area. Furthermore, during model operation, the system continuously monitors whether adjustment conditions are met. If so, at least one operation—adjusting the model fragmentation strategy and reallocating storage areas—is executed, thereby achieving a reasonable division of model parameters and dynamic optimization of model storage and operation. In this way, by differentially fragmenting and hierarchically storing model files according to the hardware configuration of the electronic device, efficient deployment and inference execution rates of large language models on AIPCs are effectively achieved. Simultaneously, by continuously monitoring adjustment conditions and dynamically optimizing the model fragmentation strategy and storage allocation, the system effectively adapts to resource changes or load fluctuations of the AIPC device, avoiding performance degradation or failures due to fixed initial configurations. Thus, while ensuring model inference performance, the deployment and inference efficiency of large language models on AIPC devices is significantly improved.

[0034] In order to accurately determine the model sharding strategy corresponding to the model file, and thus facilitate the subsequent partitioning of multiple model shards that meet the hardware configuration of the electronic device based on the model sharding strategy, in one embodiment, the step of determining the model sharding strategy corresponding to the model file of the target model according to the hardware parameters of the electronic device may include: Based on the computational granularity of processor data processing and the access granularity of memory data reading and writing, the model sharding strategy corresponding to the model file is determined.

[0035] In some embodiments, the hardware parameters may include the computational granularity of processor data processing and the access granularity of memory data reading and writing. The computational granularity of processor data processing refers to the optimal input size of the data block that a computing core, such as a GPU or Neural Processing Unit (NPU), can efficiently process during a single computational task. This size is typically represented by matrix dimensions, such as 8×8, 16×16, 64×64, 128×64, or 256×256. The computational granularity determines the optimal input size for the processor when performing matrix operations; exceeding or falling below this granularity may lead to decreased computational efficiency. Therefore, the model partitioning must match the optimal input size for the processor when performing matrix operations to maximize computational throughput.

[0036] The access granularity of memory read / write operations refers to the size of the data unit corresponding to a single read / write operation in a storage device such as a Solid State Drive (SSD). This access granularity is typically expressed in bytes or data counts, such as 4KB, 8KB, 16KB, or a 64×64 matrix. Access granularity affects data alignment within the storage medium and input / output efficiency (I / O). Therefore, model fragmentation should be an integer multiple of the access granularity to reduce read / write amplification and improve I / O efficiency.

[0037] Specifically, since the hardware parameters of the electronic device can include the computational granularity of the processor processing data and the access granularity of the memory reading and writing data, the model sharding strategy corresponding to the model file can be determined based on the computational granularity of the processor processing data and the access granularity of the memory storing data, thereby accurately meeting the model sharding strategy required by the hardware configuration of the electronic device.

[0038] In this embodiment, a model sharding strategy that satisfies the hardware configuration of the electronic device can be accurately determined based on the processor's data processing and computational granularity and the memory's read / write data access granularity in the hardware parameters. Thus, by aligning the sharding granularity of the model shards with the processor's computational granularity, it can be ensured that each model shard adapts to the processor's optimal input size when participating in computation, avoiding wasted computational resources or efficiency degradation caused by mismatch between sharding granularity and computational granularity. Simultaneously, by aligning the sharding granularity of the model shards with the memory's access granularity, the read / write alignment of data in the storage medium can be optimized, reducing additional input / output overhead caused by mismatched access granularity. Therefore, the resulting multiple model shards can fully adapt to the hardware configuration of the electronic device at both the computational and storage levels.

[0039] To obtain a more accurate model sharding strategy, thereby accurately dividing the model file into multiple model shards, in one embodiment, the step of determining the model sharding strategy corresponding to the model file based on the computational granularity of processor data processing and the access granularity of memory data reading and writing may specifically include: The initial model fragmentation strategy is determined based on the computational granularity of processor data processing and the access granularity of memory data reading and writing. The initial model sharding strategy is adjusted based on the model file of the target model to obtain the model sharding strategy corresponding to the model file.

[0040] The target model's model file refers to the file that provides the target model's own structural information. It may include the number of model layers, the type of weight matrix contained in each layer (such as attention weights, feed-forward network (FFN) weights, layer normalized weights, etc.), and the specific size of each weight matrix, which is not specifically limited here.

[0041] Specifically, an initial model partitioning strategy can be determined based on information such as the computational granularity of processor data processing and the memory read / write data access granularity in the hardware parameters of the electronic device. Based on this, the initial model partitioning strategy is further adjusted by combining the model file of the target model, thereby obtaining a more accurate model partitioning strategy.

[0042] More specifically, firstly, candidate sharding granularities are determined based on the processor's computational granularity and the memory's access granularity. For example, a sharding granularity that is both divisible by the computational granularity and aligned with the access granularity (i.e., an integer multiple of the access granularity) is selected. Next, the candidate sharding granularities are adjusted based on the target model's model file to determine the final sharding granularity used to partition the model file. This granularity is designed to be divisible by each weight matrix in the target model, resulting in multiple model shards of uniform size.

[0043] In this embodiment, based on the computational granularity of processor data processing and the access granularity of memory data reading and writing, the model file of the target model can be further combined to determine a more accurate model strategy. This enables the partitioned model fragments to meet the actual requirements in terms of computational granularity, access granularity, and model structure.

[0044] To provide a comprehensive and detailed description of the model file processing method provided in this disclosure, in one embodiment, the model file processing method further includes: Parse the model file of the target model to obtain the number of model layers and the weight matrix corresponding to each model layer.

[0045] The target model typically includes multiple model layers (i.e., network layers), and each model layer contains multiple weight matrices. The target model file refers to the file that stores the network structure configuration of the target model and the weight parameter values ​​of each layer. Therefore, parsing the target model file can yield the number of model layers and the corresponding weight matrices for each model layer, without making any specific limitations here.

[0046] Based on this, after dividing the model file into multiple model fragments according to the model fragmentation strategy corresponding to the model file, the model file processing method provided in this embodiment of the disclosure may further include: Determine the sharding information for each model shard.

[0047] In some embodiments, the fragmentation information of the model fragment may include the model layer number in the target model where the model fragment is located, the weight type of the weight matrix to which the model fragment belongs, and the fragmentation index of the model fragment in the weight matrix, which are not specifically limited here.

[0048] Specifically, by parsing the model file of the target model, the number of model layers and the weight matrix corresponding to each model layer can be obtained. Thus, after dividing the model file into multiple model fragments according to the model fragmentation strategy corresponding to the model file, the model layer number of each model fragment in the target model, the weight type of the weight matrix to which each model fragment belongs, and the fragmentation number of each model fragment in the weight matrix can be directly determined.

[0049] To illustrate the above fragmentation process more clearly, a specific example will be used to explain it in detail below.

[0050] In one example, assume the target model is an LLM model containing multiple Transformer layers. For ease of understanding, focus on Layer L in this example. Also, assume the following hardware parameters for the electronic device: the GPU's matrix computation units (such as Tensor Cores) are most efficient at processing 256×256 16-bit floating-point (FP16) matrices, and the NVMe SSD has a physical page size of 4KB (i.e., an access granularity of 4KB).

[0051] Based on the above hardware parameters, the sharding operation is performed according to the following three dimensions: 1. Hierarchical Partitioning: When partitioning along this dimension, the entire LLM model is divided into a series of independent Transformer layers (e.g., Layer0, Layer1, ..., LayerL, ..., LayerN-1), an Embedding layer, and a final LM Head layer. Thus, after partitioning along this dimension, the model layer number of each partition within the target model can be obtained.

[0052] This allows for full utilization of the sequential nature of LLM inference (i.e., computation is performed layer by layer when processing tokens), making the data prefetching access pattern highly predictable. For example, during model execution, once Layer 11 has been computed, it can be predicted that the data required for subsequent computations are all related to Layer 12, thus enabling prefetching operations for data related to Layer 12 computations to be performed in advance.

[0053] 2. Weight Type Slicing: Since the weight matrices included in each layer of the model have different functions and access modes, for ease of understanding, the following example uses a standard Transformer layer, which mainly contains the following weight matrices: (1) Weight matrices of the Self-Attention (SA) module: query projection matrix W_q, key projection matrix W_k, value projection matrix W_v, output projection matrix W_o.

[0054] (2) Weight matrices of the FFN module: up projection matrix W_up, gate matrix W_gate (e.g. in the Swish-Gated Linear Unit (SwiGLU)), and down projection matrix W_down.

[0055] (3) Weight matrix of Layer Normalization (LN) module: weight and bias of LayerNorm1, weight and bias of LayerNorm2. Among them, weight of LayerNorm1 is the weight parameter of the first layer normalization, bias of LayerNorm1 is the bias parameter of the first layer normalization, and correspondingly, weight of LayerNorm2 is the weight parameter of the second layer normalization, and bias of LayerNorm2 is the bias parameter of the second layer normalization.

[0056] It's important to note that different types of weight matrices have different access characteristics. For example, W_q, W_k, and W_v are accessed intensively when calculating attention scores, while the weights of the FFN are accessed later. Processing these weight matrices separately allows for differentiated scheduling and quantization based on their respective access sequences and computational characteristics. For instance, in layer 12, multiple independent model slices can be identified, such as L12_Attn_W_q (the query projection matrix in the layer 12 self-attention module), L12_Attn_W_k (the key projection matrix in the layer 12 self-attention module), and L12_FFN_W_up (the up projection matrix in the layer 12 feedforward network module). Thus, after partitioning according to this dimension, the weight type of the weight matrix to which each model slice belongs can be obtained.

[0057] 3. Slice Granularity (i.e., Slice Size): This dimension represents the finest granularity of the partitioning, directly impacting computational and I / O efficiency. This dimension further subdivides the model slices obtained based on weight type to arrive at the final model slices and determine the slice number for each slice.

[0058] Before partitioning based on the fragment size, it is necessary to determine the fragment size corresponding to the target model based on the processor calculation granularity and memory access granularity in the hardware parameters of the electronic device. In one example, the optimal input size of the GPU is a 256×256 FP16 matrix, and the physical page size of the NVMe SSD is 4KB. Based on these two factors, the fragment size is determined to be [256, 256]. Assuming that the size of the Layer12 W_q matrix is ​​[4096, 4096], then the W_q matrix will be divided into (4096 / 256)×(4096 / 256)=16×16=256 model fragments.

[0059] After partitioning, a unique partition number can be created for each model partition, for example, L12_Attn_W_q_CAB_r0_c0, L12_Attn_W_q_CAB_r0_c1, ..., L12_Attn_W_q_CAB_r15_c15. Here, L12_Attn_W_q_CAB_r0_c0 represents the model partition located in row 0 and column 0 of the projection matrix (W_q) in the 12th layer self-attention module. The other numbers can be deduced in the same way, that is, "r{row number}_c{column number}" respectively represent the partition blocks corresponding to different row and column index positions of the weight matrix.

[0060] It should also be noted that after the model is divided into multiple model shards, metadata for each model shard can be generated. This metadata may include information such as quantization status, the corresponding storage area in the electronic device (such as the first storage area or the second storage area), and access frequency. No specific limitations are made here.

[0061] Thus, through the above three-level sharding, precise sharding information can be determined for each model shard, including the model layer number where the model shard is located (e.g., the 12th layer), the weight type of the weight matrix to which it belongs (e.g., the query projection matrix W_q of the attention mechanism), and the sharding index in the weight matrix (e.g., the model shard in row 0 and column 0).

[0062] In one alternative embodiment, the target model is an image processing model, including but not limited to Convolutional Neural Networks (CNNs), Vision Transformers (ViTs), and their derivative architectures. Such models typically contain multi-level network structures for progressively extracting and processing image features.

[0063] In this embodiment, the three-dimensional segmentation may specifically include the following methods: 1. Layer-level Sharding: This involves sharding the image processing model according to its network structure layers. In convolutional models, this layer-level sharding can correspond to different computational stages or feature extraction layers; in visual transformer models, it can correspond to different transformer blocks or feature processing modules. Sharding by network layer facilitates the predictive scheduling of sharded data by leveraging the sequential execution of layers during image inference.

[0064] 2. Weight Type Sharding: Within each network layer, weights are further divided according to their functional type. For example, in a convolutional module, weight type sharding may include different forms of convolutional kernel weights or downsampling-related weights; in a module containing an attention mechanism, weight type sharding may include projected weights and related parameters used for feature mapping or attention calculation. By distinguishing weights of different functional types, it is beneficial to perform differentiated scheduling based on their access timing and computational characteristics.

[0065] 3. Computation Alignment Block Sharding (CAB Sharding): Building upon the hierarchical sharding and weight type sharding described above, the weight data is further divided into multiple computation alignment blocks (i.e., model shards obtained based on sharding granularity). The size of these computation alignment blocks is determined according to the computation granularity of the computing units and the data access granularity of the storage units in the electronic device. This ensures that each computation alignment block can efficiently match the input scale of the computation operators when participating in matrix or tensor operations, while reducing invalid data loading caused by mismatched storage access granularity, thereby improving the overall efficiency of the image processing model during inference.

[0066] In another alternative embodiment, the target model is a speech processing model, including but not limited to a speech recognition model or a speech understanding model based on time-series modeling. Such models are typically used for staged processing of continuous speech signals, and their parameter access exhibits clear staged characteristics in the time dimension.

[0067] In this embodiment, the three-dimensional segmentation may specifically include the following methods: 1. Layer-level Sharding: The model is divided into layers according to the speech signal processing flow. The layer-level sharding may include audio preprocessing or downsampling layers, acoustic feature encoding layers, and decoding layers. By utilizing the sequential nature of each processing stage in speech inference, pipelined scheduling of model sharding can be achieved in the time dimension.

[0068] 2. Weight Type Sharding: This involves slicing the speech model based on the functional differences of different weights in temporal modeling. Weight type slicing may include convolutional weights for extracting local temporal features, attention weights for modeling global contextual relationships, and linear projection weights related to feedforward networks or output mappings. Distinguishing between weights of different functional types facilitates dynamic management in conjunction with changes in access frequency during speech inference.

[0069] 3. Computational Alignment Block Sharding (CAB Sharding): To address the operational characteristics of speech models in streaming or near real-time inference scenarios, the weight matrix is ​​further divided into multiple computational alignment blocks. The size of these computational alignment blocks is preferably matched to the processing power of the computational pipeline and the data prefetch granularity of the storage system. This reduces the additional latency caused by frequent data reads during speech inference and improves overall operational stability and energy efficiency.

[0070] Regardless of whether the target model processes text, images, or speech, the model file can be parsed to obtain the model layer number and the weight matrix corresponding to each model layer. After the model is partitioned, the partition information of each model partition can be accurately determined. The partition information of the model partition can include the model layer number of the model partition in the target model, the weight type of the weight matrix to which the model partition belongs, and the partition number of the model partition in the weight matrix. In this way, the model file can be partitioned in three dimensions: layer, weight type, and partition size, to form multiple model partitions that are fine-grained and can be independently scheduled.

[0071] To accurately and comprehensively describe the model file processing method provided in the embodiments of this disclosure, in one embodiment, if at least one adjustment operation includes adjusting the model sharding strategy and allocating storage areas corresponding to model shards, such as Figure 2 As shown, the above-mentioned S140 may specifically include: S210, if the adjustment conditions are met, adjust the model sharding strategy, and perform re-sharding operation on multiple model shards based on the adjusted model sharding strategy to obtain multiple re-sharded model shards.

[0072] The re-sharding operation refers to the operation of re-sharding multiple existing model shards, such as changing the sharding granularity or adjusting the sharding boundaries, without specific limitations here.

[0073] S220, determine the amount of scheduling resources required for each model shard to be scheduled between different storage regions.

[0074] The amount of scheduling resources required for each model shard to schedule between different storage regions may include at least one of write amplification, time latency, or energy consumption, without specific limitations here.

[0075] S230: Select candidate model fragments from multiple model fragments whose scheduling resource amount is less than a preset value.

[0076] The preset value can be a threshold set in advance based on actual experience to determine the candidate model shards for migration between different storage areas, and no specific limitation is made here.

[0077] S240, the candidate models stored in the second storage area are sharded and scheduled to the first storage area.

[0078] It should be noted that since the second storage area may also include a first sub-storage area and a second sub-storage area, and the access performance of the model stored in the first sub-storage area is higher than that of the model fragments stored in the second sub-storage area, and the access performance of the model fragments stored in the first storage area is higher than that of the model fragments stored in the second storage area, the migration of model fragments needs to be performed in the direction of second sub-storage area, first sub-storage area, and then first storage area. Furthermore, if it is necessary to migrate model fragments from the first storage area to the second storage area, the migration needs to be performed in the direction of first storage area, first sub-storage area, and then second sub-storage area; this will not be elaborated further here.

[0079] Specifically, if the adjustment conditions are met, the model sharding strategy can be adjusted to obtain an adjusted model sharding strategy. Based on this adjusted model sharding strategy, the existing multiple model shards are re-sharded to obtain multiple re-sharded model shards. Next, for each model shard in the multiple re-sharded model shards, the amount of scheduling resources required for scheduling the model shard between different storage regions is determined. Then, candidate model shards with scheduling resource amounts less than a preset value can be selected from the multiple model shards, and the candidate model shards stored in the second storage region are scheduled to the first storage region.

[0080] The foregoing embodiments describe in detail the adjustment operations that need to be performed if the adjustment conditions are met, which may include adjusting the model sharding strategy and allocating the storage area corresponding to the model shard. In addition, when the adjustment conditions are met, the adjustment operations that need to be performed may also be adjusting the model sharding strategy, or the adjustment operation may be allocating the storage area corresponding to the model shard. The following describes these two cases in detail.

[0081] If the adjustment conditions are met, and at least one adjustment operation only involves adjusting the model sharding strategy, the model sharding strategy can be adjusted (e.g., changing the sharding granularity or adjusting the sharding boundaries). Based on the adjusted model sharding strategy, a re-sharding operation is performed on multiple model shards to obtain multiple re-sharded model shards. Since the re-sharding operation only changes the granularity and boundary division of the model shards, and does not change the access frequency of each model shard itself, there is no need to reallocate the storage area corresponding to each model shard. Thus, by only adjusting the model sharding strategy and performing the re-sharding operation, the granularity and boundary division of the model shards themselves can be optimized while keeping the existing storage areas corresponding to each model shard unchanged.

[0082] Alternatively, if the adjustment conditions are met, and at least one adjustment operation only involves allocating the storage region corresponding to the model shard, then if the current sharding granularity of the model shard already meets the actual needs, the current model sharding strategy can remain unchanged. Simultaneously, the access frequency and computational importance of each model shard during the current operation can be considered. Based on this, the storage region corresponding to each model shard is reallocated. Specifically, model shards with high access frequency or high computational importance can be scheduled to the first storage region, while model shards with low access frequency or low computational importance can be scheduled to the second storage region. In this way, by keeping the model sharding strategy unchanged and only adjusting the allocation of storage regions, there is no need to perform the costly resharding operation.

[0083] It's important to note that once a model shard within the same logical region completes a resharding operation, it enters a cooldown period. During this cooldown, no further refinement or merging operations are performed. Furthermore, after the resharding operation is completed, the multiple resharded model shards are copied and rearranged in the second storage area according to the resharding operation (merging or splitting), and a new index structure is written to generate a new layout file. Simultaneously, the multiple model shards before resharding are retained to ensure coexistence of old and new versions and facilitate safe rollback. For model shards to be migrated, a corresponding cache image is first built in the second storage area to facilitate rapid promotion to the first storage area later. After preparation, the shard directory is atomically updated based on the multiple resharded model shards, making the new shard sequence numbers effective. Simultaneously, the old shard sequence numbers are moved to the retirement queue, where resources are reclaimed after confirmation that there are no access references. When a new model shard is accessed for the first time, prefetching and promotion operations are performed according to the new layout. If tail latency is detected to worsen, a rollback mechanism is triggered to switch to the new version, while simultaneously increasing the debouncing threshold and extending the cooldown period to suppress frequent switching.

[0084] In this embodiment, the model sharding strategy can be adjusted preferentially when adjustment conditions are met, and existing model shards are re-partitioned based on the adjusted strategy. Furthermore, by selecting candidate model shards with lower scheduling resource requirements for cross-storage region migration, model shards in the second storage region can be scheduled to the first storage region with higher access performance using lower scheduling resource requirements. This not only dynamically responds to changes in the model sharding strategy but also effectively avoids the additional resource consumption caused by blindly migrating model shards with higher scheduling resources.

[0085] In order to accurately and comprehensively describe the model slicing processing method provided in the embodiments of this disclosure, in one embodiment, the adjustment conditions include at least one of the following: Condition 1: The migration frequency of the target model shard between different storage areas in multiple model shards is greater than a preset frequency threshold.

[0086] The target model segment can be any one of multiple model segments; no specific limitation is made here.

[0087] Additionally, migration frequency refers to the number of times the model fragment is scheduled to migrate between the first and second storage areas per unit of time. Correspondingly, the preset frequency threshold can be a judgment threshold pre-set based on actual conditions, used to determine whether a model fragment is migrated too frequently between different storage areas; no specific limitation is made here.

[0088] It should be noted that if the migration frequency of a certain model shard is too high, it indicates that the current model sharding strategy or storage allocation scheme may be unreasonable. For example, improper shard boundary division may cause the shard to be repeatedly swapped in and out under different access modes, affecting the overall efficiency of model inference. Therefore, this can be used as one of the conditions to trigger adjustment operations.

[0089] Condition 2: The access correlation between any two model shards in multiple model shards is greater than the first correlation threshold.

[0090] The access correlation degree is used to characterize the probability that two model parts are accessed simultaneously during model inference. For example, if two model parts are frequently accessed sequentially in time, or if the two model parts belong to the weight matrices of the same or adjacent model layers, then the access correlation degree between the two model parts is high. Correspondingly, the first correlation degree threshold can be a judgment threshold set based on the actual situation, used to determine whether the correlation degree between the two model parts reaches the level that requires adjustment operations.

[0091] It should be noted that if the access correlation between two model shards is too high, but these two model shards are currently allocated to different storage regions (e.g., one in the first storage region and the other in the second storage region), the model may need to frequently switch between the two storage regions to access the correlated shard during inference, affecting the overall efficiency of model inference. Therefore, this can be used as one of the conditions to trigger adjustment operations.

[0092] Condition 3: The resource pressure status of the electronic device meets the preset pressure triggering condition.

[0093] The resource stress status is used to characterize the current stress level of various resources in an electronic device. This can include metrics such as memory utilization and fragmentation, NVMe read latency percentiles (e.g., P95 read latency degradation), power consumption, or temperature. Correspondingly, the preset stress trigger conditions can be pre-set judgment conditions based on actual conditions, such as memory utilization exceeding 90% and high fragmentation, NVMe read P95 latency degradation exceeding a preset threshold, or device temperature exceeding a temperature threshold. No specific limitations are specified here.

[0094] It should be noted that when the resource pressure state of an electronic device changes, the original model sharding strategy or storage allocation scheme may no longer be applicable to the current resource environment of the electronic device. Therefore, changes in resource pressure state can be used as one of the conditions to trigger adjustment operations.

[0095] Condition 4: The computing configuration strategy of the electronic device is switched, and the computing configuration strategy includes at least one of the following: switching of computing units, switching of execution precision configuration of computing units.

[0096] The switching of computing units can refer to the switching of electronic devices between different processors, such as switching from CPU to GPU, or from GPU to NPU, etc., without specific limitations.

[0097] Switching the execution precision configuration of a computing unit can refer to a change in the processor's computing precision, such as switching from 32-bit floating point (FP32) to FP16, or from (8-bit Integer, INT8) integer precision to FP16 precision, etc., which will not be elaborated on here.

[0098] It should be noted that in the embodiments of this disclosure, different model sharding strategies and storage strategies correspond to different computing units and different computing precisions. Therefore, switching the computing configuration strategy of the electronic device can be used as one of the conditions for triggering the adjustment operation.

[0099] Thus, during model operation, the system continuously monitors whether at least one of the following adjustment conditions is met. If met, at least one adjustment operation is triggered. Specifically, if it is detected that the migration frequency of the target model fragment between different storage regions is greater than a preset frequency threshold among the current multiple model fragments, at least one adjustment operation is triggered to adjust the model fragmentation strategy and allocate the corresponding storage region of the model fragment; if it is detected that the access correlation between any two model fragments is greater than a first correlation threshold, at least one adjustment operation is triggered to adjust the model fragmentation strategy and allocate the corresponding storage region of the model fragment; if it is detected that the resource pressure state of the electronic device meets the preset pressure triggering conditions (e.g., the video memory is close to its limit and fragmentation is high, NVMe read P95 latency is degraded, or power consumption and temperature are increased), at least one adjustment operation is triggered to adjust the model fragmentation strategy and allocate the corresponding storage region of the model fragment; if it is detected that the computing configuration strategy of the electronic device is switched (e.g., switching of computing units, switching of execution precision configuration of computing units), at least one adjustment operation is triggered to adjust the model fragmentation strategy and allocate the corresponding storage region of the model fragment.

[0100] Additionally, if the triggering operation only involves allocating the storage region corresponding to the model shard, the triggering condition may also include whether the access popularity of the model shard meets the access popularity threshold of the current storage region. It should be noted that different storage regions have different access popularity thresholds. When the access popularity of the model shard is higher or lower than the access popularity threshold of the current storage region, an adjustment operation for allocating the storage region corresponding to the model shard is triggered.

[0101] In this embodiment, by continuously monitoring whether the migration frequency of the target model fragment between different storage areas is greater than a preset frequency threshold, whether the access correlation between any two model fragments is greater than a first correlation threshold, whether the resource pressure status of the electronic device meets the preset pressure triggering conditions, and whether the computing configuration strategy of the electronic device has been switched, the corresponding adjustment operation can be triggered in a timely and accurate manner when the adjustment conditions are met, which effectively improves the inference efficiency of the model on the AIPC device.

[0102] To describe in detail the model sharding processing method provided in the embodiments of this disclosure, in one embodiment, the above-mentioned re-sharding operation may include a merging operation, which is used to merge at least two model shards into one model shard. Based on this, the steps of performing re-sharding operations on multiple model shards may include: Get the access correlation between every two model shards in multiple model shards; Based on access relevance, at least two first model shards are selected from multiple model shards; Merge at least two first-model fragments.

[0103] The access correlation is used to characterize the probability that two model slices are accessed simultaneously during model inference, and no specific limitations are made here.

[0104] In some embodiments, the access correlation degree of any two first model fragments among the at least two first model fragments obtained through screening is greater than a second correlation threshold. The second correlation threshold can be a pre-set judgment threshold based on actual conditions, used to determine whether the correlation degree between two model fragments reaches a level requiring merging; no specific limitation is made here.

[0105] Specifically, after obtaining the access correlation between any two model fragments in multiple model fragments, all model fragments can be traversed, and a set of fragments that satisfy the condition that the access correlation between any two model fragments is greater than the second correlation threshold can be selected. This set of fragments includes at least two first model fragments, and the at least two first model fragments are merged, so that the re-fragmented model fragments can be accurately obtained.

[0106] In this embodiment, by obtaining the access correlation degree between every two model shards in multiple model shards, and since this access correlation degree can characterize the probability that the two model shards are accessed simultaneously, at least two model shards with a high degree of access correlation can be selected from the multiple model shards and merged, thereby accurately obtaining the re-sharded model shards. In this way, merging model shards with a high probability of simultaneous access into a whole can effectively avoid the resource consumption caused by these model shards being stored in different storage areas.

[0107] Based on this, in one embodiment, the step of filtering at least two first model shards from multiple model shards based on access relevance may include: Based on the access correlation between every two model shards in multiple model shards, construct a correlation graph for multiple model shards; Select connected subgraphs with edge weights greater than a preset edge weight threshold from the association graph of multiple model pieces, and determine the model pieces corresponding to at least two nodes contained in the connected subgraph as at least two first model pieces.

[0108] In some embodiments, the association graph in this disclosure can represent a graph structure of the access association degree between multiple model fragments. Specifically, the association graph can include multiple nodes and multiple edges. Each node can represent a model fragment, and the edge weight of each edge can represent the access association degree of the model fragments corresponding to the two nodes connected by the edge. No specific limitation is made here.

[0109] In addition, the preset edge weight threshold can be a judgment threshold set in advance based on the actual situation, used to determine whether the access correlation between two model fragments should be merged. No specific limitation is made here.

[0110] Specifically, after obtaining the access correlation between every two model fragments in multiple model fragments, a complete correlation graph can be constructed using each model fragment as a node and the access correlation as the edge weight between the corresponding nodes. Subsequently, the correlation graph is traversed, and all edges with edge weights greater than a preset edge weight threshold are selected. The connected subgraph formed by these edges is then determined. This connected subgraph contains at least two nodes, and the model fragments corresponding to these nodes are the first model fragments with high access correlation between them, thus facilitating the subsequent merging of these at least two first model fragments.

[0111] In this embodiment, an association graph is constructed by the access correlation between every two model shards in multiple model shards, and connected subgraphs with edge weights greater than a preset edge weight threshold are selected. This enables efficient and accurate identification of a set of model shards with high correlation among multiple model shards, which facilitates the subsequent merging of model shards in the set. This makes the merging operation more accurate and effectively reduces the resource consumption caused by model shards being stored in different storage areas.

[0112] Furthermore, to avoid uneven storage of the merged model fragments, in one embodiment, after selecting at least two first model fragments from multiple model fragments, the model file processing method provided in this disclosure further includes: Get the access popularity of each first model shard; Select at least two second model fragments from at least two first model fragments whose access popularity is greater than a preset popularity threshold; Based on this, the above steps of merging at least two first model fragments may include: Merge at least two second model fragments.

[0113] The access popularity is used to characterize the frequency or number of times a model segment is accessed within a preset time window, and is used to characterize the activity level of the model segment during the inference process. The preset time window can be determined according to the actual situation, and no specific limitation is made here.

[0114] In addition, the aforementioned preset popularity threshold can be a judgment threshold set in advance based on the actual situation, used to determine whether the access activity of the model shards has reached the level that needs to be merged, and no specific limitation is made here.

[0115] Specifically, after selecting at least two first model fragments to be merged, the fragment popularity of each model fragment in the at least two model fragments can be obtained. Subsequently, at least two second model fragments with access popularity greater than a preset popularity threshold are selected from the at least two first model fragments, and finally the at least two second model fragments are merged, so as to accurately obtain the re-sharded model fragments.

[0116] In this embodiment, based on the access correlation between each pair of model shards to obtain at least two first model shards with high access correlation, the access popularity of each first model shard is obtained, and at least two second model shards with access popularity greater than a threshold are selected from these at least two first model shards for merging. This not only effectively avoids resource consumption caused by these model shards being stored in different storage areas, but also avoids merging model shards with high access correlation but low access popularity, thus preventing the merged model shards from having uneven internal popularity and requiring further splitting, effectively reducing resource consumption.

[0117] In another embodiment, the above-described re-sharding operation may include a splitting operation, which splits a model shard into at least two model shards. Based on this, the steps of performing a re-sharding operation on multiple model shards may specifically include: Obtain the popularity distribution information for each model fragment; the popularity distribution information represents the access popularity distribution within the model fragment. The third model segment is selected from multiple model segments based on the heat distribution information of the model segments that meets the preset heat distribution conditions. Split the third model into pieces.

[0118] The aforementioned heat distribution information is used to characterize the access heat distribution within a model fragment, that is, the difference in the frequency of access to different regions (i.e., different sub-model fragments) within a model fragment during the model inference process. This heat distribution information may include the access heat variance and the access heat standard deviation, which are not specifically limited here.

[0119] In addition, the preset heat distribution condition can be a judgment condition set in advance based on the actual situation, used to determine whether the heat distribution inside the model fragment is uneven to the point that a split operation needs to be performed. For example, the preset heat distribution condition can be that the access heat variance is greater than a preset variance threshold. The preset variance threshold can be determined according to the actual situation, and no specific limitation is made here.

[0120] Specifically, after obtaining the heat distribution information of each model segment, a third model segment whose heat distribution information meets the preset heat distribution conditions is selected from multiple model segments. That is, the model segment whose internal heat distribution unevenness meets the preset heat distribution conditions is selected as the third model segment to be split. Subsequently, the splitting operation is performed on the third model segment, so as to accurately obtain multiple model segments after resegmentation.

[0121] In this embodiment, by acquiring the heat distribution information of each model fragment, and based on this information, a third model fragment with uneven internal heat distribution is selected from multiple model fragments for splitting. This breaks down the originally poorly divided model fragments into multiple model fragments with more even heat distribution. The high-heat model fragments can then be allocated to the first storage area with higher access performance, while the low-heat model fragments can be allocated to the second storage area with lower access performance, thereby effectively improving the utilization efficiency of the first storage area. Simultaneously, the split sub-fragments can be further merged to accurately determine the allocation of multiple models after resharding.

[0122] In order to accurately and comprehensively describe the model file processing method provided in the embodiments of this disclosure, in one embodiment, such as Figure 3 As shown, the steps described above for determining the amount of scheduling resources required to schedule each model shard across different storage regions may specifically include: S310, obtain the operating mode of the electronic device and the scheduling evaluation function corresponding to each operating mode.

[0123] The operating modes of electronic devices may include, but are not limited to, high-performance mode, energy-saving mode, and balanced mode, etc., without specific limitations here.

[0124] In some embodiments, for multiple evaluation metrics in the scheduling evaluation function, different weight configuration combinations correspond to the evaluation metrics under different operating modes. The scheduling evaluation function is used to evaluate the resource overhead required to schedule model shards between different storage regions. This function may include multiple evaluation metrics, such as scheduling time latency, energy consumption, write amplification factor, etc., which are not specifically limited here. The weight configuration combinations corresponding to these evaluation metrics differ under different operating modes, which are not specifically limited here.

[0125] S320 uses the scheduling evaluation function corresponding to the current operating mode to determine the amount of scheduling resources required for each model fragment to be scheduled in different storage areas.

[0126] Specifically, after obtaining the current operating mode of the electronic device, the scheduling evaluation function corresponding to that operating mode (i.e., the evaluation function with a specific weight configuration combination) can be determined. Subsequently, for each model shard, the scheduling resource amount required for scheduling each model shard in different storage regions can be determined using the scheduling evaluation function corresponding to the current operating mode.

[0127] In this embodiment, the operating mode of the electronic device and the scheduling evaluation function corresponding to each operating mode are obtained, and the scheduling evaluation function under the current operating mode is used to determine the scheduling resource amount for each model fragment. In this way, the scheduling resource amount required for scheduling each model fragment in different storage areas can be accurately calculated, so that when selecting candidate model fragments in the subsequent screening, model fragments with lower scheduling resource amounts under the current operating mode can be preferentially selected for migration.

[0128] Therefore, in one embodiment, the aforementioned evaluation metrics include at least energy consumption metrics, latency metrics, and write amplification metrics. The energy consumption metrics can be used to characterize the energy consumption required to schedule model shards across different storage regions; these metrics may include indicators such as bandwidth. The latency metrics are used to characterize the time overhead required for scheduling operations. The write amplification metrics can be used to characterize the additional storage wear caused by write operations during the scheduling process, and are not specifically limited here.

[0129] It should be noted that, for the multiple evaluation metrics in the scheduling evaluation function, different weight configuration combinations correspond to different operating modes. Specifically: When the operating mode is the first operating mode, the first weight of the energy consumption index and the third weight of the write amplification index are both less than the second weight of the delay index. When the operating mode is the second operating mode, the first weight of the energy consumption index and the third weight of the write amplification index are both greater than the second weight of the delay index. When the operating mode is the third operating mode, the weight difference between the first weight corresponding to the energy consumption index and the second weight corresponding to the delay index is less than the preset weight difference.

[0130] In one example, if the above evaluation metrics include at least energy consumption metrics, latency metrics, and write amplification metrics, then the aforementioned scheduling evaluation function can satisfy the following formula: Cost=α Latency+β Energy+δ WAF(1) Where Cost is the resource scheduling amount, Latency is the latency metric, Energy is the energy consumption metric, and WAF is the write amplification metric. Correspondingly, α, β, and δ are the weights of the latency metric, energy consumption metric, and write amplification metric, respectively.

[0131] The following section describes in detail the different weight configuration combinations of the evaluation indicators under different operating modes in the embodiments of this disclosure, based on the above formula.

[0132] First, the aforementioned first operating mode can be a high-performance mode, which aims to minimize end-to-end latency and maximize throughput, with power consumption and storage lifetime being secondary considerations. Therefore, in the first operating mode, the first weight (β) corresponding to the energy consumption index and the third weight corresponding to the write amplification index (δ) are both less than the second weight (α) of the latency index, i.e., satisfying: β < α and δ < α.

[0133] Secondly, the aforementioned second operating mode can be a power-saving mode, which aims to minimize system energy consumption and extend battery life, allowing for a certain degree of performance degradation. Therefore, in the second operating mode, the first weight (β) corresponding to the energy consumption index and the third weight (δ) corresponding to the write amplification index are both greater than the second weight (α) of the latency index, i.e., satisfying β>α and δ>α.

[0134] Furthermore, the aforementioned third operating mode can be a balanced mode, which aims to achieve the best balance between performance and power consumption, providing a smooth user experience without excessive power consumption. Therefore, in the third operating mode, the weight difference between the first weight (β) corresponding to the energy consumption index and the second weight (α) corresponding to the latency index is less than a preset weight difference. This preset weight difference can be determined based on the actual situation. For example, if a relatively balanced weight is set for the latency index and the energy consumption index, then the preset weight difference can be set to 0, that is, satisfying: α≈β.

[0135] It should be noted that the specific values ​​of the first weight (β), second weight (α), and third weight (δ) mentioned above can be set according to the operating mode of the electronic device and the actual situation, which will not be elaborated on here. It should also be noted that the performance of the electronic device in the first operating mode is higher than that in the second and third operating modes, which will not be elaborated on here.

[0136] In this embodiment, by setting differentiated weight configuration combinations for multiple evaluation indicators in the scheduling evaluation function under different operating modes, the evaluation of scheduling resource quantity can be made more in line with the current operating mode of the electronic device. This allows for accurate calculation of the scheduling resource quantity required for each model fragment to be scheduled in different storage areas. Consequently, when selecting candidate model fragments in the subsequent screening, fragments with lower scheduling costs under the current operating mode can be prioritized for migration.

[0137] To provide a comprehensive and detailed description of the model file processing method provided in this disclosure, in one embodiment, the model file processing method further includes: Obtain the operating mode of the electronic device, and the storage parameters of the corresponding storage area; Configure the storage area of ​​the electronic device based on the storage parameters corresponding to the operating mode.

[0138] It should be noted that the storage parameters differ depending on the operating mode, but no specific limitations are made here.

[0139] In some embodiments, the storage parameters include at least one of the following: the number of model shards, the shard precision, and the access heat threshold. The number of shards is used to limit the maximum or target number of model shards that can be accommodated in the storage area. The shard precision is used to characterize the data precision (e.g., FP32, FP16, INT8, etc.) used when storing model shards. The access heat threshold is used to filter the model shards stored in the storage area. That is, only model shards with an access heat greater than the access heat threshold can be stored in the corresponding storage area. No specific limitation is made here.

[0140] Specifically, it is possible to obtain the operating mode of an electronic device and the storage parameters of the storage area corresponding to that operating mode, such as the number of storage model shards, shard precision, and access heat threshold. Based on the storage parameters corresponding to that operating mode, the storage area of ​​the electronic device can be configured, for example, by adjusting the number of shards that can be stored in the storage area, the shard precision, or adjusting the access heat threshold used to determine whether the model shard stores the area.

[0141] Since the storage parameters of the storage area corresponding to different operating modes of an electronic device can be different, in this embodiment of the disclosure, the operating modes of the electronic device may include a first operating mode (high-performance mode), a second operating mode (energy-saving mode), and a third operating mode (balanced mode). Based on this, the following describes in detail the different storage parameters of the storage area corresponding to different operating modes in this embodiment of the disclosure, in conjunction with the above three operating modes of the electronic device.

[0142] First, if the electronic device is operating in the first operating mode, the storage parameters for the first storage area (e.g., video memory) are configured as follows: Regarding the number of shards for storage model shards: Set a larger number of shards, which means that model shards that will be used or may be used can be prefetched into the first storage area. In this case, the prefetch window adopts a larger look-ahead window (e.g., predicting the sharding needs of the next 3-4 layers in advance) for proactive prefetching.

[0143] Regarding the sharding precision of storage model shards: prioritize storing model shards with high precision (such as FP16 / BF16) to avoid or reduce the latency overhead caused by dequantization. Only consider storing quantized low-precision data when the capacity of the first storage area is extremely tight.

[0144] For access popularity threshold: Set a lower access popularity threshold so that migration to the first storage area can be easily triggered.

[0145] Secondly, if the electronic device operates in the second operating mode, the storage parameters for the first storage area are configured as follows: Regarding the number of storage model shards: Set a smaller number of shards, and retain the model shards that are absolutely necessary for the current computation in the first storage area. In this case, a smaller look-ahead window can be used (e.g., only predict the next layer in advance) to avoid the power consumption caused by ineffective speculative prefetching.

[0146] Regarding the fragmentation accuracy of the storage model: prioritize storing the model fragments in the second storage area with a high compression ratio and low bit width (such as INT4) format, and then partially materialize the required fragments in the first storage area. Release them immediately after the calculation is completed to maximize the saving of video memory bandwidth and capacity.

[0147] Regarding the access popularity threshold: Set a high access popularity threshold so that the model shards reside in the second storage area most of the time, and are only migrated to the first storage area when the access popularity exceeds the threshold.

[0148] Furthermore, if the electronic device operates in the second operating mode, the storage parameters for the first storage area are configured as follows: Regarding the number of shards in the storage model: the number of shards is between the first operating mode and the second operating mode, taking into account both prefetch efficiency and storage usage. Correspondingly, the prefetch window selected in this mode can be between the first operating mode and the second operating mode, and can be dynamically adjusted.

[0149] Regarding the sharding precision of the storage model sharding: a hybrid high- and low-precision storage strategy is adopted, and a dynamic decision is made on whether to dequantize in advance or materialize on the spot based on the popularity and computational characteristics of the model sharding.

[0150] For access popularity threshold: Parameters such as access popularity threshold are dynamically fine-tuned based on recent telemetry data such as system load, application request frequency and device temperature to achieve a balance between performance and power consumption.

[0151] In this embodiment, by acquiring the operating mode of the electronic device and its corresponding storage parameters, and dynamically configuring the storage area based on these parameters, the configuration of the storage area can be matched with the current operating mode of the electronic device. This effectively achieves precise adaptation between storage resources and the operating mode.

[0152] In order to fully and thoroughly describe the model file processing method provided in the embodiments of this disclosure, it should also be noted that in the embodiments of this disclosure, the processor allocation strategy and I / O strategy of the electronic device will also be different in different operating modes. Based on this, the specific configuration of processor allocation and I / O strategy in different operating modes will be described in detail, taking the operating modes of the electronic device including the first operating mode (high performance mode), the second operating mode (energy saving mode) and the third operating mode (balanced mode) as examples.

[0153] If the electronic device operates in the first operating mode, the specific configurations for processor allocation and I / O strategy in this embodiment are as follows: Processor allocation: The vast majority of computational tasks (especially computationally intensive tasks such as General Matrix Multiplication (GEMM)) will be forcibly allocated to the most powerful GPU. The NPU will only be considered when certain operators are not supported by the GPU, or when the time cost of offloading a model slice (such as INT8 quantized FFN) to the NPU and returning the result is significantly lower than the time cost of queuing execution on the GPU, as calculated precisely through the cost model. The CPU is mainly used for scheduling and I / O operations, or as a last resort option.

[0154] I / O strategy: Allows for higher NVMe SSD read bandwidth usage with fewer restrictions on I / O throttling to ensure that data can reach the first storage area as quickly as possible.

[0155] If the electronic device operates in the second operating mode, the specific configurations for processor allocation and I / O strategy in this embodiment are as follows: Processor allocation: For operators supported by the NPU (such as INT8 / INT4 integer operations), the scheduler will prioritize allocating them to the NPU because the NPU typically has higher energy efficiency. The GPU is considered a high-power unit and will only be awakened for computation if the NPU does not support the current operator or its performance is severely insufficient. In addition to handling scheduling tasks, the CPU may also handle more edge operators and dequantization tasks to keep the GPU and NPU in a sleep state as much as possible.

[0156] I / O strategy: Strictly limit SSD activity and tend to merge more read requests and perform I / O operations when the system is idle. At the same time, reduce the "thrashing" between the first sub-storage area (such as memory) and the second sub-storage area (such as SSD) included in the second storage area by a longer cooling period, thereby reducing SSD write amplification.

[0157] If the electronic device operates in the third operating mode, the specific configurations for processor allocation and I / O strategy in this embodiment are as follows: Processor allocation: Using a cost model, the combined cost (including latency and power consumption) of each model slice is evaluated in real time for execution on the GPU and NPU, and then the scheme with the lowest combined cost is selected. Furthermore, the GPU and NPU may work in parallel; for example, the GPU processes the self-attention layer while the NPU processes the feedforward network layer, achieving true heterogeneous collaborative computing.

[0158] I / O strategy: While ensuring no significant lag occurs, I / O is moderately throttled and merged to balance performance and power consumption.

[0159] Based on the same inventive concept, this disclosure also provides an electronic device, which can be specifically combined with... Figure 4 The electronic devices provided in the embodiments of this disclosure will be described in detail.

[0160] Figure 4 This is a schematic diagram of the composition structure of an electronic device provided in an embodiment of this disclosure. Figure 1 .

[0161] like Figure 4 As shown, the electronic device 400 may include at least two memories 41, multiple processors 42, and a scheduler 43, wherein the scheduler 43 is connected to the at least two memories 41 and the multiple processors 42 respectively, without being specifically limited here.

[0162] The at least two memories 41 may include different types of storage media to provide storage regions with different access performance. Multiple processors 42 may include different types of computing units such as GPUs, NPUs, and CPUs to execute model inference tasks. The scheduler 43 can be used to manage the partitioning of model slices, storage allocation, and dynamic adjustment of both; no specific limitations are specified here.

[0163] In this way, scheduler 43 can obtain the model file of the target model and divide the model file into multiple model fragments according to the model fragmentation strategy corresponding to the model file. Subsequently, scheduler 43 can determine the storage area corresponding to each model fragment in the electronic device and store the model fragment in the corresponding storage area. During model operation, scheduler 43 continuously monitors whether the adjustment conditions are met. If they are met, at least one adjustment operation is performed, including adjusting the model fragmentation strategy and allocating the storage area corresponding to the model fragment.

[0164] It should be noted that the storage area of ​​the electronic device includes storage areas corresponding to at least two memory 41. For example, the first memory (such as high-speed memory) corresponds to the first storage area, and the second memory (such as solid-state drive) corresponds to the second storage area. The access performance of the model fragments stored in the first storage area is higher than that of the model fragments stored in the second storage area. No specific limitation is made here.

[0165] By segmenting and storing model files differently according to the hardware configuration of electronic devices, the efficient deployment and inference performance of large language models on AIPCs are effectively achieved. At the same time, by continuously monitoring and adjusting conditions and dynamically optimizing model segmentation strategies and storage allocation, the system can effectively adapt to changes in the resources or fluctuations in the operating load of AIPC devices, avoiding performance degradation or failures caused by the initial configuration being fixed. Thus, while ensuring model inference performance, the deployment and inference efficiency of large language models on artificial intelligence personal computers (AIPCs) is significantly improved.

[0166] In one embodiment, the scheduler 43 described above can be specifically used for: The computational granularity of processor data processing and the access granularity of memory data read / write are obtained, and the model sharding strategy corresponding to the model file is determined based on the computational granularity and access granularity. The hardware parameters include the computational granularity of processor data processing and the access granularity of memory data read / write.

[0167] In one embodiment, the scheduler 43 described above can also be used for: The model file of the target model is parsed to obtain the number of model layers and the weight matrix corresponding to each model layer. After dividing the model file into multiple model fragments according to the model fragmentation strategy, the fragmentation information of each model fragment is determined. The fragmentation information of the model fragment includes the model layer number of the model fragment in the target model, the weight type of the weight matrix to which the model fragment belongs, and the fragmentation index of the model fragment in the weight matrix.

[0168] In one embodiment, the scheduler 43 described above can be specifically used for: If the adjustment conditions are met, the model sharding strategy is adjusted, and multiple model shards are re-sharded based on the adjusted model sharding strategy to obtain multiple re-sharded model shards. The amount of scheduling resources required for each model shard to be scheduled between different storage areas is determined. Then, candidate model shards with scheduling resource amounts less than a preset value are selected from multiple model shards. Finally, the candidate model shards stored in the second storage area can be scheduled to the first storage area.

[0169] In one embodiment, the resharding operation includes a merging operation, and the scheduler 43 described above can be specifically used for: It can obtain the access correlation degree between every two model shards in multiple model shards, and based on the access correlation degree, filter out at least two first model shards from the multiple model shards, wherein the access correlation degree between any two of the at least two first model shards is greater than a second correlation degree threshold. In this way, at least two first model shards can be merged.

[0170] In one embodiment, if the resharding operation includes a splitting operation, the scheduler 43 described above can be specifically used for: It can acquire the popularity distribution information of each model fragment, and select a third model fragment from multiple model fragments whose popularity distribution information meets the preset popularity distribution conditions. Then, the third model fragment can be split. The popularity distribution information represents the access popularity distribution within the model fragment.

[0171] In one embodiment, the scheduler 43 described above can be specifically used for: It can obtain the operating mode of electronic devices and the scheduling evaluation function corresponding to each operating mode, and use the scheduling evaluation function corresponding to the current operating mode to determine the amount of scheduling resources required for each model fragment to be scheduled in different storage areas.

[0172] Specifically, for the multiple evaluation metrics in the scheduling evaluation function, different weight configuration combinations correspond to the evaluation metrics under different operating modes. In one embodiment, the scheduler 43 described above can also be used for: It can obtain the operating mode of an electronic device and the storage parameters of the storage area corresponding to the operating mode. Then, it can configure the storage area of ​​the electronic device based on the storage parameters corresponding to the operating mode.

[0173] The storage parameters include at least one of the following: the number of storage model shards, shard precision, and access heat threshold. The access heat threshold is used to filter the model shards stored in the storage area. The storage parameters are different for different operating modes.

[0174] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this disclosure can be achieved, and this is not limited herein.

[0175] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this disclosure, "a plurality of" means two or more, unless otherwise explicitly specified.

[0176] The above description is merely a specific embodiment of this disclosure, but the scope of protection of this disclosure is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this disclosure should be included within the scope of protection of this disclosure. Therefore, the scope of protection of this disclosure should be determined by the scope of the claims.

Claims

1. A method for processing model files, the method comprising: Obtain the hardware parameters of electronic devices; Based on the hardware parameters of the electronic device, determine the model sharding strategy corresponding to the model file of the target model, and divide the model file into multiple model shards according to the model sharding strategy; The storage region corresponding to each model fragment in the electronic device is determined, and the model fragment is stored in the storage region. The storage region includes a first storage region and a second storage region. The access performance of the model fragment stored in the first storage region is higher than that of the model fragment stored in the second storage region. If the adjustment conditions are met, at least one adjustment operation is performed, including adjusting the model sharding strategy and allocating the storage area corresponding to the model shard.

2. The method according to claim 1, wherein determining the model sharding strategy corresponding to the model file of the target model based on the hardware parameters of the electronic device includes: The model sharding strategy corresponding to the model file is determined based on the computational granularity of the processor processing data and the access granularity of the memory reading and writing data; wherein, the hardware parameters include the computational granularity of the processor processing data and the access granularity of the memory reading and writing data.

3. The method according to claim 1, further comprising: The model file of the target model is parsed to obtain the number of model layers of the target model and the weight matrix corresponding to each model layer; After dividing the model file into multiple model fragments according to the model fragmentation strategy corresponding to the model file, the method includes: Determine the fragmentation information for each model fragment, which includes the model layer number in the target model, the weight type of the weight matrix to which the model fragment belongs, and the fragmentation number of the model fragment in the weight matrix.

4. The method according to claim 1, wherein performing at least one adjustment operation if the adjustment conditions are met includes: If the adjustment conditions are met, the model sharding strategy is adjusted, and a re-sharding operation is performed on the multiple model shards based on the adjusted model sharding strategy to obtain multiple re-sharded model shards. Determine the amount of scheduling resources required to schedule each model fragment across different storage regions; From the plurality of model fragments, candidate model fragments whose scheduling resource amount is less than a preset value are selected; The candidate models stored in the second storage area are sharded and scheduled to the first storage area.

5. The method according to claim 4, wherein the adjustment conditions include at least one of the following: The migration frequency of the target model fragment between different storage areas in the multiple model fragments is greater than a preset frequency threshold. The access correlation between any two model shards in the plurality of model shards is greater than the first correlation threshold. The resource pressure state of the electronic device meets the preset pressure triggering conditions; The computing configuration strategy of the electronic device is switched, and the computing configuration strategy includes at least one of the following: switching of computing units and switching of execution precision configuration of computing units.

6. The method according to claim 4 or 5, wherein the re-fragmentation operation includes a merging operation, the merging operation being used to merge at least two model fragments into one model fragment; the re-fragmentation operation on the plurality of model fragments includes: Obtain the access correlation between every two model shards in the multiple model shards; Based on the access correlation, at least two first model fragments are selected from the plurality of model fragments, and the access correlation of any two first model fragments is greater than the second correlation threshold. Merge the at least two first model fragments.

7. The method according to claim 4 or 5, wherein the re-sharding operation includes a splitting operation, the splitting operation being used to split one of the model shards into at least two model shards; The re-sharding operation on the multiple model shards includes: Obtain the popularity distribution information of each model segment, wherein the popularity distribution information represents the access popularity distribution within the model segment; The third model segment is selected from the multiple model segments whose heat distribution information meets the preset heat distribution conditions; Split the third model into pieces.

8. The method according to claim 4 or 5, wherein determining the amount of scheduling resources required for scheduling each model fragment across different storage regions includes: Obtain the operating modes of the electronic devices and the scheduling evaluation function corresponding to each operating mode; For the multiple evaluation metrics in the scheduling evaluation function, different weight configuration combinations are corresponding to the evaluation metrics under different operating modes; Using the scheduling evaluation function corresponding to the current operating mode, determine the amount of scheduling resources required for each model fragment to be scheduled in different storage regions.

9. The method according to claim 8, wherein the plurality of evaluation indicators include at least energy consumption indicators, latency indicators, and write amplification indicators; When the operating mode is the first operating mode, the first weight of the energy consumption index and the third weight of the write amplification index are both less than the second weight of the delay index. When the operating mode is the second operating mode, the first weight corresponding to the energy consumption index and the third weight corresponding to the write amplification index are both greater than the second weight of the delay index. When the operating mode is the third operating mode, the weight difference between the first weight corresponding to the energy consumption index and the second weight corresponding to the delay index is less than the preset weight difference, and the performance of the electronic device in the first operating mode is higher than that in the second and third operating modes.

10. An electronic device, comprising: At least two memory locations; Multiple processors; The scheduler is used to retrieve the hardware parameters of electronic devices. Based on the hardware parameters of the electronic device, determine the model sharding strategy corresponding to the model file of the target model, and divide the model file into multiple model shards according to the model sharding strategy; The storage region corresponding to each model fragment in the electronic device is determined, and the model fragment is stored in the storage region. The storage region includes a first storage region and a second storage region. The access performance of the model fragment stored in the first storage region is higher than that of the model fragment stored in the second storage region. If the adjustment conditions are met, at least one adjustment operation is performed. The adjustment operation includes adjusting the model fragmentation strategy and allocating the storage region corresponding to the model fragment.

Images