A method and apparatus for loading model weight parameters
By pre-splitting and parallel loading the base slices to multiple graphics processors, the problem of low loading efficiency in existing technologies is solved, achieving efficient loading and improved compatibility when hardware configuration changes.
Patent Information
- Application Number
- CN202611307878.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-26
- Publication Date
- 2026-11-10
AI Technical Summary
Existing technologies are inefficient when loading weight parameters of inference models, especially when the hardware configuration of the graphics processor changes, requiring frequent splitting of weight parameters, which leads to low loading efficiency.
By pre-determining the target hardware configuration and corresponding reorganization strategy, the pre-segmented basic slices are loaded in parallel onto multiple graphics processors, avoiding the segmentation process during each loading and improving efficiency by using a parallel loading queue.
It improves the efficiency of loading weight parameters, enhances the compatibility of basic slices to adapt to various hardware configurations, reduces the number of slices, and improves loading efficiency and compatibility.
Smart Images

Figure CN123071128A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of model loading technology, and in particular to a method and apparatus for loading model weight parameters. Background Technology
[0002] When electronic devices process data using inference models, the weight parameters of the inference models need to be loaded into the device's graphics processor so that relevant calculations can be performed using the loaded weight parameters.
[0003] Inference models typically contain a large number of weight parameters. To address this, some related technologies divide the weight parameters into several slices and load them onto multiple graphics processors in parallel to improve loading efficiency. However, this loading method is still time-consuming. Summary of the Invention
[0004] Therefore, this application discloses the following technical solution:
[0005] The first aspect of this application provides a method for loading model weight parameters, including:
[0006] Determine the target hardware configuration, which represents multiple graphics processors used for inference in the processing system;
[0007] Read the target reorganization strategy corresponding to the target hardware configuration in the base slice index and multiple reorganization strategies. The target reorganization strategy includes at least the correspondence between multiple base slices and multiple loading queues for parallel loading. The multiple base slices are obtained by dividing by the weight parameters of the inference model.
[0008] Based on the target reorganization strategy and the base slice index, the base slices corresponding to the multiple loading queues are loaded in parallel to the multiple graphics processors, so that the multiple graphics processors can perform inference based on the weight parameters.
[0009] A second aspect of this application provides an electronic device, including a memory, a central processing unit, and a plurality of graphics processors;
[0010] The memory is used to store computer programs;
[0011] The graphics processor is used to perform inference tasks based on weight parameters;
[0012] The central processing unit is used to execute the computer program to perform:
[0013] Determine the target hardware configuration, which represents multiple graphics processors used for inference in the processing system;
[0014] Read the target reorganization strategy corresponding to the target hardware configuration in the base slice index and multiple reorganization strategies. The target reorganization strategy includes at least the correspondence between multiple base slices and multiple loading queues for parallel loading. The multiple base slices are obtained by dividing by the weight parameters of the inference model.
[0015] Based on the target reorganization strategy and the base slice index, the base slices corresponding to the multiple loading queues are loaded in parallel to the multiple graphics processors, so that the multiple graphics processors can perform inference based on the weight parameters. Attached Figure Description
[0016] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.
[0017] Figure 1 This is a flowchart illustrating a method for loading model weight parameters provided in an embodiment of this application;
[0018] Figure 2 This is a schematic diagram of multiple basic slices provided in an embodiment of this application;
[0019] Figure 3 This is a schematic diagram of a basic slice cache data structure and a stored basic slice provided in an embodiment of this application;
[0020] Figure 4 This is a schematic diagram illustrating the loading of continuous weight parameters onto a graphics processor, as provided in an embodiment of this application.
[0021] Figure 5 This is a schematic diagram illustrating the loading of discontinuous weight parameters onto a graphics processor, as provided in an embodiment of this application.
[0022] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation
[0023] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0024] To perform inference on the input information, electronic devices need to first start the inference model. In related technologies, the process of starting the inference model may include stages such as process creation, model structure initialization, loading weight parameters, runtime memory initialization, and service readiness. The stage of loading weight parameters requires loading the weight parameters contained in the inference model into the cache (also known as video memory) of each graphics processor used to run the inference model. To shorten the time required to start the inference model, this application provides a method for loading model weight parameters. This method is executed during the weight parameter loading stage, aiming to complete the loading of weight parameters more quickly.
[0025] See Figure 1 The method for loading model weight parameters provided in this application embodiment may include the following steps.
[0026] S101, Determine the target hardware configuration, which represents the multiple graphics processors used for inference in the processing system.
[0027] The processing system can refer to any electronic device that has multiple graphics processing units (GPUs) and is capable of running inference models based on these GPUs. For example, the processing system can be a server host or server cluster with multiple GPUs. The execution entity of the loading method in any embodiment of this application can be the central processing unit (CPU) within the processing system. The inference model can be any model with inference capabilities in the field of artificial intelligence, such as a large language model (LLM).
[0028] The target hardware configuration can represent the number of graphics processors (GPUs) currently used by the processing system to run the inference model. This number can be less than or equal to the total number of GPUs in the processing system. For example, if the processing system has a total of 8 GPUs, the target hardware configuration could be TP4, indicating that 4 GPUs are used to run the inference model, or it could be TP2, indicating that 2 GPUs are used to run the inference model.
[0029] The target hardware configuration is related to the current hardware usage of the processing system and / or the user's usage requirements.
[0030] S102, read the base slice index and the target reorganization strategy corresponding to the target hardware configuration in multiple reorganization strategies. The target reorganization strategy includes at least the correspondence between multiple base slices and multiple loading queues for parallel loading. The multiple base slices are obtained by dividing the weight parameters of the inference model.
[0031] The total number of base slices can be the same as the total number of graphics processors in the processing system. For example, if the processing system has 8 graphics processors, the weight parameters of the inference model can be divided into 8 base slices. Each base slice contains a portion of the weight parameters of the inference model.
[0032] The target replay strategy is one of several pre-generated replay strategies (also known as Replay Plans) that corresponds to the target hardware configuration. For each hardware configuration supported by the processing system, a corresponding replay strategy can be pre-generated based on that hardware configuration, so that the corresponding target replay strategy can be read in S102 according to the target hardware configuration.
[0033] The target reorganization strategy includes multiple load queues (also known as ranks) for parallel loading, as well as the correspondence between each load queue and multiple base slices.
[0034] The number of load queues included in the target reorganization strategy can be consistent with the number of graphics processors in its corresponding hardware configuration, so that the base slice of each load queue can be loaded into the corresponding graphics processor.
[0035] As an example, the target reassembly strategy for TP2 may include load queue 0 (rank0) and load queue 1 (rank1), with the following correspondence: rank0 includes base0, base1, base2, and base3; rank1 includes base4, base5, base6, and base7. Here, base0 to base7 represent the eight pre-splitting base slices. The target reassembly strategy for TP4 may include four load queues, denoted as load queue 0 (rank0) to load queue 3 (rank3), with the following correspondence: rank0 includes base0 and base1, rank1 includes base2 and base3, rank2 includes base4 and base5, and rank3 includes base6 and base7.
[0036] The base slice index is an index file that is saved when the base slices are obtained. It contains index information of each base slice, including the source file location (i.e., storage address), data volume and data type of each base slice.
[0037] S103, based on the target reorganization strategy and the base slice index, load the base slices corresponding to multiple loading queues into multiple graphics processors in parallel, so that multiple graphics processors can perform inference based on weight parameters.
[0038] In step S103, for each loading queue, a corresponding loading process can be created. The loading process determines which basic slices correspond to (including) the loading queue based on the target reorganization strategy. For example, it determines that rank0 in the target reorganization strategy of TP4 corresponds to base0 and base1. Based on the basic slice index, these basic slices are located in the memory, and the weight parameters contained in these basic slices are loaded into the video memory of the graphics processor corresponding to the loading queue.
[0039] The loading process described above is executed in parallel across multiple loading queues, meaning that the loading processes corresponding to multiple loading queues run simultaneously. This allows the base slices corresponding to these loading queues to be loaded into their respective graphics processors at the same time, thereby reducing the time required to load the weight parameters. For example, if there are two loading queues, each containing approximately 50% of the weight parameters in the inference model, by loading the weight parameters from both loading queues simultaneously onto two different graphics processors, the loading time can be reduced to 50% of the time required to load all weight parameters sequentially.
[0040] The specific graphics processor (GPU) to which the base slice of each load queue is loaded can be arbitrarily determined, or it can be determined based on at least one of the following: GPU performance, GPU load, and the amount of data in the base slice corresponding to the load queue. For example, a load queue with a larger amount of data in its corresponding base slice can be assigned to a GPU with a lighter load, or to a GPU with better performance.
[0041] During parallel loading, once all the base slices corresponding to each loading queue have been loaded, the loading queue is marked as having completed weight loading. When each loading queue in the target reorganization strategy is marked as having completed weight loading, it can be determined that all weight parameters of the inference model have been loaded, and the subsequent stage of starting the inference model can proceed. After the start-up is complete, each graphics processor calculates the input information based on the loaded weight parameters, thereby realizing the inference capability of the inference model.
[0042] In related technologies, the hardware configuration of electronic devices that deploy inference models often changes. For example, a new graphics processor may be added, or the original graphics processor may be unable to be used for inference due to failure or excessive load. When the hardware configuration changes, the weight parameters of the inference model need to be split according to the new hardware configuration to obtain a slice corresponding to the new hardware configuration. That is, the weight parameters of the inference model need to be split every time it is loaded, which leads to low loading efficiency.
[0043] Compared with related technologies, the beneficial effects of this embodiment are as follows: On the one hand, this embodiment does not need to divide the weight parameters each time it is loaded. It only needs to determine the current target hardware configuration, and then load the pre-divided basic slices into multiple graphics processors in parallel according to the corresponding target reorganization strategy, which saves the time of dividing the weight parameters into basic slices and further improves the efficiency of loading weight parameters.
[0044] On the other hand, by setting hardware configurations and corresponding reorganization strategies, this solution can adapt multiple pre-segmented fixed basic slices to any target hardware configuration without having to specifically segment multiple slices for different hardware configurations. This improves loading efficiency and enhances the compatibility of multiple basic slices, enabling multiple fixed basic slices to be preloaded under various hardware configurations.
[0045] Optionally, multiple basic slices are obtained by dividing the inference model's weight parameters based on the processing system's reference hardware configuration and reference segmentation method. The reference hardware configuration represents the graphics processor contained in the processing system, and the reference segmentation method is related to the inference model's architecture.
[0046] The advantages of obtaining base slices in this way are as follows: First, by dividing the base slices according to the reference hardware configuration of the processing system, the obtained base slices can be reused as much as possible in various usage scenarios of the processing system, improving the compatibility of multiple base slices and avoiding repeated division of the weight parameters of the inference model due to the inability to reuse base slices; Second, by determining the reference division method according to the model architecture of the inference model, the obtained base slices can be matched with the inherent model architecture, reducing the number of reordering processes when loading base slices, which is conducive to further improving loading efficiency.
[0047] The process of splitting into multiple basic slices can be performed once before the aforementioned loading method is executed. Then, the multiple basic slices obtained after the execution can be saved and reused each time the aforementioned loading method is executed, without having to split them once every time the loading is performed.
[0048] The process of dividing the material into multiple basic slices can be performed by the aforementioned processing system or by other electronic devices different from the processing system.
[0049] The process of dividing the material into multiple base slices may include the following steps.
[0050] Step 1: Read the model configuration file and weight index file of the inference model. The model configuration file includes one or more pieces of information such as model parameter names, parameter shapes, and model structure. The weight index file includes at least the path to the weight parameter file and the offset information within the weight parameter file.
[0051] The parameter shape represents the geometric dimensions of the tensor matrix composed of the weight parameters in the inference model. For example, it might represent a tensor matrix consisting of ten 512-row, 512-column two-dimensional matrices. The model parameter name refers to the name (or other identifier) of the tensor matrix composed of the weight parameters in the inference model. The model structure indicates which tensor matrices the inference model includes and the relationships between these tensor matrices; for example, which tensor matrices belong to the first layer of the inference model and which belong to the second layer.
[0052] The weight parameter file path is the storage path of the weight parameter file, which contains the weight parameters of the inference model. Subsequent splitting involves splitting the data contained in the weight parameter file, i.e., the weight parameters of the inference model. The offset information within the weight parameter file indicates the location of each tensor matrix within the weight parameter file.
[0053] Step 2: Determine the reference segmentation method and the reference hardware configuration of the processing system. The reference hardware configuration can represent the total number of graphics processors included in the processing system.
[0054] For example, if the processing system has a total of 8 graphics processors, then the reference hardware configuration is 8. The reference hardware configuration reflects the maximum scale of parallelism supported by the processing system, that is, the maximum number of loading processes that can run in parallel.
[0055] Optionally, the reference segmentation method is one of several segmentation methods determined based on the model architecture;
[0056] Multiple segmentation methods are available, including segmentation by row, segmentation by column, segmentation by fusion parameters, segmentation by vocabulary, segmentation by layer, segmentation by expert weight, and any combination of replication parameters.
[0057] Row-based partitioning refers to dividing the tensor matrix in the inference model into multiple basic slices by rows, while column-based partitioning refers to dividing the tensor matrix in the inference model into multiple basic slices by rows. For example, the weight parameters in rows 1 to 10 can be partitioned into one basic slice, or columns 11 to 20 can be partitioned into another basic slice. Both row-based and column-based partitioning belong to tensor parallel partitioning.
[0058] Combination Figure 2 For example, suppose a 16-row, 10-column tensor matrix is divided into 8 basic slices (depending on the reference hardware configuration). The 8 basic slices would then include rows 0 and 1 (row0 and row1), rows 2 and 3 (row2 and row3) ... rows 14 and 15 (row14 and row15).
[0059] Lexicon segmentation refers to dividing the vocabulary of an inference model into multiple basic slices. The vocabulary is equivalent to a data table in the inference model, used to map each word (token) in the information input to the inference model into a corresponding vector representation.
[0060] Layer-based partitioning (i.e. pipeline parallel partitioning) refers to partitioning the weight parameters of each layer into basic slices according to the hierarchical structure of the inference model. For example, the weight parameters contained in layers 1 and 2 are used as one basic slice, and the weight parameters contained in layers 3 and 4 are used as another basic slice.
[0061] Expert weight-based segmentation (i.e., expert parallel segmentation) refers to dividing the weight parameters of each expert into base slices for a multi-expert inference model that contains multiple expert models (sub-models). For example, the weight parameters of expert 1 are segmented as base0, and the weight parameters of expert 2 are segmented as base1.
[0062] Fusion parameter splitting (also known as hybrid parallel strategy) refers to the combined use of any one or more splitting methods; for example, when layer-by-layer splitting and row-by-row splitting are combined, the weight parameters contained in the first and second layers can be split into a basic slice, and the weight parameters of the first to tenth rows of the tensor matrix in the third and fourth layers can be split into a basic slice.
[0063] Duplicated parameters refer to the practice of retaining multiple copies of certain parameters across multiple base slices when dividing the base slice. In other words, the same weight parameter is included in both base1 and base2. The advantage of this is that these duplicated weight parameters may be used during inference on multiple graphics processors. Retaining only one copy could require cross-GPU communication to access these weight parameters during inference. By retaining multiple copies, communication resources and time are saved, thus improving communication efficiency.
[0064] The reference segmentation method can be configured by the user or determined based on the model architecture of the inference model and / or the system architecture of the processing system. For example, for multi-expert inference models, segmentation by expert weight can be used; for processing systems with multiple nodes, segmentation by row or column can be used; or a fusion parameter segmentation combining row segmentation and layer segmentation can be used.
[0065] Step 3: Divide the weight parameters contained in the weight parameter file of the inference model into multiple corresponding basic slices according to the reference segmentation method, and determine the slice metadata of each basic slice. The number of basic slices (or the granularity of segmentation) matches the reference hardware configuration; that is, the number of basic slices is determined by the number of graphics processors in the processing system as indicated by the reference hardware configuration.
[0066] For example, if the reference hardware configuration is 8, the reference segmentation method is layer-by-layer segmentation, and the inference model contains 16 layers, then the weight parameters of every two layers can be segmented into a base slice, resulting in 8 base slices, denoted as base0 to base7. The segmentation process can be found in the relevant techniques for segmenting model weight parameters in the field of artificial intelligence technology, and will not be elaborated upon here.
[0067] A base slice can include multiple weight parameters. For example, a base slice can include all the weight parameters of the first layer of the inference model, the weight parameters of rows 1 to 10 in the tensor matrix of the second layer, and the weight parameters of columns 1, 3, 5 and 7 in the tensor matrix of the third layer, etc.
[0068] The metadata of a base slice may include at least one of the following: source file location, number, slice length, slice type, and associated parameter identifier. The source file location indicates the position of the weight parameters contained in the base slice within the source file, so that the weight parameters to be loaded can be found according to the source file location during loading. The source file location can be represented by a data offset index. The number is the storage address of the file to which the base slice belongs. The number is a pre-assigned identifier used to distinguish each base slice, such as the aforementioned base0 and base1. The slice length indicates the amount of data of the weight parameters contained in the base slice, for example, indicating that the base slice contains 100 megabytes (MB) of weight parameters.
[0069] The attribute identifier indicates the position of the weight parameters contained within the inference model. For example, it can represent one or more of the expert model, layer, tensor matrix, row, and column to which the weight parameters belong in the inference model. For instance, the attribute identifier of a base slice could be rows 1 to 20 of tensor matrix 1 in layer 5, indicating that the base slice contains the weight parameters in rows 1 to 20 of tensor matrix 1 in layer 5.
[0070] Loading a base slice into the graphics processor (GPU) is equivalent to writing the weight parameters contained in the base slice into the GPU's video memory. Depending on the writing method, the weight parameters contained in a base slice can be divided into two categories: continuous and non-contiguous. The slice type indicates which of the weight parameters contained in the base slice are continuous and which are non-contiguous.
[0071] For a set of weight parameters, if they can be written continuously from the base slice to the graphics processor's video memory during loading, then the weight parameters are considered continuous weight parameters. For example, if the base slice contains the weight parameters of rows 1 to 20 of tensor matrix 1, these weight parameters can be directly written from the base slice to the video memory during loading, so the weight parameters of rows 1 to 20 of tensor matrix 1 are considered continuous weight parameters.
[0072] For a given weight parameter, if loading requires concatenating and combining it with weight parameters from other base slices before writing it into the graphics processor's video memory, then that weight parameter is considered a non-contiguous weight parameter. For example, if a base slice includes weight parameters in columns 1, 3, 5, and 7 of a tensor matrix, during loading, these weight parameters need to be concatenated with weight parameters in columns 2, 4, 6, and 8 of another base slice in an interleaved column-by-column fashion to form a complete tensor matrix before being written into video memory. Therefore, the weight parameters in columns 1, 3, 5, and 7 of the tensor matrix are considered non-contiguous weight parameters.
[0073] Combination Figure 2 For example, when splitting by column, consecutive weight parameters in multiple columns within a base slice in the tensor matrix are considered consecutive weight parameters. These weight parameters can be directly written to the graphics processor during loading. Multiple weight parameters are scattered across multiple base slices. The non-consecutive weight parameters in each base slice are considered non-consecutive weight parameters. These weight parameters need to be concatenated and combined before being written to the graphics processor during loading.
[0074] To determine the slice type, after obtaining the basic slice in step 3, we can determine how the weight parameters contained in each basic slice are written during subsequent loading, and thus determine the corresponding slice type based on the writing method.
[0075] like Figure 2 For example, continuous weight parameters can be labeled C, and non-continuous weight parameters can be labeled S.
[0076] The slice type in the slice metadata can be stored in the form of a continuous data table (denoted as list C) and a non-continuous data table (denoted as list S). List C records which weight parameters are continuous, and list S records which weight parameters are non-continuous. Figure 3 shown in (2).
[0077] Furthermore, such as Figure 3 As shown in (3), the C list can also record: the tensor number of the tensor matrix to which the continuous weight parameters belong, the offset of the weight parameters (indicating the position of the continuous weight parameters in the tensor matrix), the shape of the weight parameters (indicating the number of rows and columns), the data type (dtype, indicating integer, floating-point, etc.) and the slice number of the underlying slice to which they belong.
[0078] The S list can also record: the tensor number of the tensor matrix to which the non-contiguous weight parameters belong, the stride (indicating the interval between non-contiguous weight parameters, such as an interval of 1 column or 3 columns), the shape, the group number (indicating which group the weight parameters belong to among the multiple groups of weight parameters divided from the tensor matrix), and the slice number of the underlying slice to which they belong.
[0079] Further optionally, the slice metadata may also include rearrangement parameters corresponding to non-contiguous weight parameters in the base slice. These rearrangement parameters indicate the way the non-contiguous weight parameters are reorganized. For example, in the aforementioned example, the rearrangement parameters may indicate column-wise interleaving.
[0080] Step 4: Write the base slice to the pre-allocated base slice cache to complete the persistent storage of the base slice.
[0081] In step 4, a corresponding number of basic slice caches can be pre-allocated in the processing system's memory based on the number of basic slices. For example, if there are 8 basic slices, 8 basic slice caches can be allocated. Each basic slice cache can be further divided into three partitions: metadata area, continuous data area, and non-contiguous data area. The slice metadata of the basic slice, the continuous weight parameters it contains, and the non-contiguous weight parameters are stored in the three partitions of the basic slice cache.
[0082] In other words, the base slices are stored in the base slice cache, which includes a metadata area, a contiguous data area, and a non-contiguous data area, such as... Figure 3 shown in (1);
[0083] The metadata area stores the slice metadata of the base slice. The slice metadata includes at least one of the following: the source file location of the base slice, the number (i.e., the slice number), the slice length, the slice type, and the associated parameter identifier.
[0084] The continuous data area stores the continuous weight parameters of the base slice, while the non-continuous data area stores the non-continuous weight parameters of the base slice.
[0085] For example Figure 3 In (2), the continuous data area stores m consecutive weight parameters from C1 to Cm in the basic slice, and the non-continuous data area stores k non-continuous weight parameters from S1 to Sk in the basic slice.
[0086] The advantages of storing base slices in the above manner are as follows: Firstly, each base slice is stored in an independent base slice cache. When loading a base slice to the graphics processor, the weight parameters stored in each base slice cache can be read separately, avoiding read conflicts that occur when reading the same cache simultaneously. Secondly, by storing different types of data in the base slice in partitions, when loading a base slice, the partition to which the weight parameters to be loaded belong can be accessed directly, thereby quickly reading the weight parameters that need to be loaded. In summary, storing base slices in the above manner helps to improve loading efficiency.
[0087] After obtaining multiple base slices, various recombination strategies can be generated by combining these base slices with the hardware of the processing system.
[0088] Optionally, various reorganization strategies are generated based on the hardware resources of the processing system and persistently stored in the processing system's memory;
[0089] The reorganization strategy includes correspondence and metadata. Metadata includes one or more of the following: the corresponding base slice number, slice type, cache file location, slice length, target write offset, associated parameter identifier, parameter shape, and device-side reordering parameters. The slice type indicates that the corresponding base slice contains continuous weight parameters and / or non-continuous weight parameters.
[0090] The above-mentioned various recombination strategies can be generated by the aforementioned processing system, or by other electronic devices different from the aforementioned processing system.
[0091] When generating multiple recombination strategies, the various hardware configurations of the processing system can be determined as follows: Based on the hardware resources of the processing system, determine each candidate hardware configuration that the processing system can support. For example, assuming the processing system has 8 graphics processors, it can utilize any number of graphics processors from 2 to 8 as needed to run the inference model. Therefore, the hardware configurations supported by this processing system can be determined to include TP2, TP3, TP4, TP5, TP6, TP7, and TP8, corresponding to any number of graphics processors used by the processing system from 2 to 8.
[0092] After determining multiple hardware configurations, for each hardware configuration, a reorganization strategy corresponding to that hardware configuration can be generated based on the hardware configuration and multiple pre-obtained basic slices, thereby obtaining multiple reorganization strategies that correspond one-to-one with multiple hardware configurations.
[0093] For example, for the TP2 hardware configuration, the generated reassembly strategy includes rank0 corresponding to base0+base1+base2+base3, and rank1 corresponding to base4+base5+base6+base7; for the TP4 hardware configuration, the generated reassembly strategy has rank0 corresponding to base0+base1, and rank1 corresponding to base2+base3. base0 to base7 are 8 basic slices, and rank0 and rank1 represent two loading queues in the reassembly strategy.
[0094] For each hardware configuration, the methods for generating the corresponding reconfiguration strategy can include:
[0095] Based on the number of graphics processors in the hardware configuration, establish a corresponding number of loading queues. For example, for the TP4 hardware configuration, establish 4 loading queues, denoted as rank0 to rank3.
[0096] Multiple base slices are assigned to various loading queues to determine the correspondence between the multiple base slices and the multiple loading queues, that is, to determine which base slices correspond to each rank, thereby obtaining the reorganization strategy corresponding to the hardware configuration.
[0097] There are several ways to allocate base slices, and no limitation is imposed. As some examples, multiple base slices can be evenly distributed among the loading queues according to the number of loading queues. For example, if there are 8 base slices and 4 loading queues, every two base slices can be assigned to one loading queue. For example, rank0 corresponds to base0+base1, and rank1 corresponds to base2+base3.
[0098] Alternatively, the data volume of the base slice can be allocated to the loading queues, so that the total data volume of the base slices corresponding to each loading queue is basically the same. This can make the loading time of each loading queue similar during parallel loading, which is beneficial to improving loading efficiency.
[0099] Alternatively, the basic slices can be assigned to the loading queue based on their relationship with each other. For example, if two basic slices contain weight parameters belonging to the same layer of the inference model, or contain the same expert model in the inference model, these two basic slices can be assigned to the same loading queue.
[0100] For each hardware configuration, after determining the correspondence between multiple basic slices and multiple loading queues under that hardware configuration using the method described above, the metadata of that correspondence can be further obtained. The correspondence and metadata are then persistently stored in the memory of the processing system to form the reorganization strategy corresponding to that hardware configuration.
[0101] The reorganization strategy includes the following metadata: the base slice number includes the base slice number corresponding to each loading queue; the slice type can include the aforementioned C list and S list, which record which weight parameters are consecutive and which are non-consecutive in the base slice corresponding to each loading queue; the cache file location indicates the storage address of the base slice corresponding to each loading queue; the slice length indicates the length (data volume) of each base slice corresponding to each loading queue; the target write offset indicates the address written when loading consecutive weight parameters to the graphics processor; the parameter identifier indicates the tensor matrix, layer, and / or expert model to which the weight parameter belongs in the inference model; the parameter shape indicates the number of rows and columns of the weight parameter; and the device-side reordering parameter indicates the method of reassembling and combining non-consecutive weight parameters to form the original tensor matrix of the inference model. Here, the weight parameters include the weight parameters contained in the base slice corresponding to the loading queue.
[0102] Optionally, the target hardware configuration may be determined, including at least one of the following methods:
[0103] Method 1: Determine the target hardware configuration from among multiple hardware configurations based on the current hardware usage of the processing system;
[0104] Method 2: Determine the target hardware configuration from multiple hardware options based on user requirements.
[0105] In Method 1, the current hardware usage includes, but is not limited to: the remaining available resources of each graphics processor in the current processing system (such as free video memory space, number of free computing cores), the running status of each graphics processor (i.e. whether it is running normally), and the performance indicators of each graphics processor (such as running frequency, total video memory space).
[0106] When determining the target hardware configuration based on the current hardware usage, one can determine the target hardware configuration corresponding to the number of normally operating graphics processors (GPUs). For example, if 7 GPUs are normally operating, the target hardware configuration is determined as TP7, indicating that these 7 normally operating GPUs are used to run the inference model for inference. Alternatively, one can determine the number of GPUs whose remaining available resources meet the usage conditions (e.g., greater than a preset threshold), and then determine the corresponding number of target hardware configurations. For example, if 4 GPUs have remaining available resources that meet the usage conditions, the target hardware configuration is determined as TP4. Another approach is to determine the number of GPUs whose performance indicators exceed preset indicator thresholds (e.g., operating frequency above 3000MHz), and then determine the corresponding number of target hardware configurations.
[0107] In Method Two, the user requirement information may include the expected inference time and / or the amount of data to be processed. The expected inference time indicates how long the user expects to complete the inference process. A shorter expected inference time allows for a target hardware configuration with more graphics processors, and vice versa. The amount of data to be processed indicates how much data the inference model needs to process after it is loaded and started. A larger amount of data to be processed allows for a target hardware configuration with more graphics processors, and vice versa.
[0108] Alternatively, the target hardware configuration can be determined by combining the current hardware usage and user needs. For example, if the expected inference time is short, all normally running graphics processors can be included in the target hardware configuration to call up as many graphics processors as possible for inference; if the expected inference time is long, a few graphics processors with lighter loads can be selected according to the remaining available resources to reduce resource consumption.
[0109] The advantages of determining the target hardware configuration using the above method are: it ensures that the determined target hardware configuration meets the current performance of the processing system and the user's usage needs. On the one hand, it can determine the appropriate target hardware configuration through hardware usage to achieve load balancing. On the other hand, it can comprehensively consider user needs information to meet the usage needs in different usage scenarios.
[0110] Optionally, after determining the target hardware configuration but before executing S102, the following judgment steps can be performed:
[0111] Determine whether the target hardware configuration and multiple base slices are compatible. If they are compatible, execute S102. If they are not compatible, it means that the current multiple base slices cannot be directly loaded into the multiple graphics processors of the corresponding target hardware configuration. In this case, the weight parameters of the inference model can be re-segmented based on the target hardware configuration to obtain multiple new slices, and these slices can be loaded into multiple graphics processors in parallel.
[0112] The method for re-slicing can be found in the aforementioned method for obtaining the base slice; simply replace the reference hardware configuration with the target hardware configuration.
[0113] In the case of resegmentation, the number of new slices can be the same as the number of graphics processors corresponding to the target hardware configuration. During parallel loading, the weight parameters contained in each slice can be directly loaded into the corresponding graphics processor in parallel, without the need to generate a corresponding reorganization strategy. The resulting new slices can be saved and reused in subsequent parallel loading based on the target hardware configuration.
[0114] One method to determine whether the target hardware configuration is compatible with multiple base slices is to check if the target reorganization strategy is among the generated multiple reorganization strategies. If it is, the target hardware configuration is compatible with multiple base slices; otherwise, the target hardware configuration is incompatible with multiple base slices.
[0115] As explained above, the weight parameters contained in each base slice can be divided into two categories: continuous weight parameters and non-continuous weight parameters. When the base slices are loaded into the graphics processor in parallel, the loading methods for these two types of weight parameters are different.
[0116] Optionally, for consecutive weight parameters, the underlying slices corresponding to multiple loading queues are loaded in parallel to multiple graphics processors, including:
[0117] Based on the target reorganization strategy, multiple storage regions are divided into storage areas of multiple graphics processors;
[0118] For each graphics processor, consecutive weight parameters from the base slices corresponding to multiple loading queues are loaded into multiple storage regions.
[0119] This embodiment can be applied to every graphics processor that needs to load a base slice. The storage area of the graphics processor can be the storage area of the graphics processor's own video memory.
[0120] For any graphics processing unit (GPU), when partitioning a memory region, the target write offset of the load queue corresponding to that GPU can be obtained from the metadata of the target reorganization strategy. This target write offset may include the offsets corresponding to each basic slice of the load queue, for example, including... Figure 4 The offsets from basei to basek of the base slice in the loading queue, such as offsets 0 and 1, will be used to divide the storage area for writing weight parameters, forming a sequence like... Figure 4 The multiple storage regions shown.
[0121] On the other hand, the metadata of the target recombination strategy can also include the length of the continuous weight parameters contained in each base slice, for example, including Figure 4 The lengths L0 and L2 shown indicate how long a segment of the corresponding base slice is a continuous weight parameter.
[0122] One way to load consecutive weight parameters from base slices corresponding to multiple load queues into multiple storage regions is as follows: After dividing the storage regions of a graphics processor, the processing system can determine which base slice's weight parameters should be written to each storage region based on the correspondence between base slices and offsets recorded in the metadata. For example... Figure 4 The first storage interval on the left should be filled with the weight parameter of basei, and the second storage interval should be filled with the weight parameter of basei+1. Then, multiple directional data transfers can be performed on each basic slice contained in the loading queue corresponding to the graphics processor. Each data transfer will directly write the continuous weight parameter of the corresponding length in a basic slice into the corresponding storage interval in the graphics processor, thereby completing the loading process of continuous weight parameters in each basic slice corresponding to the loading queue.
[0123] by Figure 4 For example, in multiple targeted moves, the first move can move the continuous weight parameters of length L0 in basei to the first storage area on the left, the second move can move the continuous weight parameters of length L1 in basei+1 to the second storage area on the left, and so on, until all the base slices basei to basek corresponding to the loading queue have been moved.
[0124] Optionally, the aforementioned storage area can be divided according to the tensor matrix of the inference model or according to the layering of the inference model. For example, if a graphics processor needs to load 5 tensor matrices of the inference model, then according to the data volume of each tensor matrix recorded in advance, 5 storage areas corresponding to these 5 tensor matrices are divided in the graphics processor's video memory. For each storage area, the continuous weight parameters of each basic slice are moved to the storage area in the above method to complete the movement (loading) of the entire tensor matrix.
[0125] The advantage of loading weight parameters using the above method is that it establishes a direct mapping relationship between the base slice and the graphics processor's video memory, thereby loading the weight parameters of the base slice directly into the graphics processor without going through other cache transfers and processing, which helps to improve loading efficiency.
[0126] Optionally, for non-contiguous weight parameters, the underlying slices corresponding to multiple loading queues are loaded in parallel to multiple graphics processors, including:
[0127] Based on the target reorganization strategy, the non-contiguous weight parameters in the basic slices corresponding to multiple loading queues are reordered.
[0128] The reordered weight parameters are loaded in parallel onto multiple graphics processors.
[0129] Each graphics processor can be loaded with non-continuous weight parameters according to the method of this embodiment.
[0130] For any graphics processor, the device-side reordering parameters of the loading queue corresponding to that graphics processor can be obtained from the target reordering strategy. The device-side reordering parameters record the information required to reorder the non-contiguous weight parameters of the base slice, such as the reordering method and the shape (number of rows and columns) of the weight parameters after reordering.
[0131] Simultaneously, to perform reordering, a staged cache (denoted as GPU staging buffer) can be pre-created in the graphics processor's video memory or other memory for reordering. The structure of this staged cache can be determined based on the device-side reordering parameters. For example, if the device-side reordering parameters instruct that non-contiguous weight parameters be reordered into a 10x10 tensor matrix, then a staged cache that can exactly store a 10x10 tensor matrix can be created. The advantage of this approach is that it satisfies the cache space requirements of reordering while conserving cache resources.
[0132] Having created a staged buffer, non-contiguous weight parameters can be read from the corresponding base slice in the loading queue according to device-side reordering parameters. These read weight parameters are temporarily stored in the staged buffer. Within the staged buffer, these non-contiguous weight parameters are reordered according to the device-side reordering parameters, resulting in reordered weight parameters. These reordered weight parameters are equivalent to the original tensor matrices in the inference model. Finally, the reordered weight parameters can be loaded from the staged buffer into the corresponding graphics processor.
[0133] Using the above method, this embodiment can restore the layout of discontinuously distributed weight parameters in the base slice to the original layout of these weight parameters in the inference model, and then load the restored weight parameters (i.e., the weight parameters after reordering) into the graphics processor, so that the graphics processor can complete subsequent inference based on these weight parameters.
[0134] As an example, see Figure 5 Suppose we need to load non-contiguous weight parameters from two base slices, basep and baseq, corresponding to the loading queue, into a graphics processor. Basep includes non-contiguous weight parameters [A0, A1, A2], and baseq includes non-contiguous weight parameters [B0, B1, B2]. During loading, these weight parameters can be written to a staged buffer. Within this buffer, these weight parameters undergo layout transformation to obtain, as shown below. Figure 5 The reordered weight parameters shown are [A0, B0], [A1, B1], and [A2, B2]. Finally, these reordered weight parameters are written into the graphics processor's video memory, completing the loading of this part of the non-contiguous weight parameters.
[0135] Optionally, the reordering process can include one or more of the following: spanning access, grouping, row-column interleaving, expert mapping, and layout transformation. The specific reordering methods used for each graphics processor during parallel loading are determined by the device-side reordering parameters.
[0136] Stride access refers to a method of skipping a fixed number of weight parameters in a base slice, intermittently reading weight parameters, and combining the read weight parameters.
[0137] Group Combination refers to combining multiple logically independent but physically distributed weight parameters (i.e., non-contiguous weight parameters in multiple base slices) into a larger tensor matrix during loading.
[0138] Row-column interleaving refers to the combined use of row-based and column-based partitioning when dividing a tensor matrix into multiple base slices. Correspondingly, during the reordering process, the non-contiguous weight parameters in each base slice need to be concatenated both row-wise and column-wise to form the original tensor matrix.
[0139] Expert mapping refers to mapping an expert model onto a graphics processor. In this case, it is necessary to reorganize the non-contiguous weight parameters belonging to the expert model in multiple base slices to restore the original weight parameters of the expert model, and then write these weight parameters into the video memory of the corresponding graphics processor.
[0140] Layout conversion refers to changing the storage format or arrangement of weight parameters from one to another to match the format required by the graphics processor for inference. For example, converting weight parameters that are originally stored in rows to be stored in columns, or vice versa.
[0141] For specific methods of reordering weight parameters using the above-mentioned reordering techniques, please refer to relevant technologies in the field of deep learning, which will not be elaborated here.
[0142] As explained earlier, loading a base slice into the graphics processor requires reading the weight parameters contained in the base slice and writing (or reordering and then writing) these weight parameters to the graphics processor. To improve the speed of reading weight parameters from the base slice, weight parameters located in the same cache file and in adjacent or mergeable positions can be combined into a single data block for reading at once, thus reducing the overhead of random reads of small blocks.
[0143] Based on the previous example, when reading the weight parameters A0, A1, and A2, they can be combined into a single data block [A0, A1, A2] for a single read, instead of reading A0, A1, and A2 separately three times.
[0144] This application also provides an electronic device, such as... Figure 6 As shown, the electronic device includes a memory 601, a central processing unit 602, and multiple graphics processors 603;
[0145] Memory is used to store computer programs;
[0146] Graphics processors are used to perform inference tasks based on weight parameters;
[0147] The central processing unit (CPU) is used to execute computer programs to perform:
[0148] Determine the target hardware configuration, which represents the multiple graphics processors used for inference in the processing system;
[0149] Read the base slice index and the target reorganization strategy corresponding to the target hardware configuration in multiple reorganization strategies. The target reorganization strategy includes at least the correspondence between multiple base slices and multiple loading queues for parallel loading. The multiple base slices are obtained by dividing the weight parameters of the inference model.
[0150] Based on the target reorganization strategy and the base slice index, the base slices corresponding to multiple loading queues are loaded in parallel to multiple graphics processors, so that multiple graphics processors can perform inference based on weight parameters.
[0151] The working principle of electronic devices can be found in the relevant steps of the method for loading model weight parameters mentioned above, and will not be repeated here.
[0152] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. The systems or devices described above are divided into various modules or units based on their functions. When implementing this application, the functions of each unit can be implemented in one or more software and / or hardware components.
[0153] In this document, relational terms such as first, second, third, and fourth are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0154] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.
Claims
1. A method for loading model weight parameters, comprising: Determine the target hardware configuration, which represents multiple graphics processors used for inference in the processing system; Read the target reorganization strategy corresponding to the target hardware configuration in the base slice index and multiple reorganization strategies. The target reorganization strategy includes at least the correspondence between multiple base slices and multiple loading queues for parallel loading. The multiple base slices are obtained by dividing by the weight parameters of the inference model. Based on the target reorganization strategy and the base slice index, the base slices corresponding to the multiple loading queues are loaded in parallel to the multiple graphics processors, so that the multiple graphics processors can perform inference based on the weight parameters.
2. The method according to claim 1, wherein determining the target hardware configuration includes at least one of the following: The target hardware configuration is determined from a variety of hardware configurations based on the current hardware usage of the processing system. The target hardware configuration is determined from a variety of hardware options based on user requirements.
3. The method according to claim 1, wherein the multiple reorganization strategies are generated based on the hardware resources of the processing system and persistently stored in the memory of the processing system; The reorganization strategy includes the correspondence and metadata. The metadata includes any one or more of the following: the corresponding base slice number, slice type, cache file location, slice length, target write offset, associated parameter identifier, parameter shape, and device-side reordering parameters. The slice type indicates that the corresponding base slice contains continuous weight parameters and / or non-continuous weight parameters.
4. The method according to claim 1, wherein loading the base slices corresponding to the plurality of loading queues in parallel to the plurality of graphics processors comprises: According to the target reorganization strategy, multiple storage regions are divided in the storage areas of the multiple graphics processors; For each graphics processor, the consecutive weight parameters in the base slices corresponding to the plurality of loading queues are loaded into the plurality of storage regions.
5. The method according to claim 1, wherein loading the base slices corresponding to the plurality of loading queues in parallel to the plurality of graphics processors comprises: According to the target reorganization strategy, the non-contiguous weight parameters in the basic slices corresponding to the multiple loading queues are reordered. The reordered weight parameters are loaded in parallel onto the multiple graphics processors.
6. The method according to claim 5, wherein the reordering process includes: One or more of the following: cross-access, grouping, row-column interleaving, expert mapping, and layout transformation.
7. The method according to claim 1, wherein the plurality of basic slices are obtained by the weight parameters of the inference model based on the reference hardware configuration and reference segmentation method of the processing system, wherein the reference hardware configuration represents the graphics processor contained in the processing system, and the reference segmentation method is related to the model architecture of the inference model.
8. The method according to claim 7, wherein the base slice is stored in a base slice cache, the base slice cache comprising a metadata area, a contiguous data area, and a non-contiguous data area; The metadata area stores the slice metadata of the base slice, and the slice metadata includes at least one of the following: source file location, number, slice length, slice type, and associated parameter identifier of the base slice; The continuous data area stores the continuous weight parameters of the base slice, and the non-continuous data area stores the non-continuous weight parameters of the base slice.
9. The method according to claim 7, wherein the reference segmentation method is one of a plurality of segmentation methods determined according to the model architecture; The various segmentation methods include any combination of row-based segmentation, column-based segmentation, fusion parameter segmentation, vocabulary segmentation, layer-based segmentation, and expert weight segmentation.
10. An electronic device comprising a memory, a central processing unit, and a plurality of graphics processors; The memory is used to store computer programs; The graphics processor is used to perform inference tasks based on weight parameters; The central processing unit is used to execute the computer program to perform: Determine the target hardware configuration, which represents multiple graphics processors used for inference in the processing system; Read the target reorganization strategy corresponding to the target hardware configuration in the base slice index and multiple reorganization strategies. The target reorganization strategy includes at least the correspondence between multiple base slices and multiple loading queues for parallel loading. The multiple base slices are obtained by dividing by the weight parameters of the inference model. Based on the target reorganization strategy and the base slice index, the base slices corresponding to the multiple loading queues are loaded in parallel to the multiple graphics processors, so that the multiple graphics processors can perform inference based on the weight parameters.
