Model loading and unloading methods and electronic devices

Abstract

This application discloses a model loading / unloading method and electronic device, relating to the field of server resource management technology. The method includes: preloading each feedforward neural subnetwork onto preset hardware in a preset order based on historical operation data; performing a first calculation on first resource usage information, second resource usage information, and memory usage information at preset time intervals to obtain a first resource usage function value; performing a second calculation on second resource usage information and memory usage information at preset time intervals to obtain a second resource usage function value; and unloading the feedforward neural subnetwork from a first processor to a second processor when the second resource usage function value is detected to be less than a second predefined lower bound of a resource usage function value threshold. This invention solves the problem of low model feedback efficiency caused by the uneven distribution of subnetworks in the model across different hardware components of the server in related technologies.

Description

Technical Field

[0001] This application relates to the field of server resource management technology, and in particular to a model loading / unloading method and electronic device. Background Technology

[0002] With the rapid popularization of large language models such as LLM in human-computer dialogue and content generation scenarios, their computing and storage requirements are also growing exponentially. In order to continuously improve the user experience, how to dynamically load and unload models onto corresponding server hardware has become a key research issue in the industry.

[0003] In related technologies, the main approach is to load the corresponding subnetworks from the model onto the appropriate server hardware through user-defined configurations to improve model feedback efficiency. However, a more accurate and reasonable model loading and unloading method is still needed to address the problem of uneven distribution of subnetworks across server hardware, which leads to low model feedback efficiency. Summary of the Invention

[0004] This application provides a model loading / unloading method and electronic device to at least solve the problem of low model feedback efficiency caused by the uneven distribution of sub-networks in the model across server hardware in related technologies.

[0005] This application provides a model loading / unloading method, wherein the model includes multiple feedforward neural subnetworks, and the method includes:

[0006] Obtain historical runtime data for each feedforward neural subnetwork of the model;

[0007] Based on the historical operation data, each feedforward neural subnetwork is preloaded onto a preset hardware in a preset order, wherein the preset hardware includes a first processor, a second processor, and a memory.

[0008] During the operation of the model, model information, first resource usage information of the first processor, second resource usage information of the second processor, memory usage information, and feedforward neural subnetwork weight parameter scheduling information are acquired in real time according to a preset time interval.

[0009] A first calculation is performed on the first resource usage information, the second resource usage information, and the memory usage information according to the preset time interval to obtain a first resource usage function value;

[0010] When the value of the first resource usage function is detected to be greater than the upper limit of the first predefined resource threshold, the second resource usage information and the memory usage information are calculated in a second way according to the preset time interval to obtain the second resource usage function value.

[0011] When it is detected that the value of the second resource usage function is less than the lower limit of the second predefined resource usage function value threshold, the feedforward neural subnetwork in operation is offloaded from the first processor to the second processor.

[0012] This application also provides an electronic device, including:

[0013] Memory, used to store computer programs;

[0014] A processor, used to implement the steps of any of the model loading and unloading methods described above when executing the computer program.

[0015] This application addresses the issue of low model feedback efficiency caused by the uneven distribution of feedforward neural subnetworks on the server's preset hardware based on historical operation data. During model operation, it acquires model information, first resource usage information of the first processor, second resource usage information of the second processor, memory usage information, and feedforward neural subnetwork weight parameter scheduling information in real time at preset time intervals. The loading of the feedforward neural subnetwork is then determined based on the first resource usage information, second resource usage information, and memory usage information at the preset time intervals. Through preloading and this dynamic adaptive loading / unloading method, the distribution of feedforward neural subnetwork weight parameters across the preset hardware resources is finely adjusted. Therefore, this method solves the technical problem of low model feedback efficiency due to uneven distribution of feedforward neural subnetworks on the server's preset hardware, achieving a more reasonable and uniform distribution of feedforward neural subnetworks on the server's preset hardware, thereby improving the model's feedback efficiency. Attached Figure Description

[0016] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This application provides an illustration of an application scenario for a model loading and unloading method.

[0018] Figure 2 A schematic flowchart illustrating the model loading and unloading method provided in this application embodiment;

[0019] Figure 3 This is a schematic diagram of a model loading and unloading structure provided in an embodiment of this application;

[0020] Figure 4 A schematic diagram of the structure of the electronic device provided in this application. Detailed Implementation

[0021] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0022] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0023] The following are explanations of terms used in this embodiment:

[0024] MOE (Mixture-of-Experts) is a "divide and conquer" sparse neural network architecture. Its core idea is to "break the model into a group of small experts and activate them on demand." It allows the model to grow exponentially in terms of parameter size while the computational cost increases linearly or even sublinearly, making it a key technology for current models.

[0025] LLM (Large Language Model) refers to a generative language model with over a billion parameters, based on the Transformer architecture, and trained through self-supervised pre-training on massive amounts of text. These models can perform various tasks such as question answering, translation, summarization, code generation, and dialogue, and exhibit "emergent" capabilities (such as context learning and logical reasoning).

[0026] FNN (Feedforward Neural Network) is a neural network in which information flows unidirectionally from input to output without loops. It is the most "prototype" architecture of deep learning.

[0027] HBM (High Bandwidth Memory) is a high-speed memory technology that vertically stacks multiple DRAM chips like building blocks and then solders them directly next to the GPU using an ultra-wide bus. It solves the pain points of insufficient bandwidth, high power consumption, and large area of ​​traditional GDDR, and has become the "standard video memory" for high-end GPUs in AI, HPC, graphics rendering and other scenarios.

[0028] EP (Expert Parallelism) is a distributed parallel strategy specifically designed for sparse hybrid expert (MoE) models. It splits different experts onto different GPUs / nodes, then sends the tokens precisely to the devices with the target experts for computation based on the routing results, and finally collects the results.

[0029] Tensor Parallelism (TP) is a distributed training / inference strategy that "vertically" splits a single matrix / tensor across multiple GPUs, allowing each GPU to compute only a sub-block of the entire tensor, thereby "breaking down" the hypermodel parameters and packing them into limited GPU memory.

[0030] GPU: Graphics Processing Unit.

[0031] CPU: Central Processing Unit.

[0032] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0033] This section describes the specific application environment architecture or hardware architecture that the model loading / unloading methods depend on. (References) Figure 1 , Figure 1 This diagram illustrates an application scenario for a model loading / unloading method provided in an embodiment of this application. Figure 1 As shown, it includes: terminal 101 and server 102.

[0034] Terminal 101 is used for manual operations such as starting the system in server 102 and opening models. Terminal 101 may include devices such as a mouse, keyboard, or touch screen that can generate corresponding instructions in response to user manual operations and transmit the instructions to server 102. Server 102 is used to receive instructions submitted from terminal 101 and to obtain models from other servers or the cloud, or to directly add or unload installed models, and to execute model addition and unloading methods to complete the dynamic adaptive addition and unloading of each feedforward neural subnetwork in the model.

[0035] Figure 2 This is a flowchart illustrating the model loading and unloading method provided in the embodiments of this application, as shown below. Figure 2 As shown, the embodiments of this application provide a model loading and unloading method. The execution subject of this embodiment can be... Figure 1The server 102 in the illustrated embodiment can be a physical server, a cloud server, or other computer-related devices; this embodiment does not impose any particular limitations on this. The method is described in detail below:

[0036] S201: Obtain historical runtime data for each feedforward neural subnetwork of the model.

[0037] In this embodiment, the model includes multiple feedforward neural subnetworks. For example, the model can be the MOE model mentioned above, and the feedforward neural subnetworks can be the experts mentioned above. When the system corresponding to the model is not starting for the first time, the running process data of each feedforward neural subnetwork during the model inference process will be recorded and stored as historical running process data.

[0038] In an optional embodiment of this application, the historical operation process data may include the following three parts: model_info representing the model used, hardware_info representing the hardware environment information for running the model, and expert_active_info describing the expert activation usage during the inference operation process.

[0039] The `model_info` parameter includes the model type (e.g., Tongyi Qianwen, DeepSeek, etc.), model accuracy (e.g., Int8, FP16, etc.), the number of dense and expert layers, the parameter size of each dense layer, the number of routing experts and shared experts on each expert layer, and the parameter size of each expert. Experts and inference processes differ across models, so the loading and unloading methods also differ. Furthermore, the parameter size of dense and expert layers directly affects the loading of expert weights on CPU and GPU memory / video memory.

[0040] hardware_info can include two categories: computing units (CPU and GPU) and storage units (GPU video memory vRAM, CPU memory RAM and memory). In addition, it is necessary to collect statistics on hardware usage during the model inference process from the aspects of average usage rate and maximum and minimum usage rate, so as to guide the expert weight loading and unloading during the inference process.

[0041] `expert_active_info` describes the activation status of experts during inference, including expert ID and activation frequency. The expert ID is identified by the layer number `layer_idx` and the expert's sequence number `expert_idx` within that layer. The activation frequency indicates the expert's popularity, categorized into three types: high-frequency experts (shared experts and routing experts with an activation probability greater than 90%), mid-frequency experts (routing experts with an activation probability greater than 10% but less than 90%), and low-frequency experts (routing experts with an activation probability less than 10%). Experts with higher popularity are more likely to be activated and should, in principle, be prioritized for loading onto the GPU. Here, "expert" refers to the feedforward neural subnetwork.

[0042] S202: Based on historical operation data, each feedforward neural subnetwork is preloaded onto preset hardware in a preset order, wherein the preset hardware includes a first processor, a second processor, and a memory.

[0043] In this embodiment, activation frequency data of each feedforward neural network can be extracted based on historical operation data. These activation frequencies are then sorted, and the feedforward neural networks are loaded onto their corresponding preset hardware in a predetermined order, from the first processor to the second processor and then to the memory. During the loading process, the remaining video memory space of the image processor needs to be monitored in real time. If sufficient, the data is loaded into the second processor; otherwise, it continues to be loaded into the first processor. In this embodiment, the first processor can be a graphics processing unit (GPU) or other processor with image processing capabilities. The second processor can be a central processing unit (CPU) or other processor with data processing capabilities. The memory can be a hard disk or other hardware device with data storage capabilities. This embodiment does not impose any particular limitations on these aspects.

[0044] In an optional embodiment of this application, step S202 specifically includes:

[0045] S202a: Determine the activation frequency of each feedforward neural subnetwork based on historical operating data.

[0046] In this embodiment, the activation frequency of each feedforward neural subnetwork can be determined by matching the correspondence between each feedforward neural subnetwork and its activation frequency from historical operating data in advance.

[0047] S202b: Based on the activation frequency, each feedforward neural subnetwork is divided into three types of feedforward neural subnetworks: Type I, Type II, and Type III, in descending order of activation frequency.

[0048] In this embodiment, the first type of feedforward neural subnetwork has the highest activation frequency and can be considered a "hot" feedforward neural subnetwork. The third type of feedforward neural subnetwork has the lowest activation frequency and can be considered a "cold" feedforward neural subnetwork. The lower the activation frequency, the less impact the feedforward neural subnetwork has on model inference. Therefore, the first type of feedforward neural subnetwork with the high activation frequency is preferentially loaded into the first processor so that it can be invoked at any time for inference and to generate the feedback required by the user.

[0049] S202c: Load the first type of feedforward neural subnetwork layer by layer. When the first memory space of the first processor is detected to be greater than or equal to the preset memory space space threshold, the feedforward neural subnetwork weight parameters of the first type of feedforward neural subnetwork are loaded onto the first processor until the first type of feedforward neural subnetwork is preloaded and the second memory space space of the first processor is obtained, wherein the second memory space space is less than the first memory space space.

[0050] In this embodiment, the first memory space reserve refers to the amount of remaining memory space of the first processor recorded by the system when the feedforward neural subnetwork is initially loaded into the first processor. The preset memory space reserve threshold can be the amount of memory space required for loading calculated based on the first type of feedforward neural subnetwork. When the first memory space reserve of the first processor is less than the preset memory space reserve threshold, it indicates that the first processor is unable to complete the task of loading a complete first type of feedforward neural subnetwork. At this time, the first type of feedforward neural subnetwork to be loaded can be preloaded into the second processor.

[0051] In this embodiment, when the remaining first-class feedforward neural network of the first processor is detected to be greater than or equal to a preset memory space reserve threshold, it indicates that at least one of the remaining first-class feedforward neural networks will continue to be loaded into the first processor. In this embodiment, first-class feedforward neural networks with high activation frequencies are loaded first to improve the inference efficiency of the model. After all first-class feedforward neural networks are loaded into the first processor, the system monitors and records the remaining memory space of the corresponding image processor as the second memory space reserve.

[0052] S202d: Load the second type of feedforward neural subnetwork layer by layer. When the second memory space remaining is detected to be greater than or equal to the preset memory space remaining threshold, the feedforward neural subnetwork weight parameters of the second type of feedforward neural subnetwork are loaded onto the first processor until the second type of feedforward neural subnetwork is loaded. Then, the third memory space remaining of the first processor is obtained, wherein the third memory space remaining is less than the second memory space remaining.

[0053] In this embodiment, the preset memory space reserve threshold can be the size of the memory space required for loading the second type of feedforward neural subnetwork calculated based on the second type of feedforward neural subnetwork. When the second memory space reserve of the first processor is less than the preset memory space reserve threshold, it indicates that the first processor is unable to complete the task of loading a complete second type of feedforward neural subnetwork. At this time, the second type of feedforward neural subnetwork to be loaded can be preloaded into the second processor.

[0054] In this embodiment, when the remaining second memory space of the first processor is detected to be greater than or equal to a preset memory space remaining threshold, it indicates that at least one of the remaining second-type feedforward neural subnetworks will continue to be loaded into the first processor. In this embodiment, the first-type feedforward neural subnetworks with high activation frequencies are loaded firstly to improve the inference efficiency of the model. After all the first-type feedforward neural subnetworks are loaded into the first processor, the system monitors and records the remaining memory space of the corresponding image processor as the third memory space remaining.

[0055] S202e: Load the third type of feedforward neural subnetwork layer by layer. When the third memory space margin is detected to be greater than or equal to the preset memory space margin threshold, the intelligent weight parameters of the third type of feedforward neural subnetwork are loaded onto the first processor and the fourth memory space margin of the first processor is obtained.

[0056] In this embodiment, the preset memory space reserve threshold can be the size of the memory space required for loading the third type of feedforward neural subnetwork calculated based on the third type of feedforward neural subnetwork. When the third memory space reserve of the first processor is less than the preset memory space reserve threshold, it indicates that the first processor is unable to complete the task of loading a complete third type of feedforward neural subnetwork. At this time, the third type of feedforward neural subnetwork to be loaded can be preloaded into the second processor.

[0057] In this embodiment, when the remaining third-class feedforward neural network of the first processor is detected to be greater than or equal to a preset memory space reserve threshold, it indicates that at least one of the remaining third-class feedforward neural networks will continue to be loaded into the first processor. In this embodiment, the first-class feedforward neural networks with high activation frequencies are loaded firstly to improve the inference efficiency of the model. After all the first-class feedforward neural networks are loaded into the first processor, the system monitors and records the remaining memory space of the corresponding image processor as the fourth memory space reserve.

[0058] S202f: Load the first type of feedforward neural subnetwork onto the first processor in the form of a copy according to the remaining space of the fourth video memory.

[0059] In this embodiment, the fourth memory space surplus can be compared with a preset memory space surplus threshold corresponding to the first processor. When the fourth memory space surplus is greater than or equal to the preset memory space surplus threshold, in order to further improve the inference efficiency of the model, the form of the first type of feedforward neural subnetwork with the highest activation frequency is loaded into the first processor, so that there are more first type feedforward neural subnetworks in the first processor that can play a greater role in model inference. Conversely, when the fourth memory space surplus is less than the preset memory space surplus threshold, the copy of the first type of feedforward neural subnetwork is no longer loaded.

[0060] Based on the above embodiments, in an optional embodiment of this application, step S202f includes: performing a space reserve detection operation based on the fourth video memory space reserve; when the fourth video memory space reserve is detected to be greater than or equal to the preset copy loading space reserve, then each first type of feedforward neural subnetwork is loaded into the first processor in the form of a copy layer by layer until the video memory space reserve is less than the preset copy loading space reserve.

[0061] In this embodiment, the preset copy loading space margin can be equal to the preset video memory space margin threshold, or it can be the size of the preloaded space required for the copy calculated by the system based on the first type of feedforward neural subnetwork.

[0062] In this embodiment, loading each first type of feedforward sub-neural network into the first processor layer by layer in the form of copies can be done by continuing to load each first type of feedforward neural network in the order of activation frequency as a first copy, a second copy, a third copy, and so on. During the preloading process, it is determined in real time whether the video memory space on the first processor is sufficient until there is no additional video memory space on the first processor, so as to maximize the use of the video memory resources of the first processor to load as many first type of feedforward sub-neural networks as possible.

[0063] In an optional embodiment of this application, the criteria for determining whether the video memory space on the first processor is sufficient are defined. In this embodiment, for a heterogeneous computing system with both a second processor and a first processor, in order to improve resource utilization and inference performance, a hybrid approach of expert parallelism (EP) and tensor parallelism (TP) is used to preload the weight parameters of the feedforward neural subnetwork. The feedforward neural subnetwork contains dense layers, and the loading method for the weight parameters of the dense layers can also be tensor parallelism.

[0064] The specific process is as follows:

[0065] First, the size of hardware memory or video memory occupied by the model parameters of dense layers is defined as follows:

[0066]

[0067] In the formula, dense_layer_memory_usage represents the total memory or video memory size occupied by the dense layer, dense_layer_num represents the number of dense layers, dense_layer_parameters[k] represents the number of parameters of the k-th dense layer, and precision is the preset weight parameter precision.

[0068] The memory or GPU memory size occupied by the weight parameters of each feedforward neural subnetwork is defined as follows:

[0069] per_expert_memory_usage = expert_parameters×(precision / 8)

[0070] In the formula, per_expert_memory_usage is the memory or GPU memory size occupied by each feedforward neural subnetwork weight parameter, expert_parameters is the number of feedforward neural subnetwork weight parameters in a single feedforward neural subnetwork, and precision is the preset weight parameter precision.

[0071] To ensure the quality of the model's inference service, this embodiment defines that at least 80% of the HBM (Hardware Memory) space of each first processor, after loading the weight parameters of the dense layer, needs to be reserved for storing KV cache data. Therefore, for each first processor, such as a GPU, the amount of memory space reserved for loading the weight parameters of the feedforward neural subnetwork is:

[0072] HBM_reserved_size = (HBM_size – dense_layer_data_size) ×20%

[0073] In the formula, HBM_size represents the total memory capacity of HBM, and dense_layer_data_size represents the memory size on this HBM used for loading dense layer parameters. During the loading process, the currently remaining space on the first processor's memory, HBM_reserved_now_size, is often smaller than HBM_reserved_size because the weight slices of the already loaded feedforward neural subnetwork will occupy space.

[0074] Due to the tensor parallel loading strategy, the parameters of the dense layers and the expert weights in the model are sliced ​​and loaded onto different first processors. For a server system with N first processor cards, the memory size occupied by the dense layer weight parameters and the weight parameters of each feedforward neural network on each first processor is as follows:

[0075] dense_layer_data_size = dense_layer_memory_usage / N

[0076] expert_data_slice_min = per_expert_memory_usage / N

[0077] It should be noted that the above formula expert_data_slice_min defines the size of the video memory occupied by the shards of the expert weights distributed to all the first processors in the ideal case, and at this time each shard is the smallest. However, in the actual running process, since the free video memory on each first processor card may not be the same, the shards of the feed-forward neural sub-network weight parameters can only be placed on the GPUs that can store the shards. Assuming that there are k (k < N) such first-processor GPUs in the system, then the size of each shard stored on each first-processor GPU is per_expert_memory_usage / k.

[0078] Then, to determine whether the video memory resources on a certain first processor are sufficient, it can be divided into the following situations:

[0079] a) If HBM_reserved_size on the first processor is less than or equal to 0, it means that in addition to loading the dense layer parameters and reserving the video memory for storing KV cache data, there is no extra video memory space on this first processor for storing the feed-forward neural sub-network weight parameters, so it means that the video memory resources on this first processor are insufficient.

[0080] b) If HBM_reserved_size on the first processor is greater than 0 and at the same time HBM_reserved_size < expert_data_slice_min, it means that in addition to loading the dense layer parameters, this first processor is not sufficient to load the tensor parallel parameters of the feed-forward neural sub-network weights, so it means that the video memory resources on this first processor are insufficient.

[0081] c) If HBM_reserved_size is greater than 0 on the first processor, and HBM_reserved_size > expert_data_slice_min, then it means that the first processor, besides loading dense layer parameters, is sufficient to load the tensor parallel parameters of the expert weights (i.e., it is sufficient). However, whether the first processor's memory can load the expert weights needs to be determined in conjunction with the memory usage on other first processors in the system. This is because when loading feedforward neural network weight parameters in slices, two conditions must be met simultaneously: there must be k first processors in the system with sufficient remaining space, and the remaining space of these first processors must be greater than the slice of expert weights; and k must be a power of 2, such as 2, 4, 8, 16, ... This is because the number of slices for feedforward neural network weights often needs to satisfy a power of 2, which is determined by the requirements of distributed communication and hardware optimization. By using a slice distribution method based on tensor parallelism in the first processor's memory, it is determined whether the current preset hardware resources are sufficient to load the weight parameter slices of the feedforward neural subnetwork during the preloading process, thereby improving the utilization rate of hardware resources and thus improving the inference quality of the model.

[0082] S203: During the operation of the model, model information, first resource usage information of the first processor, second resource usage information of the second processor, memory usage information, and weight parameter scheduling information of the feedforward neural subnetwork are acquired in real time according to a preset time interval.

[0083] In this embodiment, to dynamically and adaptively load and unload the feedforward neural network during the model inference process (i.e., during operation), it is necessary to monitor the hardware resource usage and the scheduling of the feedforward neural network weight parameters in real time during the server system's operation. This allows for the acquisition of model information, first resource usage information of the first processor, second resource usage information of the second processor, memory usage information, and feedforward neural network weight parameter scheduling information. This real-time monitoring can be achieved by periodically collecting and recording data based on the time intervals of the feedforward neural network's loading and unloading operations.

[0084] In this embodiment, the main aspects involved during model operation are the computing resources of the second processor and the first processor, as well as the storage resources of the second processor's memory, the first processor's video memory, and the memory. The memory is primarily used to store less frequently used feedforward neural network weight parameters when the second processor's memory is insufficient. Since storing feedforward neural network expert weight parameters in the memory affects performance, whether or not to store them in the memory requires careful consideration.

[0085] In this embodiment, the current hardware resource usage during system operation is a necessary condition for triggering the feedforward neural subnetwork loading and unloading operation. Whether the loading and unloading operation is ultimately executed depends on the actual usage of the feedforward neural subnetwork weight parameters, weighing the costs and benefits of loading and unloading. Additionally, this monitored expert scheduling information also serves as input to the historical information statistics module, providing a basis for the next preloading of the feedforward neural subnetwork from the perspective of statistical characteristics of inference tasks in the usage scenario, such as the activation frequency of the feedforward neural subnetwork.

[0086] S204: Perform a first calculation on the first resource usage information, the second resource usage information, and the memory usage information according to a preset time interval to obtain the first resource usage function value.

[0087] In an optional embodiment of this application, the process of performing a first calculation on the first resource usage information, the second resource usage information, and the memory usage information according to a preset time interval can be a process of calculating the value of the first resource usage function through a specific comparison function.

[0088] Specifically, in an optional embodiment of this application, the first resource usage information includes the first computing resource information and the first storage resource information of the first processor, the second resource usage information includes the offload process overhead and idle rate of the second processor, and the memory usage information includes the free storage space of the memory; accordingly, step S204 includes:

[0089] S204a: Determine the number of sampling times based on a preset time interval.

[0090] In this embodiment, the preset actual interval can be a fixed time difference value set in advance. For example, time node t1 and time node t2 are divided into multiple sampling time nodes according to a fixed time difference value. One sampling time node corresponds to one sampling number. For example, the number of samplings in the time range from t1 to t2 can be expressed as (t2-t1) / Δt, where Δt is the preset time interval.

[0091] S204b: Based on the first computing resource information and the first storage resource information of the first processor, the computing utilization rate, video memory utilization rate and free memory space of the first processor are obtained according to the number of samplings.

[0092] S204c: Determine the calculation result of the second processor thread based on the second processor unloaded thread and idle rate under the sampling number.

[0093] S204d: Based on the number of samplings, the weight parameters of the feedforward neural subnetwork are compared and calculated according to the free storage space of the memory to obtain the memory information calculation results.

[0094] S204e: Based on the computational utilization of the first processor, the video memory utilization, the free memory space, the calculation results of the second processor thread, and the memory information calculation results, a first resource utilization function value is obtained. In this embodiment, the first resource utilization function value can be a specific value used to compare the utilization, video memory utilization, free memory space, second processor thread calculation results, and memory information calculation results for judgment.

[0095] S205: When the value of the first resource usage function is detected to be greater than the upper limit of the first predefined resource threshold, the second resource usage information and the memory usage information are calculated according to a preset time interval to obtain the second resource usage function value.

[0096] In this embodiment, when the detected value of the first resource usage function is greater than the upper bound of the first predefined resource threshold, it means that the hardware resources corresponding to the first processor are sufficient and the feedforward neural subnetwork can be loaded. Conversely, it means that the hardware resources corresponding to the first processor are insufficient and the feedforward neural subnetwork needs to be unloaded.

[0097] Specifically, in an optional embodiment of this application, the second resource usage information includes the second computing resource information of the second processor, the second storage resource information, the offload process overhead, and the idle rate; the memory usage information includes the free storage space of the memory. Accordingly, step S205 specifically includes:

[0098] S205a: Determine the number of sampling times based on a preset time interval.

[0099] S205b: Based on the second computing resource information and the second storage resource information of the second processor, the computing utilization rate, video memory utilization rate and free memory space of the second processor are obtained according to the number of samplings.

[0100] S205c: Determine the thread calculation result of the second processor based on the unloaded thread and idle rate of the second processor under the sampling number.

[0101] S205d: Based on the number of samplings, the weight parameters of the feedforward neural subnetwork are compared and calculated according to the free storage space of the memory to obtain the memory information calculation results.

[0102] S205e: Based on the computational utilization rate of the second processor, the video memory utilization rate, the free memory space, the thread calculation results, and the memory information calculation results, the second resource usage function value is obtained.

[0103] In this embodiment, the specific calculation process of steps S204a to S204e is similar to that of steps S205a to S205e, so it will not be described again here.

[0104] S206: When it is detected that the value of the second resource usage function is less than the lower bound of the second predefined resource usage function value threshold, the feedforward neural subnetwork in operation is unloaded from the first processor to the second processor.

[0105] In this embodiment, when the detected value of the second resource usage function is less than the lower bound of the second predefined resource usage function value threshold, it indicates that the hardware resources corresponding to the second processor are sufficient, and the requirement to offload the feedforward neural subnetwork from the first processor to the second processor can be triggered. Conversely, it indicates that it cannot be directly offloaded to the second processor, but needs to be offloaded to the memory through the second processor.

[0106] Based on the above embodiments, in an optional embodiment of this application, step S206, which involves offloading the feedforward neural subnetwork from the first processor to the second processor during operation, includes:

[0107] S206a: Collect graphics processing loading information from the feedforward neural subnetwork during operation to obtain the graphics processing loading information data to be computed for each feedforward neural subnetwork on the first processor.

[0108] S206b: Calculate the graph unloading revenue ratio corresponding to each feedforward neural subnetwork based on the graph processing loading information data to be calculated.

[0109] S206c: Based on the graph offloading benefit ratio of each feedforward neural subnetwork, select the feedforward neural subnetwork with the largest graph processing benefit ratio as the feedforward neural subnetwork to be offloaded.

[0110] S206d: Unload the feedforward neural subnetwork to be unloaded to the second processor, and unload the feedforward neural subnetwork weight parameters of the feedforward neural subnetwork located on the other first processors.

[0111] In summary, the model loading and unloading method provided in this application first preloads each feedforward neural subnetwork onto preset hardware in a preset order based on historical operation data. During model operation, it acquires model information, first resource usage information of the first processor, second resource usage information of the second processor, memory usage information, and feedforward neural subnetwork weight parameter scheduling information in real time at preset time intervals. Then, it determines the loading of the feedforward neural subnetwork based on the first resource usage information, second resource usage information, and memory usage information at preset time intervals. Through preloading and this dynamic adaptive loading and unloading method, the distribution of feedforward neural subnetwork weight parameters on each preset hardware resource is finely adjusted. Therefore, it can solve the technical problem of low model feedback efficiency caused by uneven distribution of feedforward neural subnetworks in the server's preset hardware, achieving a more reasonable and uniform distribution of feedforward neural subnetworks in the server's preset hardware, thereby improving the model's feedback efficiency.

[0112] Based on the above embodiments, in an optional embodiment of this application, in step S204, the upper bound of the first predefined resource threshold may include the predefined upper limit of computing utilization WL_GPU_max and the predefined upper limit of video memory utilization WL_HBM_max of the first processor.

[0113] Whether the value of the first resource usage function is greater than the upper bound of the first predefined resource threshold can be detected by using the preset first resource usage function f (GPU).

[0114] Within the time range from time node t1 to time node t2, the number of samples is (t2-t1) / Δt. For each first processor resource to trigger an unloading operation using function f (GPU), the following conditions must be met simultaneously:

[0115] 1) The proportion of the first processor's computational utilization exceeding WL_GPU_max is greater than the upper limit of the threshold, such as 95%;

[0116] 2) The proportion of the first processor's video memory utilization exceeding WL_HBM_max is greater than the upper limit of the threshold, such as 90%;

[0117] 3) The overhead of the second processor for the feedforward neural subnetwork offloaded thread is less than the idle rate of any other second processor.

[0118] 4) There exists a second processor whose free memory space is greater than the expert weight parameter.

[0119] Correspondingly, it checks whether the value of the first resource utilization function is less than the lower bound of the first predefined resource threshold. The lower bound of the first predefined resource threshold may include the predefined lower limit of the computing utilization rate WL_GPU_min and the predefined lower limit of the video memory utilization rate WL_HBM_min of the first processor. At this time, the detection process can be represented as sampling (t2-t1) / Δt within the time range from time node t1 to time node t2. The resource utilization function f (GPU) of each first processor triggers the loading operation, which must simultaneously meet the following conditions:

[0120] 1) The proportion of the first processor whose computational utilization is lower than WL_GPU_min is greater than the upper limit of the threshold, such as 95%;

[0121] 2) The proportion of the first processor's video memory utilization exceeding WL_HBM_min is greater than the upper limit of the threshold, such as 90%;

[0122] 3) The free video memory space of the first processor satisfies the weight parameters of the feedforward neural subnetwork for piecewise loading.

[0123] 4) The overhead of the second processor in the feedforward neural subnetwork loading thread is less than the weight parameters of the dedicated feedforward neural subnetwork.

[0124] Based on the above embodiments, in an optional embodiment of this application, in step S205, the upper bound of the second predefined resource threshold may include the predefined upper limit of computing utilization WL_CPU_max and the predefined upper limit of memory utilization WL_RAM_max of the second processor.

[0125] Whether the value of the second resource usage function is greater than the upper limit of the second predefined resource threshold can be detected by using the preset second resource usage function f(CPU).

[0126] Within the time range from time node t1 to time node t2, the number of samples is (t2-t1) / Δt. Each second processor resource must meet the following conditions to trigger an unload operation using the function f(CPU):

[0127] 1) The proportion of the second processor's computational utilization exceeding WL_CPU_max is greater than the upper limit of the threshold, such as 95%;

[0128] 2) The proportion of the second processor's memory utilization exceeding WL_RAM_max is greater than the upper limit of the threshold, such as 90%;

[0129] 3) The overhead of the second processor for the feedforward neural subnetwork unloaded thread is less than the current idle rate of the second processor.

[0130] 4) The free storage space of the memory is greater than the weight parameters of the feedforward neural subnetwork.

[0131] Correspondingly, it checks whether the value of the second resource utilization function is less than the lower bound of the second predefined resource threshold. The lower bound of the second predefined resource threshold may include the predefined lower limit of computing utilization WL_CPU_min and the predefined lower limit of memory utilization WL_RAM_min of the second processor. At this time, the detection process can be represented as sampling (t2-t1) / Δt within the time range from time node t1 to time node t2. The resource utilization function f(CPU) of each second processor triggers the loading operation, which must simultaneously meet the following conditions:

[0132] 1) The proportion of the second processor whose computational utilization is lower than WL_CPU_min is less than the upper limit of the threshold, such as 95%;

[0133] 2) The proportion of the second processor's video memory utilization exceeding WL_RAM_min is greater than the upper limit of the threshold, such as 90%;

[0134] 3) The free memory of the second processor is greater than the weight parameters of the feedforward neural subnetwork.

[0135] 4) The overhead of the second processor in the feedforward neural subnetwork loading thread is less than the feedforward neural subnetwork weight parameters.

[0136] Based on the above embodiments, the model loading and unloading method provided in an optional embodiment of this application further includes:

[0137] Step A: When the value of the first resource usage function is detected to be less than the lower bound of the first predefined resource threshold, the feedforward neural subnetwork weight parameter loading information of the second processor is traversed.

[0138] Step B: Determine whether the second processor has a feedforward neural subnetwork that needs to be loaded into the first processor based on the feedforward neural subnetwork weight parameter loading information.

[0139] In an optional embodiment of the application, the feedforward neural network loading weight information includes the scale of the unloaded feedforward neural network weight parameters, the feedforward neural network unloading frequency, the feedforward neural network unloading time, the feedforward neural network residence time, the feedforward neural network activation frequency, the feedforward neural network hit rate, the feedforward neural network loading frequency, the feedforward neural network loading time, and the hardware computing power improvement; correspondingly, step B, in which the second processor is determined to have a feedforward neural network that needs to be loaded into the first processor based on the feedforward neural network weight parameter loading information, specifically includes:

[0140] Step B1: Determine the feedforward neural network unloading benefit ratio based on the loading information of the feedforward neural network, including the scale of the unloaded feedforward neural network weight parameters, the feedforward neural network unloading frequency, the feedforward neural network unloading time, the feedforward neural network residence time, the feedforward neural network activation frequency, and the feedforward neural network hit rate.

[0141] In this embodiment, the formula for calculating the offloading benefit ratio of the feedforward neural subnetwork is as follows:

[0142] The unloading benefit ratio is calculated as follows: (unloading feedforward neural network weight parameter size × feedforward neural network unloading frequency) / ((feedforward neural network unloading time + δ × expert dwell time) × feedforward neural network activation frequency × feedforward neural network hit rate). Where δ is a pre-set coefficient value.

[0143] Step B2: Determine the loading benefit ratio based on the hardware computing power improvement, feedforward neural network loading frequency, feedforward neural network activation frequency, feedforward neural network hit rate, feedforward neural network loading time, and feedforward neural network residence time.

[0144] In this embodiment, the formula for determining the loading benefit ratio is:

[0145] Loading benefit ratio = (hardware computing power improvement × feedforward neural network loading frequency × feedforward neural network activation frequency × feedforward neural network hit rate) / ((expert loading time + δ feedforward neural network dwell time)).

[0146] Step B3: Determine whether the second processor has a feedforward neural subnetwork that needs to be loaded into the first processor based on the loading benefit ratio.

[0147] In this embodiment, within a given time frame, based on the scheduling information of the feedforward neural subnetwork weight parameters obtained from model execution, the scheduling behavior of loading or unloading feedforward neural subnetwork weight parameters is quantitatively measured from both cost and benefit perspectives. As can be seen from the formula above, a feedforward neural subnetwork with a higher hit rate and activation frequency during inference should be loaded onto high-performance hardware such as the first processor for execution; conversely, it should be unloaded onto low-performance hardware such as memory as much as possible. The more time-consuming the loading / unloading operation of a feedforward neural subnetwork, and the longer the feedforward neural subnetwork resides on the current hardware, the more it should remain in its original state, i.e., without loading / unloading operations.

[0148] The more frequently a feedforward neural network is loaded, the more loading operations should be performed; conversely, the more frequently a feedforward neural network is unloaded, the more unloading operations should be performed. The computational power increase resulting from loading is the benefit gained from loading. For example, the computational power increase from loading from memory to the first or second processor is the corresponding hardware computational power increase; the computational power increase from loading from the second processor to the first processor is the difference between the computational power of the second processor and the first processor. The freed-up memory or GPU memory space on the hardware resulting from unloading the weight parameters of the feedforward neural network is the benefit gained. The greater the benefit from loading and unloading, the more appropriate the loading and unloading operations should be.

[0149] In summary, the load-to-reward ratio or unload-to-reward ratio provides a more accurate and scientific basis for determining whether to load or unload the feedforward neural subnetwork, and among which hardware to load or unload the weight parameters of the feedforward neural subnetwork, thereby making the allocation of hardware resources more balanced.

[0150] Step C: When it is detected that there is a feedforward neural subnetwork in the second processor that needs to be loaded into the first processor, the feedforward neural subnetwork in operation is loaded from the second processor into the first processor.

[0151] Specifically, in an optional embodiment of this application, step C specifically includes:

[0152] Step C1: Collect central processing loading information from the feedforward neural subnetwork during operation to obtain the first central processing loading information data to be calculated for each feedforward neural subnetwork on the second processor.

[0153] Step C2: Calculate the first central processing gain ratio corresponding to each feedforward neural subnetwork based on the first central processing loading information data to be calculated.

[0154] Step C3: Based on the first central processing benefit ratio corresponding to each feedforward neural subnetwork, select the feedforward neural subnetwork with the largest first central processing benefit ratio as the feedforward neural subnetwork to be unloaded.

[0155] Step C4: Load the feedforward neural subnetwork to be unloaded from the second processor to the first processor in a preset tensor parallel manner, and load the feedforward neural subnetwork weight parameters of the feedforward neural subnetwork to be unloaded to other first processors.

[0156] In this embodiment, the calculation process for the first central processing revenue ratio is similar to the calculation formula used in step B1, so it will not be repeated here.

[0157] Based on the above embodiments, in an optional embodiment of this application, after step B, the following step is further included:

[0158] Step D: When it is detected that the second processor does not have a feedforward neural subnetwork that needs to be loaded into the first processor, then check whether there is a feedforward neural subnetwork in the memory that needs to be loaded into the first processor.

[0159] In this embodiment, the process of detecting whether there is a feedforward neural subnetwork in the detection memory that needs to be loaded into the first processor can be referred to steps B1 to B3 of the above embodiment, and will not be repeated here.

[0160] Step E: When a feedforward neural subnetwork that needs to be loaded into the first processor is detected in the memory, the feedforward neural subnetwork in operation is loaded from the memory into the first processor.

[0161] In this embodiment, loading the feedforward neural subnetwork from memory to the first processor during operation can be achieved by loading the feedforward neural subnetwork weight parameters corresponding to the feedforward neural subnetwork into the first processor. The specific process of loading the feedforward neural subnetwork weight parameters has been described in the above embodiments, so it will not be repeated here.

[0162] Based on the above embodiments, the model loading and unloading method provided in an optional embodiment of this application further includes:

[0163] Step F: When the value of the second resource usage function is detected to be less than the lower bound of the second predefined resource usage function value threshold, determine whether to trigger uninstallation.

[0164] Step G: When an offload trigger is detected, the feedforward neural subnetwork is offloaded from the second processor to memory.

[0165] In this embodiment, detecting that the second resource usage function value is less than the lower bound of the second predefined resource usage function value threshold can refer to the process in the above embodiment based on the second resource usage function value being greater than the upper bound of the second predefined resource usage function threshold. The specific principle and implementation process are similar, so they will not be repeated here. In this embodiment, the process of unloading the feedforward neural subnetwork from the second processor to the memory can be the process of unloading the weight parameters of the feedforward neural subnetwork from the second processor to the memory.

[0166] Based on the above embodiments, in an optional embodiment of this application, step G specifically includes:

[0167] Step G1: Obtain the second central processing loading information data of each feedforward neural subnetwork on the second processor.

[0168] Step G2: Calculate the second central unloading revenue ratio corresponding to each feedforward neural subnetwork based on the second central processing loading information data to be calculated.

[0169] Step G3: Based on the second central offloading benefit ratio corresponding to each feedforward neural subnetwork, select the feedforward neural subnetwork with the largest second central processing benefit ratio and offload it to the memory.

[0170] In this embodiment, the calculation process of the first central unloading revenue ratio in steps G1 to G3 is similar to the calculation formula used in step B1, so it will not be repeated here.

[0171] Based on the above embodiments, in the model loading and unloading method provided in an optional embodiment of this application, each feedforward neural subnetwork includes a shared feedforward neural subnetwork and a routing feedforward neural subnetwork, and further includes:

[0172] Step H: When the model is detected to be running for the first time, if the initial memory space of the first processor is greater than the preset existing memory space reserve threshold, each shared feedforward neural subnetwork is loaded into the first processor layer by layer until each shared feedforward neural subnetwork is loaded and the intermediate memory space of the first processor is obtained.

[0173] Step 1: When the intermediate video memory space is detected to be greater than the preset video memory space remaining threshold, each route feedforward neural subnetwork is loaded into the first processor layer by layer until each route feedforward neural subnetwork is added to the first processor.

[0174] In this embodiment, when the system has just started running and the model has not yet run, the feedforward neural subnetwork has not performed inference. At this time, there is no historical running data as a reference, and it is impossible to identify whether the feedforward neural subnetwork in the system is a first-type, second-type, or third-type feedforward neural subnetwork based on the activation frequency of the feedforward neural subnetwork. In addition to shared experts, this preloading of feedforward neural subnetworks can be done by first loading shared experts layer by layer into the first processor. If the resources in the first processor cannot load all layers of shared feedforward neural subnetworks, then the shared feedforward neural subnetworks are loaded layer by layer until they cannot be loaded. The remaining shared feedforward neural subnetworks and all layers of routing feedforward neural subnetworks are directly loaded onto the second processor. It should be noted that if the second processor also cannot load the remaining feedforward neural subnetworks, it indicates that the system resources are severely scarce and cannot support the requirements of inference service quality. This invention does not consider this situation.

[0175] If the hardware resources of the first processor are sufficient to load the shared expert network of all layers, then the shared feedforward neural network of all layers is loaded onto the first processor first, and then the routing feedforward neural network of each layer is loaded onto the first processor layer by layer. If it is not possible to load the routing feedforward neural network of all layers onto the first processor, the remaining routing feedforward neural network is loaded onto the second processor.

[0176] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0177] This application embodiment also provides a model loading and unloading device, the execution subject of which may be... Figure 1 The server 102 shown could also be other hardware devices with the same function. This implementation does not impose any restrictions on this.

[0178] Figure 3 This is a schematic diagram of a model loading and unloading structure provided in an embodiment of this application. Figure 3 As shown, embodiments of this application also provide a model loading / unloading device, comprising: a historical information statistics module 31, a preloading module 32, a running monitoring module 33, and a loading / unloading module 34;

[0179] The historical information statistics module 31 is used to obtain historical running process data of each feedforward neural subnetwork of the model.

[0180] The preloading module 32 is used to preload each feedforward neural subnetwork onto the preset hardware in a preset order according to historical running process data. The preset hardware includes a first processor, a second processor, and a memory.

[0181] The operation monitoring module 33 is used to acquire model information, first resource usage information of the first processor, second resource usage information of the second processor, memory usage information, and feedforward neural subnetwork weight parameter scheduling information in real time according to a preset time interval during the operation of the model.

[0182] The loading / unloading module 34 is used to perform a first calculation on the first resource usage information, the second resource usage information and the memory usage information according to a preset time interval to obtain the first resource usage function value;

[0183] The loading / unloading module 34 is also used to perform a second calculation on the second resource usage information and memory usage information according to a preset time interval when the first resource usage function value is detected to be greater than the upper limit of the first predefined resource threshold, so as to obtain the second resource usage function value.

[0184] The loading / unloading module 34 is also used to unload the feedforward neural subnetwork in operation from the first processor to the second processor when it is detected that the value of the second resource usage function is less than the lower bound of the second predefined resource usage function value threshold.

[0185] In an optional embodiment of this application, the preloading module 32 is specifically used to: determine the activation frequency of each feedforward neural subnetwork based on historical running data;

[0186] Based on activation frequency, the feedforward neural subnetworks are divided into three categories: the first type, the second type, and the third type, in descending order of activation frequency.

[0187] The first type of feedforward neural subnetwork is loaded layer by layer. When the first memory space of the first processor is detected to be greater than or equal to the preset memory space space threshold, the feedforward neural subnetwork weight parameters of the first type of feedforward neural subnetwork are loaded onto the first processor until the first type of feedforward neural subnetwork is preloaded. The second memory space space of the first processor is obtained, wherein the second memory space space is less than the first memory space space.

[0188] The second type of feedforward neural subnetwork is loaded layer by layer. When the second memory space margin is detected to be greater than or equal to the preset memory space margin threshold, the feedforward neural subnetwork weight parameters of the second type of feedforward neural subnetwork are loaded onto the first processor until the second type of feedforward neural subnetwork is loaded. The third memory space margin of the first processor is obtained, wherein the third memory space margin is less than the second memory space margin.

[0189] The third type of feedforward neural subnetwork is loaded layer by layer. When the third memory space margin is detected to be greater than or equal to the preset memory space margin threshold, the intelligent weight parameters of the third type of feedforward neural subnetwork are loaded onto the first processor and the fourth memory space margin of the first processor is obtained.

[0190] Based on the remaining space in the fourth video memory, the first type of feedforward neural subnetwork is loaded onto the first processor in the form of a copy.

[0191] In an optional embodiment of this application, the preloading module 32 is specifically used to: perform a space reserve detection operation based on the fourth video memory space reserve; when the fourth video memory space reserve is detected to be greater than or equal to the preset copy loading space reserve, then load each first type of feedforward neural subnetwork into the first processor in the form of a copy layer by layer until the video memory space reserve is less than the preset copy loading space reserve.

[0192] In an optional embodiment of this application, the first resource usage information includes the first computing resource information and the first storage resource information of the first processor, the second resource usage information includes the offload process overhead and idle rate of the second processor, and the memory usage information includes the free storage space of the memory. Accordingly, the load / unload module 34 is specifically used for: determining the number of samplings according to a preset time interval; obtaining the computing utilization rate, video memory utilization rate, and free memory space of the first processor based on the first computing resource information and the first storage resource information of the first processor at the sampling number; determining the calculation result of the second processor thread based on the offload thread and idle rate of the second processor at the sampling number; performing a comparison calculation of the weight parameters of the feedforward neural subnetwork based on the free storage space of the memory at the sampling number to obtain the memory information calculation result; and obtaining the first resource usage function value based on the computing utilization rate, video memory utilization rate, free memory space, the calculation result of the second processor thread, and the memory information calculation result of the first processor.

[0193] In an optional embodiment of this application, the second resource usage information includes the second computing resource information, second storage resource information, offload process overhead, and idle rate of the second processor. The memory usage information includes the free storage space of the memory. The load / unload module 34 is specifically used for: determining the number of samplings according to a preset time interval; obtaining the computing utilization rate, video memory utilization rate, and free memory space of the second processor based on the second computing resource information and second storage resource information of the second processor at the sampling number; determining the thread calculation result of the second processor based on the offloaded threads and idle rate of the second processor at the sampling number; performing a comparison calculation of the weight parameters of the feedforward neural subnetwork based on the free storage space of the memory at the sampling number to obtain the memory information calculation result; and obtaining the second resource usage function value based on the computing utilization rate, video memory utilization rate, free memory space, thread calculation result, and memory information calculation result of the second processor.

[0194] In an optional embodiment of this application, the loading / unloading module 34 is specifically used for: collecting graphics processing loading information of the feedforward neural subnetworks during operation to obtain the graphics processing loading information data to be calculated for each feedforward neural subnetwork on the first processor; calculating the graphics unloading benefit ratio corresponding to each feedforward neural subnetwork based on the graphics processing loading information data to be calculated; selecting the feedforward neural subnetwork with the largest graphics processing benefit ratio as the feedforward neural subnetwork to be unloaded based on the graphics unloading benefit ratio corresponding to each feedforward neural subnetwork; unloading the feedforward neural subnetwork to be unloaded to the second processor, and unloading the weight parameters of the feedforward neural subnetworks located on the remaining first processors of the feedforward neural subnetwork to be unloaded.

[0195] In an optional embodiment of this application, the loading / unloading module 34 is further configured to: when the value of the first resource usage function is detected to be less than the lower bound of the first predefined resource threshold, traverse the feedforward neural subnetwork weight parameter loading information of the second processor; determine whether there is a feedforward neural subnetwork in the second processor that needs to be loaded into the first processor based on the feedforward neural subnetwork weight parameter loading information; when it is detected that there is a feedforward neural subnetwork in the second processor that needs to be loaded into the first processor, load the feedforward neural subnetwork in operation from the second processor to the first processor.

[0196] In an optional embodiment of this application, the loading / unloading module 34 is further specifically configured to: collect central processing loading information of the feedforward neural subnetworks during operation to obtain the first central processing loading information data to be calculated for each feedforward neural subnetwork on the second processor; calculate the first central processing benefit ratio corresponding to each feedforward neural subnetwork based on the first central processing loading information data to be calculated; select the feedforward neural subnetwork with the largest first central processing benefit ratio as the feedforward neural subnetwork to be unloaded based on the first central processing benefit ratio corresponding to each feedforward neural subnetwork; load the feedforward neural subnetwork to be unloaded from the second processor to the first processor in a preset tensor parallel manner, and load the feedforward neural subnetwork weight parameters of the feedforward neural subnetwork to be unloaded to other first processors.

[0197] In an optional embodiment of this application, the feedforward neural network loading weight information includes the scale of unloaded feedforward neural network weight parameters, feedforward neural network unloading frequency, feedforward neural network unloading time, feedforward neural network dwell time, feedforward neural network activation frequency, feedforward neural network hit rate, feedforward neural network loading frequency, feedforward neural network loading time, and hardware computing power improvement; correspondingly, the loading / unloading module 34 is further specifically used to: based on the feedforward neural network loading weight information including the scale of unloaded feedforward neural network weight parameters... The feedforward neural network unloading benefit ratio is determined based on the following parameters: module, feedforward neural network unloading frequency, feedforward neural network unloading time, feedforward neural network dwell time, feedforward neural network activation frequency, and feedforward neural network hit rate. The loading benefit ratio is determined based on the hardware computing power improvement, feedforward neural network loading frequency, feedforward neural network activation frequency, feedforward neural network hit rate, feedforward neural network loading time, and feedforward neural network dwell time. The loading benefit ratio is used to determine whether there is a feedforward neural network in the second processor that needs to be loaded into the first processor.

[0198] In an optional embodiment of this application, the loading / unloading module 34 is further specifically used for: when it is detected that the second processor does not have a feedforward neural subnetwork that needs to be loaded into the first processor, then detecting whether there is a feedforward neural subnetwork in the memory that needs to be loaded into the first processor; when it is detected that there is a feedforward neural subnetwork in the memory that needs to be loaded into the first processor, loading the feedforward neural subnetwork in operation from the memory into the first processor.

[0199] In an optional embodiment of this application, the loading / unloading module 34 is further specifically configured to: determine whether to trigger unloading when the second resource usage function value is detected to be less than the lower bound of the second predefined resource usage function value threshold; and unload the feedforward neural subnetwork from the second processor to the memory when unloading is detected.

[0200] In an optional embodiment of this application, the loading / unloading module 34 is further specifically configured to: acquire the second central processing load information data to be calculated for each feedforward neural subnetwork on the second processor; calculate the second central unloading benefit ratio corresponding to each feedforward neural subnetwork based on the second central processing load information data to be calculated; and select the feedforward neural subnetwork with the largest second central processing benefit ratio to be unloaded to the memory based on the second central unloading benefit ratio corresponding to each feedforward neural subnetwork.

[0201] In an optional embodiment of this application, the preloading module 32 is further configured to: when the model is detected to be running for the first time, if the initial video memory space of the first processor during the first run is greater than a preset existing memory space reserve threshold, load each shared feedforward neural subnetwork to the first processor layer by layer until each shared feedforward neural subnetwork is loaded and the intermediate video memory space of the first processor is obtained; if the intermediate video memory space is detected to be greater than a preset video memory space reserve threshold, load each routing feedforward neural subnetwork to the first processor layer by layer until each routing feedforward neural subnetwork is added to the first processor.

[0202] For a description of the features in the embodiments corresponding to the above-mentioned model loading and unloading device, please refer to the relevant descriptions in the embodiments corresponding to the model loading and unloading method, which will not be repeated here.

[0203] Figure 4 A schematic diagram of the structure of the electronic device provided in this application. Figure 4 As shown, the electronic device 40 provided in this embodiment includes at least one processor 401 and a memory 402. Optionally, the electronic device 40 further includes a communication component 403. The processor 401, memory 402, and communication component 403 are connected via a bus.

[0204] In the specific implementation process, at least one processor 401 executes computer execution instructions stored in memory 402, causing at least one processor 401 to execute the above-described model loading and unloading method embodiment.

[0205] The specific implementation process of processor 401 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0206] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0207] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0208] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0209] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above-described model loading / unloading method embodiments at runtime.

[0210] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), removable storage device, magnetic disk, or optical disk.

[0211] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described model loading / unloading method embodiments.

[0212] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described model loading / unloading method embodiments.

[0213] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0214] The above provides a detailed description of a model loading / unloading method, storage medium, and product provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only intended to help understand the method and core ideas of this application. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A model loading and unloading method characterized by, The model comprises a plurality of feedforward neural subnetworks, and the method comprises: obtaining historical running process data of each feedforward neural subnetwork of the model; loading each feedforward neural subnetwork in a preset order layer by layer onto a preset hardware according to the historical running process data, wherein the preset hardware comprises a first processor, a second processor and a memory; during the running process of the model, obtaining model information, first resource usage information of the first processor, second resource usage information of the second processor, memory usage information and feedforward neural subnetwork weight parameter scheduling information in real time according to a preset time interval; performing first calculation on the first resource usage information, the second resource usage information and the memory usage information according to the preset time interval to obtain a first resource usage function value; when it is detected that the first resource usage function value is greater than an upper limit of a first predefined resource threshold, performing second calculation on the second resource usage information and the memory usage information according to the preset time interval to obtain a second resource usage function value; when it is detected that the second resource usage function value is less than a lower limit of a second predefined resource usage function value threshold, unloading the feedforward neural subnetwork in the running process from the first processor to the second processor; the unloading of the feedforward neural subnetwork in the running process from the first processor to the second processor comprises: performing graph processing loading information collection on the feedforward neural subnetwork in the running process to obtain to-be-calculated graph processing loading information data of each feedforward neural subnetwork on the first processor; calculating a graph unloading benefit ratio corresponding to each feedforward neural subnetwork according to the to-be-calculated graph processing loading information data; selecting a feedforward neural subnetwork corresponding to a graph processing benefit ratio with the largest value among the graph unloading benefit ratios corresponding to the feedforward neural subnetworks as a to-be-unloaded feedforward neural subnetwork according to the graph unloading benefit ratios corresponding to the feedforward neural subnetworks; unloading the to-be-unloaded feedforward neural subnetwork to the second processor and unloading feedforward neural subnetwork weight parameters of the to-be-unloaded feedforward neural subnetwork on the remaining first processors; wherein a calculation formula of the feedforward neural subnetwork unloading benefit ratio is: unloading benefit ratio = (unloaded feedforward neural subnetwork weight parameter size x feedforward neural subnetwork unloading frequency) / ((feedforward neural subnetwork unloading time + δ x expert residence time) x feedforward neural subnetwork activation frequency x feedforward neural subnetwork hit rate); in the formula, δ is a preset coefficient value.

2. The method of claim 1, wherein, the loading of each feedforward neural subnetwork in a preset order layer by layer onto a preset hardware according to the historical running process data comprises: determining an activation frequency of each feedforward neural subnetwork according to the historical running process data; dividing each feedforward neural subnetwork into a first type of feedforward neural subnetwork, a second type of feedforward neural subnetwork and a third type of feedforward neural subnetwork in order of decreasing activation frequency according to the activation frequency; loading the first type of feedforward neural subnetwork layer by layer, when it is detected that the first memory space margin of the first processor is greater than or equal to a preset memory space margin threshold, then loading the feedforward neural subnetwork weight parameters of the first type of feedforward neural subnetwork onto the first processor until the first type of feedforward neural subnetwork is preloaded, and obtaining a second memory space margin of the first processor, wherein the second memory space margin is less than the first memory space margin; loading the second type of feedforward neural subnetwork layer by layer, when it is detected that the second memory space margin is greater than or equal to a preset memory space margin threshold, then loading the feedforward neural subnetwork weight parameters of the second type of feedforward neural subnetwork onto the first processor until the second type of feedforward neural subnetwork is loaded, and obtaining a third memory space margin of the first processor, wherein the third memory space margin is less than the second memory space margin; loading the third type of feedforward neural subnetwork layer by layer, when it is detected that the third memory space margin is greater than or equal to a preset memory space margin threshold, then loading the intelligent weight parameters of the third type of feedforward neural subnetwork onto the first processor and obtaining a fourth memory space margin of the first processor; loading the first type of feedforward neural subnetwork in the form of a copy onto the first processor according to the fourth memory space margin.

3. The method of claim 2, wherein, The loading of the first type of feedforward neural subnetwork in the form of a copy onto the first processor according to the fourth memory space margin comprises: performing a space margin detection operation according to the fourth memory space margin, when it is detected that the fourth memory space margin is greater than or equal to a preset copy loading space margin, then loading each first type of feedforward neural subnetwork in the form of a copy onto the first processor layer by layer until the memory space margin is less than the preset copy loading space margin.

4. The method of claim 1, wherein, The first resource usage information includes first computing resource information and first storage resource information of the first processor, the second resource usage information includes offloading trip overhead and idle rate of the second processor, and the memory usage information includes idle memory space; Accordingly, the first calculation of the first resource usage information, the second resource usage information and the memory usage information according to the preset time interval to obtain the first resource usage function value comprises: determining the sampling number according to the preset time interval; obtaining the computing utilization rate, the memory utilization rate and the idle memory space of the first processor according to the first computing resource information and the first storage resource information of the first processor under the sampling number; determining the second processor thread calculation result according to the offloading thread of the second processor and the idle rate under the sampling number; performing a feedforward neural subnetwork weight parameter comparison calculation according to the idle memory space of the memory under the sampling number to obtain a memory information calculation result; and performing a feedforward neural subnetwork weight parameter comparison calculation according to the idle memory space of the memory under the sampling number to obtain a memory information calculation result. According to the calculation utilization rate, the video memory utilization rate, the idle memory space of the first processor, the second processor thread calculation result and the memory information calculation result, a first resource usage function value is obtained.

5. The method of claim 1, wherein, The second resource usage information includes second computing resource information, second storage resource information, offloading trip overhead and idle rate of the second processor, and the memory usage information includes memory idle storage space; Accordingly, the second resource usage information and the memory usage information are calculated according to the preset time interval to obtain a second resource usage function value, including: According to the preset time interval, the sampling number is determined; According to the second computing resource information and the second storage resource information of the second processor, the calculation utilization rate, the video memory utilization rate and the idle memory space of the second processor are obtained under the sampling number; According to the offloading thread and the idle rate of the second processor, the thread calculation result of the second processor is determined under the sampling number; According to the memory idle storage space, the memory information calculation result is obtained by forward neural subnetwork weight parameter comparison calculation under the sampling number; According to the calculation utilization rate, the video memory utilization rate, the idle memory space, the thread calculation result of the second processor and the memory information calculation result, a second resource usage function value is obtained.

6. The method of claim 1, wherein, Also includes: When it is detected that the first resource usage function value is less than the first predefined resource threshold, the forward neural subnetwork weight parameter loading information of the second processor is traversed; According to the forward neural subnetwork weight parameter loading information, it is judged whether the second processor has a forward neural subnetwork that needs to be loaded into the first processor; When it is detected that the second processor has a forward neural subnetwork that needs to be loaded into the first processor, the running forward neural subnetwork is loaded from the second processor to the first processor.

7. The method of claim 6, wherein, Loading the running forward neural subnetwork from the second processor to the first processor includes: The running forward neural subnetwork is collected by central processing loading information to obtain the first to-be-calculated central processing loading information data of each forward neural subnetwork on the second processor; According to the first to-be-calculated central processing loading information data, the first central processing benefit ratio corresponding to each forward neural subnetwork is obtained by calculation; According to the first central processing benefit ratio corresponding to each forward neural subnetwork, the forward neural subnetwork corresponding to the first central processing benefit ratio with the largest value is selected as the to-be-offloaded forward neural subnetwork; The to-be-offloaded forward neural subnetwork is loaded from the second processor to the first processor in a preset tensor parallel manner, and the forward neural subnetwork weight parameters of the to-be-offloaded forward neural subnetwork are loaded to other first processors.

8. The method of claim 6, wherein, The front-end neural sub-network loading weight loading information includes unloaded front-end neural sub-network weight parameter size, front-end neural sub-network unloading frequency, front-end neural sub-network unloading time, front-end neural sub-network residence time, front-end neural sub-network activation frequency, front-end neural sub-network hit rate, front-end neural sub-network loading frequency, front-end neural sub-network loading time, and hardware computing power improvement; The judgment of whether the second processor has front-end neural sub-networks that need to be loaded into the first processor according to the front-end neural sub-network weight parameter loading information includes: According to the front-end neural sub-network loading weight loading information including unloaded front-end neural sub-network weight parameter size, front-end neural sub-network unloading frequency, front-end neural sub-network unloading time, front-end neural sub-network residence time, front-end neural sub-network activation frequency, and front-end neural sub-network hit rate, the front-end neural sub-network unloading benefit ratio is determined; According to the hardware computing power improvement, the front-end neural sub-network loading frequency, the front-end neural sub-network activation frequency, the front-end neural sub-network hit rate, the front-end neural sub-network loading time, and the front-end neural sub-network residence time, the loading benefit ratio is determined; According to the loading benefit ratio, it is judged whether the second processor has front-end neural sub-networks that need to be loaded into the first processor.

9. The method of claim 6, wherein, After the judgment of whether the second processor has front-end neural sub-networks that need to be loaded into the first processor according to the front-end neural sub-network weight parameter loading information, it further includes: When it is detected that the second processor does not have front-end neural sub-networks that need to be loaded into the first processor, it is detected whether the front-end neural sub-networks in the memory have front-end neural sub-networks that need to be loaded into the first processor; When it is detected that the memory has front-end neural sub-networks that need to be loaded into the first processor, the front-end neural sub-networks in the running process are loaded from the memory to the first processor.

10. The method according to any one of claims 1 to 9, characterized in that, It further includes: When it is detected that the second resource usage function value is less than the lower limit of the second predefined resource usage function value threshold, it is judged whether to trigger unloading; When it is detected that unloading is triggered, the front-end neural sub-networks are unloaded from the second processor to the memory.

11. The method of claim 10, wherein, Unloading the front-end neural sub-networks from the second processor to the memory includes: Obtaining second to-be-computed central processing loading information data of each front-end neural sub-network on the second processor; According to the second to-be-computed central processing loading information data, the second central unloading benefit ratio corresponding to each front-end neural sub-network is obtained by calculation; According to the second central unloading benefit ratio corresponding to each front-end neural sub-network, the front-end neural sub-network corresponding to the second central processing benefit ratio with the largest value is selected and unloaded to the memory.

12. The method of claim 2, wherein, The method further includes: When detecting that the model is first run, in a case that an initial video memory space of the first processor at the first run is greater than a preset video memory space margin threshold, each shared feedforward neural subnetwork is loaded to the first processor layer by layer until each shared feedforward neural subnetwork is loaded completely and an intermediate video memory space of the first processor is obtained; In a case that the intermediate video memory space is greater than the preset video memory space margin threshold, each routing feedforward neural subnetwork is loaded to the first processor layer by layer until each routing feedforward neural subnetwork is loaded to the first processor.

13. An electronic device, comprising: The method comprises: a memory for storing a computer program; a processor for executing the computer program to implement the steps of the model loading and unloading method according to any one of claims 1 to 12.

Images