Model loading and unloading method and electronic equipment

Abstract

The invention discloses a model loading and unloading method and electronic equipment, and relates to the technical field of server resource management.The method comprises the steps that feedforward neural sub-networks are preloaded to preset hardware layer by layer according to a preset sequence and historical operation process data; performing first calculation on the first resource use information, the second resource use information and the memory use information according to a preset time interval to obtain a first resource use function value; performing second calculation on the second resource use information and the memory use information according to a preset time interval to obtain a second resource use function value; when it is detected that the second resource use function value is smaller than a second predefined resource use function value threshold value lower bound, the feedforward neural sub-network in the running process is unloaded from the first processor to the second processor. According to the invention, the problem of low model feedback efficiency caused by unbalanced distribution of each sub-network in each hardware of the server in the model in the related technology is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of server resource management, and in particular to a model loading and unloading method and an electronic device. BACKGROUND

[0002] With the rapid popularization of large language models (LLMs) and other models in human-computer dialogue, content generation, and other scenarios, the computing and storage requirements thereof are also growing exponentially. In order to continuously improve user experience, how to dynamically load and unload models into corresponding server hardware has become a key problem in the industry.

[0003] In related technologies, a subnetwork in a corresponding model is loaded into corresponding hardware of a server through user configuration customization to improve feedback efficiency of the model. However, a more accurate and reasonable model loading and unloading method is still needed to solve the problem of uneven distribution of subnetworks in the model among the hardware of the server, which leads to low feedback efficiency of the model. SUMMARY

[0004] Embodiments of the present application provide a model loading and unloading method and an electronic device to at least solve the problem of low feedback efficiency of a model caused by uneven distribution of subnetworks in the model among the hardware of a server in related technologies.

[0005] The present application provides a model loading and unloading method, wherein the model includes a plurality of feedforward neural subnetworks, and the method includes:

[0006] obtaining historical running process data of each feedforward neural subnetwork of the model at runtime;

[0007] loading each feedforward neural subnetwork in a preset order layer by layer onto a preset hardware according to the historical running process data, wherein the preset hardware includes a first processor, a second processor, and a memory;

[0008] during running of the model, obtaining model information, first resource usage information of the first processor, second resource usage information of the second processor, memory usage information, and feedforward neural subnetwork weight parameter scheduling information in real time according to a preset time interval;

[0009] performing first calculation on the first resource usage information, the second resource usage information, and the memory usage information according to the preset time interval to obtain a first resource usage function value;

[0010] when it is detected that the first resource usage function value is greater than an upper limit of a first predefined resource threshold, performing second calculation on the second resource usage information and the memory usage information according to the preset time interval to obtain a second resource usage function value;

[0011] when it is detected that the second resource usage function value is less than a second predefined resource usage function value threshold, offloading the running feedforward neural subnetwork from the first processor to the second processor.

[0012] The application further provides an electronic device, comprising:

[0013] a memory for storing a computer program;

[0014] a processor for implementing the steps of the model loading and offloading method as described above when executing the computer program.

[0015] According to the application, each feedforward neural subnetwork is preloaded onto the preset hardware in a preset order according to historical running process data, and during the running of the model, model information, first resource usage information of the first processor, second resource usage information of the second processor, memory usage information and feedforward neural subnetwork weight parameter scheduling information are acquired in real time according to a preset time interval, and then the first resource usage information, the second resource usage information and the memory usage information are used to determine the loading of the feedforward neural subnetwork according to the preset time interval. Through the preloading and this dynamic adaptive loading and offloading method, the reasonable distribution of the feedforward neural subnetwork weight parameters on each preset hardware resource is fine-tuned. Therefore, the technical problem of low feedback efficiency of the model caused by the uneven distribution of each feedforward neural subnetwork in the preset hardware of the server can be solved, and the technical effect of improving the more reasonable and uniform distribution of each feedforward neural subnetwork of the model in the preset hardware of the server is achieved, thereby improving the feedback efficiency of the model. BRIEF DESCRIPTION OF DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the application, the drawings needed in the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor.

[0017] Figure 1 An application scenario diagram of the model loading and offloading method provided by the embodiments of the application;

[0018] Figure 2 A flowchart of the model loading and offloading method provided by the embodiments of the application;

[0019] Figure 3 A structural diagram of the model loading and offloading provided by the embodiments of the application;

[0020] Figure 4 A structural diagram of the electronic device provided by the application. DETAILED DESCRIPTION

[0021] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0022] It should be noted that, in the description of the present application, the terms "comprise", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device comprising a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. The terms "first", "second" and the like in the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence.

[0023] The term explanation related to the embodiments of the present application:

[0024] MOE (Mixture-of-Experts, mixed expert model) is a "divide and conquer" sparse neural network architecture, and the core idea is "divide the model into a group of small experts and activate them on demand". It makes the model grow in parameter size while the computational complexity grows linearly or even sub-linearly, becoming a key technology of current models.

[0025] LLM (Large Language Model, large language model) refers to a generative language model with a parameter size of more than one billion (Billion), based on the Transformer architecture, obtained by self-supervised pre-training of massive text. They can complete multiple tasks such as question answering, translation, summarization, code generation, and dialogue, and show "emergence" capabilities (such as context learning and logical reasoning).

[0026] FNN (Feedforward Neural Network, feedforward neural network) is a neural network with unidirectional information flow from input to output and no loop, which is the most "original" architecture of deep learning.

[0027] HBM (High Bandwidth Memory, a high-speed memory technology that vertically stacks multiple DRAM chips like a building, and then directly welds them next to the GPU using a super-wide bus). It solves the pain points of traditional GDDR, such as insufficient bandwidth, high power consumption, and large area, and becomes the "standard memory" of high-end GPUs in AI, HPC, and graphics rendering scenarios.

[0028] EP (Expert Parallelism) is a distributed parallel strategy specifically designed for sparse hybrid expert (MoE) models. It splits different experts onto different GPUs / nodes, then sends the tokens precisely to the devices with the target experts for computation based on the routing results, and finally collects the results.

[0029] Tensor Parallelism (TP) is a distributed training / inference strategy that "vertically" splits a single matrix / tensor across multiple GPUs, allowing each GPU to compute only a sub-block of the entire tensor, thereby "breaking down" the hypermodel parameters and packing them into limited GPU memory.

[0030] GPU: Graphics Processing Unit.

[0031] CPU: Central Processing Unit.

[0032] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0033] This section describes the specific application environment architecture or hardware architecture that the model loading / unloading methods depend on. (References) Figure 1 , Figure 1 This diagram illustrates an application scenario for a model loading / unloading method provided in an embodiment of this application. Figure 1 As shown, it includes: terminal 101 and server 102.

[0034] Terminal 101 is used for manual operations such as starting the system in server 102 and opening models. Terminal 101 may include devices such as a mouse, keyboard, or touch screen that can generate corresponding instructions in response to user manual operations and transmit the instructions to server 102. Server 102 is used to receive instructions submitted from terminal 101 and to obtain models from other servers or the cloud, or to directly add or unload installed models, and to execute model addition and unloading methods to complete the dynamic adaptive addition and unloading of each feedforward neural subnetwork in the model.

[0035] Figure 2 This is a flowchart illustrating the model loading and unloading method provided in the embodiments of this application, as shown below. Figure 2 As shown, the embodiments of this application provide a model loading and unloading method. The execution subject of this embodiment can be... Figure 1The server 102 in the illustrated embodiment can be a physical server or a cloud server, or other computer-related devices, and the present embodiment is not particularly limited herein. The method is described in detail as follows.

[0036] S201: Obtain historical running process data of each feedforward neural subnetwork in the model.

[0037] In the present embodiment, the model includes multiple feedforward neural subnetworks, for example, the model can be the MOE model in the foregoing, and the feedforward neural subnetworks can be the experts in the foregoing. When the system corresponding to the model is started for the first time, the running process data of each feedforward neural subnetwork in the model inference process is recorded as historical running process data.

[0038] In an optional embodiment of the present application, the historical running process data can include the following three parts: model information model_info indicating the use, hardware environment information hardware_info indicating the use of the model, and expert active use information expert_active_info used to describe the inference running process.

[0039] The model_info includes the type of the model (such as MoE, DeepSeek, etc.), the model precision (such as Int8, FP16, etc.), the number of dense layers and expert layers, the parameter size of each dense layer, the number of routing experts and shared experts on each expert layer, and the parameter size on each expert. Experts and inference processes on different models are different, so the loading method is also different; in addition, the parameter size of the dense layer and the expert layer directly affects the loading of the expert weight in the CPU and GPU memory / video memory.

[0040] The hardware_info can include two types of computing units (CPU and GPU) and storage units (GPU video memory vRAM, CPU memory RAM, and storage), and in addition, the hardware usage in the model inference process needs to be counted from the average usage and the maximum and minimum usage, for guiding the expert weight loading in the inference process.

[0041] expert_active_info describes the expert activation usage in the inference running process, including expert ID and activation frequency, wherein the expert ID is identified by layer number layer_idx and sequence number expert_idx of the expert in the layer, and the activation frequency represents the hot degree of the expert, including high-frequency experts (shared experts and routing experts with an activation probability greater than 90%), medium-frequency experts (routing experts with an activation probability greater than 10% and less than 90%), and low-frequency experts (routing experts with an activation probability less than 10%). The higher the hot degree of the expert, the greater the possibility of being activated, so in principle, the expert should be preferentially loaded onto the GPU. Here, the expert refers to a feedforward neural subnetwork.

[0042] S202: According to the historical running process data, each feedforward neural subnetwork is preloaded on the preset hardware in a preset order, wherein the preset hardware includes a first processor, a second processor and a memory.

[0043] In this embodiment, the activation frequency data of each feedforward neural network can be extracted from the historical running process data, and each feedforward neural network is sorted according to the activation frequency of each feedforward neural network, and then loaded on the corresponding preset hardware in a preset order from the first processor to the second processor to the memory. During the loading process, it is also necessary to monitor the remaining video memory space of the image processor in real time. When it is sufficient, it is loaded into the second processor, otherwise it is continuously loaded into the first processor. In this embodiment, the first processor can be a graphics processing unit (GPU) or other processor with image processing function. The second processor can be a central processing unit (CPU) or other processor with data processing function. The memory can be a hard disk or other hardware device with data storage function. For this, this embodiment does not make special limitation.

[0044] In an optional embodiment of the present application, step S202 specifically includes:

[0045] S202a: According to the historical running data, the activation frequency of each feedforward neural subnetwork is determined.

[0046] In this embodiment, the activation frequency of each feedforward neural subnetwork can be determined by matching the corresponding relationship between each feedforward neural subnetwork and the activation frequency from the historical running data by data pre-processing.

[0047] S202b: According to the activation frequency, each feedforward neural subnetwork is divided into a first type of feedforward neural subnetwork, a second type of feedforward neural subnetwork and a third type of feedforward neural subnetwork in order of activation frequency from high to low.

[0048] In this embodiment, the first type of feedforward neural subnetwork has the highest activation frequency and can be regarded as a hot feedforward neural subnetwork. The third type of feedforward neural subnetwork has the lowest activation frequency and can be regarded as a cold feedforward neural subnetwork. The lower the activation frequency, the smaller the role of the feedforward neural subnetwork in model inference. Therefore, the first type of feedforward neural subnetwork with high activation frequency is preferentially loaded into the first processor, so as to be called at any time for inference and to generate feedback required by the user.

[0049] S202c: The first type of feedforward neural subnetwork is loaded layer by layer. When it is detected that the first memory space margin of the first processor is greater than or equal to the preset memory space margin threshold, the feedforward neural subnetwork weight parameters of the first type of feedforward neural subnetwork are loaded onto the first processor until the first type of feedforward neural subnetwork is preloaded, and the second memory space margin of the first processor is obtained, wherein the second memory space margin is less than the first memory space margin.

[0050] In this embodiment, the first memory space margin refers to the remaining memory space size of the first processor recorded by the system when the feedforward neural subnetwork is loaded into the first processor at the beginning. The preset memory space margin threshold can be the memory space size required for loading the first type of feedforward neural subnetwork calculated according to the first type of feedforward neural subnetwork. When the first memory space margin of the first processor is less than the preset memory space margin threshold, it indicates that the first processor cannot complete the task of loading a complete first type of feedforward neural subnetwork. At this time, the first type of feedforward neural subnetwork to be loaded can be preloaded into the second processor.

[0051] In this embodiment, when it is detected that the first memory space margin of the first processor is greater than or equal to the preset memory space margin threshold, at least one first type of feedforward neural subnetwork in the remaining first type of feedforward neural subnetwork is still loaded into the first processor. In this embodiment, the first type of feedforward neural subnetwork with high activation frequency is preferentially loaded in order to improve the inference efficiency of the model. After the first type of feedforward neural subnetwork is loaded into the first processor, the system monitors and records the remaining memory space size of the corresponding image processor as the second memory space margin.

[0052] S202d: The second type of feedforward neural subnetwork is loaded layer by layer. When it is detected that the second memory space margin is greater than or equal to the preset memory space margin threshold, the feedforward neural subnetwork weight parameters of the second type of feedforward neural subnetwork are loaded onto the first processor until the second type of feedforward neural subnetwork is loaded, and the third memory space margin of the first processor is obtained, wherein the third memory space margin is less than the second memory space margin.

[0053] In this embodiment, the preset GPU space margin threshold value can be the GPU space size required for loading the second type of feedforward neural subnetwork calculated according to the second type of feedforward neural subnetwork. When the second GPU space margin of the first processor is less than the preset GPU space margin threshold value, it indicates that the first processor cannot complete the task of loading a complete second type of feedforward neural subnetwork. At this time, the second type of feedforward neural subnetwork to be loaded can be preloaded into the second processor.

[0054] In this embodiment, when it is detected that the second GPU space margin of the first processor is greater than or equal to the preset GPU space margin threshold value, it indicates that at least one of the remaining second type of feedforward neural subnetworks is still loaded into the first processor. In this embodiment, the first type of feedforward neural subnetwork with a high activation frequency is preferentially loaded to improve the inference efficiency of the model. After all the first type of feedforward neural subnetworks are loaded into the first processor, the system monitors and records the remaining GPU space size of the corresponding image processor as a third GPU space margin.

[0055] S202e: Layer by layer loading the third type of feedforward neural subnetwork, when detecting that the third GPU space margin is greater than or equal to the preset GPU space margin threshold value, loading the intelligent weight parameters of the third type of feedforward neural subnetwork into the first processor and obtaining a fourth GPU space margin of the first processor.

[0056] In this embodiment, the preset GPU space margin threshold value can be the GPU space size required for loading the third type of feedforward neural subnetwork calculated according to the third type of feedforward neural subnetwork. When the third GPU space margin of the first processor is less than the preset GPU space margin threshold value, it indicates that the first processor cannot complete the task of loading a complete third type of feedforward neural subnetwork. At this time, the third type of feedforward neural subnetwork to be loaded can be preloaded into the second processor.

[0057] In this embodiment, when it is detected that the third GPU space margin of the first processor is greater than or equal to the preset GPU space margin threshold value, it indicates that at least one of the remaining third type of feedforward neural subnetworks is still loaded into the first processor. In this embodiment, the first type of feedforward neural subnetwork with a high activation frequency is preferentially loaded to improve the inference efficiency of the model. After all the first type of feedforward neural subnetworks are loaded into the first processor, the system monitors and records the remaining GPU space size of the corresponding image processor as a fourth GPU space margin.

[0058] S202f: Loading the first type of feedforward neural subnetwork in the form of a copy into the first processor according to the fourth GPU space margin.

[0059] In this embodiment, the fourth memory space margin can be compared with the preset memory space margin threshold corresponding to the first processor. When the fourth memory space margin is greater than or equal to the preset memory space margin threshold, in order to further improve the inference efficiency of the model, the first type of feedforward neural network with the highest activation frequency is loaded into the first processor in the form of a copy, so that there are more first type of feedforward neural networks in the first processor that can play a greater role in model inference. On the contrary, when the fourth memory space margin is less than the preset memory space margin threshold, the copy of the first type of feedforward neural network is no longer loaded.

[0060] Based on the above embodiment, in an optional embodiment of the present application, step S202f includes: performing a space margin detection operation according to the fourth memory space margin, and when it is detected that the fourth memory space margin is greater than or equal to a preset copy loading space margin, each first type of feedforward neural network is loaded into the first processor in the form of a copy layer by layer until the memory space margin is less than the preset copy loading space margin.

[0061] In this embodiment, the preset copy loading space margin can be equal to the preset memory space margin threshold, or can be calculated by the system according to the preloading space size required by the copy of the first type of feedforward neural network.

[0062] In this embodiment, loading each first type of feedforward neural network into the first processor in the form of a copy layer by layer can be to continue to load each first type of feedforward neural network into the first processor in the order of the activation frequency in the form of a first copy, a second copy, a third copy, and so on. In the preloading process, it is judged in real time whether the memory space on the first processor is sufficient, until there is no extra memory space on the first processor, so as to maximize the use of the memory resources of the first processor to load as many first type of feedforward neural networks as possible.

[0063] In an optional embodiment of the present application, the judgment standard for judging whether the memory space on the first processor is sufficient is defined. In this embodiment, for a heterogeneous computing system with both a second processor and a first processor, in order to improve resource utilization and inference performance, an expert parallel EP and a tensor parallel TP hybrid method is used for preloading the weight parameters of the feedforward neural network, wherein the feedforward neural network includes a dense layer, and the loading method of the dense layer weight parameters can also use the tensor parallel method.

[0064] The specific process is as follows:

[0065] First, the model parameter size of the dense layer occupies the hardware memory or memory size, which is defined as:

[0066]

[0067] In the formula, dense_layer_memory_usage represents the total memory or video memory size occupied by the dense layer, dense_layer_num is the number of dense layers, dense_layer_parameters[k] represents the number of parameters of the kth dense layer, and precision is the preset weight parameter precision.

[0068] The memory or video memory size occupied by each feedforward neural network weight parameter is defined as:

[0069] per_expert_memory_usage = expert_parameters × (precision / 8)

[0070] In the formula, per_expert_memory_usage is the memory or video memory size occupied by each feedforward neural network weight parameter, expert_parameters is the number of feedforward neural network weight parameters in a single feedforward neural network, and precision is the preset weight parameter precision.

[0071] To ensure the inference service quality of the model, the embodiment defines that, in addition to loading the weight parameters of the dense layer, at least 80% of the space on the HBM of each first processor is required to store KV cache data. Therefore, for each first processor such as a graphics processing unit (GPU), the video memory space size of the first processor reserved for loading the feedforward neural network weight parameters is:

[0072] HBM_reserved_size = (HBM_size – dense_layer_data_size) ×20%

[0073] In the formula, HBM_size represents the total video memory capacity of the HBM, and dense_layer_data_size represents the video memory size of the HBM used for loading the dense layer parameters. During the loading process, the current remaining space HBM_reserved_now_size on the first processor video memory is often less than HBM_reserved_size, because the weight slices of the loaded feedforward neural network occupy space.

[0074] Because the tensor parallel loading strategy is adopted, the parameters of the dense layer and the expert weights in the model are sliced and loaded onto different first processors. In a server system with N first processor cards, the video memory size occupied by the dense layer weight parameters and each feedforward neural network weight parameter on each first processor is:

[0075] dense_layer_data_size = dense_layer_memory_usage / N

[0076] expert_data_slice_min = per_expert_memory_usage / N

[0077] It should be noted that the above formula expert_data_slice_min defines the size of the slice occupied by the expert weight distribution to all first processors in the ideal case, at which time each slice is the smallest. However, in the actual running process, since the free video memory on each first processor card is not necessarily the same, the feedforward neural network weight parameter slice can only be placed on the GPU that can store the slice. Assuming that there are k (k < N) such first processor GPUs in the system, then the size of the slice stored on each first processor GPU is per_expert_memory_usage / k.

[0078] Then, whether the video memory resource on a certain first processor is sufficient can be divided into the following cases:

[0079] a) If the HBM_reserved_size on the first processor is less than or equal to 0, it means that the first processor has no additional video memory space for storing the feedforward neural network weight parameters except for loading the dense layer parameters and reserving for storing KV cache data, so the video memory resource on the first processor is insufficient.

[0080] b) If the HBM_reserved_size on the first processor is greater than 0, and HBM_reserved_size < expert_data_slice_min at the same time, it means that the first processor is not sufficient to load the tensor parallel parameters of the feedforward neural network weight except for loading the dense layer parameters, so the video memory resource on the first processor is insufficient.

[0081] c) If HBM_reserved_size on the first processor is greater than 0, and HBM_reserved_size > expert_data_slice_min, it means that the first processor is sufficient to load the tensor parallel parameters of the expert weight (i.e. sufficient) in addition to loading the dense layer parameters. But at this time, whether the first processor memory can load the expert weight also needs to be judged in combination with the use of the memory of other first processors in the system. Because when the slice of the forward neural network weight parameter is loaded, two conditions need to be met at the same time: there are k first processors with sufficient remaining space in the system, and the remaining space of these first processors is greater than the slice of the expert weight; At the same time, k must satisfy that it is a power of 2, such as 2, 4, 8, 16, …… Because the number of slices of the forward neural network weight often needs to meet the form of the power of 2, which is jointly determined by the distributed communication and hardware optimization requirements. Through the slice distribution method based on tensor parallelism under the first processor memory, it is judged whether the current preset hardware resources are sufficient to load the weight parameter slice of the forward neural network in the preloading process of the forward neural network, which further improves the utilization rate of hardware resources and improves the inference quality of the model.

[0082] S203: In the running process of the model, the model information, the first resource usage information of the first processor, the second resource usage information of the second processor, the memory usage information and the forward neural network weight parameter scheduling information are acquired in real time according to the preset time interval.

[0083] In this embodiment, in order to perform dynamic adaptive loading and unloading operation on the forward neural network in the model inference process, that is, the running process, the hardware resource usage in the running process of the server system and the scheduling situation of the forward neural network weight parameter need to be monitored in real time, so as to obtain the model information, the first resource usage information of the first processor, the second resource usage information of the second processor, the memory usage information and the forward neural network weight parameter scheduling information. The real-time monitoring here can be timing acquisition and recording based on the time interval of the loading and unloading operation of the forward neural network.

[0084] In this embodiment, in the running process of the model, the computing resources of the second processor and the first processor, and the memory of the second processor, the memory of the first processor and the storage resource of the memory are mainly involved. Among them, the memory is mainly used for storing the less frequently used forward neural network weight parameter when the memory of the second processor is insufficient. Because the memory stores the forward neural network expert weight parameter, it will affect the performance, so whether to store the memory needs to be considered.

[0085] In this embodiment, the current use of hardware resources during system operation is a necessary condition for triggering the feedforward neural subnetwork to perform load balancing operations. Whether to perform load balancing operations or not also depends on the actual use of the feedforward neural subnetwork weight parameters, and the trade-off between the cost of load balancing and the benefits brought by load balancing. In addition, these monitored expert scheduling information also has the effect of being input to the historical information statistical module, providing a basis for the preloading of the next feedforward neural subnetwork from the perspective of statistical characteristics of inference tasks in the use scenario, such as the activation frequency of the feedforward neural subnetwork.

[0086] S204: performing first calculation on the first resource usage information, the second resource usage information, and the memory usage information according to a preset time interval to obtain a first resource usage function value.

[0087] In an optional embodiment of the present application, the process of performing first calculation on the first resource usage information, the second resource usage information, and the memory usage information according to a preset time interval can be a process of calculating the first resource usage function value through a specific comparison function.

[0088] Specifically, in an optional embodiment of the present application, the first resource usage information includes first computing resource information and first storage resource information of the first processor, the second resource usage information includes thread offloading overhead and idle rate of the second processor, and the memory usage information includes memory idle storage space. Correspondingly, step S204 includes:

[0089] S204a: determining the sampling times according to the preset time interval.

[0090] In this embodiment, the preset actual interval can be a fixed time difference value set in advance, for example, the time nodes t1 and t2 are divided into a plurality of sampling time nodes by a fixed time difference value, and one sampling time node corresponds to one sampling time. For example, the sampling times in the time range from t1 to t2 can be represented as (t2-t1) / Δt, where Δt is the preset time interval.

[0091] S204b: obtaining the computing utilization rate, the video memory utilization rate, and the idle memory space of the first processor according to the first computing resource information and the first storage resource information of the first processor at the sampling times.

[0092] S204c: determining the second processor thread calculation result according to the second processor thread offloading and the idle rate at the sampling times.

[0093] S204d: performing feedforward neural subnetwork weight parameter comparison calculation according to the memory idle storage space at the sampling times to obtain a memory information calculation result.

[0094] S204e: obtaining a first resource usage function value according to the calculation utilization of the first processor, the video memory utilization, the idle memory space, the second processor thread calculation result and the memory information calculation result. In this embodiment, the first resource usage function value can be a specific value used for judging the utilization, the video memory utilization, the idle memory space, the second processor thread calculation result and the memory information calculation result.

[0095] S205: when it is detected that the first resource usage function value is greater than the first predefined resource threshold upper limit, performing a second calculation on the second resource usage information and the memory usage information according to a preset time interval to obtain a second resource usage function value.

[0096] In this embodiment, when it is detected that the first resource usage function value is greater than the first predefined resource threshold upper limit, it indicates that the hardware resource corresponding to the first processor is sufficient, and the feedforward neural subnetwork can be loaded, otherwise, it indicates that the hardware resource corresponding to the first processor is insufficient, and the feedforward neural subnetwork needs to be unloaded.

[0097] Specifically, in an optional embodiment of the present application, the second resource usage information includes the second computing resource information, the second storage resource information, the unloading trip overhead and the idle rate of the second processor, and the memory usage information includes the memory idle storage space. Accordingly, step S205 specifically includes:

[0098] S205a: determining the sampling number according to the preset time interval.

[0099] S205b: obtaining the calculation utilization, the video memory utilization and the idle memory space of the second processor according to the second computing resource information and the second storage resource information of the second processor under the sampling number.

[0100] S205c: determining the thread calculation result of the second processor according to the unloading thread and the idle rate of the second processor under the sampling number.

[0101] S205d: performing a feedforward neural subnetwork weight parameter comparison calculation according to the memory idle storage space under the sampling number to obtain a memory information calculation result.

[0102] S205e: obtaining a second resource usage function value according to the calculation utilization, the video memory utilization, the idle memory space, the thread calculation result and the memory information calculation result of the second processor.

[0103] In this embodiment, the specific calculation processes of steps S204a to S204e are similar to those of steps S205a to S205e, and thus will not be described here.

[0104] S206: When it is detected that the second resource usage function value is less than the second predefined resource usage function value threshold lower limit, the running feedforward neural subnetwork is offloaded from the first processor to the second processor.

[0105] In the embodiment, when it is detected that the second resource usage function value is less than the second predefined resource usage function value threshold lower limit, it indicates that the hardware resources corresponding to the second processor are sufficient, and the requirement of offloading the feedforward neural subnetwork from the first processor to the second processor can be triggered. Otherwise, it indicates that the feedforward neural subnetwork cannot be directly offloaded to the second processor, and needs to be offloaded to the memory through the second processor.

[0106] Based on the above embodiment, in an optional embodiment of the present application, the step S206 of offloading the running feedforward neural subnetwork from the first processor to the second processor comprises:

[0107] S206a: The running feedforward neural subnetwork is subjected to graph processing loading information collection to obtain to-be-calculated graph processing loading information data of each feedforward neural subnetwork on the first processor.

[0108] S206b: According to the to-be-calculated graph processing loading information data, the graph offloading benefit ratio corresponding to each feedforward neural subnetwork is calculated.

[0109] S206c: According to the graph offloading benefit ratio corresponding to each feedforward neural subnetwork, the feedforward neural subnetwork corresponding to the graph processing benefit ratio with the largest value is selected as a to-be-offloaded feedforward neural subnetwork.

[0110] S206d: The to-be-offloaded feedforward neural subnetwork is offloaded to the second processor, and the feedforward neural subnetwork weight parameters of the to-be-offloaded feedforward neural subnetwork on the remaining first processors are offloaded.

[0111] In summary, the model loading and unloading method provided in the embodiments of the present application first preloads each feedforward neural subnetwork to a preset hardware in a preset order according to historical running process data, and in the running process of the model, acquires model information, first resource usage information of the first processor, second resource usage information of the second processor, memory usage information and feedforward neural subnetwork weight parameter scheduling information in real time according to a preset time interval, and then judges the loading of the feedforward neural subnetwork according to the first resource usage information, the second resource usage information and the memory usage information in the preset time interval. Through the preloading and the dynamic adaptive loading and unloading method, the reasonable distribution of the feedforward neural subnetwork weight parameters on each preset hardware resource is fine-tuned. Therefore, the problem of low feedback efficiency of the model caused by the uneven distribution of each feedforward neural subnetwork in the preset hardware of the server can be solved, and the technical effect of improving the more reasonable and uniform distribution of each feedforward neural subnetwork of the model in the preset hardware of the server is achieved, thereby improving the feedback efficiency of the model.

[0112] Based on the above embodiments, in an optional embodiment of the present application, in step S204, the first predefined resource threshold upper limit can include a first predefined calculation utilization upper limit value WL GPU max and a predefined video memory utilization upper limit value WL HBM max of the first processor.

[0113] The detection of whether the first resource usage function value is greater than the first predefined resource threshold upper limit can be performed by using a preset first resource usage function f (GPU).

[0114] The sampling frequency in the time range from time node t1 to time node t2 is (t2-t1) / Δt, and each first processor resource usage function f (GPU) triggering the unloading operation needs to meet the following conditions at the same time:

[0115] 1) The proportion of the calculation utilization of the first processor exceeding WL GPU max is greater than the threshold upper limit, such as 95%;

[0116] 2) The proportion of the video memory utilization of the first processor exceeding WL HBM max is greater than the threshold upper limit, such as 90%;

[0117] 3) The second processor overhead of the feedforward neural subnetwork unloading thread is less than the idle rate of some other second processor.

[0118] 4) There is some second processor whose idle memory space is greater than the expert weight parameter.

[0119] Correspondingly, it is detected whether the first resource usage function value is less than a first predefined resource threshold lower limit, wherein the first predefined resource threshold lower limit can include a predefined calculation utilization lower limit value WL GPU min and a predefined video memory utilization lower limit value WL HBM min of the first processor, and the detection process can be represented as that a sampling number in a time range from a time node t1 to a time node t2 is (t2-t1) / Δt, and the resource usage function f(GPU) of each first processor triggers the loading operation to meet the following conditions at the same time:

[0120] 1) a proportion of the calculation utilization of the first processor being less than the WL GPU min is greater than a threshold upper limit such as 95%;

[0121] 2) a proportion of the video memory utilization of the first processor being greater than the WL HBM min is greater than a threshold upper limit such as 90%;

[0122] 3) the idle video memory space of the first processor meets the slice loading feedforward neural network weight parameter.

[0123] 4) the overhead of the second processor of the feedforward neural network loading thread is less than the feedforward neural network weight parameter.

[0124] Based on the above embodiment, in an optional embodiment of the present application, in step S205, the second predefined resource threshold upper limit can include a predefined calculation utilization upper limit value WL CPU max and a predefined memory utilization upper limit value WL RAM max of the second processor.

[0125] The detection of whether the second resource usage function value is greater than the second predefined resource threshold upper limit can be performed by using a preset second resource usage function f(CPU).

[0126] In a time range from a time node t1 to a time node t2, a sampling number is (t2-t1) / Δt, and the resource usage function f(CPU) of each second processor triggers the unloading operation to meet the following conditions at the same time:

[0127] 1) a proportion of the calculation utilization of the second processor being greater than the WL CPU max is greater than a threshold upper limit such as 95%;

[0128] 2) a proportion of the memory utilization of the second processor being greater than the WL RAM max is greater than a threshold upper limit such as 90%;

[0129] 3) the overhead of the second processor of the feedforward neural network unloading thread is less than the idle rate of the current second processor.

[0130] 4) the idle memory space of the memory is greater than the feedforward neural network weight parameter.

[0131] Correspondingly, it is detected whether the second resource usage function value is less than a second predefined resource threshold lower limit, wherein the second predefined resource threshold lower limit can include a predefined calculation utilization lower limit value WL CPU min and a predefined memory utilization lower limit value WL RAM min of the second processor, and at this time the detection process can be represented as that the sampling times in the time range from time node t1 to time node t2 are (t2-t1) / Δt, and the resource usage function f(CPU) of each second processor triggers the loading operation to simultaneously satisfy the following conditions:

[0132] 1) The proportion of the calculation utilization of the second processor being lower than WL CPU min is less than a threshold upper limit such as 95%;

[0133] 2) The proportion of the video memory utilization of the second processor being higher than WL RAM min is greater than a threshold upper limit such as 90%;

[0134] 3) The idle memory of the second processor is greater than the feedforward neural subnetwork weight parameter.

[0135] 4) The overhead of the second processor of the feedforward neural subnetwork loading thread is less than the feedforward neural subnetwork weight parameter.

[0136] Based on the above embodiment, the model loading and unloading method provided in an optional embodiment of the application further includes:

[0137] Step A: when it is detected that the first resource usage function value is less than the first predefined resource threshold lower limit, the feedforward neural subnetwork weight parameter loading information of the second processor is traversed;

[0138] Step B: it is judged according to the feedforward neural subnetwork weight parameter loading information whether the second processor has the feedforward neural subnetwork that needs to be loaded into the first processor.

[0139] In an optional embodiment of the application, the feedforward neural subnetwork loading weight loading information includes the unloaded feedforward neural subnetwork weight parameter size, the feedforward neural subnetwork unloading frequency, the feedforward neural subnetwork unloading time, the feedforward neural subnetwork residence time, the feedforward neural subnetwork activation frequency, the feedforward neural subnetwork hit rate, the feedforward neural subnetwork loading frequency, the feedforward neural subnetwork loading time and the hardware computing power improvement; correspondingly, the step B of judging according to the feedforward neural subnetwork weight parameter loading information whether the second processor has the feedforward neural subnetwork that needs to be loaded into the first processor specifically includes:

[0140] Step B1: Load the information including the unloaded weight parameter size of the feedforward neural subnetwork, the feedforward neural subnetwork unloading frequency, the feedforward neural subnetwork unloading time, the feedforward neural subnetwork residence time, the feedforward neural subnetwork activation frequency, and the feedforward neural subnetwork hit rate according to the weight loading of the feedforward neural subnetwork, and determine the unloading benefit ratio of the feedforward neural subnetwork.

[0141] In this embodiment, the calculation formula for determining the unloading benefit ratio of the feedforward neural subnetwork is:

[0142] The unloading benefit ratio = (the unloaded weight parameter size of the feedforward neural subnetwork x the feedforward neural subnetwork unloading frequency) / ((the feedforward neural subnetwork unloading time + δ x the expert residence time) x the feedforward neural subnetwork activation frequency x the feedforward neural subnetwork hit rate). In the formula, δ is a pre-set coefficient value.

[0143] Step B2: Determine the loading benefit ratio according to the hardware computing power improvement, the feedforward neural subnetwork loading frequency, the feedforward neural subnetwork activation frequency, the feedforward neural subnetwork hit rate, the feedforward neural subnetwork loading time, and the feedforward neural subnetwork residence time.

[0144] In this embodiment, the calculation formula for determining the loading benefit ratio is:

[0145] The loading benefit ratio = (the hardware computing power improvement x the feedforward neural subnetwork loading frequency x the feedforward neural subnetwork activation frequency x the feedforward neural subnetwork hit rate) / ((the expert loading time + δ the feedforward neural subnetwork residence time)).

[0146] Step B3: Determine whether there is a feedforward neural subnetwork that needs to be loaded into the first processor according to the loading benefit ratio.

[0147] In this embodiment, within a given time range, the scheduling behavior of the loading or unloading of the feedforward neural subnetwork weight parameters is quantitatively measured from the aspects of cost and benefit based on the feedforward neural subnetwork weight parameter scheduling information obtained by model running. As can be seen from the above formula, the higher the hit rate and activation frequency of a feedforward neural subnetwork in the inference process, the more it should be loaded onto a high-power hardware such as the first processor for execution, and vice versa. The more time-consuming the loading or unloading operation of a feedforward neural subnetwork is, and the longer the feedforward neural subnetwork resides in the current hardware, the more it should remain unchanged, i.e., no loading or unloading operation is taken.

[0148] The more frequently a feedforward neural subnetwork is loaded, the more it should take loading operation; the more frequently a feedforward neural subnetwork is unloaded, the more it should take unloading operation. The computing power improvement brought by loading is the benefit obtained by loading, such as the computing power improvement from the memory to the first processor or the second processor is the corresponding hardware computing power improvement, and the computing power improvement of the second processor to the first processor is the difference between the computing power of the second processor and the computing power of the first processor; the space release of the memory or the video memory on the hardware brought by the unloading of the feedforward neural subnetwork weight parameters is the benefit. The greater the benefit brought by loading and unloading, the more it should take the corresponding loading and unloading operation.

[0149] In summary, the loading benefit ratio or the unloading benefit ratio provides a more accurate and scientific basis for judging whether to perform loading and unloading of the feedforward neural subnetwork, and for determining which hardware to load or unload the feedforward neural subnetwork weight parameters, thereby making the allocation of hardware resources more balanced.

[0150] Step C: When it is detected that the second processor has a feedforward neural subnetwork that needs to be loaded into the first processor, the running feedforward neural subnetwork is loaded from the second processor to the first processor.

[0151] Specifically, in an optional embodiment of the present application, step C specifically includes:

[0152] Step C1: Collecting central processing loading information of the running feedforward neural subnetwork to obtain first to-be-calculated central processing loading information data of each feedforward neural subnetwork on the second processor.

[0153] Step C2: Calculating according to the first to-be-calculated central processing loading information data to obtain a first central processing benefit ratio corresponding to each feedforward neural subnetwork.

[0154] Step C3: According to the first central processing benefit ratio corresponding to each feedforward neural subnetwork, selecting the feedforward neural subnetwork corresponding to the first central processing benefit ratio with the largest value as a to-be-unloaded feedforward neural subnetwork.

[0155] Step C4: Loading the to-be-unloaded feedforward neural subnetwork from the second processor to the first processor in a preset tensor parallel manner, and loading the feedforward neural subnetwork weight parameters of the to-be-unloaded feedforward neural subnetwork to other first processors.

[0156] In this embodiment, the calculation process of the first central processing benefit ratio is similar to the calculation formula used in step B1, so this embodiment will not be described here.

[0157] Based on the above embodiments, in an optional embodiment of the present application, after step B, it further includes:

[0158] Step D: When it is detected that the second processor does not exist the feedforward neural subnetwork that needs to be loaded into the first processor, then it is detected whether the feedforward neural subnetwork in the memory exists the feedforward neural subnetwork that needs to be loaded into the first processor.

[0159] In the embodiment, the process of detecting whether the feedforward neural subnetwork in the detection memory exists the feedforward neural subnetwork that needs to be loaded into the first processor can refer to the processes of steps B1 to B3 in the above embodiment, which will not be described here again in the embodiment.

[0160] Step E: When it is detected that the feedforward neural subnetwork in the memory exists the feedforward neural subnetwork that needs to be loaded into the first processor, the feedforward neural subnetwork in the running process is loaded from the memory to the first processor.

[0161] In the embodiment, the process of loading the feedforward neural subnetwork in the running process from the memory to the first processor can be loading the feedforward neural subnetwork weight parameters corresponding to the feedforward neural subnetwork to the first processor. The specific process of loading the feedforward neural subnetwork weight parameters has been described in the above embodiment, which will not be described here again in the embodiment.

[0162] Based on the above embodiments, in an optional embodiment of the present application, the model loading and unloading method further includes:

[0163] Step F: When it is detected that the second resource usage function value is less than the lower limit of the second predefined resource usage function value threshold, it is judged whether to trigger unloading.

[0164] Step G: When it is detected that the unloading is triggered, the feedforward neural subnetwork is unloaded from the second processor to the memory.

[0165] In the embodiment, the process of detecting that the second resource usage function value is less than the lower limit of the second predefined resource usage function value threshold can refer to the process of detecting that the second resource usage function value is greater than the upper limit of the second predefined resource usage function threshold in the above embodiment, and the specific principle and implementation process are similar, which will not be described here again in the embodiment. In the embodiment, the process of unloading the feedforward neural subnetwork from the second processor to the memory can be unloading the feedforward neural subnetwork weight parameters from the second processor to the memory.

[0166] Based on the above embodiments, in an optional embodiment of the present application, step G specifically includes:

[0167] Step G1: Obtain the second to-be-calculated central processing loading information data of each feedforward neural subnetwork on the second processor.

[0168] Step G2: According to the second to-be-calculated central processing loading information data, the second central unloading benefit ratio corresponding to each feedforward neural subnetwork is obtained by calculation.

[0169] Step G3: According to the second central unloading benefit ratio corresponding to each feedforward neural subnetwork, the feedforward neural subnetwork corresponding to the second central processing benefit ratio with the maximum value is selected to be unloaded to the memory.

[0170] In this embodiment, the calculation process of the first central unloading benefit ratio in steps G1 to G3 is similar to the calculation formula used in step B1, so this embodiment will not be described here.

[0171] Based on the above embodiments, in the model loading and unloading method provided in an optional embodiment of the present application, each feedforward neural subnetwork includes a shared feedforward neural subnetwork and a routing feedforward neural subnetwork, and further includes:

[0172] Step H: When it is detected that the model is running for the first time, in the case where the initial video memory space of the first processor at the first time of running is greater than the preset video memory space margin threshold, each shared feedforward neural subnetwork is loaded to the first processor layer by layer until each shared feedforward neural subnetwork is loaded and the intermediate video memory space of the first processor is obtained.

[0173] Step I: In the case where the intermediate video memory space is greater than the preset video memory space margin threshold, each routing feedforward neural subnetwork is loaded to the first processor layer by layer until each routing feedforward neural subnetwork is loaded to the first processor.

[0174] In this embodiment, when the system is just started and the model has not run, the feedforward neural subnetwork has not performed inference, at this time, there is no historical running data as a reference, and the feedforward neural subnetwork in the system cannot be identified as the first type of feedforward neural subnetwork, the second type of feedforward neural subnetwork or the third type of feedforward neural subnetwork according to the activation frequency of the feedforward neural subnetwork. In addition to the shared expert, this preloading of the feedforward neural subnetwork can load the shared expert layer by layer to the first processor first. If the resources in the first processor cannot load all layers of the shared feedforward neural subnetwork, the shared feedforward neural subnetwork is loaded layer by layer until it cannot be loaded. The remaining layers of the shared feedforward neural subnetwork and all layers of the routing feedforward neural subnetwork are directly loaded to the second processor. It should be noted that if the second processor also cannot load the remaining feedforward neural subnetwork, it means that the system resources are severely deficient and cannot support the demand for inference service quality. The present application does not consider this case.

[0175] If the hardware resources of the first processor are sufficient to load all layers of the shared expert, all layers of the shared feedforward neural subnetwork are loaded to the first processor first, and then each layer of the routing feedforward neural subnetwork is loaded to the first processor layer by layer. If all layers of the routing feedforward neural subnetwork cannot be loaded to the first processor, the remaining routing feedforward neural subnetwork is loaded to the second processor.

[0176] Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be realized by means of software on a general hardware platform as necessary, and of course can also be realized by hardware, but in many cases the former is a better embodiment.

[0177] The embodiment of the present application also provides a model loading and unloading device, and the execution subject of the device can be the server 102 as shown in the figure, or other hardware devices with the same specific functions. In this regard, the present embodiment does not make any limitation. Figure 1

[0178] Figure 3 A structural schematic diagram of model loading and unloading provided by the embodiment of the present application. As shown in the figure, the embodiment of the present application also provides a model loading and unloading device: comprising a historical information statistical module 31, a preloading module 32, a running monitoring module 33 and a loading and unloading module 34. Figure 3

[0179] The historical information statistical module 31 is used for acquiring historical running process data of each feedforward neural subnetwork when running.

[0180] The preloading module 32 is used for preloading each feedforward neural subnetwork on a preset hardware in a preset order according to the historical running process data, wherein the preset hardware comprises a first processor, a second processor and a memory.

[0181] The running monitoring module 33 is used for acquiring model information, first resource usage information of the first processor, second resource usage information of the second processor, memory usage information and feedforward neural subnetwork weight parameter scheduling information in real time according to a preset time interval during the running process of the model.

[0182] The loading and unloading module 34 is used for performing first calculation on the first resource usage information, the second resource usage information and the memory usage information according to the preset time interval, so as to obtain a first resource usage function value.

[0183] The loading and unloading module 34 is also used for performing second calculation on the second resource usage information and the memory usage information according to the preset time interval when it is detected that the first resource usage function value is greater than an upper limit of a first predefined resource threshold, so as to obtain a second resource usage function value.

[0184] The loading and unloading module 34 is also used for unloading the feedforward neural subnetwork in the running process from the first processor to the second processor when it is detected that the second resource usage function value is less than a lower limit of a second predefined resource usage function value threshold.

[0185] ​​In an optional embodiment of the present application, the preloading module 32 is specifically configured to determine the activation frequencies of the respective feedforward neural subnetworks according to the historical operation data;

[0186] According to the activation frequencies, the respective feedforward neural subnetworks are divided into the first type of feedforward neural subnetworks, the second type of feedforward neural subnetworks and the third type of feedforward neural subnetworks in the order of the activation frequencies from high to low;

[0187] The first type of feedforward neural subnetworks are loaded layer by layer, and when it is detected that the first GPU memory space margin of the first processor is greater than or equal to the preset GPU memory space margin threshold, the feedforward neural subnetwork weight parameters of the first type of feedforward neural subnetworks are loaded onto the first processor until the first type of feedforward neural subnetworks are preloaded completely, and the second GPU memory space margin of the first processor is obtained, wherein the second GPU memory space margin is less than the first GPU memory space margin;

[0188] The second type of feedforward neural subnetworks are loaded layer by layer, and when it is detected that the second GPU memory space margin is greater than or equal to the preset GPU memory space margin threshold, the feedforward neural subnetwork weight parameters of the second type of feedforward neural subnetworks are loaded onto the first processor until the second type of feedforward neural subnetworks are loaded completely, and the third GPU memory space margin of the first processor is obtained, wherein the third GPU memory space margin is less than the second GPU memory space margin;

[0189] The third type of feedforward neural subnetworks are loaded layer by layer, and when it is detected that the third GPU memory space margin is greater than or equal to the preset GPU memory space margin threshold, the intelligent weight parameters of the third type of feedforward neural subnetworks are loaded onto the first processor and the fourth GPU memory space margin of the first processor is obtained;

[0190] The first type of feedforward neural subnetworks are loaded onto the first processor in the form of copies according to the fourth GPU memory space margin.

[0191] In an optional embodiment of the present application, the preloading module 32 is specifically configured to perform a space margin detection operation according to the fourth GPU memory space margin, and when it is detected that the fourth GPU memory space margin is greater than or equal to a preset copy loading space margin, the respective first type of feedforward neural subnetworks are loaded onto the first processor in the form of copies layer by layer until the GPU memory space margin is less than the preset copy loading space margin.

[0192] In an optional embodiment of the present application, the first resource usage information includes first computing resource information and first storage resource information of the first processor, the second resource usage information includes offloading thread overhead and idle rate of the second processor, and the memory usage information includes free memory space; accordingly, the offloading module 34 is specifically configured to: determine a sampling number according to a preset time interval; obtain computing utilization rate, video memory utilization rate, and free memory space of the first processor according to the first computing resource information and the first storage resource information of the first processor at the sampling number; determine thread computing result of the second processor according to the offloading thread and the idle rate of the second processor at the sampling number; perform forward neural subnetwork weight parameter comparison calculation according to the free memory space of the memory at the sampling number to obtain a memory information calculation result; and obtain a first resource usage function value according to the computing utilization rate, the video memory utilization rate, the free memory space of the first processor, the thread computing result of the second processor, and the memory information calculation result.

[0193] In an optional embodiment of the present application, the second resource usage information includes second computing resource information and second storage resource information of the second processor, offloading thread overhead and idle rate, the memory usage information includes free memory space of the memory, and the offloading module 34 is specifically configured to: determine a sampling number according to a preset time interval; obtain computing utilization rate, video memory utilization rate, and free memory space of the second processor according to the second computing resource information and the second storage resource information of the second processor at the sampling number; determine thread computing result of the second processor according to the offloading thread and the idle rate of the second processor at the sampling number; perform forward neural subnetwork weight parameter comparison calculation according to the free memory space of the memory at the sampling number to obtain a memory information calculation result; and obtain a second resource usage function value according to the computing utilization rate, the video memory utilization rate, the free memory space of the second processor, the thread computing result, and the memory information calculation result.

[0194] In an optional embodiment of the present application, the offloading module 34 is specifically configured to: collect first processing loading information data of each forward neural subnetwork on the first processor by performing graphic processing loading information collection on the forward neural subnetwork in a running process; calculate a graphic offloading benefit ratio corresponding to each forward neural subnetwork according to the first processing loading information data; select a forward neural subnetwork corresponding to a graphic processing benefit ratio with a maximum value as a to-be-offloaded forward neural subnetwork according to the graphic offloading benefit ratio corresponding to each forward neural subnetwork; offload the to-be-offloaded forward neural subnetwork to the second processor, and offload forward neural subnetwork weight parameters of the to-be-offloaded forward neural subnetwork on each remaining first processor.

[0195] In an optional embodiment of the present application, the loading and unloading module 34 is further configured to: when it is detected that the first resource usage function value is less than the first predefined resource threshold, traverse the front feed neural subnetwork weight parameter loading information of the second processor; determine whether there is a front feed neural subnetwork in the second processor that needs to be loaded into the first processor according to the front feed neural subnetwork weight parameter loading information; and when it is detected that there is a front feed neural subnetwork in the second processor that needs to be loaded into the first processor, load the front feed neural subnetwork in the running process from the second processor to the first processor.

[0196] In an optional embodiment of the present application, the loading and unloading module 34 is further configured to: collect the central processing loading information of the front feed neural subnetwork in the running process to obtain first to-be-calculated central processing loading information data of each front feed neural subnetwork on the second processor; calculate according to the first to-be-calculated central processing loading information data to obtain a first central processing benefit ratio corresponding to each front feed neural subnetwork; select a front feed neural subnetwork corresponding to a first central processing benefit ratio with the largest value as a to-be-unloaded front feed neural subnetwork according to the first central processing benefit ratio corresponding to each front feed neural subnetwork; load the to-be-unloaded front feed neural subnetwork from the second processor to the first processor in a preset tensor parallel manner, and load the front feed neural subnetwork weight parameters of the to-be-unloaded front feed neural subnetwork to other first processors.

[0197] In an optional embodiment of the present application, the front feed neural subnetwork loading weight loading information includes a front feed neural subnetwork weight parameter scale of unloading, a front feed neural subnetwork unloading frequency, a front feed neural subnetwork unloading time, a front feed neural subnetwork residence time, a front feed neural subnetwork activation frequency, a front feed neural subnetwork hit rate, a front feed neural subnetwork loading frequency, a front feed neural subnetwork loading time, and a hardware computing power improvement. Correspondingly, the loading and unloading module 34 is further configured to: determine a front feed neural subnetwork unloading benefit ratio according to the front feed neural subnetwork loading weight loading information including the front feed neural subnetwork weight parameter scale of unloading, the front feed neural subnetwork unloading frequency, the front feed neural subnetwork unloading time, the front feed neural subnetwork residence time, the front feed neural subnetwork activation frequency, and the front feed neural subnetwork hit rate; determine a loading benefit ratio according to the hardware computing power improvement, the front feed neural subnetwork loading frequency, the front feed neural subnetwork activation frequency, the front feed neural subnetwork hit rate, the front feed neural subnetwork loading time, and the front feed neural subnetwork residence time; and determine whether there is a front feed neural subnetwork in the second processor that needs to be loaded into the first processor according to the loading benefit ratio.

[0198] In an optional embodiment of the present application, the loading and unloading module 34 is further configured to: when it is detected that the second processor does not have a feedforward neural subnetwork that needs to be loaded into the first processor, then detect whether the memory has a feedforward neural subnetwork that needs to be loaded into the first processor; and when it is detected that the memory has a feedforward neural subnetwork that needs to be loaded into the first processor, load the feedforward neural subnetwork in the running process from the memory to the first processor.

[0199] In an optional embodiment of the present application, the loading and unloading module 34 is further configured to: when it is detected that the second resource usage function value is less than the second predefined resource usage function value threshold lower limit, then determine whether to trigger unloading; and when it is detected that unloading is triggered, unload the feedforward neural subnetwork from the second processor to the memory.

[0200] In an optional embodiment of the present application, the loading and unloading module 34 is further configured to: obtain second to-be-calculated central processing loading information data of each feedforward neural subnetwork on the second processor; perform calculation according to the second to-be-calculated central processing loading information data to obtain a second central unloading benefit ratio corresponding to each feedforward neural subnetwork; and according to the second central unloading benefit ratio corresponding to each feedforward neural subnetwork, select the feedforward neural subnetwork corresponding to the second central processing benefit ratio with the largest value to unload to the memory.

[0201] In an optional embodiment of the present application, the preloading module 32 is further configured to: when it is detected that the model is running for the first time, in the case that the initial video memory space of the first processor at the first time of running is greater than a preset video memory space margin threshold, load each shared feedforward neural subnetwork to the first processor layer by layer until each shared feedforward neural subnetwork is loaded and the intermediate video memory space of the first processor is obtained; and in the case that the intermediate video memory space is greater than the preset video memory space margin threshold, load each routing feedforward neural subnetwork to the first processor layer by layer until each routing feedforward neural subnetwork is loaded to the first processor.

[0202] The above-mentioned features of the embodiments of the model loading and unloading device can be referred to the related descriptions of the embodiments of the model loading and unloading method, which will not be repeated here.

[0203] Figure 4 The structure schematic diagram of the electronic device provided in the present application is shown in FIG. 4. As shown in FIG. 4, the electronic device 40 provided in the present embodiment includes at least one processor 401 and a memory 402. Optionally, the electronic device 40 further includes a communication component 403. The processor 401, the memory 402 and the communication component 403 are connected through a bus. Figure 4

[0204] ​In the implementation process, the at least one processor 401 executes the computer-executable instructions stored in the memory 402, so that the at least one processor 401 performs the model loading and unloading method embodiments described above.

[0205] The specific implementation process of the processor 401 can refer to the method embodiments described above, which have similar implementation principles and technical effects, and details are not described here.

[0206] In the above embodiments, it should be understood that the processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor, etc. The steps of the method disclosed in the application can be directly embodied as hardware processor execution, or executed by a combination of hardware and software modules in the processor.

[0207] The memory can include a random access memory (RAM), and can also include a non-volatile memory (NVM), such as at least one disk memory.

[0208] The bus can be an industry standard architecture (ISA) bus, a peripheral component (PCI) bus, or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, the bus in the drawings of the present application does not limit to only one bus or one type of bus.

[0209] The embodiments of the present application also provide a computer readable storage medium, which stores a computer program, wherein the computer program is configured to execute the steps in any of the above model loading and unloading method embodiments when running.

[0210] In an example embodiment, the computer readable storage medium described above can include, but is not limited to, a U disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a mobile memory, a magnetic disk or an optical disk, and various media that can store computer programs.

[0211] Embodiments of the present application also provide a computer program product, which comprises a computer program, and the computer program, when executed by a processor, implements the steps in any of the model loading and unloading method embodiments described above.

[0212] Embodiments of the present application also provide another computer program product, which comprises a non-volatile computer readable storage medium, and the non-volatile computer readable storage medium stores a computer program, and the computer program, when executed by a processor, implements the steps in any of the model loading and unloading method embodiments described above.

[0213] The skilled in the art can further realize that the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be realized in electronic hardware, computer software or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of the examples have been described in the above description in general terms. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. The skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0214] The above describes in detail a model loading and unloading method, a storage medium and a product provided by the present application. The principles and implementation modes of the present application are described by applying specific examples in this paper, and the above description of the examples is only applicable to help understand the method and core idea of the present application. It should be noted that for those skilled in the art, without departing from the principles of the present application, some improvements and modifications can be made to the present application, and these improvements and modifications also fall within the protection scope of the claims of the present application.

Claims

1. A model loading and unloading method characterized by, The model comprises a plurality of feedforward neural subnetworks, and the method comprises: obtaining historical running process data of each feedforward neural subnetwork of the model; loading each feedforward neural subnetwork in a preset order layer by layer onto a preset hardware according to the historical running process data, wherein the preset hardware comprises a first processor, a second processor and a memory; during the running process of the model, obtaining model information, first resource usage information of the first processor, second resource usage information of the second processor, memory usage information and feedforward neural subnetwork weight parameter scheduling information in real time according to a preset time interval; performing first calculation on the first resource usage information, the second resource usage information and the memory usage information according to the preset time interval to obtain a first resource usage function value; when it is detected that the first resource usage function value is greater than an upper limit of a first predefined resource threshold, performing second calculation on the second resource usage information and the memory usage information according to the preset time interval to obtain a second resource usage function value; when it is detected that the second resource usage function value is less than a lower limit of a second predefined resource usage function value threshold, unloading the feedforward neural subnetwork in the running process from the first processor to the second processor.

2. The method of claim 1, wherein, The loading each feedforward neural subnetwork in a preset order layer by layer onto a preset hardware according to the historical running process data comprises: determining an activation frequency of each feedforward neural subnetwork according to the historical running process data; dividing each feedforward neural subnetwork into a first type of feedforward neural subnetwork, a second type of feedforward neural subnetwork and a third type of feedforward neural subnetwork in order of decreasing activation frequency according to the activation frequency; loading the first type of feedforward neural subnetwork layer by layer, and when it is detected that a first video memory space margin of the first processor is greater than or equal to a preset video memory space margin threshold, loading feedforward neural subnetwork weight parameters of the first type of feedforward neural subnetwork onto the first processor until the first type of feedforward neural subnetwork is completely loaded, and obtaining a second video memory space margin of the first processor, wherein the second video memory space margin is less than the first video memory space margin; loading the second type of feedforward neural subnetwork layer by layer, and when it is detected that the second video memory space margin is greater than or equal to the preset video memory space margin threshold, loading feedforward neural subnetwork weight parameters of the second type of feedforward neural subnetwork onto the first processor until the second type of feedforward neural subnetwork is completely loaded, and obtaining a third video memory space margin of the first processor, wherein the third video memory space margin is less than the second video memory space margin; loading the third type of feedforward neural subnetwork layer by layer, and when it is detected that the third video memory space margin is greater than or equal to the preset video memory space margin threshold, loading intelligent weight parameters of the third type of feedforward neural subnetwork onto the first processor and obtaining a fourth video memory space margin of the first processor; loading the first type of feedforward neural subnetwork in the form of a copy onto the first processor according to the fourth video memory space margin.

3. The method of claim 2, wherein, The loading of the first type of feedforward neural subnetwork in the form of a copy onto the first processor according to the fourth GPU space margin comprises: According to the fourth GPU space margin, a space margin detection operation is performed. When it is detected that the fourth GPU space margin is greater than or equal to a preset copy loading space margin, each first type of feedforward neural subnetwork is loaded in the form of a copy onto the first processor layer by layer until the GPU space margin is less than the preset copy loading space margin.

4. The method of claim 1, wherein, The first resource usage information comprises first computing resource information and first storage resource information of the first processor, the second resource usage information comprises offloading trip overhead and idle rate of the second processor, and the memory usage information comprises memory idle storage space; Accordingly, the first calculation of the first resource usage information, the second resource usage information and the memory usage information according to the preset time interval to obtain a first resource usage function value comprises: According to the preset time interval, a sampling number is determined; According to the first computing resource information and the first storage resource information of the first processor, the computing utilization rate, the GPU utilization rate and the idle memory space of the first processor are obtained under the sampling number; According to the offloading thread of the second processor and the idle rate, the thread calculation result of the second processor is determined under the sampling number; According to the memory idle storage space, a feedforward neural subnetwork weight parameter comparison calculation is performed to obtain a memory information calculation result under the sampling number; According to the computing utilization rate, the GPU utilization rate, the idle memory space of the first processor, the thread calculation result of the second processor and the memory information calculation result, a first resource usage function value is obtained.

5. The method of claim 1, wherein, The second resource usage information comprises second computing resource information and second storage resource information of the second processor, offloading trip overhead and idle rate, and the memory usage information comprises memory idle storage space; Accordingly, the second calculation of the second resource usage information and the memory usage information according to the preset time interval to obtain a second resource usage function value comprises: According to the preset time interval, a sampling number is determined; According to the second computing resource information and the second storage resource information of the second processor, the computing utilization rate, the GPU utilization rate and the idle memory space of the second processor are obtained under the sampling number; According to the offloading thread of the second processor and the idle rate, the thread calculation result of the second processor is determined under the sampling number; According to the memory idle storage space, a feedforward neural subnetwork weight parameter comparison calculation is performed to obtain a memory information calculation result under the sampling number; According to the computing utilization rate, the GPU utilization rate, the idle memory space of the second processor, the thread calculation result and the memory information calculation result, a second resource usage function value is obtained.

6. The method of claim 1, wherein, The offloading of the running feedforward neural subnetwork from the first processor to the second processor comprises: The running feedforward neural subnetworks are subjected to graphic processing loading information collection, and first processor each feedforward neural subnetwork's to-be-calculated graphic processing loading information data is obtained; According to the to-be-calculated graphic processing loading information data, the graphic offloading benefit ratio of each feedforward neural subnetwork is calculated; According to the graphic offloading benefit ratio of each feedforward neural subnetwork, the feedforward neural subnetwork corresponding to the graphic processing benefit ratio with the largest value is selected as a to-be-offloaded feedforward neural subnetwork; The to-be-offloaded feedforward neural subnetwork is offloaded to the second processor, and the feedforward neural subnetwork weight parameters of the to-be-offloaded feedforward neural subnetwork on each remaining first processor are offloaded.

7. The method of claim 1, wherein, Further comprising: When it is detected that the first resource usage function value is less than a first predefined resource threshold, the feedforward neural subnetwork weight parameter loading information of the second processor is traversed; According to the feedforward neural subnetwork weight parameter loading information, it is judged whether the second processor has a feedforward neural subnetwork that needs to be loaded into the first processor; When it is detected that the second processor has a feedforward neural subnetwork that needs to be loaded into the first processor, the running feedforward neural subnetwork is loaded from the second processor to the first processor.

8. The method of claim 7, wherein, Loading the running feedforward neural subnetwork from the second processor to the first processor comprises: The running feedforward neural subnetworks are subjected to central processing loading information collection, and first processor each feedforward neural subnetwork's first to-be-calculated central processing loading information data is obtained; According to the first to-be-calculated central processing loading information data, the first central processing benefit ratio of each feedforward neural subnetwork is calculated; According to the first central processing benefit ratio of each feedforward neural subnetwork, the feedforward neural subnetwork corresponding to the first central processing benefit ratio with the largest value is selected as a to-be-offloaded feedforward neural subnetwork; The to-be-offloaded feedforward neural subnetwork is loaded from the second processor to the first processor in a preset tensor parallel manner, and the feedforward neural subnetwork weight parameters of the to-be-offloaded feedforward neural subnetwork are loaded to other first processors.

9. The method of claim 7, wherein, The feedforward neural subnetwork loading weight loading information comprises offloaded feedforward neural subnetwork weight parameter size, feedforward neural subnetwork offloading frequency, feedforward neural subnetwork offloading time, feedforward neural subnetwork residence time, feedforward neural subnetwork activation frequency, feedforward neural subnetwork hit rate, feedforward neural subnetwork loading frequency, feedforward neural subnetwork loading time, and hardware computing power improvement; The judgment of whether the second processor has a feedforward neural subnetwork that needs to be loaded into the first processor according to the feedforward neural subnetwork weight parameter loading information comprises: According to the feedforward neural subnetwork loading weight loading information comprising offloaded feedforward neural subnetwork weight parameter size, feedforward neural subnetwork offloading frequency, feedforward neural subnetwork offloading time, feedforward neural subnetwork residence time, feedforward neural subnetwork activation frequency, and feedforward neural subnetwork hit rate, the feedforward neural subnetwork offloading benefit ratio is determined; The judgment of whether the second processor has a feedforward neural subnetwork that needs to be loaded into the first processor according to the feedforward neural subnetwork weight parameter loading information comprises: determine a loading benefit ratio according to the hardware computing power improvement, the feedforward neural subnetwork loading frequency, the feedforward neural subnetwork activation frequency, the feedforward neural subnetwork hit rate, the feedforward neural subnetwork loading time, and the feedforward neural subnetwork residence time; determine whether the second processor has a feedforward neural subnetwork that needs to be loaded into the first processor according to the loading benefit ratio.

10. The method of claim 7, wherein, after determining whether the second processor has a feedforward neural subnetwork that needs to be loaded into the first processor according to the feedforward neural subnetwork weight parameter loading information, further comprising: when it is detected that the second processor does not have a feedforward neural subnetwork that needs to be loaded into the first processor, then determine whether the memory has a feedforward neural subnetwork that needs to be loaded into the first processor; when it is detected that the memory has a feedforward neural subnetwork that needs to be loaded into the first processor, then load the feedforward neural subnetwork in the running process from the memory to the first processor.

11. The method according to any one of claims 1 to 10, characterized in that, further comprising: when it is detected that the second resource usage function value is less than the second predefined resource usage function value lower threshold, then determine whether to trigger unloading; when it is detected that unloading is triggered, then unload the feedforward neural subnetwork from the second processor to the memory.

12. The method of claim 11, wherein, unloading the feedforward neural subnetwork from the second processor to the memory comprises: obtain second to-be-computed central processing loading information data of each feedforward neural subnetwork on the second processor; perform computation according to the second to-be-computed central processing loading information data to obtain a second central unloading benefit ratio corresponding to each feedforward neural subnetwork; according to the second central unloading benefit ratio corresponding to each feedforward neural subnetwork, select the feedforward neural subnetwork corresponding to the second central processing benefit ratio with the largest value to unload to the memory.

13. The method of claim 2, wherein, each feedforward neural subnetwork comprises a shared feedforward neural subnetwork and a routing feedforward neural subnetwork, and the method further comprises: when it is detected that the model is running for the first time, if the initial video memory space of the first processor when it is running for the first time is greater than a preset video memory space margin threshold, then load each shared feedforward neural subnetwork layer by layer to the first processor until each shared feedforward neural subnetwork is loaded completely and the intermediate video memory space of the first processor is obtained; if it is detected that the intermediate video memory space is greater than the preset video memory space margin threshold, then load each routing feedforward neural subnetwork layer by layer to the first processor until each routing feedforward neural subnetwork is loaded to the first processor.

14. An electronic device, comprising: comprise: a memory for storing a computer program; a processor for executing the computer program to implement the steps of the model loading and unloading method according to any one of claims 1 to 13.

Images