Artificial intelligence model deployment method, computer system, computer readable storage medium and computer program product

Abstract

The invention discloses an artificial intelligence model deployment method, a computer system, a computer readable storage medium and a computer program product, and relates to the field of artificial intelligence. According to the parameter quantity and the information of available storage resources of the computing system deployed by the artificial intelligence model, determining the storage resources for storing the parameters according to the parameter quantity and the information of the available storage resources of the computing system; and when the artificial intelligence model is deployed, storing the parameters to the determined storage resources. Wherein the information of the storage resources comprises multiple types of storage resources, and each type of storage resources in the multiple types of storage resources has different access performance. Therefore, the storage positions of the parameters are flexibly allocated, the resources of the computing system are fully utilized, the utilization rate of the resources is effectively improved, the parameters are obtained from the determined storage resources as soon as possible, computing is accelerated, and the system performance is fully exerted.

Description

[0001] This application is a divisional application. The application number of the original application is 202410054568.5, and the original application date is January 12, 2024. The entire content of the original application is incorporated herein by reference. Technical Field

[0002] This application relates to the field of artificial intelligence, and particularly to an artificial intelligence model deployment method, a computer system, a computer-readable storage medium, and a computer program product. Background Art

[0003] Currently, general-purpose processors and dedicated processors perform the calculations for training and inference of artificial intelligence models to improve computational efficiency. For artificial intelligence models such as recommendation systems and natural language processing, in a single calculation, only some parameters (e.g., sparse parameters) in the artificial intelligence model are required to participate in the calculation. For example, when parameterizing (embedding), dedicated processors process vector data. Generally, due to the different data volumes of vector data of artificial intelligence models of different scales, storing the vector data of artificial intelligence models of different scales based on a single storage scheme results in poor storage flexibility, low resource utilization, and inability to fully utilize the system performance. Summary of the Invention

[0004] This application provides an artificial intelligence model deployment method, a computer system, a computer-readable storage medium, and a computer program product, thereby effectively improving resource utilization and system performance.

[0005] In a first aspect, an artificial intelligence model deployment method is provided. The method includes obtaining the number of parameters of the parameters used when the artificial intelligence model performs inference or training, and information on the available storage resources of the computing system on which the artificial intelligence model is deployed, determining the storage resources for storing the parameters according to the number of parameters and the information on the available storage resources of the computing system; and when deploying the artificial intelligence model, storing the parameters in the determined storage resources. Among them, the information on the storage resources includes multiple types of storage resources, and each type of storage resource among the multiple types of storage resources has different access performances.

[0006] Compared with storing the parameters of artificial intelligence models of different scales based on a single storage solution, for example, storing the parameters of a relatively small-scale artificial intelligence model in a remote storage device, or storing the parameters of an ultra-large-scale artificial intelligence model in a dedicated processor, it will lead to low resource utilization and inability to fully utilize the system performance. The artificial intelligence model deployment method provided in this application is that when deploying an artificial intelligence model, for artificial intelligence models of different scales, the number of parameters of the parameters used when the artificial intelligence model performs inference or training is different, then the parameters are stored in at least one storage medium that meets the number of parameters of the parameters according to the information of the available storage resources of the computing system where the artificial intelligence model is deployed. Thus, the storage location of the parameters is flexibly allocated according to multi-dimensional information such as the number of parameters and storage resource information, fully utilizing the resources of the computing system, effectively improving the resource utilization rate, and enabling the dedicated processor to obtain the parameters as soon as possible from the determined storage resources when training or inferring the artificial intelligence model, and accelerating the execution of model training calculations or model inference calculations based on the parameters, giving full play to the system performance. Avoiding storing the parameters of the artificial intelligence model using only one storage medium, restricting the storage location of the parameters, resulting in the dedicated processor being unable to obtain the parameters in time and reducing the training or inference speed of the artificial intelligence model.

[0007] Among them, the parameters described in this application can be data after vectorizing (embedding) sparse parameters. For example, converting sparse parameters into dense data. Thus, due to the relatively small number of parameters, the storage resources and computing resources occupied are reduced, the computing resource utilization rate and storage resource utilization rate are improved, and the computing speed is increased.

[0008] In a possible implementation, multiple types of storage resources include: a first storage medium associated with a dedicated processor, a second storage medium associated with a general-purpose processor, and a remote storage device.

[0009] In another possible implementation, the information of the available storage resources of the computing system includes the remaining storage capacity; determining the storage resources for storing the parameters according to the number of parameters and the information of the available storage resources of the computing system includes: when the remaining storage capacity of the first storage medium is greater than the number of parameters of the parameters, determining that the determined storage resources include the first storage medium.

[0010] In another possible implementation, storing the parameters in the determined storage resources includes: when the remaining storage capacity of the first storage medium is greater than the number of parameters of the parameters, storing the parameters in the first storage medium.

[0011] When the remaining storage capacity of the first storage medium associated with the dedicated processor meets the quantity of parameters, that is, the first storage medium has sufficient storage space to store the parameters related to the training calculation or inference calculation of the artificial intelligence model, the parameters are stored in the first storage medium. When the dedicated processor trains or infers the artificial intelligence model, the parameters are retrieved from the determined storage resources as soon as possible, and the model training calculation or model inference calculation is accelerated based on the parameters, giving full play to the system performance.

[0012] In another possible implementation, the storage resources for storing the parameters are determined according to the quantity of parameters and the information on the available storage resources of the computing system, including: when the remaining storage capacity of the first storage medium is less than the quantity of parameters, it is determined that the determined storage resources include the second storage medium.

[0013] In another possible implementation, storing the parameters in the determined storage resources includes: when the remaining storage capacity of the first storage medium is less than the quantity of parameters, storing the parameters in the second storage medium.

[0014] When the remaining storage capacity of the first storage medium associated with the dedicated processor does not meet the quantity of parameters, that is, the first storage medium does not have sufficient storage space to store the parameters related to the training calculation or inference calculation of the artificial intelligence model, the parameters are stored in the second storage medium associated with the general-purpose processor. Thus, according to the storage requirements and storage resource characteristics, the storage location of the parameters is flexibly allocated, making full use of the resources of the computing system and effectively improving the resource utilization rate.

[0015] In another possible implementation, determining that the determined storage resources include the second storage medium includes: when the remaining storage capacity of the second storage medium is greater than the quantity of parameters, determining that the determined storage resources include the second storage medium.

[0016] In another possible implementation, storing the parameters in the determined storage resources includes: when the remaining storage capacity of the second storage medium is greater than the quantity of parameters, storing the parameters in the second storage medium.

[0017] When the remaining storage capacity of the second storage medium associated with the general-purpose processor meets the quantity of parameters, that is, the second storage medium has sufficient storage space to store the parameters related to the training calculation or inference calculation of the artificial intelligence model, the parameters are stored in the second storage medium.

[0018] In another possible implementation, determining that the determined storage resources include the second storage medium includes: when the remaining storage capacity of the second storage medium is less than the quantity of parameters, determining that the determined storage resources include the second storage medium and the remote storage device.

[0019] In another possible implementation, storing the parameters into the determined storage resource includes: when the remaining storage capacity of the second storage medium is less than the quantity of the parameters, storing the first quantity of the parameters into the second storage medium and storing the second quantity of the parameters into the remote storage device.

[0020] When the remaining storage capacity of the second storage medium associated with the general-purpose processor does not meet the quantity of the parameters, that is, the second storage medium does not have enough storage space to store the parameters related to the training calculation or inference calculation of the artificial intelligence model, the parameters are divided into two parts. The first quantity of the parameters is stored into the second storage medium, and the second quantity of the parameters is stored into the remote storage device. Thus, according to the storage requirements and the characteristics of the storage resources, the storage location of the parameters is flexibly allocated, the resources of the computing system are fully utilized, and the utilization rate of the resources is effectively improved.

[0021] In another possible implementation, determining the storage resource for storing the parameters according to the quantity of the parameters and the information of the available storage resources of the computing system includes: when the remaining storage capacity of the first storage medium is less than the quantity of the parameters and the remaining storage capacity of the second storage medium is less than the quantity of the parameters, determining that the determined storage resource includes the remote storage device.

[0022] In another possible implementation, storing the parameters into the determined storage resource includes: when the remaining storage capacity of the first storage medium is less than the quantity of the parameters and the remaining storage capacity of the second storage medium is less than the quantity of the parameters, storing the parameters into the remote storage device.

[0023] When the remaining storage capacity of the first storage medium associated with the dedicated processor does not meet the quantity of the parameters and the remaining storage capacity of the second storage medium associated with the general-purpose processor does not meet the quantity of the parameters, storing the parameters into the remote storage device. Thus, according to the storage requirements and the characteristics of the storage resources, the storage location of the parameters is flexibly allocated, the resources of the computing system are fully utilized, and the utilization rate of the resources is effectively improved.

[0024] In another possible implementation, the parameters include multiple feature tables related to the artificial intelligence model; determining the storage resource for storing the parameters according to the quantity of the parameters and the information of the available storage resources of the computing system includes: when the remaining storage capacity of the first storage medium is less than the quantity of the parameters, determining that the determined storage resource includes the first storage medium and the second storage medium; storing the parameters into the determined storage resource includes: when the remaining storage capacity of the first storage medium is less than the quantity of the parameters, storing the first quantity of the feature tables into the first storage medium and storing the second quantity of the feature tables into the second storage medium.

[0025] In another possible implementation, the parameter includes multiple feature tables related to the artificial intelligence model; determining the storage resources for storing the parameter according to the number of parameters and the information of the available storage resources of the computing system, including: when the remaining storage capacity of the first storage medium is less than the number of parameters of the parameter, determining that the determined storage resources include the first storage medium and the remote storage device; storing the parameter in the determined storage resources, including: when the remaining storage capacity of the first storage medium is less than the number of parameters of the parameter, storing the first number of feature tables in the first storage medium and storing the second number of feature tables in the remote storage device.

[0026] Thus, the multiple feature tables are divided into two parts, one part is stored in the first storage medium associated with the dedicated processor, and the other part is stored in the second storage medium or the remote storage medium associated with the general-purpose processor, that is, the first storage medium is used as a cache layer to cache some of the feature tables in the multiple feature tables, so that the dedicated processor can obtain the parameters as soon as possible, accelerate the execution of model training calculation or model inference calculation based on the parameters, improve the training or inference speed of the artificial intelligence model, and give full play to the system performance.

[0027] In another possible implementation, the method further includes: obtaining the performance requirements of the user during the inference or training of the artificial intelligence model; determining the storage resources for storing the parameter according to the number of parameters and the information of the available storage resources of the computing system includes: determining the storage resources for storing the parameter according to the number of parameters, the performance requirements and the information of the available storage resources of the computing system.

[0028] In another possible implementation, obtaining the parameter from the determined storage resources, and the dedicated processor performs training calculation or inference calculation on the artificial intelligence model according to the obtained parameter.

[0029] In a second aspect, an artificial intelligence model deployment device is provided. The artificial intelligence model deployment device includes each module for executing the artificial intelligence model deployment method in the first aspect or any possible design of the first aspect. For example, the artificial intelligence model deployment device is used to implement the functions of the general-purpose processor, and the artificial intelligence model deployment device includes a communication module and a processing module.

[0030] The communication module is used to obtain the number of parameters of the parameters used during the inference or training of the artificial intelligence model to be deployed, and the information of the available storage resources of the computing system on which the artificial intelligence model is deployed. The information of the storage resources includes multiple types of storage resources, and each type of storage resource in the multiple types of storage resources has different access performances.

[0031] A processing module, configured to determine storage resources for storing the parameters according to the number of the parameters and information about available storage resources of the computing system; and, when deploying the artificial intelligence model, store the parameters into the determined storage resources.

[0032] In a possible implementation manner, the multiple types of storage resources include: a first storage medium associated with a dedicated processor, a second storage medium associated with a general-purpose processor, and a remote storage device.

[0033] In another possible implementation manner, the information about available storage resources of the computing system includes the remaining storage capacity; when the processing module determines the storage resources for storing the parameters according to the number of the parameters and the information about available storage resources of the computing system, it is specifically configured to: when the remaining storage capacity of the first storage medium is greater than the number of the parameters, determine that the determined storage resources include the first storage medium.

[0034] In another possible implementation manner, when the processing module stores the parameters into the determined storage resources, it is specifically configured to: when the remaining storage capacity of the first storage medium is greater than the number of the parameters, store the parameters into the first storage medium.

[0035] In another possible implementation manner, when the processing module determines the storage resources for storing the parameters according to the number of the parameters and the information about available storage resources of the computing system, it is specifically configured to: when the remaining storage capacity of the first storage medium is less than the number of the parameters, determine that the determined storage resources include the second storage medium.

[0036] In another possible implementation manner, when the processing module stores the parameters into the determined storage resources, it is specifically configured to: when the remaining storage capacity of the first storage medium is less than the number of the parameters, store the parameters into the second storage medium.

[0037] In another possible implementation manner, when the processing module determines that the determined storage resources include the second storage medium, it is specifically configured to: when the remaining storage capacity of the second storage medium is greater than the number of the parameters, determine that the determined storage resources include the second storage medium.

[0038] In another possible implementation manner, when the processing module stores the parameters into the determined storage resources, it is specifically configured to: when the remaining storage capacity of the second storage medium is greater than the number of the parameters, store the parameters into the second storage medium.

[0039] In another possible implementation manner, when the processing module determines that the determined storage resources include the second storage medium, it is specifically configured to: when the remaining storage capacity of the second storage medium is less than the number of the parameters, determine that the determined storage resources include the second storage medium and the remote storage device.

[0040] In another possible implementation, when the processing module stores the parameters in the determined storage resource, it is specifically used for: when the remaining storage capacity of the second storage medium is less than the number of parameters, storing the first quantity of parameters in the second storage medium and storing the second quantity of parameters in the remote storage device.

[0041] In another possible implementation, when the processing module determines the storage resource for storing the parameters according to the number of parameters and the information on the available storage resources of the computing system, it is specifically used for: when the remaining storage capacity of the first storage medium is less than the number of parameters and the remaining storage capacity of the second storage medium is less than the number of parameters, determining that the determined storage resource includes the remote storage device.

[0042] In another possible implementation, when the processing module stores the parameters in the determined storage resource, it is specifically used for: when the remaining storage capacity of the first storage medium is less than the number of parameters and the remaining storage capacity of the second storage medium is less than the number of parameters, storing the parameters in the remote storage device.

[0043] In another possible implementation, the parameters include multiple feature tables related to the artificial intelligence model; when the processing module determines the storage resource for storing the parameters according to the number of parameters and the information on the available storage resources of the computing system, it is specifically used for: when the remaining storage capacity of the first storage medium is less than the number of parameters, determining that the determined storage resource includes the first storage medium and the second storage medium; when the processing module stores the parameters in the determined storage resource, it is specifically used for: when the remaining storage capacity of the first storage medium is less than the number of parameters, storing the first quantity of feature tables in the first storage medium and storing the second quantity of feature tables in the second storage medium.

[0044] In another possible implementation, the parameters include multiple feature tables related to the artificial intelligence model; when the processing module determines the storage resource for storing the parameters according to the number of parameters and the information on the available storage resources of the computing system, it is specifically used for: when the remaining storage capacity of the first storage medium is less than the number of parameters, determining that the determined storage resource includes the first storage medium and the remote storage device; when storing the parameters in the determined storage resource, it is specifically used for: when the remaining storage capacity of the first storage medium is less than the number of parameters, storing the first quantity of feature tables in the first storage medium and storing the second quantity of feature tables in the remote storage device.

[0045] In another possible implementation, the communication module is further configured to obtain the performance requirements of the user during the inference or training of the artificial intelligence model; when the processing module determines the storage resources for storing the parameters according to the number of parameters and the information of the available storage resources of the computing system, it is specifically configured to: determine the storage resources for storing the parameters according to the number of parameters, the performance requirements, and the information of the available storage resources of the computing system.

[0046] In a third aspect, a computing system is provided, which includes a general-purpose processor and multiple dedicated processors, and the general-purpose processor and the multiple dedicated processors jointly execute the operation steps of the method in the first aspect or any possible implementation manner of the first aspect.

[0047] In a fourth aspect, a computer system is provided, which includes a storage node and multiple computing nodes. The storage node is used to store the parameters indicated by the computing nodes, and the computing nodes jointly execute the operation steps of the method in the first aspect or any possible implementation manner of the first aspect.

[0048] In a fifth aspect, a computer system is provided, which includes a memory and multiple processors. The memory is used to store a set of computer instructions; when the processors execute the set of computer instructions, the multiple processors jointly execute the operation steps of the method in the first aspect or any possible implementation manner of the first aspect.

[0049] In a sixth aspect, a computer-readable storage medium is provided, including: computer software instructions; when the computer software instructions run in a processor, the processor is caused to execute the operation steps of the method described in the first aspect or any possible implementation manner of the first aspect.

[0050] In a seventh aspect, a computer program product is provided, which, when running on a computer, causes the computer to execute the operation steps of the method described in the first aspect or any possible implementation manner of the first aspect.

[0051] For the technical effects brought by any of the design manners in the second aspect to the seventh aspect, reference can be made to the technical effects brought by the first aspect or different design manners in the first aspect, which will not be elaborated here.

[0052] Based on the implementation manners provided in the above aspects of the present application, further combinations can be made to provide more implementation manners. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 It is a schematic structural diagram of a neural network provided by the present application;

[0054] Figure 2 It is a schematic diagram of sparse parameters provided by the present application;

[0055] Figure 3 A schematic diagram of one-hot encoding and vectorization provided for this application;

[0056] Figure 4 A schematic diagram of the architecture of a parameter server provided for this application;

[0057] Figure 5 A schematic diagram of the architecture of a computer system provided for this application;

[0058] Figure 6 A schematic diagram of the architecture of a computing system provided for this application;

[0059] Figure 7 A schematic diagram of the architecture of a computer system provided for this application;

[0060] Figure 8 A schematic diagram of the architecture of a computing system provided for this application;

[0061] Figure 9 A schematic diagram of parameter storage for a medium-sized model provided for this application;

[0062] Figure 10 A schematic diagram of parameter storage for a large model provided for this application;

[0063] Figure 11 A schematic diagram of parameter storage for an extra-large model provided for this application;

[0064] Figure 12 A schematic diagram of the process flow of a method for deploying an artificial intelligence model provided for this application;

[0065] Figure 13 A schematic diagram of the process flow of a method for deploying an artificial intelligence model provided for this application;

[0066] Figure 14 A schematic diagram of the structure of a device for deploying an artificial intelligence model provided for this application;

[0067] Figure 15 A schematic diagram of the structure of a computer device provided for this application. Detailed implementation manners

[0068] For ease of understanding, the main terms involved in this application are first explained.

[0069] Artificial Neural Network (ANN): Abbreviated as Neural Network (NN) or Neural-like Network. In the fields of machine learning and cognitive science, it is a mathematical model or computational model that mimics the structure and function of a biological neural network (the central nervous system of an animal, especially the brain). A neural network is a network formed by connecting multiple single neurons together, that is, the output of one neuron can be the input of another neuron. The input of each neuron can be connected to the local receptive field of the previous layer to extract the features of the local receptive field, and the local receptive field can be a region composed of several neurons.

[0070] Each node represents a specific output function, called the activation function. The connection between every two nodes represents a weighted value for the signal passing through this connection, called the weight, which is equivalent to the memory of the artificial neural network. The output of the neural network varies according to the connection method, weight value, and activation function of the network. The neural network itself is usually an approximation of a certain algorithm or function in nature, or it may be an expression of a logical strategy.

[0071] As Figure 1 shown, it is a schematic structural diagram of a neural network provided by this application. The neural network 100 includes N processing layers, where N is an integer greater than or equal to 3. The first layer of the neural network 100 is the input layer 110, which is responsible for receiving input signals, and the last layer of the neural network 100 is the output layer 130, which is responsible for outputting the processing results of the neural network. The other layers except the first layer and the last layer are intermediate layers 140, and these intermediate layers 140 together form the hidden layer 120. Each intermediate layer 140 in the hidden layer 120 can both receive input signals and output signals. The hidden layer 120 is responsible for the processing process of the input signals. Each layer represents a logical level of signal processing, and through multiple layers, the data signal can be processed through multiple levels of logic.

[0072] In some feasible embodiments, the input signal of the neural network can be various forms of signals such as video signals, voice signals, text signals, image signals, temperature signals, and engineering signals that can be processed by a computer.

[0073] Model training: Refers to training an artificial intelligence model using a training set so that the artificial intelligence model can predict or classify unknown data. During the training process of the artificial intelligence model, it learns according to the features and target values in the training set, and generates an artificial intelligence model after training. This artificial intelligence model can be used to predict or classify unknown data. Artificial intelligence model training is one of the most important links in machine learning and affects the accuracy and reliability of the artificial intelligence model.

[0074] Parameters: Variables or weights that need to be learned or adjusted in an artificial intelligence model. Parameters can affect the prediction ability and performance of the artificial intelligence model. During the training process of the artificial intelligence model, the artificial intelligence model tries different parameter combinations to optimize its performance. Common parameters include weights, biases, learning rates, and regularization coefficients, etc. When using the artificial intelligence model for prediction, these parameters are used to calculate the output results.

[0075] Sparse parameters: Parameters that only partially participate in the calculation during a single training process. For example, the parameters participating in forward calculation and backward update.

[0076] Exemplarily, Figure 2 is a schematic diagram of sparse parameters provided by this application. As Figure 2 shown, in a single training step, the weight part participating in the calculation only includes one-half of all weights.

[0077] Usually, the scale of sparse parameters is very large. For example, taking a production-level recommendation system as an example, the number of sparse parameters reaches the level of 10TB - 30TB.

[0078] Recommendation system: An application that is built on the basis of massive data mining, provides users with personalized decision-making support and information services, and judges the items or services that users currently need or are interested in according to information such as users' historical behaviors, social relationships, interests, and the context environment they are in.

[0079] In some artificial intelligence models such as recommendation systems and natural language processing, the input data usually contains discrete features. The embedding table (Embedding table) parameters are used to convert the input data into continuous vector data, and then the vector data is processed. During a single training process, only some of the parameters in the embedding table participate in the calculation and are trained and updated.

[0080] Embedding technology: A representation form of dense vectors. That is, converting sparse parameters into dense vector data. Embedding can represent the features of an object. The object can be, for example, height, gender, name, or item, etc.

[0081] Exemplarily, as Figure 3 shown, is a schematic diagram of one-hot encoding and Embedding provided by this application. Embedding is equivalent to performing smoothing on one-hot encoding, and one-hot encoding is equivalent to performing max pooling on Embedding.

[0082] During the training process of the artificial intelligence model, parameters (such as sparse parameters) are continuously adjusted to make the predicted value calculated from the input data and the parameters approximate the actual value. The parameters can be loaded into the storage medium of a dedicated processor, and the dedicated processor calculates and updates the parameters. For example, the dedicated processor includes, but is not limited to, a graphics processing unit (GPU), a data processing unit (DPU), a neural processing unit (NPU), and an embedded neural network processor (NPU). The storage medium of the dedicated processor includes High Bandwidth Memory (HBM).

[0083] Exemplarily, as Figure 4 As shown in (a) of [reference], a schematic diagram of the architecture of a parameter server provided by this application is shown. The Embedding table is entirely stored in the storage medium of the GPU (such as: HBM). The storage medium of the GPU is divided into two parts. One part is used to store data during the training process of the artificial intelligence model; the other part is used to store the Emb table. During the training process of the artificial intelligence model, the required Embedding vectors are pulled to the specified GPU in an All2All manner to achieve the purpose of sharing the Emb table. The Embedding table is abbreviated as the Emb table. The Emb table contains vector data of sparse parameters.

[0084] However, this solution has a large demand for GPUs. For example, it requires about 300 GPU cards to store a 10TB Emb table, resulting in a huge cost for training the artificial intelligence model. In addition, due to the large number of GPUs required, the linearity of the system is poor, the performance of the entire system is not fully utilized, and the computing reliability is also poor. All GPUs participate in training using All2All. If one GPU fails, it is easy to lose the data of the Emb table stored in the faulty GPU.

[0085] The storage capacity of the dedicated processor is only 16GB - 32GB, which is a drop in the bucket for tens of TB of sparse parameters. Therefore, the sparse parameters can also be stored in storage media such as the volatile memory (such as: memory) or non-volatile memory (such as: solid state drive (SSD)) of the server.

[0086] In some embodiments, the sparse parameters are stored in a server including a general-purpose processor, and the general-purpose processor converts the sparse parameters into vector data. Then, the vector data is transmitted to the dedicated processor for calculation.

[0087] Exemplarily, as Figure 4As shown in (b), it is a schematic diagram of the architecture of a parameter server provided by this application. The Emb table is entirely stored in the CPU cluster; the GPU is used to train the artificial intelligence model. The CPU provides the functions of storing, querying, and updating the Emb table, which is called the parameter server. The storage capacity of the GPU is extended through the parameter server, and the CPU and GPU cooperate to perform the forward calculation and backward update of the artificial intelligence model; during the forward calculation, the required Emb table is pulled from the parameter server through the pull interface; the GPU trains the artificial intelligence model based on the Emb table to obtain gradients; the gradients are pushed back to the parameter server through the push interface, and the Emb table is updated in the parameter server.

[0088] Since a large amount of Emb tables are transmitted through the network every time the artificial intelligence model is trained, that is, the pull operation and the push operation are executed, the computing efficiency is low.

[0089] In some other embodiments, when the server cannot store the complete Emb table, the Emb table is stored in the storage medium of the server and the remote storage device.

[0090] Exemplarily, as Figure 4 As shown in (c), it is a schematic diagram of the architecture of a parameter server provided by this application. When the storage capacity of the memory is insufficient, the SSD is also used to store the Emb table. The CPU can pull the required Emb table from the SSD and transmit the Emb table to the GPU.

[0091] To solve the problem that the utilization rate of resources is low and the system performance cannot be fully exerted due to storing the parameters of artificial intelligence models of different scales based on a single storage scheme, this application provides an artificial intelligence model deployment method, that is, obtaining the number of parameters of the parameters used when the artificial intelligence model is inferred or trained, and the information of the available storage resources of the computing system where the artificial intelligence model is deployed, and determining the storage resources for storing the parameters according to the number of parameters and the information of the available storage resources of the computing system; and when deploying the artificial intelligence model, storing the parameters in the determined storage resources. Among them, the information of the storage resources includes various types of storage resources, and each type of storage resource among the various types of storage resources has different access performances.

[0092] When deploying an artificial intelligence model, for artificial intelligence models of different scales, the number of parameters used by the artificial intelligence model during inference or training is different. Then, according to the information on the available storage resources of the computing system on which the artificial intelligence model is deployed, the parameters are stored in at least one storage medium that can accommodate the number of parameters. Thus, the storage location of the parameters is flexibly allocated based on multi-dimensional information such as the number of parameters and storage resource information, making full use of the resources of the computing system, effectively improving the resource utilization rate, and enabling the dedicated processor to obtain the parameters as soon as possible from the determined storage resources when training or inferring the artificial intelligence model, and accelerating the execution of model training calculations or model inference calculations based on the parameters, giving full play to the system performance. Avoiding storing the parameters of the artificial intelligence model using only one storage medium, restricting the storage location of the parameters, resulting in the dedicated processor being unable to obtain the parameters in time and reducing the training or inference speed of the artificial intelligence model.

[0093] The following will describe in detail the implementation manners of the artificial intelligence model deployment method provided in this application with reference to the accompanying drawings.

[0094] Figure 5 The architecture diagram of a computer system provided in this application. As Figure 5 shown, the computer system 500 includes a client 510, a computing cluster 520, and a storage cluster 530.

[0095] The computing cluster 520 includes multiple computing nodes 521. The multiple computing nodes 521 can be connected based on high-speed interconnect technology through network devices (such as switches, network cards, etc.) to enable communication between the multiple computing nodes 521.

[0096] In some embodiments, the computing node 521 can include computing units with computing capabilities such as a graphics processing unit (GPU), a data processing unit (DPU), a neural processing unit (NPU), and an embedded neural-network processing unit (NPU) to provide high-performance computing.

[0097] The computing cluster 520 further includes a control node 522. The control node 522 is used to manage and allocate tasks, and multiple tasks are executed in parallel by multiple computing nodes to improve the data processing rate.

[0098] Among them, the computing nodes with computing capabilities (such as GPUs, DPUs, and NPUs) can be referred to as devices, training cards, or acceleration cards. The control node can be referred to as a host. Optionally, the computing node can also be referred to as a dedicated processor. The control node can be referred to as a general-purpose processor.

[0099] In this application, the control node 522 is further configured to perform preprocessing operations on the parameters required for model training or model inference, and cooperate with multiple computing nodes 521 to manage the parameters. For example, the control node 522 vectorizes the sparse parameters and converts them into dense vector data (such as: Emb table). Furthermore, when the control node 522 obtains a model training task or a model inference task, it determines the storage resources for storing the parameters according to the number of parameters and the information of the available storage resources of the computing system. The determined storage resources include at least one of the storage media associated with the computing node, the storage media associated with the control node, or the storage media associated with the storage node. That is, for artificial intelligence models of different scales, due to the different numbers of parameters to be loaded, the control node 522 performs hierarchical storage on the parameters with different storage requirements, that is, stores the parameters related to model training or model inference in the storage media of at least one of the computing node, the control node, or the storage node.

[0100] When the computing node 521 performs training calculation or inference calculation on the artificial intelligence model according to the parameters, if the parameters are stored in the storage media of the computing node 521, the computing node 521 obtains the parameters related to model training or model inference from the storage media of the computing node 521.

[0101] If the parameters are stored in the storage media of the control node 522 or the storage node 531, the control node 522 obtains the parameters from the storage media of the control node 522 or the storage node 531, loads the parameters into the computing node 521, and the computing node 521 performs training calculation or inference calculation on the artificial intelligence model according to the parameters.

[0102] Optionally, if the parameters are stored in the storage media of the control node 522 or the storage node 531, the computing node 521 can also obtain the parameters related to model training or model inference from the storage media of the control node 522 or the storage node 531.

[0103] The storage cluster 530 includes multiple storage nodes 531. A storage node 531 includes one or more controllers, network cards, and multiple hard disks. The hard disks are used to store data. The hard disks can be magnetic disks or other types of storage media, such as solid state drives or shingled magnetic recording hard disks, etc. The network cards are used to communicate with the computing nodes 521 included in the computing cluster 520. The controller is configured to write data to the hard disks or read data from the hard disks according to the read / write data requests sent by the computing nodes 521. During the process of reading and writing data, the controller needs to convert the addresses carried in the read / write data requests into addresses that the hard disks can recognize.

[0104] The storage cluster 530 includes multiple storage nodes 531, and the computing cluster 520 includes multiple computing nodes 521. They are connected based on high-speed interconnection technology through network devices (such as switches, network cards, etc.), enabling communication between multiple computing nodes 521 and multiple storage nodes 531.

[0105] In this application, the storage nodes 531 included in the storage cluster 530 can serve as the remote storage devices described in this application. For example, a storage node can be a server, and the storage node includes a CPU and a storage medium. In the case where the remaining storage capacity in the computing cluster is insufficient, the storage cluster 530 can provide a storage expansion function, that is, store parameters related to model training or model inference. For example, the Emb table required for model training or model inference.

[0106] This application does not limit the deployment forms of the above-mentioned computing nodes 521 and control nodes 522. For example, the computing nodes 521 and control nodes 522 can be deployed on the same server or multiple servers.

[0107] In some other embodiments, the computing nodes 521 and control nodes 522 can each be an independent server, etc. The computing node 521 can be a server providing a heterogeneous computing architecture to provide high-performance computing. The computing node 521 includes a dedicated processor and a general-purpose processor. For example, the general-purpose processor can include a central processing unit (CPU). The dedicated processor includes computing nodes with computing capabilities such as GPUs, DPUs, and NPUs to provide high-performance computing. The control node 522 can be an ordinary application server, and the control node 522 includes a CPU and a storage medium.

[0108] The control node 522 is used to manage and allocate tasks, instructing multiple computing nodes to execute multiple tasks in parallel to improve the training or inference speed of the artificial intelligence model.

[0109] The general-purpose processor in the computing node 521 is used to determine the storage resources for storing parameters based on the number of parameters and the information of the available storage resources of the computing system. The dedicated processor in the computing node 521 is used to perform training calculations or inference calculations on the artificial intelligence model according to the parameters.

[0110] Optionally, a server used to store the parameters required for model training or model inference can be called a parameter server. A server used to perform model training calculations or model inference calculations can be called a training server. The training server can include multiple acceleration cards. For example, Figure 6As shown in the figure, it is a schematic architecture diagram of a computing system provided by this application. The computing system includes 3 parameter servers and 3 training servers. Each training server contains 2 NPUs. The parameter server is used to store the parameters required for model training or model inference (such as: Emb table). The training server is used to perform training calculations or inference calculations on the artificial intelligence model according to the parameters.

[0111] The client 510 communicates with the computing cluster 520 and the storage cluster 530 through the network 540. For example, the client 510 sends a request to the computing cluster 520 through the network 540, requesting the computing cluster 520 to perform model training or model inference. The network 540 can refer to an enterprise internal network (such as: Local Area Network (LAN)) or the Internet. The client 510 can refer to a computer connected to the network 540, and can also be called a workstation. Different clients can share resources on the network (such as: computing resources, storage resources).

[0112] In some embodiments, the client 510 installs the client program 511. The client 510 runs the client program 511 to display a user interface (UI). The user 550 operates the user interface to submit a request. For example, the user 550 operates the user interface to submit a model training request or a model inference request. After the control node 522 obtains the request, it can obtain the parameters required for model training or model inference from the storage cluster 530, and determine the storage resources for storing the parameters according to the number of parameters and the information of the available storage resources of the computing system.

[0113] Optionally, the system administrator 560 can configure system information, etc. through the client 510 by calling the application platform interface (API) 512 or the command-line interface (CLI) interface 513. For example, the system information includes the storage policy of the parameters required for model training or model inference.

[0114] Figure 5 It is only a schematic diagram. The embodiments of this application do not limit the device connection method, the number of devices, and the device form in the computer system.

[0115] For example, Figure 7Schematic diagram of an architecture of a computer system provided by this application. A dedicated server and a general server are interconnected through a network device based on a high-speed interconnection technology. The computer system may include a dedicated server and a general server. The dedicated server and the general server may each be an independent server. The dedicated server includes a storage medium, a general-purpose processor, and multiple dedicated processors. The dedicated server may be an artificial intelligence server. The artificial intelligence server includes a general-purpose processor, a storage medium, and multiple dedicated processors. Multiple artificial intelligence servers are interconnected through a network, and multiple artificial intelligence servers form an AI cluster to implement model training calculation and model inference calculation. The general server includes a general-purpose processor.

[0116] Among them, the general server is used to manage and allocate tasks, and multiple dedicated servers execute multiple tasks in parallel to improve the training or inference speed of the artificial intelligence model.

[0117] The general-purpose processor in the dedicated server is used to determine the storage resources for storing parameters according to the number of parameters and the information of the available storage resources of the computing system. The determined storage resources include at least one of the storage media of the dedicated processors, the storage media of the dedicated server, and the storage media of the general server. The storage medium is used to store parameters related to the training calculation or inference calculation of the artificial intelligence model, such as the storage medium is used to store the Emb table or part of the data in the Emb table.

[0118] The dedicated processors in the dedicated server are used to perform training calculation or inference calculation on the artificial intelligence model according to the parameters.

[0119] It should be noted that the specific form of the storage medium described in this application is not limited. The storage medium includes a volatile memory pool or a non-volatile memory pool, or may include both volatile and non-volatile memories. For example, the storage medium of the dedicated processor includes at least one of HBM or SSD.

[0120] The following describes the hierarchical storage of the parameters required for model training calculation or model inference calculation provided by this application based on a multi-level parameter server.

[0121] Figure 8 Schematic diagram of an architecture of a computing system provided by this application. The computing system provides a unified architecture of a multi-level parameter server. As Figure 8As shown in the figure, the computing system 800 includes one or more local servers 810 and one or more storage servers 820. The local server 810 includes a general-purpose processor and multiple dedicated processors. The general-purpose processor and the multiple dedicated processors can be jointly deployed on the same server. The storage medium in the dedicated processor serves as the first-level storage layer, and the storage medium in the general-purpose processor serves as the second-level storage layer, that is, the storage medium in the general-purpose processor serves as the Local Parameter Server (LocalPS Server), and the storage server 820 serves as the third-level storage layer, that is, the storage server 820 serves as the Remote Parameter Server (Remote PS Server).

[0122] The storage medium in the dedicated processor is used to store the parameters during model training or model inference, as well as all or part of the parameters of the Emb table.

[0123] The storage medium in the general-purpose processor is used to store all or part of the parameters of the Emb table, gradient accumulation, optimizer data, cache data, etc.

[0124] The storage server is used to store all or part of the parameters of the Emb table, gradient accumulation, optimizer data, cache data, etc.

[0125] The general-purpose processor is used to perform preprocessing operations on the parameters required for model training or model inference. For example, vectorize the sparse parameters and convert them into dense vector data to obtain the Emb table. The general-purpose processor is also used to store the Emb table to the determined storage resources according to the storage requirements of the Emb table and the storage resource characteristics of the system. The determined storage resources include the storage media of at least one of the above-mentioned first-level storage layer, second-level storage layer, and third-level storage layer.

[0126] The general-purpose processor can run multiple worker processes. One worker process can control one dedicated processor. For example, when the worker instructs the dedicated processor to perform model training calculations or model inference calculations, it can pull the Emb table and transfer it to the dedicated processor, and update the Emb table according to the gradients pushed back from the dedicated processor.

[0127] Optionally, the model described in this application may refer to a recommendation model.

[0128] Based on the above unified architecture of the multi-level parameter server, unified hierarchical storage solutions are proposed for medium-scale models, large-scale models, and extra-large-scale models respectively.

[0129] In the first possible implementation, the number of parameters of the Emb table corresponding to the medium-scale model is not large, and the required storage capacity is not large. The Emb table can be stored in the first-level storage layer, that is, the Emb table is stored in the storage medium in the dedicated processor.

[0130] In some embodiments, the Emb table can be split into multiple data blocks according to the number of dedicated processors, and the multiple data blocks of the Emb table are respectively stored in the storage media of multiple dedicated processors. The splitting method of the Emb table is not limited in this application. The number of data blocks of the Emb table can also be less than the number of dedicated processors, and the data blocks of the Emb table are stored in some of the multiple dedicated processors. The number of data blocks of the Emb table can also be greater than or equal to the number of dedicated processors, and the data blocks of the Emb table are stored in the storage media of multiple dedicated processors. One dedicated processor stores one data block, or one dedicated processor can also store multiple data blocks.

[0131] Exemplarily, as Figure 9 shown, storing the Emb table of a medium-sized model in the storage medium (such as, HBM) of a dedicated processor is called Form One. Each time the dedicated processor iteratively trains the model or infers the model, it obtains the Emb vectors in the Emb table from the storage medium of the dedicated processor. After the training is completed, the Emb vectors in the HBM are updated according to the gradient. Or, after the training is completed, the general-purpose processor can update the Emb table according to the gradient.

[0132] In some other embodiments, by expanding the dedicated processor, that is, adding dedicated processors in the local server, it is convenient to store the Emb table based on the storage media in multiple dedicated processors.

[0133] In the second possible implementation, the number of parameters of the Emb table corresponding to a large-scale model is large, and the required storage capacity is also very large. For example, the number of parameters of the Emb table can reach 10TB. If the Emb table is stored in the storage medium of a dedicated processor, taking the storage capacity of each dedicated processor as 32GB as an example, 300 dedicated processors are required to start training, and the demand for dedicated processors is too large. The Emb table can be stored in the secondary storage layer, that is, the Emb table is stored in the storage medium in the general-purpose processor, that is, the Emb table is stored in the local parameter server.

[0134] Exemplarily, as Figure 10 shown, the general-purpose processor can store the Emb table of the large-scale model in the storage medium of the general-purpose processor (such as, DDR), that is, store the Emb table in the local parameter server, which is called Form Two.

[0135] After the general-purpose processor obtains a model training task request or a model inference task request, the worker can obtain the identifier of the parameter set from the request, and the identifier can indicate the Emb vectors in the Emb table. The general-purpose processor obtains the Emb vectors corresponding to the identifier and transmits them to the dedicated processor.

[0136] Optionally, the HBM in the dedicated processor is used to cache a part of the Emb vectors in the full Emb table. The storage medium in the general-purpose processor is used to store another part of the Emb vectors in the full Emb table.

[0137] When the dedicated processor trains or infers an artificial intelligence model, it reads the required Emb vectors from the HBM. After training is completed, the Emb vectors in the HBM are updated according to the gradients. Alternatively, after training is completed, the general-purpose processor can update the Emb table according to the gradients.

[0138] In some embodiments, triggered by a period / event, the dedicated processor periodically writes the Emb vectors in the HBM to the local parameter server.

[0139] Optionally, the Emb table of a large-scale model can also be stored in a remote parameter server. The general-purpose processor can obtain the Emb vectors corresponding to the identifiers from the remote parameter server.

[0140] In the third possible implementation, the number of parameters of the Emb table corresponding to an ultra-large-scale model is very large. For example, the number of parameters of the Emb table can reach 100TB. If the Emb table is stored in the dedicated processor, thousands of dedicated processors are needed to store the Emb table and thousands of dedicated processors are used to train the ultra-large-scale model. The demand for dedicated processors is too high and it is almost impossible to implement. The Emb table can be stored in a tertiary storage layer, that is, the Emb table is stored in the storage medium in the storage server, that is, the Emb table is stored in the remote parameter server.

[0141] Exemplarily, as Figure 11 shown, the general-purpose processor can store the Emb table of the ultra-large-scale model in the storage server, that is, the Emb table is stored in the remote parameter server, which is called Form Three.

[0142] The general-purpose processor reads the Emb vectors corresponding to the identifiers from the remote parameter server and transfers them to the dedicated processor.

[0143] The HBM in the dedicated processor is used to cache a part of the Emb vectors in the full Emb table.

[0144] Optionally, when there are more workers, the workers can be grouped, and the requests of the workers in a group are handed over to the general-purpose processor for processing. The local parameter server can provide deduplication and caching services for the workers; finally, the Emb vectors are transferred to the storage medium in the dedicated processor.

[0145] Thus, the unified architecture of the multi-level parameter server provides a hierarchical storage path, expands the storage method of the Emb table, supports the hierarchical storage of the Emb tables of models of different scales, and various different hardware deployment forms, realizing a naturally scalable Emb table storage solution. One set of architecture supports three recommendation models: medium-scale Emb recommendation model, large-scale Emb recommendation model, and extra-large-scale Emb recommendation model, without the need to develop and deploy related software for each scenario.

[0146] For example, for a medium-scale model, by expanding dedicated processors, the Emb table is stored in chunks in the storage media (Embedding Storage) of multiple dedicated processors, and multiple Embedding Storages are combined into a complete Emb table.

[0147] Another example is that for a large-scale model, the Emb table is stored in the storage media of a general-purpose processor, that is, the local parameter server. The storage media in the dedicated processor serves as a cache layer, and the Emb table is read from the general-purpose processor as needed for model training or model inference.

[0148] Another example is that for an extra-large-scale model, the Emb table is stored in a remote storage device, that is, the remote parameter server. The storage media in the dedicated processor serves as a cache layer, and the Emb table is read from the remote storage device as needed for model training or model inference.

[0149] In some embodiments, the unified architecture of the multi-level parameter server provides a unified programming interface externally, and the interface definition is as follows:

[0150]

[0151]

[0152] Among them, the SparseEmbeddingManager is used to manage the Emb table. Such as: creating an Emb table, setting parameter sets, etc.

[0153] SparseEmbedding corresponds to an Emb table, and provides corresponding methods for lookup and update according to the storage type (storage_type). The external interfaces include: embedding_lookup for Embedding vector lookup; embedding_update for Embedding vector update; save for Emb table storage. The key attributes include the storage type (storage_type) and the storage method of the Emb table.

[0154] The unified architecture of the multi-level parameter server presents a set of unified programming interfaces externally. For models of different scales, the user code does not need to be modified.

[0155] Next, the process of deploying the artificial intelligence model will be described in detail with reference to the accompanying drawings.

[0156] Figure 12 It is a schematic flowchart of a method for deploying an artificial intelligence model provided by this application. Here, the storage of sparse parameters related to the training calculation or inference calculation of the artificial intelligence model will be mainly described. As Figure 12 shown, the method includes the following steps 1210 to 1230.

[0157] Step 1210: The general-purpose processor obtains the number of parameters used when the artificial intelligence model performs inference or training, and the information of the available storage resources of the computing system where the artificial intelligence model is deployed.

[0158] This application does not limit the original storage location of the parameters to be stored. The general-purpose processor can obtain the parameters to be stored from storage devices such as a storage server or a storage array. The information of the storage resources includes various types of storage resources, and each type of storage resource in the various types of storage resources has different access performances.

[0159] Step 1220: The general-purpose processor determines the storage resources for storing the parameters according to the number of parameters and the information of the available storage resources of the computing system.

[0160] Step 1230: When the general-purpose processor deploys the artificial intelligence model, it stores the parameters in the determined storage resources.

[0161] The parameters to be stored include sparse parameters related to the training calculation or inference calculation of the artificial intelligence model. For example, the parameters include one or more Emb tables.

[0162] The information of the storage resources of the computing system includes the remaining storage capacity of the storage resources. For example, the information of the storage resources of the computing system includes Figure 8 the remaining storage capacity of the storage resources in the primary storage layer shown, the remaining storage capacity of the storage resources in the secondary storage layer, or at least one of the remaining storage capacity of the storage resources in the secondary storage layer. The remaining storage capacity may refer to the storage capacity of the storage space where no data is stored or the storage capacity of the storage space that can be used.

[0163] In the first possible implementation manner, the general-purpose processor determines the storage resources for storing the parameters according to the number of parameters and the information of the available storage resources of the computing system. The determined storage resources include at least one of the first storage medium associated with the dedicated processor, the second storage medium associated with the general-purpose processor, or the remote storage device.

[0164] The first storage medium associated with the dedicated processor includes the storage medium in the dedicated processor. The first storage medium includes volatile memory, non-volatile memory, or both volatile memory and non-volatile memory. The volatile memory includes at least one of HBM or DDR. The non-volatile memory may include an SSD.

[0165] The second storage medium associated with the general-purpose processor includes the storage medium in the general-purpose processor and the storage medium located in the same server as the general-purpose processor. The second storage medium includes volatile memory, non-volatile memory, or both volatile memory and non-volatile memory. For example, the second storage medium associated with the general-purpose processor is a local parameter server.

[0166] The remote storage device includes the storage medium belonging to a different server from the general-purpose processor. The remote storage device is mainly used to store parameters. For example, the remote storage device includes a storage array. For example, the remote storage device may be a remote parameter server.

[0167] The general-purpose processor determines whether the remaining storage capacity of the storage resources meets the storage capacity required for storing the parameters. If the remaining storage capacity of the storage resources meets the storage capacity required for storing the parameters, the determined storage resources are determined. Understandably, the general-purpose processor determines the storage medium that meets the parameter quantity from the system, and the storage medium that meets the parameter quantity can store the parameters. The storage medium that meets the parameter quantity includes at least one storage medium in the system. For example, the general-purpose processor determines whether the remaining storage capacity of the storage resources is greater than the parameter quantity of the parameters. If the remaining storage capacity of the storage resources is greater than the storage capacity required for storing the parameters, the determined storage resources are determined.

[0168] For example, the general-purpose processor determines whether the remaining storage capacity of the first storage medium associated with the dedicated processor is greater than the parameter quantity of the parameters. If the remaining storage capacity of the first storage medium is greater than the parameter quantity of the parameters, the first storage medium associated with the dedicated processor is determined as the determined storage resources, and the parameters are stored in the first storage medium associated with the dedicated processor.

[0169] Optionally, the general-purpose processor may divide the Emb table into multiple data blocks and store the multiple data blocks in multiple storage media in the dedicated processor. For example, one storage medium stores one data block, and the multiple data blocks stored in the multiple storage media can form a complete Emb table. Another example is that one storage medium can store multiple data blocks.

[0170] Another example is that if the remaining storage capacity of the first storage medium is less than the parameter quantity of the parameters, it is determined that the determined storage resources include at least one of the second storage medium or the remote storage device. That is, at least one of the second storage medium associated with the general-purpose processor or the remote storage device is used to assist the dedicated processor in storing the Emb table.

[0171] Exemplarily, a general-purpose processor may divide the Emb table into a first parameter set and a second parameter set, store the first parameter set in a storage medium in the dedicated processor, and store the second parameter set in at least one of a second storage medium associated with the general-purpose processor or a remote storage device.

[0172] Exemplarily again, a general-purpose processor may divide multiple Emb tables into a first number of Emb tables and a second number of Emb tables, store the first number of Emb tables in a storage medium in the dedicated processor, and store the second number of Emb tables in at least one of a second storage medium associated with the general-purpose processor or a remote storage device.

[0173] Optionally, if the remaining storage capacity of the first storage resource associated with the dedicated processor is equal to the number of parameters of the parameter, the parameter may be stored in the first storage medium associated with the dedicated processor.

[0174] Optionally, if the remaining storage capacity of the second storage resource associated with the general-purpose processor is equal to the number of parameters of the parameter, the parameter may be stored in the second storage medium associated with the general-purpose processor.

[0175] For the storage process of sparse parameters related to the training calculation or inference calculation of the artificial intelligence model, see Figure 13 。

[0176] Figure 13 It is a schematic diagram of the storage process of the sparse parameters provided by this application.

[0177] Judge whether the remaining storage capacity of the first storage medium (e.g., HBM) is greater than the number of parameters of the parameter. For example, whether the HBM can store the Emb table (step 1310). When the remaining storage capacity of the first storage medium is greater than the number of parameters of the parameter, store the parameter in the first storage medium, that is, store the Emb table in the card (step 1320). When the remaining storage capacity of the first storage medium is less than the number of parameters of the parameter, judge whether the number of Emb tables is 1 (step 1330).

[0178] If there is an Emb table, determine whether the remaining storage capacity of the second storage medium (e.g., local parameter server) is greater than the number of parameters of the parameter. For example, whether the local parameter server can store the Emb table (step 1340). When the remaining storage capacity of the second storage medium is greater than the number of parameters of the parameter, store the parameter in the second storage medium, that is, store the Emb table in the local parameter server (step 1350). When the remaining storage capacity of the second storage medium is less than the number of parameters of the parameter, determine whether the number of workers is small (step 1360). If the number of workers is large, store the parameter in the remote storage device and the second storage medium, that is, store the Emb table in the remote parameter server and the local parameter server (step 1370). For example, store the first quantity of parameters in the second storage medium and the second quantity of parameters in the remote storage device. If the number of workers is small, store the parameter in the remote storage device, that is, store the Emb table in the remote parameter server (step 1380). For example, when the remaining storage capacity of the second storage medium is less than the number of parameters of the parameter, store the parameter in the remote storage device.

[0179] If there are multiple Emb tables, when the remaining storage capacity of the first storage medium is less than the number of parameters of the parameter, store the first quantity of Emb tables in the first storage medium, that is, store the first quantity of Emb tables in the card (step 1390). Determine whether the remaining storage capacity of the second storage medium (e.g., local parameter server) is greater than the remaining number of parameters, that is, whether the local parameter server can store the remaining Emb tables (step 13100). If the local parameter server can store the remaining Emb tables, store the second quantity of feature tables in the second storage medium, that is, store the second quantity of Emb tables in the local parameter server (step 13110).

[0180] When the remaining storage capacity of the first storage medium is less than the number of parameters of the parameter, store the first quantity of Emb tables in the first storage medium (step 1390). If the local parameter server cannot store the remaining Emb tables, store the second quantity of feature tables in the remote storage device, that is, store the second quantity of Emb tables in the remote parameter server (step 13120).

[0181] In the second possible implementation, the general-purpose processor determines the determined storage resource according to the number of parameters of the parameter, the remaining storage capacity and storage type of the storage resource. The general-purpose processor determines the storage medium that meets the number of parameters of the parameter according to the remaining storage capacity and storage type of the storage resource, determines the storage medium that meets the number of parameters of the parameter as the determined storage resource, and stores the parameter in the determined storage resource. The storage type may refer to the storage medium type. The storage medium type includes both volatile memory and non-volatile memory.

[0182] In some embodiments, the general - purpose processor may first select the volatile memory from the system and determine whether the remaining storage capacity of the volatile memory meets the number of parameters of the parameter. If the remaining storage capacity of the volatile memory meets the number of parameters of the parameter, the volatile memory is determined as the determined storage resource. If the remaining storage capacity of the volatile memory does not meet the number of parameters of the parameter, the non - volatile memory is determined as the determined storage resource.

[0183] In other embodiments, the general - purpose processor may also obtain the performance requirements of the user during the inference or training of the artificial intelligence model; determine the storage resource for storing the parameters according to the number of parameters, the performance requirements, and the information of the available storage resources of the computing system.

[0184] Due to different services, the performance requirements for model training or model inference are different. For example, the performance requirements include data - processing speed requirements and data - access speed requirements.

[0185] For example, for a service with high requirements for data - processing speed and data - access speed, select a storage medium that meets the performance requirements from the system. For example, the access speed of the storage medium meets the data - access speed requirements, and the dedicated processor to which the storage medium belongs meets the data - processing speed requirements. Then, determine whether the remaining storage capacity of the storage medium that meets the performance requirements is greater than the number of parameters. If the remaining storage capacity of the storage medium that meets the performance requirements is greater than the number of parameters, determine the determined storage resource as the storage medium that meets the performance requirements, and store the parameters in the storage medium that meets the performance requirements. If the remaining storage capacity of the storage medium that meets the performance requirements is less than the number of parameters, other storage media assist the dedicated processor to store the parameters.

[0186] Optionally, the general - purpose processor may also jointly determine the determined storage resource based on multi - dimensional features such as the ratio of the performance and cost of the dedicated processor or the remaining storage capacity. Select a storage medium with a relatively high cost - performance ratio to store the Emb table under the factors of meeting performance requirements and storage requirements. Thus, combining the hardware resources required by the model, the available hardware resources, and the processing performance expected by the user, automatically select the appropriate hardware resources to store the parameters, without the need for complex manual selection and testing, improving the utilization rate of resources and the training or inference speed of the artificial intelligence model, and giving full play to the system performance.

[0187] Further, after storing the parameters in the determined storage resource, when performing training calculation or inference calculation on the artificial intelligence model, obtain the parameters from the determined storage resource. This application may also include step 1240.

[0188] Step 1240: Obtain the parameters from the determined storage resource, and the dedicated processor performs training calculation or inference calculation on the artificial intelligence model according to the parameters.

[0189] If the Emb table is stored in the storage medium of a dedicated processor, the Emb table is obtained from the storage medium of the dedicated processor, and then the dedicated processor performs model training calculation or model inference calculation according to the data in the Emb table.

[0190] If the Emb table is stored in the storage medium of a general-purpose processor, the Emb table is obtained from the storage medium of the general-purpose processor, and then the dedicated processor performs model training calculation or model inference calculation according to the data in the Emb table.

[0191] If the Emb table is stored in a remote storage device, the Emb table is obtained from the storage medium of the remote storage device, and then the dedicated processor performs model training calculation or model inference calculation according to the data in the Emb table.

[0192] Optionally, the data in the Emb table can be obtained from the determined storage resource by a general-purpose processor or a dedicated processor, the data in the Emb table is loaded into the dedicated processor, and the dedicated processor performs model training calculation or model inference calculation according to the data in the Emb table.

[0193] Therefore, the Emb table storage solution provided by this application, that is, when storing the Emb table required for model training or model inference, the storage media provided by the dedicated processor, general-purpose processor, and storage device are used to jointly store the Emb table. When the storage medium of the dedicated processor can store the Emb table, the complete Emb table is divided into data blocks and stored in the dedicated processor. During model training or model inference, each dedicated processor pulls the Emb vectors it needs for calculation as required.

[0194] When the storage medium of the dedicated processor cannot store the Emb table, the Emb table is stored in the storage medium of the general-purpose processor. A part of the Emb table is cached in the storage medium of the dedicated processor as needed. During model training or model inference, the required Emb vectors are pulled from the cache for calculation.

[0195] When the storage medium of the general-purpose processor cannot store the Emb table, a remote storage device is used to store the Emb table. Also, a part of the Emb table is cached in the storage medium of the dedicated processor as needed. During model training or model inference, the required Emb vectors are pulled from the cache for calculation.

[0196] A unified multi-level parameter server architecture is adopted to support the above three scenarios, and a unified interface is provided externally. Moreover, an automatic deployment component is provided to automatically select the optimal hardware facilities for deployment according to the data volume of the Emb table, hardware resources, whether the user pursues extreme performance or the best cost performance.

[0197] It can be understood that, in order to implement the functions in the above embodiments, a computer device includes corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should easily realize that, in combination with the units and method steps of each example described in the embodiments disclosed in this application, this application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a certain function is executed in the way of hardware or computer software driving hardware depends on the specific application scenarios and design constraints of the technical solution.

[0198] In the foregoing, in combination with Figures 1 to 13 , the method for deploying an artificial intelligence model provided by this application has been described in detail. Next, in combination with Figure 14 , the devices provided by this application will be described. These devices can be used to implement the functions of the dedicated processor or general-purpose processor in the above method embodiments, and thus can also achieve the beneficial effects possessed by the above method embodiments. In this embodiment, the device can be a dedicated processor or general-purpose processor as shown in Figure 12 , and can also be a module (such as a chip) applied to a computer device.

[0199] As Figure 14 shown, the artificial intelligence model deployment device 1400 includes a communication module 1401, a processing module 1402, and a storage module 1403.

[0200] The artificial intelligence model deployment device 1400 is used to implement the functions of the general-purpose processor in the method embodiments shown in the above Figure 12 or Figure 13 .

[0201] The communication module 1401 is used to obtain a request, and the request is used to indicate the execution of training calculation or inference calculation of the artificial intelligence model.

[0202] The communication module 1401 is further used to obtain the number of parameters of the parameters used when the artificial intelligence model performs inference or training, and the information of the available storage resources of the computing system on which the artificial intelligence model is deployed. For example, the communication module 1401 is used to execute Figure 12 in step 1210.

[0203] The processing module 1402 is used to determine the storage resources for storing the parameters according to the number of parameters and the information of the available storage resources of the computing system, and store the parameters in the determined storage resources, and when deploying the artificial intelligence model, store the parameters in the determined storage resources. The determined storage resources include at least one of a first storage medium associated with a dedicated processor, a second storage medium associated with a general-purpose processor, or a remote storage device. For example, the processing module 1402 is used to execute Figure 12 in step 1220 and step 1230. For example, the processing module 1402 is used to execute Figure 13Steps 1310 to 13120.

[0204] The artificial intelligence model deployment device 1400 is used to implement the functions of the dedicated processor in the method embodiments shown above Figure 12 or Figure 13 in the above.

[0205] Optionally, a communication module 1401, configured to obtain a request for indicating to perform training calculation or inference calculation of an artificial intelligence model.

[0206] Optionally, a processing module 1402, configured to obtain parameters from the determined storage resource and perform training calculation or inference calculation on the artificial intelligence model according to the parameters. For example, the processing module 1402 is configured to execute Figure 12 step 1240 in the above.

[0207] A storage module 1403 is used to store Emb tables, etc., for facilitating the training model or the inference model.

[0208] It should be understood that the artificial intelligence model deployment device 1400 according to the embodiments of the present application may be implemented by an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The above PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof. It may also be implemented by software Figure 12 or Figure 13 When implementing the method shown above, and its respective modules may also be software modules. The artificial intelligence model deployment device 1400 and its respective modules may also be software modules.

[0209] The artificial intelligence model deployment device 1400 according to the embodiments of the present application may correspond to executing the methods described in the embodiments of the present application, and the above and other operations and / or functions of each unit in the artificial intelligence model deployment device 1400 are respectively for implementing Figure 12 or Figure 13 the corresponding processes of each method in the above. For the sake of brevity, they will not be described in detail here.

[0210] Figure 15 FIG. is a schematic structural diagram of a computer device 1500 provided by the present application. As Figure 15As shown, the computer device 1500 includes a processor 1510, a bus 1520, a memory 1530, a communication interface 1540, a memory 1550 (which can also be referred to as the main memory unit), and a processor 1560. The processor 1510, the processor 1560, the memory 1530, the memory 1550, and the communication interface 1540 are connected through the bus 1520.

[0211] It should be understood that in this embodiment, the processor 1510 can be a CPU, and the processor 1510 can also be other general-purpose processors, digital signal processors (DSPs), ASICs, FPGAs, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0212] The computer device 1500 may also include a graphics processing unit (GPU), a neural network processing unit (NPU), a microprocessor, an ASIC, or one or more integrated circuits for controlling the execution of the program of the solution of the present application. For example, the processor 1560 can be a GPU or an NPU.

[0213] The communication interface 1540 is used to implement the communication between the computer device 1500 and external devices or components.

[0214] In the present application, when the computer device 1500 is used to implement Figure 12 or Figure 13 the functions of the general-purpose processor as shown, the communication interface 1540 is used to obtain requests, etc., so that the processor 1560 can determine the storage resources for storing parameters according to the information of the parameter quantity and the available storage resources of the computing system, and when deploying the artificial intelligence model, store the parameters in the determined storage resources.

[0215] When the computer device 1500 is used to implement Figure 12 or Figure 13 the functions of the dedicated processor as shown, the communication interface 1540 is used to transmit data, etc., so that the processor 1510 or the processor 1560 can instruct to obtain parameters from the determined storage resources, and the processor 1560 performs training calculations or inference calculations on the artificial intelligence model according to the parameters.

[0216] The bus 1520 may include a path for transmitting information between the above components (such as the processor 1510, the memory 1550, and the storage 1530). In addition to the data bus, the bus 1520 may also include a power bus, a control bus, a status signal bus, etc. However, for the sake of clarity, all kinds of buses are labeled as the bus 1520 in the figure. The bus 1520 may be a Peripheral Component Interconnect Express (PCIe) bus, or an extended industry standard architecture (EISA) bus, a unified bus (Ubus or UB), a Compute Express Link (CXL), a Cache Coherent Interconnect for Accelerators (CCIX), etc. The bus 1520 may be divided into an address bus, a data bus, a control bus, etc.

[0217] As an example, the computer device 1500 may include multiple processors. The processor may be a multi-CPU processor. Here, the processor may refer to one or more devices, circuits, and / or computing units for processing data (such as computer program instructions).

[0218] It is worth noting that Figure 15 only the case where the computer device 1500 includes 1 processor 1510 and 1 storage 1530 is taken as an example. Here, the processor 1510 and the storage 1530 are respectively used to indicate a type of device or equipment. In specific embodiments, the number of each type of device or equipment can be determined according to business requirements. For example, the computer device 1500 includes multiple GPUs or multiple NPUs.

[0219] The memory 1550 can be a volatile memory pool or a non-volatile memory pool, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchlink DRAM (SLDRAM), and direct rambus RAM (DR RAM). The memory 1550 is used to store the Emb table and the like.

[0220] The memory 1530 can correspond to the storage medium used to store information such as the Emb table in the above method embodiments. For example, a disk, such as a mechanical hard disk or a solid-state drive.

[0221] The above computer device 1500 can be a general-purpose device or a dedicated device. For example, the computer device 1500 can also be a server or other devices with computing capabilities.

[0222] It should be understood that the computer device 1500 according to this embodiment can correspond to the artificial intelligence model deployment device 1400 in this embodiment, and can correspond to the corresponding subject executing any of the methods according to Figure 12 or Figure 13 The above and other operations and / or functions of each module in the artificial intelligence model deployment device 1400 are respectively for implementing Figure 12 or Figure 13 The corresponding processes of each method in, for the sake of brevity, will not be described in detail here.

[0223] The method steps in this embodiment can be implemented in a hardware manner or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, and the software modules can be stored in a random access memory (RAM), flash memory, read-only memory (ROM), programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), register, hard disk, removable hard disk, CD-ROM, or any other form of storage medium well-known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC. Additionally, the ASIC can be located in a computing device. Of course, the processor and the storage medium can also exist as discrete components in a computing device.

[0224] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are executed in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user device, or other programmable devices. The computer program or instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer program or instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center in a wired or wireless manner. The computer-readable storage medium can be any available medium that the computer can access or a data storage device such as a server or data center integrating one or more available media. The available medium can be a magnetic medium, such as a floppy disk, a hard disk, or a magnetic tape; it can also be an optical medium, such as a digital video disc (DVD); or it can be a semiconductor medium, such as a solid state drive (SSD). As described above, the above are only specific embodiments of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art within the technical scope disclosed in the present application can easily think of various equivalent modifications or substitutions, and these modifications or substitutions should all be covered by the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. An artificial intelligence model deployment method, characterized in that, Including: obtaining the number of parameters used when an artificial intelligence model to be deployed performs inference or training, and information on the available storage resources of the computing system on which the artificial intelligence model is deployed, where the storage resources include multiple types of storage resources, and each type of storage resource among the multiple types of storage resources has different access performances, and the information on the available storage resources includes the storage capacity that can be used for each type of storage resource; determining the storage resource for storing the parameters according to the number of parameters and the information on the available storage resources; and when deploying the artificial intelligence model, storing the parameters in the determined storage resource.

2. The method according to claim 1, wherein The multiple types of storage resources include: a first storage medium associated with a dedicated processor in the computing system, a second storage medium associated with a general-purpose processor in the computing system, and a remote storage device connected to the server where the dedicated processor and the general-purpose processor are located through a network.

3. The method according to claim 2, wherein the determining the storage resource for storing the parameters according to the number of parameters and the information on the available storage resources includes: when the remaining storage capacity of the first storage medium is greater than the number of parameters, determining that the determined storage resource includes the first storage medium.

4. The method according to claim 2 or 3, characterized in that, The determining the storage resource for storing the parameters according to the number of parameters and the information on the available storage resources includes: when the remaining storage capacity of the first storage medium is less than the number of parameters, determining that the determined storage resource includes the second storage medium.

5. The method according to claim 4, wherein The determining that the determined storage resource includes the second storage medium includes: when the remaining storage capacity of the second storage medium is greater than the number of parameters, determining that the determined storage resource includes the second storage medium.

6. The method according to claim 4 or 5, characterized in that, The determining that the determined storage resource includes the second storage medium includes: when the remaining storage capacity of the second storage medium is less than the number of parameters, determining that the determined storage resource includes the second storage medium and the remote storage device.

7. The method according to claim 2 or 3, characterized in that, The determining the storage resource for storing the parameters according to the number of parameters and the information on the available storage resources includes: when the remaining storage capacity of the first storage medium is less than the number of parameters and the remaining storage capacity of the second storage medium is less than the number of parameters, determining that the determined storage resource includes the remote storage device.

8. The method according to claim 2 or 3, characterized in that, The parameters include multiple feature tables related to the artificial intelligence model; The determining the storage resource for storing the parameters according to the number of parameters and the information on the available storage resources includes: when the remaining storage capacity of the first storage medium is less than the number of parameters, determining that the determined storage resource includes the first storage medium and the second storage medium; The storing the parameters in the determined storage resource includes: storing a first number of feature tables in the first storage medium and storing a second number of feature tables in the second storage medium.

9. The method according to claim 2 or 3, characterized in that The parameters include multiple feature tables related to the artificial intelligence model; Determining the storage resources for storing the parameters according to the parameter quantity and information on the available storage resources of the computing system, includes: When the remaining storage capacity of the first storage medium is less than the parameter quantity, determining that the determined storage resources include the first storage medium and the remote storage device; Storing the parameters into the determined storage resources, includes: Storing a first quantity of feature tables into the first storage medium and storing a second quantity of feature tables into the remote storage device.

10. The method according to claim 1, characterized in that, The method further includes: Obtaining the performance requirements of the user during the inference or training of the artificial intelligence model; The determining the storage resources for storing the parameters according to the parameter quantity and information on the available storage resources of the computing system includes: Determining the storage resources for storing the parameters according to the parameter quantity, the performance requirements, and information on the available storage resources of the computing system.

11. A computer system, characterized in that, The computer system includes a memory and a processor, the memory is used for storing a set of computer instructions; when the processor executes the set of computer instructions, the processor is caused to execute the operation steps of the method according to any one of claims 1-10 above.

12. A computer-readable storage medium, characterized in that, Includes: Computer software instructions; when the computer software instructions run in the processor, the processor is caused to execute the operation steps of the method according to any one of claims 1-10 above.

13. A computer program product, characterized in that, Includes: When the computer program product runs on a computer, the computer is caused to execute the operation steps of the method according to any one of claims 1-10 above.

Images